Category: Geospatial Projects

  • Visualizing all of the World’s 258 Countries and Regions

    Bookstores have a lot of printed world atlases, which are nice to look at but have the trade off that each country is shown either once with a tradeoff of physical and political attributes, or in multiple maps that are not overlaid. Online maps often have impressive visualizations of particular places, but rarely cover all of the world’s countries at a consistent level. Using online vector and raster data along with QGIS and PyQGIS enables viewing of each of the world’s 258 countries and regions with a variety of combinations of data. Dask-Leaflet enables these per-country views to be displayed in a webapp in the browser.

    The top-level approach taken to achieve this desired view of all 258 countries and regions is to overlay-and-clip:

    • Select a vector map containing country/region borders
    • Overlay desired raster and vector data
    • Clip to the desired country/region

    This can be done in the QGIS GUI to find suitable datasets, then automated using PyQGIS to run for all 258 countries/regions.

    Data

    We want to map all the countries, so key requirements for the data are

    • Every dataset used covers all countries: no gaps or subsets
    • Available as raster or vector usable by QGIS/PyQGIS
    • (Also) Open source, not paid

    Following these requirements, the project’s current selection of datasets consists of

    The idea is that, for the displayed country, these data layers can be overlaid in various combinations to gain interesting learning, insights, or exploration of that country, and that the same can be done for every country. An example would be population density overlaid on natural features + climate zones.

    Any single map will be a simplification of the full story of a country, but the available insight from that map is always multiplied by 258.

    Automate

    Once the datasets are assembled, loading the layers, clipping to the desired country, and giving the layers the desired properties is fairly straightforward in the QGIS GUI without writing code. However, to do this for all 258 countries/regions requires automation.

    QGIS is built upon GDAL, so the GDAL command line interface commands like gdalwarp could be used for some of the processing such as clipping to countries. However, the Python integration PyQGIS is more convenient end-to-end. We therefore use this within a Python virtual environment in the VS Code IDE, and the QGIS Python console.

    Using an IDE in this way opens up AI-assistance in writing the code, which is beneficial. While most code generated by the AI had to be modified manually to work correctly, it noticeably sped its creation. The AI was particularly good at providing the syntax and classes from QGIS’s large selection to correctly set raster and vector layer properties.

    (QGIS can also provide the Python equivalent to a command executed in the GUI, another valid approach.)

    Getting it to work

    Of course, this is GIS using real data, so it won’t “just work” out of the box 😀 . There are various issues and details with the datasets that need to be overcome to obtain a useful result. Other issues encountered are suitable for future work and improving the results further.

    The most significant issues found and resolved are

    • Invalid geometries: Artifacts like self-intersecting polygons in country borders render a vector layer invalid. It may be surprising that the supplied data has this with a dataset as general as country borders, but setting the QGIS property Invalid features filtering to Do not filter resolves the problem. Better would be to make the geometries valid using an algorithm like QGIS’s fix geometries or PostGIS’s ST_MakeValid, but it was not needed at the project’s current level of development.
    • Country shapes: The QGIS default coordinate reference system (CRS) EPSG:4326 is good for showing the whole world. (Equal Earth 8857 would also be good.) However, countries far from the equator have their shapes distorted, and since we want to view countries individually, their shapes should be correct while their relative sizes do not matter. This makes the Mercator projection EPSG:3857 appropriate.
    • NoneType objects: This usually occurs when trying to set a property on a layer that has no pixels due to the country being very small. Checking the raster is valid removes this.

    The most significant issues found and not yet resolved are

    • Small countries are low resolution: with a globe-spanning dataset of fixed pixel size such as Natural Earth, small countries are pixellated, which limits the usefulness of their views. Currently we have categorized them into ok = works well, can see pixels = works well but can see some pixellation, and small, meaning to few pixels to be useful. 112 countries are ok, 61 are can-see-pixels, and 79 are small.
    • 6 countries still fail outright: Antarctica (unsurprisingly), Côte d’Ivoire (character encoding), and 4 more with a list index out of range. Antarctica can likely be handled as a special case using a suitable CRS that covers 90° south latitude. The others just need more error or special case handling in the code.
    • Some countries have areas separated by sea distances much larger than the sizes of their land areas, for example some Pacific island nations, and France (French Guyana). Generally these areas will fall under different regions in the admin/regions dataset and so could be separated and viewed that way. Similarly countries that span 180° latitude that have a too-far-out default zoom could be better projected.

    Webapp

    Making the country visualizations viewable is key to the results being useful. There are a number of options, including

    • Have the user run QGIS itself (instructions or plugin)
    • QGIS Web Client
    • Dash-Leaflet in the browser
    • Others: Folium, ipyleaflet, Leafmap, Google Earth Engine

    We chose Dash-Leaflet, because it provides straightforward overlaying of raster and vector data, appearance customization, and on/off layer switches, the main items needed for this visualization. It does so directly in the browser via localhost or a shareable URL. QGIS itself also works, as a byproduct of writing the code. The others were not attempted.

    Aside from setting up the webpage and the various elements, the main other webapp code in Dash-Leaflet is using Javascript functions to set the layer properties. This was another area where the IDE AI assistance considerably sped the work, writing the functions largely correctly from descriptions of the desired properties.

    Webapp

    Interesting countries and views

    As hoped for, the country visualizations provide some interesting views and insights. The screenshots in this section are taken from the QGIS interface.

    For reference, the Köppen-Geiger colors and climate zones are

    Köppen-Geiger colors and climate zones (from Wikipedia)

    Uzbekistan

    Here we overlay the population of Uzbekistan on the terrain (monochrome to avoid color mixing) with road + rail infrastructure, alongside the Köppen-Geiger climate zones. We see that the west of the country is barren (climate type BWk dry arid cold), flatter, and mostly depopulated, while the east has more people. But why is there population in some parts of the dry areas?

    If we show the land cover layer, a possible answer is cropland.

    Uzbekistan with land cover

    And indeed, if we add population back over the map, we see that it tracks the cropland areas almost exactly:

    Uzbekistan with population overlaid on land cover

    Other countries showing this population-follows-land-use pattern include Tanzania, and ones that partially extend into a desert such as Niger, Tunisia, and Yemen.

    Indonesia

    This is an example where broad patterns within the country are immediately apparent.

    Indonesia with population and infrastructure

    While there are many islands spread over a large area, the island of Java is clearly dominant in terms of both population and infrastructure. Next is Sumatra to the west of Java, followed by Sulawesi to the northeast. Only Java and Sumatra show railroads, and large parts of Borneo and Papua are sparsely inhabited.

    Nigeria

    Some countries are well known to have large populations, such as China and India. But there are others that have a great deal of people that not everyone is aware of. A good example of this is Nigeria. By far the most populous country in Africa, the extent of people is seen immediately when the population layer is activated:

    Nigeria with population

    A possible reason, broad areas of crops, trees, and grass across the whole country, is also clear to see:

    Nigeria with land cover

    Pakistan is another example of a highly populous country whose map shows it:

    Pakistan with population

    Indonesia is too, although in its case it is less obvious because of the large area of the country compared to the population centers. The United States is similar in that regard.

    Oman

    Finally for population, some countries have almost all people living only in very specific areas. Often these are inhabitants following rivers or being by the coast. Oman is a striking example:

    Oman with population

    Others where the people follow the rivers, the topography, or some other pattern in all or part of the country, are readily found, such as Syria (in the east), and Taiwan.

    Peru

    Moving on from population, some countries have wide ranges of topography, land use, and climate. By spanning the arid coast, through the Andes mountains and into the Amazon rainforest, Peru is one of the best examples, visible if we view the topography and climate zones:

    There are plenty of others showing variety that may be less well-known, such as the more subtle but still-there regions of Italy, or the colder north of Japan.

    Japan climate types

    South Africa

    Finally, let’s view one more country that well exemplifies the overview seen by accessing a range of data: South Africa.

    South Africa: topography, climate, and population

    The topography shows that most of the country is highlands, with lower-lying coasts. But the landscape and climate vary widely, being more barren in the northwest and temperate in the east. The population largely follows this, with the main cities being Pretoria (and Johannesburg), Durban, and Cape Town.

    More

    This page highlights a small fraction of what can be found by exploring these datasets. All the stories and insights seen here can be deepened by further research on a given country. The point of course is to give the level of overview seen for every country, in one place: multiply every layer combination shown above by 258.

    Summary and Future Work

    We have created a process that visualizes each of the world’s 258 countries and regions and allows interesting information to be overlaid. The process of gathering raster and vector layers and clipping them to the desired country is generic and extensible to other data. The results are viewable in QGIS or in a webapp. The processing is done in QGIS/PyQGIS, and the webapp is in Dash-Leaflet.

    While better views may exist for given countries, the key driver here is to get them all in one place, with all layers always available, so the user can explore and learn without bias towards parts of the world that are covered by greater amounts of geospatial data, projects, or companies.

    There are a number of obvious ways this work can be improved further

    • Webapp legend
    • Higher resolution data for smaller countries, especially the borders and digital elevation models (DEM)
    • Cloud-native workflow to tile and serve such larger data at an appropriate resolution for the chosen country being viewed, using formats like cloud-optimized GeoTIFF (COG)
    • Other datasets, e.g., Satellite Embedding V1 from DeepMind’s AlphaEarth Foundations GeoAI model
    • Other information such as industries, geology, mark the capital city, some photos, major attractions, etc.
    • Country-specific CRSs: some countries such as Canada are still distorted in Mercator projection (Ellesmere Island is not that big), and might benefit from a more “looking down at the globe” view, or a country-specific CRS
    • Enable printer-friendly or PDF views for each country
    • Enhance the webapp with pictures, text, etc.
    • Project as a QGIS plugin
    • Combine DEM data and QGIS 3D for a 3D view of each country

    In addition to these, there a range of smaller or technical details in the code, data, and presentation presented in the project’s GitHub repository.

  • Finding the Steepest Streets in Millbrae

    Finding the Steepest Streets in Millbrae

    OpenStreetMap and the OSMnx add-on for viewing streets as a network can be used to find the steepest streets in a neighborhood. Loosely based on their tutorial, we find the steepest streets in my home San Francisco Bay Area suburb of Millbrae, make a map of street grades that turned out to be useful for walking the area, and show views looking up the streets via the Google Street View API. The same code can be applied to any named area present in the well-known Nominatim geocoding tool.

    The steepest street in Millbrae?

    Why the steepest streets?

    The main motivations for finding the steepest streets in Millbrae are

    • Curiosity: I live there, and there are a lot of hills to walk or cycle on. Which is steepest?
    • Use OpenStreetMap, OSMnx, and Street View
    • The resulting code is applicable to any named neighborhood

    OpenStreetMap + OSMnx

    OpenStreetMap is the well-known and widely used open source project to provide street maps of the world. Here is provides a suitable base map for us, and is integrated with the second tool we need, OSMnx (docs, citation).

    OSMnx allows you to work with the streets, or more accurately street segments between intersections, to be worked with as a NetworkX MultiDiGraph of nodes and connections, each of which can have properties.

    When the locations of nodes are combined with elevation data, the grade of each street segment can be obtained using OSMnx’s function for the grade along the line segments corresponding to the street segment between 2 nodes. This gives us grades for every street segment in the network, and segment with the largest value is the steepest street.

    OSMnx has an extensive set of tutorials, one of which includes finding steepest streets. The code in this project is loosely based on it, but is mostly original. Some of the lines are:

    G = ox.graph_from_place(place_query, network_type="drive")
    G = ox.elevation.add_node_elevations_google(G, api_key=api_key)
    G = ox.elevation.add_edge_grades(G)
    
    grades_gdf = gpd.GeoDataFrame([d for _, _, d in ox.convert.to_undirected(G).edges(data=True)])
    
    grades_gdf_sorted = grades_gdf.sort_values(by='grade_abs', ascending=False)

    I found some detail improvements in OSM that could be submitted for Millbrae, such as speed limits information and presence of sidewalks.

    Finding the steepest

    OSMnx allows a map of the street network with the street segments colored by grade to be produced straightforwardly:

    Street grades in Millbrae: Yellow are the steepest, through orange, magenta, to purple as the flattest. The hillside running northwest-southeast on the left and the flatter areas next to the San Francisco Bay on the right are visible.

    Interestingly, when first running this project, I was considering doing more walking around Millbrae after dropping our 1-year-old at daycare, and seeing the areas nearby to the northwest of the neighborhood in the above plot with graded but not-too-steep streets encouraged me to do so.

    According to this analysis, the top 10 steepest streets in Millbrae are:

    Geodataframe of Millbrae’s top 10 steepest streets, according to this analysis. The main columns are name and grade_abs, the grade in % (0.18 = 18%, etc.).

    Now, though, comes a reminder that your analysis is only as good as your data. The steepest street is not, in fact, Mullins Court. The street segment in question happens to be one of very few in the neighborhood that at the time of writing are mislabeled on Google Maps. Having now walked there, I can confirm that, while it is definitely steep, it is part of Aura Vista. This makes sense, as Mullins Court is the no-through street to the west of the intersection. The rest of the top 10 are correct as far as I know.

    We can see the location of the Aura Vista segment as well, either on the street network:

    or with an OSM basemap (with the Mullins Court again mislabeled at the time of writing):

    See it on Street View

    To view the steepest street on Google Street View, we need two steps

    • Access the Street View API at the location of the street
    • Rotate the view direction and angle to be looking up or down the street in question

    After setting up the required API key, accessing the Street View location is achieved by constructing the correct URL:

    URL = (f'https://maps.googleapis.com/maps/api/streetview?size=600x300&location={lat_origin},{lon_origin}'
           f'&heading={heading:.0f}&pitch={pitch}&key={api_key}')

    which for Millbrae looks like this (API key not shown)

    'https://maps.googleapis.com/maps/api/streetview?size=600x300&location=37.5911979,-122.4058984&heading=254&pitch=8.527488864617514&key=<API key>'

    To rotate the view direction and angle, we need to extract and transform the direction and grade of the street segment from the OSMnx network into direction and angle above horizontal to pass to Street View. This is done by considering latitude and longitude for the direction, along with the x and y distances and elevations for the viewing angle.

    Here we always select the view of the bottom of the street looking up. This gives us our view looking up the steepest street segment in Millbrae, as at the start of the text.

    The steepest street in Millbrae, at 18% grade, is Aura Vista. At the time of writing the segment was mislabeled as Mullins Court.

    The approach fails if the street segment has a sharp curve between the two nodes, because you are looking directly towards the next node and not necessarily the direction the street immediately heads. This could be solved by decomposing the segment into its component linestrings and aligning with the first of them, but in practice this rarely comes up in Millbrae.

    Other neighborhoods

    The code is in principle generic to any neighborhood worldwide that can be named in the widely-used OpenStreetMap Nominatim geocoding tool. Simply swap out the name in the settings and explore. Here are some others nearby in the Bay Area:

    Brisbane (San Mateo Lane, 20% grade)
    Burlingame (Martinez Drive, 13%)
    Palo Alto (Mockingbird Lane, 10%)
    San Bruno (Rollingwood Drive, 16%)

    Limits of the data: San Francisco

    As is common in GIS, however, the real world has a way of bringing on limitations on workflows that naively seem straightforward, often in instructive ways. In this project, we see it if we set the place considered to be not a medium-sized suburb, but the whole of San Francisco.

    Because of its many steep hills, streets navigating them, and large size, San Francisco is particularly prone to street segments with spurious grades:

    • Limited latitude + longitude accuracy of the elevation data results in street node points on hillsides close to the street, not on the street
    • Very short segments of only a few meters connecting one-way halves of streets at an intersection (common in San Francisco)

    The grades plot itself looks good:

    San Francisco street grades

    but it thinks that this wall is the steepest street (71% grade!):

    “The steepest street in San Francisco”. This is a spurious steep grade created by slight inaccuracies in the locations of the elevation nodes compared to the local topography.

    Appropriately the name of this “street” is NaN.

    There are some detail ways that these artifacts could be reduced:

    • Filter out any segments of obviously spurious grade, say over 40%, and too-short length, say less than 10 meters
    • Manually inspect remaining Street View images
    • Manually adjust street node points so they correspond to the satellite image of the intersection (being sure that the network and raster are precisely aligned using a local CRS projection)
    • Remove the OSMnx default consolidation of intersections, which may over-simplify some short steep segments

    Ultimately addressing all of these in detail would constitute a full research project.

    Summary & Future work

    We found the steepest streets in my local neighborhood of Millbrae using OpenStreetMap and OSMnx, and viewed each one using Google Street View. The steepest street is Aura Vista, with a grade of 18%.

    The code can in principle be run on any neighborhood worldwide that is named in the well-known Nominatim geocoding tool.

    The main improvements to the results here would be

    • Make a webapp so users can see their neighborhood without writing code
    • Refine the analysis to be more robust in places such as San Francisco, using the ideas detailed in the main text above
    • Share it directly to target communities, e.g., OSM User Diaries
    • Add an option to use Open Topo data instead of Google elevation data so that the code can optionally be run without requiring the user to set up a Google API key

    There are smaller improvements as well. For example, normalizing the grades colormap to be absolute and not calibrated to only the range of grades in that neighborhood. As currently coded, for example, somewhere very flat like Foster City would still show yellow for the streets with the highest grades, even though those grades are not high. The GitHub repository has the full set of ideas.

    Interestingly, one consequence of these improvements might be, if a minimum threshold length for consistent grade is introduced to remove short street segments, such as the short ~ 10m segments of Hillcrest in Millbrae that place second and third, this might result in the 65m Aura Vista getting dropped, and the much longer (307m) but almost-as-steep Larkspur Drive taking the title. The whole segment between Pinehurst Court and Crestview Drive has an average grade of just under 16%, and in a way is a much worthier steepest street. Ultimately it depends on what we would like to highlight as a steep street.

    Larkspur Drive: the real winner?
  • Mapping a Historical Tour of Sheffield’s Don Valley, where the World’s Steel Industry Began

    Mapping a Historical Tour of Sheffield’s Don Valley, where the World’s Steel Industry Began

    My brother Andrew wrote a walking tour of the historical roots of the modern steel industry, Sheffield’s Don Valley for his blog. The open-source software uMap with an OpenStreetMap base can be used to turn this into a visual tour with a map, walking route, text and photographs. uMap can be used similarly for any piece of writing that involves traveling around an area. The map is available here .

    Don Valley walking tour route and locations on uMap with OpenStreetMap basemap

    Don Valley

    Extending northeast from Sheffield City Centre, Don Valley is an area of global historic importance because of the history of the Sheffield steel industry. This encompasses its origins, the invention of stainless steel, and the manufacture of many products, the most well-known of which is cutlery.

    Why map it?

    Sheffield is a large modern city that has moved on since these industrial days, and so viewing such history in-place today requires knowing where to look, and what you are looking at. What you need, in other words, is a guide, and Andrew’s walking tour provides this.

    The original tour is a text document that is nice to read and well researched. It is, however, only text, and so unless you know Sheffield very well already, or walk the tour in-person, there is a certain amount of abstraction to the history described.

    For this reason it benefits from adding a map:

    • Visualize the locations
    • Show the walking route
    • Incorporate text into the map
    • Add pictures

    Furthermore, the approach taken is generic: any text that involves describing places could be mapped in the same way.

    uMap

    uMap provides an open source solution that allows geospatial locations (points, lines, and polygons) to be overlaid on a street map base. It is built on top of the widely used Leaflet mapping software.

    uMap is generally used in a no-code fashion, although GIS knowledge, use of markup, etc., is useful for more advanced features.

    Making the map

    We start by selecting the basemap, OpenStreetMap in English, and an appropriate center and zoom level. The center is the start of the walk at Sheffield Cathedral, and the zoom is to Don Valley.

    Then we use the points datatype to add the locations mentioned in the text, formatting with various properties such as name, icon type, color, etc.

    Editing a point: Kelham Island Museum

    The walking route is added using the line datatype, following paths where they are marked, and placed in the road center where they are not. Sidewalks are not generally marked on the basemap used, likely because its maximum zoom level is not as high as the original OSM. The linestring resulting also has nodes in some places with more than two lines intersecting, because the route loops and returns to places already passed.

    Editing a line: walking route west of Tinsley locks

    We did not use the polygon datatype on the current map. It could be added, for example to show the outlines of buildings of interest, but there was sufficient content already for our purposes.

    Likewise we used the simple default symbology available in the uMap GUI, without, e.g., addons of icon types or symbols.

    A nice feature of uMap is that it allows points to be labeled with media such as text, pictures, or video, activated in various ways such as clicking on the point. This allows pictures of the locations to be added, or even the entire original blog text appropriately spaced. Currently each point has a simple label and the text is the original blog entry.

    Details, details…

    So this is all fairly easy, right? Just read a few pages and add some map points. Well, nope. As you might expect with the real world (and especially GIS), some things come up.

    A particular example from this walking route was on the Five Weirs Walk. Andrew’s original route had you following it for some distance, but since the text was written part of the walk between East Coast Road and Attercliffe Road has collapsed into the river. If you like climbing steep grass embankments with trees and old pieces of concrete it might still be possible, but for normal people we need a reroute round the streets.

    The first step was to confirm that it was closed. Google Maps shows the path as still there, labeled Five Weirs Walk. Apple Maps shows the path, unlabeled. But the OSM base map shows closed gates and the segment of the path no longer present. On the satellite images, on both Google and Apple the collapsed section is visible, with the river extending close to the wall on the west bank, and pieces of concrete in the river. Not much is visible from Street View.

    To reroute round the closed section, from Google Maps you might naively choose Windsor Street, but Apple Maps hints you may not want to because the street is marked narrowly as not a regular street, and indeed OSM and StreetView show that it passes through the premises of Thessco Ltd. At the other end there is a gate blocking the way to Princess Street, and it kind of goes near the Five Weirs Walk again but is behind the gate on that path. So maybe you can still walk through, but simpler is to route across the river around Faraday Road and Washford Road, which are regular streets.

    The obvious step if you really wanted to go along Windsor would be to go there (or have Andrew go there), but it’s not necessary to do that here.

    Addressing details like this routing ideally results in a route you can walk in-whole or in-part.

    Some places along the route

    The full route is on the map, but a few highlights are below.

    Bower Spring: These are the best preserved remains of a mid-19th century cementation furnace, the process that enabled production of steel in quantity and put Sheffield on the map as a world center. It was recently restored and removed from the Heritage at Risk list in Yorkshire, which was reported on by the BBC in this article.

    Bower Spring cementation furnace. From the BBC news article.

    Kelham Island Industrial Museum: Showcases the history of the steel industry in Sheffield, along with other exhibits. The 12,000 horsepower River Don Engine that was used to roll steel plate is regularly run and has the shortest time to reverse direction of a machine its size anywhere.

    Kelham Island Industrial Museum. From Google Street View.

    Ball Street bridge: Aside from being my namesake, this is a scenic bridge over the River Don, giving good views of the river and former works on the banks such as Alfred Beckett and Sons.

    Ball Street bridge. From Google Street View.

    Birthplace of stainless steel: This building used to house Brown-First Research, and was the world birthplace of stainless steel, one of the materials that revolutionized manufacturing in the 20th century due to its lack of rusting.

    Birth place of stainless steel. From Google Street View.

    Terry Shellby, canal terrapin: Not officially part of the tour, but found along the way.

    Terry Shellby canal terrapin. From Google Maps.

    Can AI make this map?

    The short answer is no, not yet. While the latest foundation models and coding tools (Google’s Gemini 3 and Claude Code at the time of writing) can help make maps from a prompt, and do other agentic work, the material here is too esoteric to work well. Places such as the sites of old steelworks not now named on the basemap need to be appropriately placed, and actually-possible routes like not taking the part of the Five Weirs Walk that has collapsed into the river need to be given, plus the suitable rerouting round a street mentioned above.

    What AI could likely provide is a rough draft map or list of locations from the text, and help with creating a map with the uMap API if you prefer that route to the no-code one. It could also provide a handy summary of the text, an interface to answer questions about it, further pictures or links about places of interest, and how to travel there. But it is not going to outdate the full manual work to create this project just yet.

    Summary & Future work

    We have made a map of a walking tour of Don Valley showcasing the globally important history of the steel industry present in the area. The original walking tour text is here, and the map is here.

    The map is made in a no-code fashion using the open source software uMap using an OpenStreetMap base. The emphasis is on the content and making a visualization of sufficient accuracy that it can be used as a guide to walk the route.

    Some improvements to the work include

    • Add pictures and text to the points
    • More extensive symbology
    • Have someone walk the route to confirm it is 100% correct

    The uMap approach is generic, meaning that any text that describes travel around locations could be mapped in the same way.