Imagico.de

blog

Weaving roads #2 – creative compositioning

September 15, 2024
by chris
1 Comment

Weaving roads #2 – creative compositioning

In the previous post i explained one of the main design issues of road depiction in OpenStreetMap based maps and identified the most viable approach to address this. Here i want to demonstrate and explain how this can be practically done.

Just like in the case of viewpoint rendering i discussed earlier, this requires extending the functionality of Mapnik. If you want to re-produce what i show here you will, therefore, need to build Mapnik yourself using the modifications i have published.

To recapitulate – the problem i intend to address here is that in normal road rendering, the roads (including road polygons, like pedestrian areas) are rendered above linear features like waterways. Therefore, those line features do not render properly, even if they are diligently cut out in mapping from the road polygon (which – according to broadly established mapping conventions – is not strictly necessary). Also, road tunnels are fully covered underneath larger road polygons, which hides important information on underground road and path connections from the map user.

Current layering of pedestrian road lines and polygons relative to other road and line features

Current layering of pedestrian road lines and polygons relative to other road and line features

We want to address these issues without actually changing the road drawing order overall. Linear road tunnels are still supposed to be rendered below linear surface roads and linear road bridges above them. And surface roads, as well as tunnels and bridges of the same layer, should remain visually connected to one another to correctly communicate the connectivity in the roads system.

Impossible waterfalls

Working this problem is a bit like drawing an impossible waterfall like the one in the famous drawing by M.C. Escher.

M.C. Escher: Waterfall, 1961

M.C. Escher: Waterfall, 1961

This kind of drawing is consistent in the 2d depiction sketched and locally consistent also in three dimensions. It, however, becomes self contradicting and physically absurd in total. Therefore you cannot create this kind of drawing strictly from physical principles alone, you have to explicitly break with these principles. The challenge is to do so in a way that remains locally consistent.

The map rendering task at hand is similar, what we need to explicitly break with is the strict adherence to the vertical order of features in drawing and the rule that the styling of a feature is only defined by its own attributes. And we want to – as much as possible – maintain local consistency in drawing, giving the correct impression about connectivity and vertical order of features in reality.

This is a fairly complex task so i limited my proof-of-concept here to just the surface level road polygons. For the bridges and tunnels the setup presented here would essentially need to be duplicated once more for every layer (because for each of them a separate set of linear road features will need to be treated specially). Since bridge and tunnel road polygons are rare and large ones substantially covering other roadwork are even less common this limitation is practically not very big.

What we will need to do is rendering some road features (specifically road tunnels and linear features otherwise drawn before the road layers) differently when and for the area where they would normally be covered by a surface level road polygon. The procedure to do that is:

  • Render the road layer as usual with the drawing order chosen according to the vertical oder of elements in reality as tagged, including the surface level road polygons.
  • Render separately a second version of the road layers without the surface level road polygons.

    the linear features that are visible in this version because they are not hidden by the road polygons need to be drawn here in the design in which they should ultimately show up above the road polygons. The layer implementing this is called roads_noareas.

  • Render a mask of the line features that are to be shown like in roads_noareas in the area where they are normally covered by the surface level road polygons. The layer implementing this is called roads-line-mask.
  • Render a mask of the surface level road polygons visible in the standard road display. That is essentially the surface level road polygon footprints minus the bridge road polygons. The layer implementing this is called roads-area-mask.
  • Combine these four renderings by composing roads_noareas over roads only in areas covered by both roads-area-mask and roads-line-mask.

Advanced compositioning with GMIC

This is currently not possible to do with the rather limited compositioning capabilities of Mapnik. The only thing Mapnik can do out-of-the-box is composing what is newly drawn (either on the feature or on the layer level) with what has been previously drawn in a two component compositioning operation. The kind of multi-component compositioning operation described here is not available.

To overcome this i added support for raster post processing of the rendering using the GMIC image processing framework to Mapnik. GMIC is doing image processing using a dedicated scripting language. This framework can be invoked for every layer of the map style in addition to Mapnik’s internal per-style compositioning operation, either before or after. GMIC perfoms its processing based on an image buffer stack. The way i integrated this framework into Mapnik this stack is retained between layers with the current layer (either before or after Mapnik’s internal compositioning) being added to that stack for further processing.

To understand how to make use of that feature it is useful to understand how Mapnik does its own per-style compositioning. If you don’t use either style level opacity or comp-op (meaning no opacity or comp-op being set for the whole layer in Carto-CSS) Mapnik plainly renders everything into a single rendering buffer one element after the other. If opacity or comp-op is used Mapnik renders the layer into an empty (i.e. originally fully transparent) buffer and then composes this into the buffer retained from the previous layers using the chosen opacity and comp-op. If you use the new gmic style property the compositioning is automatically activated as well, the layer is rendered into a new, empty buffer and that buffer is then added to the GMIC image buffer stack. Afterwards the script specified in the gmic property is run.

But by default nothing is composed into the main rendering buffer. For that to happen the script has to assign the name use to the last image buffer in the stack. Then Mapnik uses this buffer as the source for drawing the layer (potentially using the specified comp-op/opacity) and removes it from the GMIC image buffer stack for the next layer. In other words: the no-op GMIC command is (in CartoCSS syntax)

gmic: '-name. use';

To store the rendering of this layer for use in processing one of the next layers in addition to using it as is you can use

gmic: '+to_rgba. -name. use';

which duplicates the last image on the stack (the current layer rendering) into another rgba buffer and then sets that up to be used. Since only the buffer used will get removed from the stack by Mapnik, the second version of the current buffer will remain for later re-use. Finally, if you just want to store what has been rendered for future use and not directly use the current layer at all, use something like:

gmic: '-to_rgba.';

This converts the last buffer in the stack to rgba – which it already is, hence: it essentially does nothing. Since the name of the buffer is unchanged, Mapnik will neither remove not use it and will simply move on to the next layer.

The other important thing to know is the naming of the buffers. All buffers added by Mapnik are named after the layer they are generated from – with any dash (-) in the layer name being replaced by an underscore (_) since dashes are special characters in the GMIC scripting language.

This whole principle of operation is just designed as i saw it fit for the moment – it is highly probable this is not ideal and needs adjustment in the future.

Back to the actual problem: The normal road layer (roads) and the second variant without the surface level road polygons (roads_noareas) are rendered from the same SQL code and share most of the CartoCSS code. The roads layer is rendered normally, for reference: here is how the rendering looks like after that layer:

Starting base before the roads-area-mask layer - after rendering the roads in classical ordering

Starting base before the roads-area-mask layer – after rendering the roads in classical ordering

After that comes the roads_noareas in addition has the GMIC parameter

#roads-noareas { gmic: '-to_rgba.'; }

The roads-line-mask uses the same, i.e. stores the rendering rather than using it. The actual compositioning happens in roads-area-mask where we use the following GMIC script, written here – for clarity – with one command per line:

+channels[roads_noareas] 3
-name. roads_noareas_mask
-channels[roads_line_mask] 3
-channels[roads_area_mask] 3
-min[roads_area_mask] [roads_noareas_mask]
-min[roads_area_mask] [roads_line_mask]
-to_rgb[roads_noareas]
-append[roads_noareas] [roads_area_mask],c
-keep[roads_noareas]
-name. use

This is explained step by step in the following. I link to the resulting images of the different processing steps. These you can generate yourself by adding suitable -output commands in the script. When you invoke the rendering via Nik4 or similar means, this will generate snapshots of the processing – which is useful for debugging purposes.

  • duplicate the alpha channel of roads_noareas into a new buffer (result)
  • name that buffer roads_noareas_mask
  • reduce roads_line_mask to its alpha channel (result)
  • reduce roads_area_mask to its alpha channel (result)
  • calculate the minimum of roads_area_mask and roads_noareas_mask and store it into roads_area_mask (result)
  • calculate the minimum of roads_area_mask and roads_line_mask and store it into roads_area_mask (result)
  • strip the alpha channel from roads_noareas
  • append roads_area_mask as new alpha channel to roads_noareas (result – with transparent parts rendered as dark gray)
  • remove all buffers except for roads_noareas
  • use the remaining buffer for composing this layer

The final results on the sample is here:

Final results of rendering the sample setup

Final results of rendering the sample setup – link goes to double resolution version

This, of course, not only works for pedestrian areas, but also for all the other polygon features rendered in the road layers:


highway=pedestrian/service/platform


highway=residential/aeroway=taxiway/highway=track

Note that highway=steps lines are rendered above highway polygons, just like barriers and tunnels, in disregard of the z-order because open air steps starting from within a pedestrian area, like at a subway entrance, are often just drawn onto the polygons. This is not a very descriptive mapping practice though – despite the convention that linear features supersede polygons (see previous post). Better options for mapping are sketched below.

Subway entrance on a pedestrian area: (1) plainly mapped onto the pedestrian polygon, (2) cut out and tagged with width, (3) in addition mapping of barrier=wall around the edge

Subway entrance on a pedestrian area: (1) plainly mapped onto the pedestrian polygon, (2) cut out and tagged with width, (3) in addition mapping of barrier=wall around the edge

That’s it essentially. This probably sounds much easier than it was though. The design of the roads_line_mask and roads_area_mask is pretty delicate to ensure the results are free of artefacts.

The whole exercise was about the display of line features overlapping with surface level road polygons. One remaining question in light of this is how to treat tunnel road polygons. They could reasonably stay hidden by the surface level rendering in line with the general principle of rendering according to the vertical ordering in reality. I decided instead to render them with outline only. My thought is in particular about subway platforms mapped with polygons – which are useful to be shown underneath pedestrian areas. A bit problematic is the strong difference between the outline rendering of the polygon version and the filled rendering of the linear version, making recognition as the same type of feature unlikely.

Outline rendering of tunnel road polygons overlapping surface level road polygons

Outline rendering of tunnel road polygons overlapping surface level road polygons

Real world examples

A few practical examples of how this looks like with real world data. First a case featuring barrier lines:

Note that the new approach not only shows barriers within the pedestrian area, the fence in this case, but also better shows the walls and retaining walls at the edge of the pedestrian area. The next example features waterways in Freiburg – the famous Bächle:

This well demonstrates that the drawing order change only applies to the road polygons and not to the linear roads – which here leads to partially visible waterway line signature within the pedestrian gray. That is somewhat confusing, but ultimately is caused by the inconsistent mixing of polygon and linear mapping of the pedestrian roads in this case. Another example from Lyon shows a partially mapped subway station:

You can see the underground platform and the subway lines as well as the entrances to the subway. Not mapped so far are the connecting footways in between. A similar case in Marseille:

Mapping is, likewise, incomplete here. And the underground platforms are mapped with linear ways so they are rendered with a fill color. Finally the example from Prague i already showed in the first post of this series:

As far as the tunnel rendering is concerned – more elaborate examples of subway stations or other underground structures underneath pedestrian areas are a bit difficult, because mapping consistency is often not that good. While mappers tend to try to apply tagging with layer diligently, the lack of good visual feedback combined with the counterproductive incentive, that existing mainstream map rendering often has, tends to lead to issues in the data. Common examples are:

  • location=underground/tunnel=yes/covered=yes not being applied consistently to underground platforms.
  • roads underneath pedestrian areas being deliberately not tagged tunnel=yes/covered=yes to make them show up.

An overall discussion of the rendering technique demonstrated here will follow in the next and last part of this series.

Weaving roads #1 - polygons and lines

September 13, 2024
by chris
1 Comment

Weaving roads #1 – polygons and lines

When discussing the rendering of roads in detail 3.5 years ago i showed how OpenStreetMap-Carto uses the different options offered by CartoCSS and Mapnik to draw the road network in a consistent fashion but, ultimately, still falls short of this in some aspects. I also showed the new system of rendering the roads in a single layer, used now in the AC-Style, and how this can help overcome some of these limitations.

Road layering in OSM-Carto with inconsistencies

Road layering with tunnel, ground level and bridge elements represented by linear ways and polygons – as shown in OSM-Carto with inconsistencies – link goes to double resolution version

Improved road rendering as used in the AC-Style so far

Improved road rendering as used in the AC-Style so far – link goes to double resolution version

All of this was done by defining a consistent drawing order of the different features, in many cases in several instances – casing, background, fill, centerline etc. This fully relies on every aspect of the physical vertical ordering being explicitly represented in the data (splitting the geometries as required for that during mapping and, where necessary, using the well known layer=* tag). That works relatively well since mappers in OpenStreetMap often quite diligently record the relevant information on the roads network explicitly. As an additional bonus, it provides clear direct feedback to mappers on how things have been mapped – which helps maintain this high quality mapping.

There are, however, still fundamental issues with this approach that can be pretty severe in the concrete cartographic reality. Most notably regarding the road polygons, where we have two big (and related) problems that i am going to explain below.

Contradicting mapping conventions

Both problems are related to the fact that road polygons (that is for example highway=pedestrian or highway=service polygons – which are used to map pedestrian or vehicle use areas where no single direction of navigation is dominant) are treated exactly like linear roads in terms of drawing order and this is strictly based on the physical vertical order in reality. Although this logic is easy to understand and leads to a clear and easy to interpret rendering (see above), it is also somewhat unexpected in some cases for mappers – which creates the first problem:

It is common that mappers expect features intersecting a highway=pedestrian polygon to be shown above the pedestrian area – like for example a small pond around a fountain or a kiosk building. They would not have the same expectation for a linear pedestrian road intersecting such features, but for highway=pedestrian polygons this is quite common. Sometimes, mappers add layer tags to such features to explicitly indicate the kiosk/pond is located on the pedestrian area. Semantically, this is questionable though, since a surface water area or a ground level building overlapping a non-bridge, non-tunnel highway=pedestrian polygon is incompatible with the established meanings of these tags – independent of the presence of a layer tag. Or in other words: These things are not actually overlapping. In short: It is quite clear consensus among mappers that road polygons should be limited to the area of actual navigation with no implicit (or explicit – through layer=*) resolution of ambiguous overlaps with other polygons.

Pedestrian areas with non-walkable parts (like flowerbeds, fountain ponds) being excluded from mapping as highway=pedestrian.

Pedestrian areas with non-walkable parts (like flowerbeds, fountain ponds) being excluded from mapping as highway=pedestrian. Not universally though – see here

The situation is different though for overlaps of road polygons with other linear features like waterways or barriers. The overarching convention in OpenStreetMap for linear and polygon elements in general is that overlaps are acceptable in mapping, at least in most cases, even if they are semantically contradicting. A highway=track crossing a landuse=orchard or a landuse=farmland polygon is considered correct mapping. Same for a waterway crossing a natural=scrub or natural=bare_rock. Although no crops are grown on the highway=track and no scrub grows within the river, the mapping convention here is that the linear mapping of the track/waterway implicitly supersedes the polygon landcover/landuse mapping. Data users have to interpret the data accordingly – which, in map rendering, is typically accomplished by drawing the line signatures of the linear map elements on top of the polygon elements.

Variants of mapping pedestrian polygons: (1) gross area including non-walkable polygons 'within' (2) only effective pedestrian area but covering linear features and (3) cutting out both polygon and linear features within

Variants of mapping pedestrian polygons: (1) gross area including non-walkable polygons ‘within’ (2) only effective pedestrian area but covering linear features and (3) cutting out both polygon and linear features within

And here you probably notice the dilemma in map rendering since we have two conventions clashing:

  • That linear mapping (of features like barriers, waterways, roads and paths) supersede polygon mapping (like landcover polygons) in cases of overlaps between the two for the area implicitly belonging to the linear feature for where there would be semantic contradictions.
  • That in road layering both polygon and linear mapping are together ordered based on physical vertical location relative to each other according to the explicit tagging with bridge=*, tunnel=* and layer=*.

So far, the AC-Style for the roads has been following exclusively the latter principle, which is – as mentioned – frequently irritating for mappers with regard to the road polygons because of the former principle. And giving up on the former principle and cutting out even linear features from the road polygons in mapping does not help practically either, because the gap in the road polygon mapping based on actual ground width of the linear feature does not provide enough space for the line signature of the linear feature to be rendered. You will see something on the map (like in the sample 3 above) but not the recognizable line signature of the feature in question.

Rendering so far of overlaps between ground level highway=pedestrian features and line features

Rendering so far of overlaps between ground level highway=pedestrian features and line features (barrier=wall on top, waterway=stream on bottom) – (1) linear road with plain intersections, (2) linear road with explicited intersections (barrier=entrance, ford=yes), (3) road polygon

You could now get the idea of rendering all line features of the style after rendering the road layers. That would, however, not work well, because the road network would become difficult to read since it gets frequently overlapped by other line features, especially at the lower zoom levels. The other commonly mentioned idea is to sort everything in rendering according to the layer=* tagged – essentially giving mappers control over the drawing order. That is also not a good idea, because it would effectively sabotage OpenStreetMap as a collection of semantically meaningful information about the geography. Right now layer=* has a fairly well defined meaning regarding the relative physical ordering of close-by or overlapping elements. If mappers instead would start using the layer tag instead to define the drawing order of a map or to define override rules in semantically contradicting mapping that would create a lot of damage to OpenStreetMap as a whole.

Hidden tunnels

The other unresolved aspect of the road layering stems from the fact that defining the drawing order strictly after the vertical ordering in the geographic reality does not universally produce the best readable map – even when you only look at the road features. A drawing order following strictly the vertical ordering in reality means that tunnels are universally covered by surface features. For the linear road features this is rarely an issue (at least for the roads themselves – the road labels and oneway arrows are a different matter). For the road polygons, however, if you have a larger pedestrian (or other road type) polygon, that essentially covers all the underground road infrastructure underneath.

Underground infrastructure hidden by pedestrian area

Underground infrastructure hidden by pedestrian area – links to same area in OSM-Carto.

Bottom line: For a well readable map you ideally do not stick to strictly defining the drawing order by vertical ordering for the roads. For elements that would otherwise be fully hidden by what is drawn above them the better approach is to try visualizing them out of order in some form. This avoids creating substantial gaps in the depiction of the road system.

Solutions

Tackling this second issue is not possible with the methods so far deployed in road rendering in either OSM-Carto or the AC-Style. As mentioned above, so far these styles base the drawing order and styling decisions fully on information explicitly stored in the data per feature. What is necessary here is interpreting the context – a tunnel feature needs to be drawn differently (in either order of drawing or styling or both) depending on if they are underneath a road polygon or not. This can be done either by

  • Intersecting the geometries in SQL. That is computationally expensive and would require adding substantial code complexity to the already complex road layers.
  • Using raster compositioning. This seems the more elegant approach here. But it is beyond the limited compositioning abilities of current Mapnik.

The good news is that both approaches will equally allow addressing the first problem of properly displaying non-road line signatures on road polygons – dealing with those differently within road polygons is not much different from the dedicated display of tunnel features within road polygons.

In the next part i am going to demonstrate and explain an implementation of the second method offering a partial solution to the problem discussed.

September 10, 2024
by chris
2 Comments

A short State of the Map 2024 comment

State of the Map 2024 took place in Nairobi last weekend. I did not attend, although i tried to get an idea by watching some of the video streams. With tried being the operative word here – the streaming infrastructure did not seem to be quite up to the task. That was clearly not the fault of the local team though (with the exception of the microphoning and audio levels – which was severely challenging for the viewer).

Not going to write an in depth commentary here. To recapitulate the background: The year before (2023) State of the Map was meant to take place in Cameroon but the OSMF board threatened to intervene for political reasons so the SotM working group scuttled their plan – meaning no SotM in 2023. Bottom line: A lot of people were pissed, but for very different reasons.

When, for 2024, the SotM working group selected Nairobi, the opposition was less severe and the OSMF board kept quiet – so the conference went ahead. It was nice to see the African communities got some broader recognition. And given the severe lack of more substantial communication of the OSMF with the larger OSM community otherwise, the program items from the OSMF gave valuable insights into the inner-OSMF mindset that are otherwise not available to the ordinary OSM community member.

Beyond that i want to point interested readers to the commentary and impressions from Severin and Ilya from the conference. If you compare the official public communication from the OSMF to Ilya’s comments, that is truly a night-and-day difference – predominantly PR phrases and random pointers to communications of others vs. thoughtful personal impressions and critical commentary based on concrete individual experiences.

The OSMF has now for quite a few years tried to organize some streamlined synthetic corporate PR style communication (with fairly limited results even by corporate standards). At the same time they are essentially ignoring the substantial grassroots efforts and talents the OSM community has developed in the field of public communication because they do not fit into the OSMF corporate culture. The people who try to help the OSMF writing mastodon and twitter posts or entries on the official OSMF blog are surely well meaning and enthusiastic. But the idea to compete through engineered PR style communication with people freely and independently writing about the topics they are involved and experienced in is simply not a winning proposition. And, as a side effect, it alienates everyone in the wider OSM community who publicly shares their thoughts on OpenStreetMap topics with a wider audience and who has the ambition to do so independently and in a reflected way without being affected by organizational interests.

Ironically, this year’s SotM also showed how much talent and eagerness there is also in the African OSM communities in public communication, which is sadly not going to find the support from the OSMF to thrive and to develop the experience and self confidence to write and talk independently and critically about matters of interest for the global OSM community with a distinctly African perspective. My hope is that such support will at least to some extent come from the broader OSM community outside the OSMF.

Next year’s conference is announced to be in Manila, Philippines – from former British colony to former US colony i might critically add. But i am happy for the Philippine OSM community. They are a very active local community in OpenStreetMap with a very dynamic development of the map in the area. It is unlikely i am going to be there though. I might reconsider that if someone was interested enough in me talking there about map design, generalization, open data satellite imagery or any other topic of interest to finance me visiting the conference. But as is, there is neither enough use for my business in such a visit nor am i interested in becoming part of the international OSM jet set.

A short final remark regarding diversity of locations of SotM conferences. It has occurred to me that up to and including Manila 2025 all SotM conferences with the exception of Japan (2012 and 2017) will always have taken place in predominantly Christian countries. Coincidence?

Viewpoints in Provence, France

September 6, 2024
by chris
1 Comment

More dynamic symbols: Viewpoints

Two years ago i showed a new design concept for rendering trees in maps. This used a combination of hand designed components with automatic geometry processing to depict different types of trees of different sizes in a way the symbols naturally overlap in a well readable way when the trees are close to each other.

Non-blocking dynamic tree symbols

Non-blocking dynamic tree symbols – click for the blog post discussing these in more depth

Technically, this was done by rendering the symbols from a polygon representation in the rendering database. This facilitates both the construction of the symbols from their hand drawn design elements, the dynamic scaling according to the tree diameter, and the cutting of intersecting tree symbols for the natural display of overlapping symbols.

One thing you might have noticed with the tree symbols is that they are non-blocking. Other symbols and labels overlap with the tree symbols freely. This matches the tree display in OpenStreetMap-Carto and is a prudent choice for tree display. But it also was a practical necessity, since polygons rendered from the database in Mapnik are inherently non-blocking. Collision detection and blocking in Mapnik is traditionally only available for point, marker and text symbolizers. The technique i demonstrated for the trees was therefore limited in its application to situations where non-blocking symbols are the goal.

I have since then extended the capabilities of Mapnik to customize dependencies between symbols drawn using the concept of anchors. I here want to show how that feature can be further extended to allow the use of symbols drawn from the database for blocking applications.

Displaying viewpoints

Viewpoints are locations that offer an extraordinary view of the environment and are therefore often a popular destination to visit. Such are tagged in OpenStreetMap with tourism=viewpoint. A significant percentage of those (>12 percent) are also tagged with a direction tag, indicating the direction in which the viewpoint offers a good view.

OSM-Carto has rendered viewpoints with a static point symbol for a long time, but is not taking into account the direction tag. Depiction of viewpoint directions has been pioneered by the OpenTopoMap style.

I am using SQL functions to calculate azimuth direction and viewing angle from the direction tag. Then i use those to pick and suitably rotate the best fitting symbol from a pre-generated table of symbols in the rendering database. Here is how this looks like for different variants of the direction tag:

Viewpoint display with direction visualization

Viewpoint display with direction visualization – click to see the corresponding direction values as label

And here are the pre-generated symbols from the symbols table these are rendered from:

Pre-generated symbols stored in the rendering database

Pre-generated symbols stored in the rendering database

There are a number of aspects about this rendering of the individual viewpoint symbols i want to point out. First: Viewpoints without a direction tag remain shown with the generic traditional symbol to distinguish them clearly from the direction=0-360 viewpoints. Second: The size of the symbols is slightly increased as you zoom in – to not take too much space at the lower zoom levels and be more clearly readable at the higher ones.

Symbol size change with zoom level

Symbol size change with zoom level – click for double resolution version

But also note that the size of the symbol varies depending on the angle depicted – for the narrow angles this is increased while for the 360 degree it is decreased. This makes the weight of the symbols of viewpoints with different angles more similar. Finally, also note the positioning of the name labels is adjusted to the symbol bounds so there is no excessive gap between symbol and label for the northward looking viewpoints.

In contrast to most other point symbols in the style, the viewpoints are rendered with a relatively subtle bright halo. This ensures a good readability of the fairly fine grained symbol design above strongly structured backgrounds – which are common near viewpoints with elements like natural=cliff, barrier=retaining_wall or natural=bare_rock polygons.

Cutting and blocking

The real innovation, however, comes from the way symbols are cut and interacting with other symbols. Like in case of trees, close-by viewpoints are cut away in their overlaps, ensuring a clean display without fully dropping the display of some of the symbols. At the same time, symbols that have priority over the viewpoints (because they are displayed at earlier zoom levels) are blocking the display of close-by viewpoints. Also viewpoints are blocking the display of lower priority symbols close-by. Mountain peaks (natural=peak) are treated differently, because they are commonly close to viewpoints or mapped in combination on a single node. Here symbols do not block each other, but the peak symbol is cut out from the viewpoint depiction to allow both to be displayed together. More on that later.

Interaction of viewpoints with other symbols

Interaction of viewpoints with other symbols: (1) non-overlapping close-by viewpoints, (2) overlapping viewpoint symbols being cut out from each others, (3) with peaks overlapping as well, either mapped separately or both tagged on the same node, (4) viewpoint symbol being blocked by a higher priority symbol and (5) viewpoint symbols blocking other lower priority symbols – click for double resolution version

There are two modifications of Mapnik necessary to accomplish this:

  • To support blocking between viewpoints and other symbols, support for anchor-cond was added for polygon symbolizers. This allows implementation of simple blocking by rendering a zero opacity marker symbolizer for the viewpoint like you’d do for an SVG symbol and then tie the rendering of the actual symbols to that via anchor-cond. That, however, does not work in combination with the cutting of the viewpoint symbols with one another, because even symbols that are blocked by other non-viewpoint symbols are getting included in the cutting operation – which they should not. To fix that we need to
  • add a way to access the list of anchors (the identifiers of symbols rendered successfully so far) from within SQL, where we are doing the cutting of the symbols.

While the first of these was fairly straightforward to implement, the second was a bit more tricky. This was mainly because of the way Mapnik handles database queries and rendering. Instead of running the query of each of the layers of the map style and then rendering its results before moving on to the next layer, Mapnik has the ability to asynchronously start all the queries up-front and then start rendering the map as soon as the queries return with the data. But this procedure, of course, does not work when the query of a layer depends on the rendering results of the layers rendered before. So, to use this feature of allowing access to the anchors from within SQL, we have to move to a strictly synchronous query and rendering procedure.

Practically the whole process looks as follows:

  • The main POI (amenity-points) layer is rendered with the viewpoints being represented by zero opacity placeholder markers, that have a suitable anchor-set parameter to document successful symbol placement.
  • A second layer (viewpoints) is set up to display the actual viewpoint symbols. It also performs the intersection with the other viewpoints. This layer is rendered after the POI layer and has an additional parameter in the Datasource: anchors_table: carto_anchors. This tells Mapnik to make available the list of anchors previously set in a temporary table in the database with the specified name. That table is then used in the SQL query of the layer in the form of an additional WHERE condition like AND EXISTS (SELECT 1 FROM carto_anchors WHERE "name" = 'viewpoint_' || osm_id::text).

Viewpoints and peaks

The combined rendering of viewpoints and peaks required some further modifications to the point symbol and label rendering system. The peak symbols are cut out from the viewpoint symbols just like the other viewpoints. In addition, the label position is adjusted as necessary to avoid overlap/blocking with either the peak or the viewpoint symbol in all the different direction variants.

Viewpoints and peaks tagged on the same node

Viewpoints and peaks tagged on the same node – click for double resolution version

Real world examples

While there are quite a lot of viewpoints mapped with direction tags, this tagging is usually patchy – not many regions have all or even most viewpoints tagged this way. None the less, here are a few examples of how viewpoints with specified direction look like in real world contexts. A double resolution version is linked from the images.

Reunion, Indian Ocean at z16

Reunion, Indian Ocean at z16

Provence, France at z17

Provence, France at z17

Kaiserstuhl, Germany at z18

Kaiserstuhl, Germany at z18

Conclusions

What i showed here is how dynamic symbol design can be implemented in maps with various forms of collision and overlap handling. In the presented case of viewpoints, this entails in particular the following components:

  • The different viewpoint symbols are not blocking each other, but instead their symbols are cut out to avoid overlaps in a manner similar to what i have demonstrated for trees in the past. Symbol priorities in cutting are based on elevation (when tagged) and view angle.
  • The viewpoint symbols are blocking and get blocked by other point symbols based on their priorities. Except for
  • peak symbols, which do not block viewpoints, but have their symbols cut out from the viewpoint depiction like viewpoints are cut out from each other. This includes cases where a node is tagged both as a viewpoint and a peak.
  • Labels are shown under the symbol with an offset dynamically adjusted to the geometry of the viewpoint visualization used.

To accomplish these things additional functions were added to Mapnik and Carto. One allows conditional rendering of polygon symbolizers, depending on the successful placement of other symbolizers, which are subject to collision detection. The other provides access to information on what symbolizers were rendered successfully in previous layers from within SQL via temporary tables. Combined with the use of zero opacity placeholder symbolizers this allows the implementation of the design features described.

The Mapnik modifications are available on Github. To build Carto with support for these new features you also need the modified mapnik-reference.

The modifications of the Alternative-colors style implementing the dynamic viewpoint rendering have been published as well.

The map design concepts demonstrated here and the modifications of Mapnik that enable these further contribute to implementing point 5 on my list of critical features map design work requires from the tools it uses. But keep in mind i am not a software developer, i do not aim for these features to be usable more generally, they are just a proof-of-concept.

The Musaicum West Asia

August 18, 2024
by chris
1 Comment

The Musaicum West Asia – extending 10m resolution image coverage

Last year i introduced the Musaicum EU-plus, a new 10m resolution satellite image mosaic of Europe, which pioneered a new, more automated classical mosaic production technique. I am happy to now introduce a new regional mosaic of West Asia with similar specifications.

The Musaicum West Asia

The Musaicum West Asia – click for larger version

The new mosaic covers the region that is commonly called West Asia ranging from the Suez Canal to the eastern border of Iran. In the north the image not only includes the whole Caucasus region but also parts of Ukraine, Russia and Kazachstan up to about 50 degrees latitude – Essentially the southern parts of far eastern Europe that are not included in the Musaicum EU-plus.

Suez Canal, Egypt

Suez Canal, Egypt

Rostov-on-Don, Russia

Rostov-on-Don, Russia

Golestan, Iran

Golestan, Iran

Quite a bit of work went into further refining the production technique to achieve good results in the diverse geographic settings that can be found in West Asia. In the Caucasus region you can largely find climate and vegetation conditions similar to mountains in Europe. Across much of the region, however, you find predominantly winter rain and a vegetation maximum in early spring, somewhat similar to southern Europe, but much more variable from year to year. Hence i used a significantly broader data basis here than for Europe ranging from 2019 to 2024 to be able to approximately depict the multi-year vegetation maximum. Compared to the Green Marble, which shows a multi-year average of the season of maximum vegetation this more strongly emphasizes the presence of vegetation. In a region like this where dried sparse vegetation is not well discernible from bare ground and where many parts will only see significant greening every few years this is highly beneficial to well work out the differences in ground cover and vegetation.

Sakaka, Saudi Arabia

Sakaka, Saudi Arabia

Euphrates Valley, Syria

Euphrates Valley, Syria

Towards Central Asia this seasonal characteristic is further constrained by winter frost which leads to a very short growth period in spring. In addition, in the south of the Arabian Peninsula, we have an influence from the Indian Ocean Monsoon leading to summer rain and a late summer/autumn vegetation maximum.

Ibb, Yemen

Ibb, Yemen

Most other satellite image mosaic productions ignore all of that and do not regard West Asia as a particularly challenging region. Cloud incidence is generally low so there is plenty of data available, overall, that can be used to assemble a cloud free coverage without much effort. The result is a dry season image which displays most of the region in fairly uniform and structureless brown-gray colors. This, however, creates a wrong impression of a region where there are significant nuances in natural vegetation even in many drier parts.

So the Musaicum West Asia is not only the highest quality image of the region in that resolution class, it is also, to my knowledge, the only one that consistently shows a vegetation maximum rendering and not predominantly a dry season depiction.

You can find the product description and more sample images on the Musaicum West Asia product page.

Gaza Strip

Gaza Strip

Bahrain

Bahrain

Caucasus mountains, Georgia/Russia

Caucasus mountains, Georgia/Russia

Dark area in the Caribbean Sea in the Green Marble 4

July 21, 2024
by chris
1 Comment

Color representation and noise in satellite images

In my announcement of the Green Marble 4 global satellite image mosaic i mentioned that i am moving to a 32 bit per channel color representation in processing of the data. I here want to explain the background of this development a bit.

Color representation basics

Color images in computer systems, for example on websites like this, are commonly represented with 8 bit per channel. That allows for 256 different levels of intensity or for 16.7 million different colors in a color image as it is commonly advertised. That is fairly coarse and only works reasonably well because these intensity levels are defined non-linearly in a way that happens to roughly match the physiology of human color perception. I won’t go into the details of that here – it has to do with the physical characteristics of the CRTs which were used as computer displays.

Anyway – for recording images in cameras this has long been insufficient and most digital cameras use 12 or 14 bit color representations (equivalent to 4096 or 16384 levels), older models sometimes also 10 bit (1024 levels). Earth observation satellites roughly match this development.

These raw values are typically cast into a 16 bit representation for further processing – both in digital photography and with earth observation satellites. When you download optical satellite imagery today for analytic applications this is almost always in the form of 16 bit per channel data.

Reflectance representation conventions

The most common form of distributing optical satellite imagery is as reflectance values. Reflectance is a unit-less quantity where a value of 1.0 means at a certain point in the image as much light is recorded as you’d expect to come from a horizontal surface that is a perfect diffuse reflector under the lighting conditions the image is recorded at.

By almost universal convention these values are scaled with a factor of 10000 for representation in 16 bit values. Sometimes, in addition, an offset is applied as well to be able to represent negative reflectances in unsigned 16 bit values – but that is not of much interest here.

Many readers might ask: Why use factor of only 10000 when the full range of 16 bit values (65536 levels) is available. The reason is that reflectance values routinely exceed 1.0. This can be easily understood based on the definition i gave above. If you have a low sun position a mountain slope facing the sun will reflect significantly more light than a horizontal surface. So even if it is not a perfect diffuse reflector it will frequently exceed a reflectance of 1.0. Hence you practically need significant headroom above a reflectance of 1.0 – which is why a scale factor of 10000 makes sense.

Noise

The next question you might ask: Is this representation of reflectance values (integers with a scale factor of 10000) sufficient for an accurate representation of the recorded data?

The answer to that is yes – as long as

  • we are talking about individual images.
  • we are talking about data in the visible range of the spectrum.

And most importantly: This is also still going to apply in the future with further improvements in sensor technology.

The reason for that lies in the Earth atmosphere. Whenever a satellite image is recorded it will inevitably contain not only light from the Earth surface but also from the Earth atmosphere. We can try to compensate for the atmosphere part when processing the images – but that compensation is for the bulk effect only. It does not eliminate the noise.

All signal recorded by a satellite image sensor, whether it comes from the Earth surface or from the atmosphere, is subject to noise. And with noise here i do not mean noise from the sensor or from the signal processing in the satellite, i am talking about noise that is already present in the light before it reaches the satellite. This noise is unavoidable and inherently limits the dynamic range of satellite image data. Because of that a dynamic range of 10000 in the data representation is – under the constraints i listed – more than sufficient for an accurate representation of satellite image data.

Aggregation

That is not the end of the story of course. The photon shot noise i discussed follows a well known mathematical characteristic: It is proportional to the square root of the signal. In other words: You can reduce the amount of noise relative to the signal and thereby improve the signal-to-noise ratio and the dynamic range by recording more light. But because of the square root relationship you need a lot more light to have a substantial effect.

There are two potential ways to exploit this possibility:

  • You can build larger satellites with larger optics. That is rather costly of course.
  • You can combine multiple images.

The latter is what i do when producing a pixel statistics mosaic like the Green Marble. And when combining thousands of individual images in areas with very low surface reflectance you can reach the limits of the standard integer representation of reflectance values with a scale factor of 10000. Here is a practical example to illustrate that. First the area in standard tone mapping.

Caribbean Sea rendering in the Green Marble 4 with standard tone mapping

Caribbean Sea rendering in the Green Marble 4 with standard tone mapping

This shows several fairly dark reefs in an even darker sea area in the Caribbean Sea between Jamaica and Nicaragua/Honduras. With a brighter tone mapping this becomes better visible.

Caribbean Sea rendering in the Green Marble 4 with brighter tone mapping

Caribbean Sea rendering in the Green Marble 4 with brighter tone mapping

The open ocean away from the reefs is of a very dark blue color with an extremely low red color reflectance. And if we further contrast emphasize this area we can actually get to see the residual noise and work out the difference between the Green Marble 3 and 4 here.

Caribbean Sea rendering in the Green Marble 4 with strongly emphasized contrast

Caribbean Sea rendering in the Green Marble 4 with strongly emphasized contrast

Caribbean Sea rendering in the Green Marble 3 with strongly emphasized contrast

Caribbean Sea rendering in the Green Marble 3 with strongly emphasized contrast

In comparison the Green Marble 4 has a lower noise level overall but in particular note it lacks the posterization in contrast to the Green Marble 3. The banding visible in both images is the result of images with different viewing direction being combined with less-than-perfect compensation for the differences in view geometry.

Conclusions

Practically reaching the limit of standard integer surface reflectance representation in visible range satellite image aggregation – as i have demonstrated here – marks a significant milestone in satellite image processing methodology. So far practical relevance for users of the Green Marble is small. Even users of the linear surface reflectance data who do their own custom color processing will in most cases be able to work with the 16 bit version as before without measurable disadvantages.

What this, however, demonstrates is that pixel statistics mosaicing methods – in addition to the main function of assembling a uniform, representative depiction of the Earth surface from individually incomplete and flawed images due to less-than-perfect viewing conditions – are in principle capable to generate images of higher inherent quality than the original individual images that are used as a source.

Green Marble 4 with synthetic relief shading

July 20, 2024
by chris
0 comments

Announcing the Green Marble 4

I am happy to announce another update of my global satellite image mosaic – the Green Marble.

The Tian Shan mountains in the Green Marble 4

The Tian Shan mountains in the Green Marble 4

Version 4 of the Green Marble does not differ as much visually from its predecessors as version 2 and 3 since it continues to use the same principal data sources as before, that is Sentinel-3 OLCI data and MODIS images. It still represents a significant improvement in quality, in particular with substantially reduced noise levels in the water depiction. It updates and expands the source data for both the water and land depiction until late 2023.

Since the range of dates of source images processed is quite large to ensure high quality results, short term changes on the Earth surface do not usually manifest strongly in the Green Marble. So the data update is primarily visible in the form of long terms trends, which are often relatively subtle. Here an example of irrigation farming in southern Egypt:

Southern Egypt in Green Marble 3Southern Egypt in Green Marble 4

Changes in irrigation farming in southern Egypt (large: GM 2, GM 3, GM 4)

In addition to the data update there are also significant changes in data processing, fixing various smaller flaws and improving color uniformity. The biggest change is the move to a 32 bit per channel color representation in assembly of the mosaic. This avoids accumulation of rounding errors that has led to some color distortion in the previous versions on the water depiction and it avoids running into limitations of the standard 16 bit integer reflectance value representation in satellite image processing. I am going to discuss the background of this in a separate blog post.

Amazon river mouth in Green Marble 3Amazon river mouth in Green Marble 4

Improvements of coastal water color depiction between Green Marble version 3 and 4 (large: GM 3, GM 4)

The Green Marble 4 global satellite image mosaic is available for licensing for use in your projects now. Have a look at the Green Marble product page for more information and sample images.

The demo maps with the Green Marble in different rendering variants are updated as well:

Eastern India and Southern China in the Green Marble 4

Eastern India and Southern China in the Green Marble 4

Patagonia in the Green Marble 4

Patagonia in the Green Marble 4

Twenty years OpenStreetMap - revisiting observations and predictions

July 17, 2024
by chris
3 Comments

Twenty years OpenStreetMap – revisiting observations and predictions

Five years ago i wrote about the 15th anniversary of OSM and my outlook into the next fifteen years. We now have the 20th anniversary. This is a bit early to review my predictions (with merely 1/3 of the time span covered having passed) – still i want to newly consider some of the topics i contemplated five years ago.

My big topic for the 15th anniversary was the idea that a separation might occur in the future between the original core idea of OpenStreetMap, the sharing and communication of local geographic knowledge by a cross cultural community of mappers through egalitarian, self determined cooperation and the project with the name of OpenStreetMap.

This scenario consisted of two components:

  • the (back then increasingly visible) trend for OpenStreetMap to move away from the fundamental ideas and values it was created on.
  • the possibility that this development might lead people to share and exchange their geographic knowledge increasingly through means outside of OpenStreetMap.

I want to look at both parts here a bit through the somewhat different perspective i have today, five years later.

OpenStreetMap changing

Previously my hypothesis that OpenStreetMap is – at large – moving clearly away from its original ideas and values was primarily backed by the observable trends in mapping, that the data in OpenStreetMap as well as the edits made were increasingly not founded in personal local knowledge of the people making the edits and their own, intrinsic desire to share that knowledge. It seems quite clear to me today, five years later, that this trend has not reverted, although i also have the distinct impression that is has not accelerated either. OpenStreetMap continues to attract mappers sharing their local geographic knowledge and it is good to see that this happens with an increasingly diverse geographic and cultural distribution. Bulk data additions and systematic volume edits not backed by local geographic knowledge have expanded significantly as well – hence no reversal of the trend – but it can’t be said that they are substantially further displacing local knowledge based mapping at this time. Or in other words: You could say the situation has stabilized a bit.

So much for the actual development of mapping in OpenStreetMap. Things look very different when you look at how OpenStreetMap is communicated in public and hence how the wider public perceives OpenStreetMap these days. This means public communication by individual community members, by the OpenStreetMap Foundation as well as – and this is the largest part probably – by third party organizations, be that businesses of all sizes, public institutions or other organizations of all kinds.

If you study this body of public communication about OpenStreetMap these days then the traditional ideals and values of OpenStreetMap are almost completely absent. You can find these being discussed in communications of individual community members with no organizational affiliations as well as some academic studies from the social sciences sector. But the vast majority of organizational communication about OpenStreetMap (and that is what tends to have the largest reach) presents OpenStreetMap these days as a collection of useful geodata that is largely produced by volunteers (with volunteers meaning unpaid workers and not implying either local knowledge or self-determination in what they map and how they map it).

Recent public communication of the OpenStreetMap Foundation is a good example for that – the page created for the 20th anniversary is framing OpenStreetMap exactly along these lines.

Normally you’d expect such a clear and almost universal trend in public communication to lead to mappers increasingly internalizing this framing of OpenStreetMap. Interestingly this does not seem to be the case though. As mentioned OpenStreetMap these days continues to recruit new mappers who map in the traditional way sharing their local knowledge of the areas around them in a self determined fashion and in egalitarian cooperation with the other mappers around them. They must have learned this either

  • from legacy documentation that still exists
  • from other mappers by imitation/direct communication
  • from their local community through decentralized small group communication
  • through their own bootstrapping process as the way to approach mapping in a way that seems natural to them.

That this is possible and practically happening at scale despite organized public communication in most cases drawing a very different picture of OpenStreetMap is a very positive trend.

The separation

The second element of my five years old prediction was the idea that the people who value the original idea of OpenStreetMap of cooperative collection and sharing of local knowledge might move to doing this increasingly outside of OpenStreetMap. So far there is not much concrete tendency in that direction visible. And, as discussed already, at the moment the traditional mapping values seem astonishingly strong in practical mapping in OpenStreetMap.

What this, however, indicates – and this is different from what i discussed five years ago – is that there is a widening schism within OpenStreetMap between how practical mapping and the social interactions that facilitate it actually happen and the perception of OpenStreetMap and mapping in the organizational world around OpenStreetMap.

It is best not to see this schism as a complete separation of communication bubbles with one being completely disconnected from the other, it is more like a strong cultural divide.

If this schism is the precursor to a true separation as discussed in the past or if it can be a stable way to combine the social necessities of cross cutural cooperation in the form of the traditional values of mapping with the economic necessities of capitalism is not yet decided. It will probably largely depend on if people on the organizational side of the schism develop enough respect and appreciation for the social necessities on the other side. I am sceptical here (as are probably many of my readers). But i also like to point out that a lot of the organizations around OpenStreetMap have a lot of smart people working for them who are, in principle, capable of understanding these fragile inter-relationships and accordingly are able to make responsible decisions. The question is going to be: Will they?

An experiment in social cooperation

The ultimate long term question (and long term here means this is not a matter that can be determined reliably in a 15 or 20 years time frame) is if the cooperative collection of local geographic knowledge across cultures in an egalitarian and self determined fashion is something that can work at scale. As hinted at in the comments of my 15th anniversary post this is not clear, even if it has worked amazingly well during much of the past 20 years. The null hypothesis on this would be: It can’t, OpenStreetMap will necessarily either vanish or turn into a classical organization with a social hierarchy (i.e. become non-egalitarian) – at best with a framework of representative democracy of some sort.

I would very much like to see the thoughts of others on this question, especially if they differ from the null hypothesis.

Trends for the future of map design in OpenStreetMap

April 20, 2024
by chris
5 Comments

Trends for the future of map design in OpenStreetMap

After having looked at the history of digital map design in general in the first part of this short series and at the current situation of map design in OpenStreetMap in the second part, i am now, in this last part, going to specifically look at what the OpenStreetMap Foundation seems to be pursuing in that field at the moment and in what direction that is likely going to go.

The OSMF’s history in providing maps

The OSMF has for some time followed a quite erratic course with regards to rendering and providing maps for the OSM community. Historically, a significant percentage of the OSMFs IT resources have gone into rendering the standard map layer on openstreetmap.org. And while there has been discussion and ultimately a policy was developed on who is allowed to use that tile service under what conditions, the OSMF has never really stated a clear purpose for that tile service or the conditions for choosing OpenStreetMap Carto as the style for it. There is a set of requirements for tile layers to be included on osm.org – which are mostly hosted externally – but nothing to this date defines what qualifies OSM-Carto to be the style the OSMF invests substantial resource in to render as a map. I have, as an OSM-Carto maintainer, repeatedly criticized this omission, because it also means the OSMF takes no responsibility for the decision to do that.

Back in 2021 a poll the OSMF board made of the OSM community included the question about the future of map services provided by the OSMF, for the first time indicating to the OSM community that they are contemplating making changes in that field. The question was very vague and ill-defined (which i criticized back then) and the answers were accordingly inconclusive. The term vector tiles had already become a magic word back then without much of a well defined meaning but where everyone seemed to project their hopes and dreams into.

Later that year the OSMF announced that they would – on the operational level – do some testing in that field. But even upon inquiry no overall strategy was mentioned this initiative would be part of and it became clear that this was an initiative from operations and not part of a larger strategy of the OSMF at that time. The interesting part here is that, while this announcement was only about a relatively small operational investment into testing with nothing in terms of money being involved at all, there were quite immediate reactions from economically interested parties substantially lobbying for their pet frameworks being considered by the OSMF as well.

Long story short: Back in 2021 it was clear that (a) the OSMF had no substantial strategy for the future of the map services provided by the OSMF, (b) there were already a lot of people projecting their hopes into the vague concept of vector tiles and loudly calling for the OSMF to make these hopes to magically come true while the whole matter became further political because economically interested parties began to see the chance to profit and started lobbying the OSMF to adjust plans in their favor – and (c) that actual map design was not going to play even the most marginal role in the whole process. After that there was not much publicly visible progress for some time. The strategic plan decided on in July 2021 included only a development platform for vector tiles by volunteers – which seems to reflect the board’s reading of the survey results back at that time to leave that to the community to pursue.

Something changed at some point in 2022. In November there was a short mentioning of the matter in one of the meeting minutes. From then on a fairly remarkable timeline in discussions and decisions can be reconstructed from the minutes that paints quite a rich tapestry of the inner workings of the OSMF. Although this is of little relevance for the topic of this blog post i highly recommend taking the time to read through all of that.

The meaningful information w.r.t. the topic of this post is that, when the OSMF contracted Paul Norman to do the work that these days seems to be coming to an end, they still apparently did not actually have a strategy on their own regarding the OSMF’s map services. And it is unclear if they have de facto adopted what Paul sketched in his proposal (published here, which is rather vague in terms of strategic goals) as the OSMF’s de facto strategy or if they have meanwhile developed some other kind of plan. What is, however, quite self evident is that they will certainly want to roll out the results of spending EUR 42k in some form. And this is what i will base the following considerations on, i am not making any further assumptions beyond that.

The map serving process

What i am going to do here is discussing what that likely means for the OSM community. To explain that, here first how the current setup of the standard tile layer on openstreetmap.org is run (based on OSM-Carto):

The practical map rendering process and mapper feedback loop with OSM-Carto (or similar styles) - link goes to larger version

The practical map rendering process and mapper feedback loop with OSM-Carto (or similar styles) – link goes to larger version

I illustrated which parts of this setup are controlled by who: The gray rectangle titled Map design is controlled by the OSM-Carto map designers and maintainers. The Mappers using the map decide which parts to view and what to map and the OSMF (Operations) is in control of the OSM database and the map rendering chain until the rendered tiles are delivered to the map user. Some might be irritated because i split up the rendering process with the tile data (internal only) to make the process better comparable to the vector tiles one shown below. This split is not visible to operations, it exists only internally in Mapnik. It is, however, visible to the map designer who separately designs the query to generate the tile data and the rules to draw the tile from this data as formulated in the CartoCSS code.

The important thing to note is that the rendering database in case of OSM-Carto is deliberately kept generic. This allows you to render several completely different styles from the same database, to make almost any style changes without reloading the rendering database and it also simplifies systematically testing the style as a whole because the database import does not need to be part of this testing.

In case of OSM-Carto the processes within the map design domain are relatively simple. For comparison: Here is how this looks like with the Alternative Colors style.

Differences with a more complex style like the AC-Style - link goes to larger version

Differences with a more complex style like the AC-Style – link goes to larger version

Like a lot of other map styles the AC-Style extends the rendering database with some style specific data. In my case this is some SQL functions, mostly for tag interpretation and geometric processing, and the tree symbology (which needs to be in the database to work around limitations in Mapnik – see the blog post discussing tree rendering). But this is static data, it does not change when the main database content changes.

Here is now how the whole process would change if Paul’s work would be rolled out as developed (based on my understanding from what has been communicated – i have not actually tested this setup myself).

The expected map rendering process and mapper feedback loop with minutely updated vector tiles (as currently pursued by the OSMF) - link goes to larger version

The expected map rendering process and mapper feedback loop with minutely updated vector tiles (as currently pursued by the OSMF) – link goes to larger version

From the operational side this means – in a nutshell – that the last part of the traditional rendering chain as shown for OSM-Carto above is cut off and outsourced to the map user. This substantially reduces the resources required for serving the tiles and at the same time allows customizing the actual rendering of the tiles without this requiring additional resources.

That is – in short – the whole magic of vector tiles. In reality it is of course not all that simple – for multiple reasons.

First: For the outsourcing of rendering to the map viewer to work you need to massively reduce the volume of data transferred and involved in rendering. In the diagrams shown i indicated high volume data with the wide red arrows. And the tile data internally processed by Mapnik is typically even substantially larger volume than what is in the rendering database (which tends to be many hundred GB for a full planet import) because the queries do geometry processing that frequently adds data volume. In case of the client side rendered vector tiles this volume needs to be massively boiled down with the help of lossy data compression. And despite these efforts (and their negative side effects) the bandwidth requirements for viewing one of these styles is usually much higher than for a traditional raster tile based map.

Second: The rendering database is not generic any more. That is not inevitable but practically i am not aware of any client side rendered vector tile setup that uses a generic rendering database.

Third: Map design as a domain in control of parts of the whole process is gone. That requires a more detailed discussion of course.

Above, i explained that one of the main promises of client side rendered vector tiles is, that the actual rendering can be customized for every map viewer. But the ability to customize the actual rendering does not mean that you can fully customize the map design. In reality, most of the map design decisions are implemented in the vector tile generation, partly because of the need to keep the tile sizes small. This means that everything that is not necessary for the specific map design chosen needs to be stripped from the data. Or seen the other way round: Trying to introduce any level of map design genericity into client side rendered vector tiles will often have massive impact on bandwidth requirements.

Practically, all client side rendered maps i have seen so far have a division between the people/organization in control of the vector tile generation and those developing the client side part of the map design (which i labeled style design in the diagram). And the vector tile generation furthermore tends to be under control of software developers and not map designers – hence i labeled that part software development. In the setup Paul has developed this part is further split into multiple components under control of different people.

What this means is that map designers would, in such a setup, get marginalized to mere style designers who can play a bit with colors and line width selection and designing point symbols. The real control over map design would sit with the software developers in control of schema and vector tile production, whose main concern is not going to be map design but operational efficiency and keeping the size of the vector tiles small – the technocratic direction as i described it in my discussion of OSM-Carto. Theoretically, you could argue, you could put the style designers hierarchically on top of the software developers in control of the vector tile schema and require the latter to adhere to the wishes of the former. But in today’s OSMF this has zero chance to happen as evidenced by the fact that map design aspects played no role at all in the whole process of developing this setup. As i wrote in the discussion of OSM-Carto, the only way from my perspective for sustainable progress in map design would be to bring the technocratic and the artistic direction together in a single cooperative project with clear common goals everyone subscribes to.

Beyond the magic

So what remains of the magic of vector tiles and its promises in light of this? The lower resource requirements for operating a tile server are indeed there. But only if you look at it purely from the micro-economic perspective of the tile server operator. From a macro-economic perspective the resource requirements are actually higher in sum because the actual rendering is done separately for every map user. Client side vector tiles is not Tiles@home.

But don’t get me wrong, there are of course benefits of client side rendered maps. One major one is the ability to implement more interactive maps based on the semantic data transferred to the client. And if you invest the bandwidth to include additional data in the tiles, customizing the map can be a real benefit. But there is no magic here, this all comes with costs.

The real question, however, is, what this means for map design. If i put aside the organizational division between style design and vector tiles generation for the moment what remains is what changes if you design for a client side renderer compared to a Mapnik based rendering chain. There are still several really big issues i am afraid. I already explained those in more detail in my post on what rule based map design requires from the tools it uses.

But now comes the twist some readers probably don’t expect: All of this does not mean the OSMF did something wrong by investing into implementing a toolchain for minutely updated vector tiles. Of course apart from the board epically failing in the procurement process for this on so many different levels. If i ignore that for the moment then investing in this is a completely sound decision. Client side rendered vector tiles are – independent of their practical relevance for the OSM community – currently the main technique for producing interactive web maps used by commercial map providers. And these commercial actors, while having invested substantially in the toolchains for this style of map production, have very little interest in minutely updates. These are a relatively unique interest of the OSM community. Hence it is a reasonable idea that the OSMF should chime in to serve this specific need.

If the OSMF now feels that this reasoning is not sufficient to justify the project and feels instead the need to roll out a cheaply thrown together postmodern OpenMapTiles/Mapbox clone with minutely updates as the main map on openstreetmap.org that would be regrettable. But it would not really mean much in the grand scheme of things. It would surely not solve the structural problems in practical map design in and around the OSM community that i explained in the previous post.

The current state of map design in OpenStreetMap

April 18, 2024
by chris
5 Comments

The current state of map design in OpenStreetMap

In the previous post i discussed a bit the history of digital map design, in particular also outside of OpenStreetMap. I also wrote in the past about the history of OpenStreetMap Carto, the cartographic face of OpenStreetMap and without doubt the most influential map design project in OpenStreetMap – today as well as historically. But it is neither the only map design endeavor in and around OpenStreetMap, nor is it going to be in its current position forever. In this post i am going to discuss various other map design projects – historic and present – that exist in and around OpenStreetMap and that substantially have shaped or are shaping map design in OpenStreetMap.

Beyond OSM-Carto

I am going to leave a more concrete discussion of commercial OSM based maps out of this post. As discussed in the previous post about the history of digital map making, the domain of web maps is strongly dominated by big tech companies and their digital platforms. These have mostly optimized their maps for cost efficiency – in terms of operational costs for serving the maps just as in terms of development costs. Summarily, these maps are largely simplistic in design, dominated by relatively coarse visual elements and with a lack of conscious attention to detail and semantic nuances. Seen as art they feature elements in particular of primitivism and dadaism – as well as a kind of digital technological surrealism. It is fitting in my opinion to say that these maps form the core of post modern cartography.

Maptiler Mapbox

Examples of commercial OpenStreetMap based web map styles: Maptiler and Mapbox

Because of their dominance in today’s practical use of maps they have a significant influence on contemporary map design but at the end of the day they contribute very little to the advancement of the state of the art in map design in general. The big problem is – as i have pointed out in the past – that the technological basis of today’s digital map design is largely financed by these companies, which hampers the development of map design beyond that.

What i do want to mention in that context is the probably most important precursor of today’s corporate cartography, the avant-garde of post modern cartography you might say. That is quite clearly the web maps work of Stamen from – in parts – more than 10 years ago.

Stamen Watercolor Stamen Terrain Stamen Toner

Early map styles by Stamen

There is another quite substantial set of maps that are commercially developed, usually by smaller businesses, for specific use cases that i am also going to skip here. For most of these, the main connection to OpenStreetMap is that they use OSM data. Collectively, this body of map design work is quite significant for the OSM community as it demonstrates a broad range of use cases for OpenStreetMap data to the wider public and the mapper community. But individually, these maps tend to be of rather limited reach, compared to the maps of the big tech platforms, their individual significance for OpenStreetMap design wise is rather small, especially also since the vast majority are not open source. Reviewing this set of map styles from a map design perspective would probably also be interesting, but i consider this out of scope here.

And finally i am also going to leave out various specialized QA maps and overlays produced by the OSM community – some of these feature noteworthy design techniques – but overall, their purpose is so much different from that of normal maps that they are not really that relevant here.

Historically the origin of OSM based map design was Osmarender. Osmarender was the primary platform of visualization of OSM data and mapper feedback in the early days of the project and has enormously shaped mapping in OpenStreetMap as well as OSM based map design – which can still to some extent be observed in present day maps. Unfortunately, documentation of these early days of OpenStreetMap is generally still very poor.

Historic example of the Osmarender map design

Historic example of the Osmarender map design

Especially also together with the Tiles@home framework facilitating distributed rendering of maps, Osmarender was a fairly revolutionary project which anticipated many later developments and already tried to address some problems that we still struggle with today. It was also the first attempt of the OSM community at cooperative map design and the experiences made with that significantly influenced the approach to subsequent projects.

I won’t discuss OSM-Carto and its XML predecessor style here – for that please refer to my previous posts. I am going to look at the various OSM-Carto derivatives a bit though that form a significant part of the OSM map design ecosystem.

I will start with the German OSM-Carto fork, maintained by Sven Geggus. This OSM-Carto variant features various color and design differences to the parent project, in particular

  • A road coloring scheme derived from traditional German road maps
  • Various different symbols specific to German cartographic culture
  • Green instead of purple boundaries
  • Rendering tracktype=1 tracks similar to service roads
  • Different rendering of footways/cycleways/paths

In addition, the German style features a sophisticated label language selection and transliteration system. This is implemented during database import, meaning the style uses a different database schema than OSM-Carto.

The German OSM-Carto fork is the only of the larger forks that is regularly kept in sync with upstream changes. This constrains the ability to implement larger and more invasive design changes because such would make synchronizing with upstream changes more difficult. Some of the color changes in land cover and water coloring, that have been implemented in the style at some time, have been dropped again because of that.

German OSM-Carto fork

German OSM-Carto fork

The French map style, in contrast to the German one, is a hard fork that has split off from OSM-Carto a longer time ago. It therefore retains in particular many of the original colors of early OSM-Carto for buildings, landcover and roads. It is maintained by Christian Quest and, apart from some symbology adjustments to French cartographic tradition, comes with a number of innovative feature extensions and rendering techniques. Some have been adopted by OSM-Carto in the meantime – like the depiction of trees and the rendering of details on golf courses. Others remain unique to the french style. Specifically:

  • The rendering of street lamps using circular symbols similar to those for trees.
  • Rendering of pedestrian crossings on roads.
  • Rendering of sport specific linework on sport pitches.
  • Normalization and abbreviation of French names.
  • Low zoom landcover depiction based on scaled raster rendering.
French OSM-Carto fork (high zoom level)

French OSM-Carto fork (high zoom level)

French OSM-Carto fork (low zoom level)

French OSM-Carto fork (low zoom level)

Then there is the AJT-Style by Andy Townsend as an OSM-Carto fork with a distinct British and pedestrian focus. The AJT-Style departed from OSM-Carto design wise even before the French one. It is noteworthy for being likely the most feature rich style in OpenStreetMap in total, with a strongly differentiated rendering of various attributes of roads and paths (like British rights of way) and a huge number of different classes of point features being shown with point symbols.

Design wise the style is relatively traditional with pixel painted raster symbols. And much of the symbology uses relatively cryptic and not necessarily very intuitive encodings to represent semantic differences. In contrast to OSM-Carto, which is explicitly designed to be intuitively usable without a legend, the AJT style deliberately accepts to require a legend to be used and puts formal clarity in differentiation above intuitive readability.

Technically, the most remarkable aspect of the AJT-Style is the way tag interpretation is performed at data import through successive conditional in-place manipulation of attributes in a more than 10k lines long LUA script.

Britisch/Rural pedestrian focused OSM-Carto fork by Andy Townsend

Britisch/Rural pedestrian focused OSM-Carto fork by Andy Townsend

And for completeness: There is of course also my own Alternative Colors style – which i have discussed extensively here in past posts so i will only mention it in passing in this post.

Finally, i want to mention one OSM-Carto fork that is not strictly open source – which i cover here primarily because it is meanwhile featured on the OpenStreetMap website. I am talking about the Tracestrack style. This is an OSM-Carto derived collage of design components from OSM-Carto and OpenTopoMap remixed/overlayed with separately designed additional features and design modifications. The add-on features and design modifications have a distinct urban European cultural focus – contrasting quite clearly with the global orientation of OSM-Carto.

I call the Tracestrack style not strictly open source because, although they have published some code components of the style and licensed them under an open license, this entails only parts of the style. The openly licensed parts do not allow you to independently reproduce the map they offer. While this is in principle totally legitimate as far as OSM-Carto is concerned (which is CC-0), the only partial open licensing is potentially at odds with the more constraining license of OpenTopoMap (CC-BY-SA).

OSM-Carto derived style by Tracestrack

OSM-Carto derived style by Tracestrack

So far these were OSM-Carto forks – which are implementation wise based on the OSM-Carto code. There is another class of OSM-Carto derivatives that are mere OSM-Carto look-alikes which try to imitate the OSM-Carto design appearance-wise to some extent but which do so with an independent implementation. The depth of imitation in those varies, some merely try to resemble the color scheme in broad strokes while others go into more depth to resemble the design of OSM-Carto. The ESRI client side rendered demo is one of the earliest and the most ambitious projects in that regard.

Client side rendered OSM-Carto look-alike by ESRI based on a proprietary tool chain

Client side rendered OSM-Carto look-alike by ESRI based on a proprietary tool chain

The trend to imitate OSM-Carto design in technically independently developed map design projects is testimony to two things:

  • That the visual design of OSM-Carto has turned into a kind of reference standard in OSM map design meanwhile independent of the concrete technical implementation.
  • That designing a rich OSM based map similar in scope to OSM-Carto is a lot of work that requires significant map design skills and therefore avoiding this substantial investment and building on what OSM-Carto has developed over many years is an attractive option (which does not seem to become less attractive with the increasing age of the project and the design).

Outside the range of OSM-Carto derivatives we have another particularly noteworthy project: OpenTopoMap. OpenTopoMap is being developed and maintained by Stefan Erhardt and Max Berger. Technologically, the project is remarkable because it is the only actively maintained style written directly in Mapnik XML. This makes editing the style rather awkward but compared to CartoCSS based styles it has the advantage that it can make full use of the Mapnik feature set while CartoCSS styles are limited to the subset of Mapnik features supported by Carto.

Design wise OpenTopoMap is different from other OSM map styles because it aims to resemble and follow many of the design paradigms of traditional topographic maps. This means in particular deliberately using only a limited set of colors. But it goes beyond a mere imitation of traditional maps, it extends the traditional design paradigms with OSM specific ideas.

OpenTopoMap also features a number of other substantial innovations:

  • context dependent orientation of train station and saddle symbols
  • visualization of view directions of viewpoints
  • importance rating of peaks and parkings based on isolation
  • curved labeling of polygons
OpenTopoMap at low zoom levels

OpenTopoMap at low zoom levels

OpenTopoMap at higher zoom levels

OpenTopoMap at higher zoom levels

In context of maps with relief depiction it is also important to mention the oldtimer in that context – TopOSM by Lars Ahlzen. The map is no more operational and its development ceased more than ten years ago, but it was a significant step in the development of OSM based maps with relief depiction.

The historic TopOSM style

The historic TopOSM style

Another noteworthy contributor to the OSM map design ecosystem is the style of freemap.sk – an outdoor oriented map style that is written in a unique mixture of TypeScript and Mapnik XML by Martin Ždila. It is one of the most carefully designed OSM based map styles with relief shading (which is otherwise often not very well executed in Mapnik based styles)

The freemap.sk outdoor map style

The freemap.sk outdoor map style

Then we have as a relative newcomer to the field of OSM based maps, the OpenStreetMap Americana project. This is a client side rendered style based on the commercially developed and maintained OpenMapTiles data schema. Because OpenStreetMap Americana has been newly created and not derived from a pre-existing map style it is not yet really a complete map style. Technically, it is fairly unique as a style written in Javascript – some would even say it is therefore strictly seen not really a map style but a piece of software implementing map rendering on top of Maplibre GL as a rendering library.

Its use of the OpenMapTiles data schema and client side rendering through Maplibre GL design wise puts the project relatively close to the post modern ideas of corporate maps discussed above. Since the US cartographic tradition that Americana tries to connect to in its design even in the pre-digital manifestations has substantial post modern elements (much more than European cartography at least) this is a relatively good match. But it is also visible, that despite Americana being a relatively young and not yet feature complete project by its own goals, it already struggles significantly with the technical limitations of the map rendering framework used and the commercially developed data schema it is built on.

OpenStreetMap Americana style

OpenStreetMap Americana style

One particularly noteworthy design innovation of OpenStreetMap Americana is the highway shield library it contains. Pictorial highway shields are an important component of the US cartographic tradition and Americana meanwhile contains the most extensive collection of highway shield designs in digital form, with coverage way beyond the US. This work is unfortunately fairly strongly tied to the Javascript based framework of the style and therefore not easy to reuse in other map design projects.

I also want to mention Map Machine by Sergey Vartanov – which is both a map rendering engine and a map style and is noteworthy in particular because of the extensive set of point symbols for a highly differentiated depiction of features in the map that competes with the AJT-Style mentioned above in that regard.

Map Machine rendering example

Map Machine rendering example

And finally this list of maps is closed with another rather unusual project – the Straßenraumkarte Neukölln by Supaplex030. This is kind of a personal niche project so far, being limited to a small area in Berlin and – while it is in principle open source – not really being suitable for an independent deployment. But it is – map design wise – pretty innovative. It is more of a feasibility study on how large scale map design based on OSM data could look like.

Straßenraumkarte Neukölln

Straßenraumkarte Neukölln

Conclusions

So much as a glimpse into the current state of map design in and around the OpenStreetMap project. This is not meant as a complete list, more like a cross section with the aim to include most of the projects where major innovation in map design is happening or has happened during the past years in direct connection to OpenStreetMap. Still: If anyone feels that i have missed an important project please mention it in the comments below.

What does this tell us? That there is a reasonable diversity of map design work to be found around OpenStreetMap if you look carefully enough, although much of this is in a pretty precarious situation. Many of the projects mentioned show no or very little activity in recent years and practically all of them in some way struggle with the technical limitations of the tools used and quite a few resort to pretty exotic methods of script generating style rules to be able to develop the design they use. Also worth pointing out is of course that map design work in and around OpenStreetMap is massively dominated by European and North American projects and designers – much more so than mapping and also software development.

This situation is quite definitely not due to the lack of people interested in and with the talent and ambition to work in map design. The problem is that of the countless people that start looking into designing OSM based maps every month very few reach a level where they actually become innovative and produce something that is expanding the state of the art in OSM based map design and could therefore realistically make it on the list here. The reason for that is the map design tools we have are universally insufficient for that. Or in other words: Essentially, everything that can be reasonably done with the tools at hand for map design and map rendering has been already done a long time ago and to actually do something new you typically have to navigate around an obstacle course of limitations and flaws in rendering engines and styling languages – a requirement that a large percentage of to-be map designers are typically going to pass on.

What the main limitations are that, from my perspective, stand in the way of moving map design in OpenStreetMap to the next level and to practically get talented and motivated people into OSM map design, i have explained some time ago.

I am not going to speculate when and how something along these lines might actually happen. Obviously i would very much like to see progress here. But if such progress should come out of the OSM community (and not – like most of the currently used map design and map rendering tools – originate largely from the outside from the investments of commercial map producers) that would require a broader realization from within the OSM community of the necessity to invest here. And that is clearly a long shot.

What i am going to do in the next post (which is going to conclude this short series on map design) is explaining a bit where the short term trend in map design in OpenStreetMap might be heading in light of the recent activities of the OpenStreetMap Foundation around map rendering.