Imagico.de | blog

Augmented symbols for bus stops with shelters

October 9, 2023
by chris
4 Comments

Symbols and labels #3 – augmenting symbols

In the previous post i discussed how a relatively simple feature addition to Mapnik called anchors, which allows conditionally connecting the rendering of some visual elements to that of other elements in the map in a freely customizable way, can help solving the basic dilemma of combined symbol and label visualizations of point features.

But these anchors are useful for more than that – among other things for augmenting symbols.

I have already demonstrated augmented symbols in the past in a very rudimentary fashion for water features with a drinking water attribute. The technique as shown there was rather limited and essentially this was just a basic proof of concept for the visual design idea.

rendering of water sources introduced back in 2018

Augmenting symbols with anchors

The idea of symbol augmentation is that you have a pictorial/geometric symbol in the map visualizing a certain point of interest and that you add a supplementary design element to that symbol to illustrate either additional attributes of the feature in question or to provide information about other data in the geometric context.

The difference to a symbol variation, where you vary the symbol used from a common base design – like it is shown for example in the example above for intermittent and hot springs – is, that symbol augmentations are optional, they are rendered only when there is space for them and the non-augmented symbol, while providing less information, still illustrates the fundamental nature of the feature in question.

The idea with the water POIs is to indicate that a spring/well/fountain provides drinking water with a supplemental beaker symbol, as it is used also for the generic primary tag amenity=drinking_water (see above).

In many cases the design of base symbol and augmentation allow for a number of different relative placements – which can be prioritized based on where other symbols block space – as shown here:

or based on other geometric context – like in case of connected springs:

Augmentation of springs connected to streams at z19

Augmentation of springs connected to streams at z18

Augmentation of springs connected to streams at z17

Augmentation of springs connected to streams at z16

Augmentation of springs connected to streams at z15

Augmentation of springs connected to streams at z14

Technically, a number of things are necessary to make this work well and be flexible in terms of symbol design:

Priority of the augmentation needs to be adjustable independent of the base symbol.
Overlap of the augmentation and base symbol needs to be possible without blocking.
It needs to be possible to specify the relative placement of the augmentation in all the variants precisely.
It needs to be possible to use different symbol designs in different relative placements.

Existing possibilities in Mapnik do not provide the means for this. How anchors can be used to accomplish that i will show in the following.

Like with the symbols and labels rendering, the base symbol is rendered with

marker-anchor-set: '[osm_id]';

The rendering of the beaker symbol in a separate lower priority layer naturally in the first placement variant gets the already known

marker-anchor-cond: '[osm_id]';

but in addition also

marker-allow-overlap-anchor: '[osm_id]';
marker-anchor-set: "'dw_'+[osm_id]";

marker-allow-overlap-anchor is a selective allow-overlap, meaning that rendering allows overlaps of this symbol with any already rendered symbols with the specified anchor. The second line sets another anchor for the augmentation symbol. This is then used as a condition in subsequent placement variants:

dir2/marker-anchor-cond: "[osm_id]+',!dw_'+[osm_id]";
dir2/marker-allow-overlap-anchor: '[osm_id]';
dir2/marker-anchor-set: "'dw_'+[osm_id]";

This means that rendering of this second placement variant is conditional to (a) the successful rendering of the base symbol and (b) the first placement variant of the augmentation symbol not having been successfully placed.

In addition, for connected spring rendering, the SQL query sorts the different placement variants in priority based on the direction of the connecting waterway – resulting in the behavior illustrated above.

Here a few real data examples illustrating how this looks practically:

Well at z19

Well at z18

Connected spring at z19

Fountain at z19

Aggregating features with augmented symbols

That was how symbol augmentation can be implemented with anchors in the very simple case that the augmentation visualizes exclusively a certain secondary tagging on the features in question (amenity=drinking_water or drinking_water=yes on natural=spring/man_made=water_well/amenity=fountain). I will now show and explain how this concept can be extended to aggregate different close-by and semantically connected features.

The case i want to demonstrate this on is bus stops and shelters. Bus stops in OpenStreetMap are tagged with highway=bus_stop. If the bus stop has a shelter for protection of waiting passengers against the weather this can be tagged with

a separate feature (node or polygon) representing the shelter tagged amenity=shelter (optionally with shelter_type=public_transport)
or a secondary tag shelter=yes on the highway=bus_stop node
or both

In many cases the bus stop and the shelter are very close to each other so there is no room at the lower zoom levels to render both with a dedicated symbol without displacement of either. In OSM-Carto in addition amenity=shelter has priority over highway=bus_stop in rendering so you see a lot of shelters but not a lot of bus stops at z16/z17.

A viable solution is again to use symbol augmentation. Here is how this looks like at z16:

From left to right: (1) amenity=shelter (2) amenity=shelter + shelter_type=public_transport (3) highway=bus_stop (4) highway=bus_stop with either shelter=yes or separate amenity=shelter nearby

and at z17 and above:

Bus stop rendering with augmented symbols at z17 and above – see text for detailed explanation

From left to right this shows:

amenity=shelter
amenity=shelter + shelter_type=public_transport
highway=bus_stop
highway=bus_stop with separate node tagged amenity=shelter nearby
highway=bus_stop with separate node tagged amenity=shelter + shelter_type=public_transport nearby
highway=bus_stop with separate node tagged amenity=shelter + shelter_type=public_transport nearby and nearby other symbol blocking default placement of symbol augmentation
highway=bus_stop with secondary tag shelter=yes and with separate node tagged amenity=shelter + shelter_type=public_transport nearby

The last four are shown on top with the shelter node so close that there is no space for both symbols, hence resulting in the augmented symbol rendering, and on bottom with the shelter node far enough away for a separate rendering. In the last variant on the right this version shows the non-augmented bus stop symbol because the shelter is close enough to be recognized as the shelter of this bus stop explicitly representing the implicitly tagged shelter=yes – while in the third variant below, the shelter node is so far away that it is no more recognized as belonging to the bus stop – hence the shelter=yes leads to the augmentation.

Implementing this with anchors in Mapnik requires the query for the add-on symbols to provide both the osm_id of the bus stop and that of the closest nearby shelter (as osm_id_shelter). This is then used for the first placement variant of the augmentation symbol like

marker-anchor-cond: "[osm_id]+',!'+[osm_id_shelter]";
marker-allow-overlap-anchor: '[osm_id]';
marker-anchor-set: "[osm_id]+'_'+[osm_id_shelter]";

meaning to

only render the augmentation symbol if the base symbol has been successfully rendered and if the standalone symbol for the shelter has not. For only implicitly mapped shelters, osm_id_shelter is set to zero so this second condition is always met in those cases if no explicitly mapped shelter is nearby.
Allow the augmentation symbol to overlap with the base symbol.
Set a new anchor concatenating the bus stop id and the shelter id.

For the second placement variant (augmentation on the right) the anchor condition is then obviously extended to

alt/marker-anchor-cond: 
  "[osm_id]+',!'+[osm_id_shelter]+',!'+[osm_id]+'_'+[osm_id_shelter]";

meaning that this marker is only rendered if

the corresponding base symbol has been successfully rendered,
the shelter has not been successfully rendered as a standalone symbol,
the previous placement variant(s) have been non-successful

Why do we include both the bus stop id and the shelter id in the anchor? Because we want the same shelter to potentially serve for several bus stops and they need to have different anchors in both augmentations for that to work. For bus stops this is practically not a common arrangement but for tram stops and railway halts this is quite common when both directions of transport share a common platform with a single shelter (the 2+1 variant below).

Augmented symbols for railway halts and tram stops

And here finally a few example of how this looks like with real world data.

Bus and tram stops in Strasbourg at z16

Bus and tram stops in Strasbourg at z17

Bus stops in Strasbourg at z18

Conclusion

What i showed here is how symbol augmentation can be used as a technique to transport additional information in a map in an intuitive way in cases where space is scarce and there is not enough room to depict everything through atomic and static point symbols. Doing this in a proper way requires support from the rendering tools and i demonstrated how the low level feature of anchors i added to Mapnik can help with that.

As mentioned in the previous post already these anchors implement only part of point 5 in my list of features map design would like to have from the tools it uses. At the same time what i showed in this and the previous blog post barely scratches the surface of what can be done with this relatively minor additional feature. This should give you a bit of an idea how we are in automated rule based map design – figuratively speaking – barely at the threshold towards the Neolithic.

October 7, 2023
by chris
0 comments

Symbols and labels #2 – making a connection

I could not be at the Karlsruhe Hack Weekend this time, but i did some hacking work during that and the following week i wanted to write about here. Back in May i discussed in depth the problem of visualizing points of interest in automated rule based maps with symbols and labels in combination. I want to revisit that problem now.

In a nutshell – what i discussed back then was, that you can approach the problem primarily in two ways:

separately rendering the symbols and labels, typically with the symbols having higher priority than the labels.
connecting the label and the symbol for every feature and render them with the same priority (typically via shield symbolizer in Mapnik)

Both approaches have advantages and disadvantages – as discussed and demonstrated in that post. The bottom line is – quoting what i wrote back then:

What we would really like instead is rendering the symbols as in the first approach but then render the labels only for those features for which a symbol has been successfully rendered. This, however, is not possible with the current tools.

I have tried to work on this dilemma now by extending Mapnik with the possibility to connect two different independently rendered elements in the map – making the rendering of an element (like a label or a marker) conditional in some form on if other elements have been (or have not been) successfully placed before.

Introducing anchors

The reason why this kind of functionality is not available in Mapnik so far is probably because Mapnik mostly aims to provide higher level features to connect different elements in the form of what is called group and shield symbolizers (to tie several elements to be either rendered together or not at all) and placements (to try several alternative labeling/symbol placement options). But those are practically severely limited in the level of control they provide to the map designer and they do not allow connecting elements independent of their priorities.

Mapnik’s internal design is fairly complicated and very sparsely documented so what i implemented is surely not very elegant and in many ways not how it should be done. Be that as it may – i will try to explain the feature and how it works in its CartoCSS syntax. For setting up Carto with the new feature you also need an updated mapnik-reference.

The feature is called an anchor – which is a string you can assign to any blocking element you render (point, marker and text symbolizers) – either statically or based on data properties via expressions. With an osm2pgsql PostGIS database the most straight away attribute to use is the osm_id.

So with the separately rendered POI symbols and labels the symbols get something like

marker-anchor-set: '[osm_id]';

which means: When this marker is successfully placed (i.e nothing previously rendered is blocking it) record its osm_id attribute in the list of elements successfully placed. osm_id of course has to be added to the SQL query for the layer.

When you later, after having placed all the symbols, want to place the matching offset labels as far as there is space left and want to avoid placing labels for features for which no symbol has been successfully placed you use

text-anchor-cond: '[osm_id]';

which means: Only place this label if its osm_id attribute is in the list of anchors of elements successfully placed before. If you want to – in case no symbol could be placed – try to render a centered non-offset label for the feature in question you use

textonly/text-anchor-cond: "'!'+[osm_id]";

which means: Only place this label if its osm_id attribute is not in the list of anchors successfully placed before.

Practical results

As you can see this is pretty simple to use – and it solved our symbol and label rendering dilemma quite nicely. For reference here the two variants offered by the AC-style so far – first the symbol and label separate version (similar to OSM-Carto, but with consistent priorities):

AC-Style with separately rendered symbols and labels

and in the combined version:

AC-Style with combined symbol and label placement

which has the main disadvantage that the labels of high priority features block the symbols for lower priority features – hence no shelters on the right side in the second sample image.

With the help of the new Mapnik feature the separate version combines the benefits of both:

separately rendered symbols and labels connected via anchors

On the very right you can see that the higher priority wilderness hut does not prevent the shelter symbol from showing because its label is not rendered together with the symbol but placed later with lower priority. At the same time it avoids the problem of stray offset labels without a corresponding symbol as it happened previously when rendering symbols and labels separately.

That’s it essentially – a relatively simple extension of Mapnik solves the dilemma of combined point symbols and labels that bugged OSM-Carto for almost exactly ten years now.

In terms of the list of desirable features of map design tools i published recently this implements parts of point 5 by the way.

But of course the concept of anchors in Mapnik and making the rendering of elements conditional on one another in a freely definable way can be used for other things as well. More on that in the next post.

September 25, 2023
by chris
5 Comments

Map style licensing – another comment

Earlier this year i wrote a bit about map style licensing. The OSMF has just added a new featured tile layer on the openstreetmap.org website – and i wanted to quickly point out an interesting quirk in that regard here.

First – the tile layer added is an interesting mashup of OSM-Carto and OpenTopoMap – and individual styling ideas integrated from other map styles and from map development discussion on public channels. If i had to specify a unique and interesting aspect of the style in terms of the guidelines for new tile layers that would be the mashup aspect. It is not the first style that combines design elements developed independently into a distinctly and deliberately heterogeneous design, but it is definitely a noteworthy example of that.

However, the style is not open source and as you might know (i mentioned it in January) OpenTopoMap is licensed under CC-BY-SA. So unless Tracestrack has obtained an explicit license for non-open use from the OpenTopoMap developers we are having a clear license violation here.

The freedom to recombine different design elements and to create mashups is an important and attractive aspect of Open Source map design. But you still have to respect the licenses.

September 16, 2023
by chris
8 Comments

What rule based map design requires from the tools it uses

I have in the past explained on several occasions that the free and open-source software (FOSS) tools available to map designers developing rules for automatically rendered maps are severely lacking and that it would be of high importance for the OSM community to invest strategically in that field because commercial map providers develop tools for fairly narrow short term needs and focus very strongly on technical aspects and have little interest in substantial innovation in actual map design.

But i have never actually explained more in detail what the concrete needs of state-of-the-art rule based map design are regarding tools to be able to produce innovative and high quality maps. The main reason for that is that this is not something you can put into a simple list of requirements a software developer can work through and then ends up with a product that meets the requirements. Most of the needs i will list below are matters that require an in depth discussion between designers and software engineers with a solid background in computer graphics and an open mind towards the work of map designers. Me trying to provide a concise summary of some of the main requirements below is not a substitute for such discussion – although i will try to provide some hints to examples that can help initiating a more in depth contemplation.

The other reason i have been hesitant in discussing this in public is that the most likely practical beneficiaries of such discussion are proprietary software developers. Professional FOSS development in the field of map rendering seems unfortunately very focused on the already mentioned commercial map providers (understandably, because that is what at the moment largely pays for their work) and is fairly detached from the world of actual map design. The vast majority of actual innovation in FOSS tools for map design of the last 5-10 years has happened not by professional software developers but by map designers trying to address their specific needs on their own. Most of these attempts are fairly awkward, too specialized and of low technical quality because most map designers of course lack the software development skills to produce something more solid.

I like to emphasize that this list of requirements is not meant to be complete in any way. Map design is a highly diverse field and my personal areas of focus in map design work of course influence the priorities i see. Other map designers will have additional needs that i do not discuss here. At the same time i tried to keep this limited to clear needs and left out out stuff that would be just nice to have or things that would be worth exploring as potential technical innovations without there being a clear practical necessity visible so far.

I also left out any specific higher level feature requirements for the renderer. I did that because (a) such a list would be highly specific to the design use cases you have in mind and (b) higher level feature can be built directly into the renderer or they could be implemented by the users based on more fundamental drawing functions and a flexible and efficient framework for defining rendering rules. Bottom line: High level rendering features are not strictly required at all. They can, but they do not have to be, an integrated part of a well usable map rendering framework.

I also like to emphasize that this only covers map design in a strict sense. Many map providers these days regard their business as providing an overall map viewing experience with interactivity and integration with other services in the foreground. This is often the most important selling point, not the quality of the map design used. This whole domain i deliberately do not cover.

So here is the list of what – from my perspective – rule based map design requires from the tools it uses to be innovative and produce high quality maps:

A rendering engine capable of precision rendering. In other words: The rendering engine should faithfully execute the instructions of the map designer on how to draw things. I had previously analyzed a number of rendering engines in that regard in a rudimentary fashion. So far Mapnik (and maybe Mapserver) are the only practically usable map rendering engines that come close to meeting this requirement.
A map design centered language/file format allowing the map designer to implement their map rendering rules in a way that matches their work and thought process and that scales well with design complexity. The interpreters of this language/file format should support the full feature set of the rendering engine. CartoCSS was fairly revolutionary w.r.t. being map design centered. At least in the form supported by Carto it has, however, only very rudimentary options for the parametrization of design rules and only very limited modularization and code reuse options and therefore is not that well suitable for more complex map design work. And Carto does not support the full feature set of Mapnik. Examples exist in the broader field of graphics design that can provide valuable cues – i for example like to point at POV-SDL, which supports both geometric modeling and defining styling together in a design centered language (for the most part – user defined functions, which were a late addition, are not particularly well usable for designers).
Practically usable means to define scale dependent precision geometry processing. Practical use cases for this can be found in the AC-Style (like trees, water barriers, fords). Support for this is excellent in the OSM-Carto toolchain but completely lacking in any of the client side rendered frameworks. Even more: This is practically not feasible to add there because the inevitable lossy geometry data compression makes client side precision geometry processing impossible and scale dependent geometry processing on the server side would lead to massive increases of the data volume.
The ability to define drawing order independent of the way the data is structured. This is currently not possible with Mapnik without unifying all the involved data into a single layer (as done in the AC-Style for the road layers). This is one of the main hard limits of all Mapnik based toolchains.
The design rules should have access to detailed information on what has been drawn in the map so far. Currently this is limited to simple blocking in most map rendering frameworks, meaning that you can draw something independent of what has been drawn before or you only allow it to be drawn if no other blocking feature has been drawn before at the same location.
The ability to define drawing order independent of the prioritization of competing elements. Classic example where this is needed is contour line labeling: Contour line labels typically have the lowest priority of all labels and are not to be placed overlapping most of the other features in a map (including line work like roads, water lines etc.) At the same time contour line labels are traditionally cut out from the contour lines with a buffer for good readability and harmonic integration into the map. And the contour lines are drawn below almost everything else in the map. Doing this (drawing the contour lines early but cutting out the low priority labels) is not possible with most automated map rendering frameworks available these days so you will practically not see contour line labels with cutouts in such maps (with very few exceptions).
Flexible raster compositioning options in the rendering engine. Mapnik has rudimentary support for that (feature and layer based compositioning of the current feature/layer relative to what has been drawn so far). More flexibility here would be very important, in particular regarding rendering performance, because it would reduce the need for expensive geometry processing and for drawing features several times and it would practically allow developing solutions for problems that are currently not solvable in an efficient way (like the road line/polygon layering issue). Cues for this could be taken from common raster processing frameworks like ImageMagick and G’MIC.
A data processing chain that provides access to the full data model of the source data to the map designer. This has so far not been the case for the OSM-Carto tool chain for OSM data because osm2pgsql only supported multipolygon and route relations and only in a rudimentary way (without exposing relation membership information). The osm2pgsql flex backend improved this a lot though.
A text rendering engine that provides information on the would-be geometry of labels in the design rules without actually placing the label in the map.
The practical possibility to do automated integration testing of the full rendering tool chain (from the raw source data to the final raster image).

The list is roughly sorted by importance with the most important requirements on top. The first three points i regard as essential. Without meeting these points at least in a rudimentary fashion the map design work i do is practically impossible. That is the main reason why so far i have very little interest in working with any of the client side map rendering frameworks that are popular these days – because they substantially lack in all three of these points without a clear perspective of improvement in any of them.

September 14, 2023
by chris
0 comments

Flooding in eastern Libya

Supplementing the previous post about the extreme rain and resulting flooding in the eastern Mediterranean i also prepared an image from the other side of the sea at the north African coast in Libya, where the same storm that caused the flooding in Greece also lead to extraordinarily intense rainfall and flooding a few days later.

Most of the damage and fatalities due to the flooding happened at the coast in the north in the steep and narrow valleys, indirectly visible in this image mostly due to the sediment carried by the streams in these valleys into the sea and coloring it.

More impressive in this picture is, however, the extensive flooding in the desert further south. In general the flatland regions of much of northern Africa have been very dry during the past few thousand years compared to the long term average before. The more humid long term history of northern Africa is still well visible in many places in the form of valleys shaped by water flow and extensive lakes which are permanently dry in more recent history. In a way the flooding we can observe now in the desert of northeastern Libya gives us a bit of an idea of how the conditions might have been in the Sahara desert thousands of years ago.

September 12, 2023
by chris
0 comments

Flooding in the Thessalian plain

A few days ago a storm has brought extraordinarily intense rain to some parts of the Eastern Mediterranean, leading to substantial flooding in some regions. I prepared a rendering of a satellite image recorded on September the 10th of the Thessalian plain in Greece, showing the flooding a few days after the event around the course of the Pineios River that i wanted to show here.

You can see both the primary flooding in the western part of the plain around Trikala and the secondary flooding further downstream to the east caused by the huge amounts of water raising the level of the Pineios River, right up to the coast were an impressive sediment plume can be observed in the Mediterranean Sea.

I don’t know how useful that is – but i put up the image in my images for mapping in case it is useful to update OpenStreetMap in the area (like indicating impassable roads). Note, however, that mapping temporary water cover itself in OSM is not a good idea.

September 3, 2023
by chris
3 Comments

A note on OSM-Carto

I like to make another note on OSM-Carto here. Background is that there is another wave of negativity (aiming to find a relatively neutral, yet at least a bit descriptive term here) being expressed by some people from the OSM community on some channels regarding the project.

It is not the first such articulation and it was pretty much to be expected that this would become a recurring pattern. The background i already explained at length back in May 2022. Back around that time i made the decision (and i indicated that in the text) that i would continue trying to do my best to support the idea of OSM-Carto, to be an openly cooperative map design project, governed by consensus of a diverse group of maintainers.

What i do is I provide guidance to contributors how to make changes to the style in a way that complies with the documented goals of the project as i understand them and with the unwritten conventions of the style. I explain the historic context how OSM-Carto came to render things the way it does today, explain how things connect design wise creating a harmonic overall result and provide ideas what possibilities exist to resolve the various technical and design problems the style is faced with. And i listen to arguments that challenge the views i articulate and, when arguments are convincing, re-evaluate these views. At the same time i explicitly avoid standing in the way of consensus forming among the other maintainers even if i disagree with those ideas. And i merge changes when they are suitable for the style in my opinion. What i don’t do is approving changes that i don’t stand behind merely because of pressure from others to have them in the style.

I took this approach despite knowing that the chances of OSM-Carto finding its way back to a sustainable strategic consensus among the maintainers would be very small. The alternative would have been to try essentially staging a coup and to attempt single-handedly reshaping the governance of the project in a way that i would consider more sustainable. Or i could have abandoned the project. This would either have killed the project or it would at least have lead to an end of the diversity in governance. And i was not willing to do either of these things because that would have clearly betrayed the idea of OSM-Carto as described. What i did instead was summarizing some of the lessons learned in terms of governance and social dynamics and some key points that should – from my perspective – be taken into account in case someone would want to bootstrap a new cooperative map design project with a similar paradigm as OSM-Carto. It was unclear if something along these line was going to happen of course (and it still is) or if the OSM-Community is going abandon the idea of consensus based cooperative map design and move back to the more common, more traditional single benevolent dictator paradigm or to something governed by a homogeneous group of like-minded people with common interests. But from today’s perspective i stand firmly by my decision to continue holding up the ideals of OSM-Carto but decidedly not trying to actively shape where OSM community map design is going beyond OSM-Carto while continuing to share my experience and thoughts on the matter with everyone willing to listen – both within OSM-Carto development discussion and on this blog.

The recent wave of negativity towards OSM-Carto is remarkable in the level of resentment that is articulated towards the values of the project, its maintainers and me specifically as the person most outspoken about the project and its problems. I have read quite a bit about and also have seen the effects of people radicalizing themselves in closed groups, reaffirming each other in their preconceptions and simplified friend-foe images. But i had previously never seen this process in a setting that is on public record (mostly – at least one of the most vile comments has been silently removed – not before the author received cheers from his peers though).

The whole thing is also telling in context of the bold claims of the OSMF that the channel in question is moderated with the aim that everyone feels safe and confident to take part. I criticized that idea on ethical grounds in the past but it is visible here that this claim also simply fails to live up to its promise. The level of resentment, bad faith insinuations as well as evidently and ostentatiously false claims (including the title of the topic) is remarkable. Personally i don’t really care – it is not the first time people have articulated this kind of spiteful thoughts about me. But it well demonstrates that the whole top-down moderation idea of the OSMF is an empty promise. And there are plenty of people who – when seeing this kind of conversation happening unchallenged in the OSM community – will think twice before they engage actively in this kind of social context. Note this is not me projecting, other people have articulated that very sentiment to me on quite a few occasions. And the problem here is not the lack of moderators sanctioning people breaking the rules (or weaseling around them as it is also openly done here – mit Ansage as we say in German). The problem is that there is apparently no one left on that channel having the courage to push back and calling out the insults, disrespect and plainly false claims, leading to the mentioned radicalization and reaffirmation of preconceptions in a closed group.

I like to emphasize that the sentiment of people, that when things develop in a bad direction they are trying to identify the reason and who is responsible for it, is understandable and often helpful – but also misguided here as far as assigning blame is concerned. And i think i made it clear back in May 2022 already and i will do so again here: The OSM-Carto maintainers having failed to find back to a consensus direction back in 2018 and the years after – that is our failure and we have to collectively accept responsibility for that. Now you could try to assign individual blame to individual maintainers based on how you assess their efforts to achieve consensus within the group, if you like their ideas and preferences regarding map design work or whether you approve of their style of communication or simply who you can relate to better on a personal level. I would not do that. Keep in mind that the OSM-Carto maintainer team was deliberately put together as a heterogeneous group of people, selected not by like-mindedness and common interests but because of their abilities and the diverse viewpoints and ideas they are able to contribute. This diverse composition of the maintainer team is the very essence of the project that made it unique and was largely responsible for its success. Everyone within this heterogeneous group did what they could within their unique abilities for the success of the project. So we own this collective failure as a group.

What is getting lost – and that is the key point – is that the ultimate problem is not OSM-Carto, it is the lack of anything other than OSM-Carto in the field of openly cooperative map design with an ambition of a balanced global representation. I have been pointing out that this is a problem for many years, practically every time i write about OSM-Carto. I have also pointed out that strategically investing in development of software that facilitates openly cooperative map design on that level is imperative for the OSM community. The most recent plans of the OSMF in the context of map rendering are highly disappointing in that regard. Not a single word, let alone a single cent, is invested there into the question what the needs and requirements of community map design are regarding tools.

September 1, 2023
by chris
0 comments

Map style modularization

I have finally gotten around completing the modularization of my Alternative Colors map style and i want to introduce that here.

I already discussed this briefly in context of the new symbol and label rendering system. Introducing that change depended on splitting project.mml into separate files – one for each layer – which then get combined into a single file for use by carto with a python script. I already sketched the idea that this system could also be used to have different style variants and make some more computationally expensive features of the style optional. But this was still incomplete back then.

What was missing was in particular the option to split and further modularize the lengthy SQL query of the roads layer. I added this now and also implemented the option to turn off

entrance cutouts in line barriers
implicit sidewalks on roads
sophisticated rendering of junctions of roads in ground units
contextualized rendering of fords and mountain passes
preprocesed low zoom waterbody rendering
sophisticated tree rendering
contextualized water barrier rendering

How does it work? layers.yaml defines tags for each of the layers configured and when running assemble_project.py you can select which tags you want to use. For example running

scripts/assemble_project.py -t ac

generates the standard AC-Style setup with all the sophisticated and expensive layers while

scripts/assemble_project.py -t simple

leaves out all the more complex stuff or replaces it with a simplified version.

Within SQL code different variants can be implemented with specific directives. Like:

@@if(ac)
  @@include(roads/sidewalks.sql)
@@else
  @@include(roads/sidewalks-dummy.sql)
@@end

A few notes on the purpose of the AC-Style

Since people are sometimes confused about what the purpose of the AC-Style is and why there is no interactive map based on the style available i wanted to add a few notes on that.

In my last update on the OpenStreetMap-Carto project i explained a bit the history of the AC-Style. After starting as a simple demonstrator of how a more holistic and systematic color design in the style can lead to a more harmonic, better readable and better maintainable map, it changed over time into a much broader test bed for case studies in advanced map design.

What has not changed since the beginning though is that i decidedly do not invest into producing an interactive map based on the style. I would support anyone who is interested in deploying a map based on the AC-Style with advice and if necessary adjustments to the style. But i don’t want to spend my time on doing something that i am not good at and administering a map deployment is not my field of expertise.

Why do i not actively try to get any of the developments from the AC-Style into OSM-Carto? Because for many of the core design changes there is no consensus support among the OSM-Carto maintainers. Joseph Eisenberg invested quite a lot of time and energy to get some of the fundamental color changes i developed into OSM-Carto – but without success. And many of the other changes in the AC-Style design wise depend on these changes. There are still some improvements and feature additions that could be integrated and some actually are – at times with modifications, like the allotments rendering or the ridge/arete patterns.

Do i think it would be good to replace OSM-Carto in deployments with the AC-Style? Hell no. OSM-Carto is foremost a cooperative community project where decisions are made through consensus of a fairly diverse team of maintainers. As i have pointed out in the previously mentioned update on OSM-Carto the project has been struggling with the practical difficulties of this approach for some time (for reasons i discussed there in more depth). But any project run by a single person – no matter how benevolent they are – is not a suitable substitute.

As i have pointed out in the past the OSM-Community could very much use a larger variety of independently developed map styles. Quite a bit of progress has been made on that front in the past years, but unfortunately exclusively on styles targeting the needs and wishes of specific local communities and mostly under control of a single decision maker. Nothing is in sight so far even remotely aiming to create a map style for a broad global audience in a similar fashion as OSM-Carto though.

Would it in my eyes makes sense if someone was to start off a new cooperative map design project with such a scope to base it on the AC-Style rather than OSM-Carto? Design wise a strong argument could be made for that but i would still advise against it because of the technical complexity and the massive hurdle that would pose to potential contributors. What might be worth considering though is to have some of the fundamental design changes from the AC-Style, that – as explained – do not have consensus support in OSM-Carto right now, from the start in such a project because it would make further design work so much easier.

August 27, 2023
by chris
0 comments

Watching the watchers

A bit less than two years ago the OpenStreetMap Foundation (OSMF) started centrally imposing behavior regulation on OSMF provided communication channels – channels which were previously largely individually self governed by their users. Originally this applied only on two mailing lists but since then it was successively extended more and more. I had commented on that quite in detail already back then (and before) and it seems a good time now to look at how this turned out so far.

Back in 2021 i had expected the practical consequence of this to be primarily that

community members less aligned to the Anglo-American-European Leitkultur of the OSMF would be more reluctant to actively participate on the channels in question or would withdraw from participation.
remaining participants would engage in preemptive self-alignment, adjusting their communication to diminish the risk of it being interpreted to clash with what is considered acceptable by the powers that be.

The first point is probably something that indeed happened – though it is hard to reliably measure because other changes in communication culture (like the emergence of various new platforms and channels as means of communication of the OSM community, including – among others – the OSMFs discourse based platform), evidently have an effect on that as well that is hard to separate.

The second point is something where i seem to have been wrong. Since the behavior rules imposed are largely vague and often cryptic and difficult to interpret objectively, it seems that contributors on the channels in question largely ignore them. At the same time we see a remarkable volume of punitive measures being imposed by the OSMF-board appointed moderators based on perceived violations of the rules. To me this is a bit surprising because this is not how behavior regulation typically works in Anglo-American style tech communities.

What i think shines through here is what i have pointed out in the past: that OpenStreetMap is not a tech project and local OSM communities have more recently developed a quite robust self confidence in their specific cultural styles of communication and social interaction. Nudging towards self-alignment does not work as well in OSM as it would in a group that is culturally more homogeneous from the start with more effective peer pressure towards conformity.

The way behavior regulation is enforced now contrasts quite strongly in many ways with what during the discussion in 2021 was communicated how it is supposed to work, namely through moderators also otherwise active on the channel in question defusing conflicts through counseling and being a moderating voice in the true sense of the word. This still happens, but in most cases not through the official OSMF moderators. It is positively remarkable and pleasantly surprising how well quite a few people in the OSM community manage to address even strong expressions of emotions in many cases with empathy and sensitivity. But what you can see quite frequently now is that after that, when the participants of a channel have already made substantial progress to defuse the situation and foster understanding and respect for different views among the participants in a heated discussion, the official moderators swoop in (typically without having been involved in the discussion to that point), identify a single culprit and dispense some sort of punishment on them – either merely through an official reprimand (and this way implicitly exonerating everyone else from their responsibility) or through a ban from participation. I like to emphasize this is not universally the case, there are also situations where the OSMF moderators defuse conflicts through counseling in a sensitive manner. But the described pattern is quite common in those cases where ultimately substantial punitive measures are imposed.

On the positive side – official moderation activities are quite decently documented. I can recommend everyone to look a bit through this documentation to get your own impression on how centrally imposed behavior regulation is implemented on OSMF channels these days. A bit of warning though when you look at the official record of the communications on the discourse platform referenced in these incident reports: These are often incomplete since messages have been removed afterwards and are selectively quoted in the incident reports. If you are subscribed to the mailing list mode the picture you get from some of these conversations is substantially different from that you get on the web interface where messages are not only hidden (with there still being a marker indicating where there formerly was a message) but also fully removed from the view of both the general public and logged in members. And the management of this hiding/removing of messages is done not by the official moderators but by a separate self appointed governance team, without either independent oversight or meaningful public reporting. Using the mailing list mode can help with that and with maintaining a more complete record of communication but there is apparently a built in 20 minutes delay in the mailing list mode which makes this not fully reliable either.

Looking at the incident reports and the moderation activities you can make some in my opinion quite remarkable observations. First: The OSMF-board appointed moderation team consists formally of five people. Practically, however, it seems only two of them are actively pursuing moderation. These two (both of them Americans) act as rapporteurs and the other three merely confirm the decisions prepared by these two. Second: eight out of nine documented cases of moderation activity are about sanctioning non-native English speakers for communication activities in English language.

Now i want to keep this blog post relatively brief so i will not discuss the merits of the individual cases here. As said – everyone is invited to read up on these cases for themselves. If anyone is in need of any of the messages that have been removed from public record related to any of these incidents feel free to contact me directly about them. I also won’t further analyze what the statistical observations made mean in this post. Feel welcome to share your own thoughts on this in the comments below.

Apart from that it is worth noting that the original promise that local communities will be allowed to individually self govern their own channels on the discourse platform is not kept. According to comments made, bans on the discourse platform are always implemented globally. Practically that seems to mean someone who runs into conflict with the OSMF rules on the channels under control of the official OSMF moderators will also be banned from participating in their local community channel, even if what they did was perfectly acceptable under their local community’s social standards and conventions. The interesting question is of course if this will practically also work the other way round – that someone violating the local community standards anywhere in the world also will get banned from all the other channels on the OSMF platform.

One other thing that might be food for thought: At least one of the people that have been officially banned by the OSMF-board appointed moderation team so far has made a public statement in the past indicating that they might have a learning disability.

How are things going to develop in the future? I don’t really know of course. There are quite clearly two trends active here pointing in opposite directions. On the one hand the efforts to induce cultural homogenization on OSMF provided communication channels are quite clearly successful to some extent. On the other hand we can quite clearly also observe a trend towards diversification in terms of communication channels and platforms used outside of OSMF control. And while this cannot be reliably proven of course, it is likely that the latter at least partly happens as a reaction to the former – local communities, which are denied true self governance on the OSMF managed channels, opt to establish and use channels they have more control over.

But as much as i welcome and applaud initiatives for true cultural diversity and self determined bottom-up cooperation and think this is essential for OpenStreetMap to function in the long term, it is important to realize that cultural homogenization is an attractive strategy for many as a means to facilitate and simplify cooperation in larger groups – and not only for those whose culture in being imposed on others, even for quite a few of those who need to adjust and denounce their culture in the process. True cultural diversity is hard, even for those who strongly depend on it.

deutsch The Musaicum EU-plus satellite image mosaic

July 26, 2023
by chris
0 comments

Announcing the Musaicum EU-plus 10m resolution image of Europe

I am pleased to here introduce a new satellite image product that i have been working on for a few months. Development of this has been co-financed by Geofabrik. They are going to offer tile services for web maps based on this image in combination with the Green Marble.

Background

Most of the readers of this blog will be familiar with the Green Marble – my global satellite image product offering the highest quality rendering of the whole earth surface available at a resolution of 250m. The Green Marble is produced with a pixel statistics approach, that means an analysis of all observations available is done independently for every pixel of the image to estimate the surface color at this point. This kind of technique is very popular because of its principal simplicity and because processing can be implemented in a very efficient fashion.

But this method has two main disadvantages:

It requires a significant amount of data to actually get to a point where the final product is in visual quality equal to or better than an individual good quality image. How much depends on the algorithm used and its convergence characteristics and obviously this will differ a lot between different parts of the planet. For the Green Marble version 3 for example about 1PB of raw data was processed – which means more than 100kB of data per pixel.
It does not scale well with increasing spatial resolution. I discussed this matter in more depth already back in 2018. This has multiple underlying reasons, the one that is most straight away to understand is that the higher the spatial resolution is you are looking at the more volatile the content of an image becomes. This mean the higher in terms of spatial resolution you go the less you have – on average – a long term stable state of the Earth surface your pixel statistics can converge to.

Long story short: Pixel statistics work very well at a resolution of around 250m if you have a large data basis to work with. They work poorly at much higher resolutions, even if you have a lot of data (which you typically don’t – but that is a different issue). This has not prevented various companies over the last 5-10 years to invest substantial resources in naively trying pixel statistics approaches on Landsat and Sentinel-2 data – with the expected mediocre results.

The alternative to pixel statistics for aggregating satellite imagery into a homogeneous larger area visualization is using classical mosaicing techniques where individual images are essentially put together in the form of a patchwork or mosaic. If you want a concise definition: You have a classical mosaicing techniques when the color at any point in the image is in most cases (a) primarily derived from a single source image and (b) the surrounding pixels are primarily derived from the same image. This is evidently not the case for a pixel statistics process where the processing of a pixel is not correlated to that of its neighbor.

Classical mosaicing techniques are the dominant method for aggregating very high resolution satellite and aerial imagery and for high quality images based on Landsat and Sentinel-2 data. The problem here is that achieving good quality with this approach requires fairly complex processing techniques and there are certain key steps that are notoriously difficult to automatize because the quality of the results depends a lot on competent human judgement.

Hence most satellite image based visualizations using classical mosaicing techniques either are relatively poor quality (high cloud incidences, poor consistency in colors across the seams between images) or are based on fairly old data because updates are costly.

I myself have been producing images using classical mosaicing techniques for nearly 20 years now (an early discussion of this can be found on this blog in 2013) and both improved and streamlined the methods i use over the years. But also for me hand work was so far always a key component in production of these images and as a result in many cases updates are very costly to do. Therefore i had been looking for some time at strategies to eliminate the remaining manual processing steps in the production of my higher resolution mosaics without sacrificing too much in terms of quality. With the help from Geofabrik i was now able to practically implement and evaluate some of these ideas and here i am going to present and discuss the results.

The Musaicum EU-plus – click for larger version

The image

At low resolution the image looks very similar to the Green Marble – which is not astonishing since it is assembled with the same aim – to depict the local vegetation maximum and snow minimum. If you look closely you can see the appearance is not quite as uniform as the Green Marble – with some inhomogenities that are clearly non-natural. Part of this is due to the low volume of data used (put simply: Not everywhere was there a source image available in the time period used that exactly represents the vegetation maximum). Another part is that there is still room for improvement in the data processing – after all this is a first attempt.

Swiss Alps

Western Greece

What you will quickly notice if you look through the sample images is that there are no clouds (or almost none – with very keen eyes you might spot a few, and yes, if you systematically screen the whole image you will find some more). This is a point where the results substantially exceeded my expectations. I was hoping to accomplish a cloud incidence substantially better than what is available on the market otherwise but i was expecting it to be significantly worse than in my manually produced local mosaics. The ultimate result is pretty much at par with the manually produced local mosaics with less than about one in 100k pixels severely affected by clouds. Most of these are small, isolated convective clouds.

Vlieland, Netherlands

The focus of the project was on land area visualization so water surfaces were not a particular concern. Since waterbodies tend to be quite variable in appearance and often not in a strict seasonal pattern, the results in that domain are not always ideal, in particular rivers often change in coloring along their course quite erratically. Sun glint is also a problem at lower latitudes, in particular on smaller lakes.

Gdansk, Poland

Istanbul, Turkey

A few words on the data used for producing the image. In analogy to the numbers i presented above for the Green Marble: The volume of original Sentinel-2 data processed for this project was about 20TB, which means it needed less than 250 bytes per pixel. This is extremely sparse considering that the volume of Sentinel-2 data collected for Europe within a single year alone is much higher. A low volume of data to process helps keeping the processing costs low and it also allows using more expensive processing techniques. And in contrast to pixel statistics methods where adding more data always helps, there is no advantage per se with classical mosaicing techniques to use more data, it is more a quality over quantity thing.

What has been more challenging is that i wanted to keep the time-frame from which to source data from relatively short. Preferably just three years (2020-2022), where necessary another year (2019) and only in rare cases also data from 2018 is used. In areas where the earth surface appearance is very volatile – either across the seasonal cycle or between years – this makes it difficult to produce homogeneous results over larger areas.

Paris, France

Cordoba, Spain

What you have to work with

One of the things that made work on this quite substantially harder than you would naively expect it to be is the poor quality of some of the data supplied.

Sentinel-2 data can be obtained in two different forms:

L1C data – which is the original Top-of-Atmosphere reflectance values as recorded by the satellite
L2A data – which is as estimate of the surface reflectance data based on the TOA measurements

Now most experienced users of satellite imagery will understand that the L2A data is – as i characterized it – just an estimate of the surface reflectance. And while this should reduce the variance due to the variable influence of the Earth atmosphere it will also introduce variance in the form of various forms of noise and systematic and random errors in the estimate. What, however, most data users will not realize is that the Sentinel-2 L2A data distributed also attempts to compensates for illumination differences (shading) and that this compensation – to put it bluntly – is incredibly bad. So bad that it is practically not usable for visualization purposes. Here an example – larger version is linked:

S2A L1C from 2021-07-17

S2A L2A from 2021-07-17

For comparison here the new Europe mosaic (which in the default rendering is an atmosphere but not shading compensated version) and my own shading compensated version. For most of the sample area the mosaic is based on the same recording – except for the top right where the July image contains residual snow so the mosaic uses a later season image.

The new Europe mosaic – Tena Valley, Pyrenees

Shading compensated version

The overall color difference is not the main point here (the tone mapping i applied to the L1C/L2A data is somewhat arbitrary). The main point is that the illumination compensation in the L2A data massively over-compensates, leading to the shaded slopes often being brighter than the sun facing slopes. Also it is geometrically fairly inaccurate, leading to a strong emphasis of ridges and the introduction of substantial amounts of high frequency noise.

There seem to be quite a few people using Sentinel-2 L2A data in visualization applications. That is not a good idea. You should only use shading compensated imagery in visualizations if you know exactly what you are doing and in that case you should use a proper method and not the hatchet job offered by ESA here.

I have discussed the matter of shading compensation, in particular for use in 3d rendering, previously. A shading compensated version is available for the Musaicum EU-plus as well – but i have not had the time for a proper evaluation yet. It is available on request.

What remains

As i have mentioned earlier – this is a first attempt at a more highly automated classical mosaicing process. As often with this kind of work many things remain unclear during process development and only become clear once you run the process at scale. This might sound a bit like undue perfectionism considering the results are pretty good. But this is not just about the quality of the results but also about the robustness and efficiency of the process. As a general rule of thumb for a project like this i would say: Putting as much work into process development after the first use at scale as before is reasonable if your aim is to make full use of the potential of the method used.

Svartisen, Norway

Where to go from here

One thing that i am sure quite a few of my readers are going to ask is: Does this also work beyond Europe? In principle the answer is yes, but there is more to it than just running it on another volume of data. Quite a significant part of the development work that went into this project was for tuning and adjusting methods and parameters for the specific settings in terms of climate and surface characteristics that can be found in the processed area. I know from experience with the Green Marble that the variety of different settings on Earth is larger than people usually expect. This needs to be taken into account and that was not part of this project.

There are a number of other additions and extensions of the project that i would like to work on, for example producing a vegetation map similar to the ones available for various local mosaics is is on the list. And of course over time there will be the question of updating the mosaic with newer data. I don’t know if and when i will have the capacity to do any of this. If any of the readers are interested in supporting further work on any of this please feel welcome to get in touch.

You can find the product description and more sample images on the Musaicum EU-plus product page.

Many thanks to Geofabrik for co-financing this project. If you are interested in tile services based on the the Musaicum EU-plus or the Green Marble, people at Geofabrik will be happy to help you. If you are interested in licensing the image for other purposes you are welcome to contact me.

Tödi, Switzerland

Zakynthos, Greece