Imagico.de | blog

September 25, 2023
by chris
4 Comments

Map style licensing – another comment

Earlier this year i wrote a bit about map style licensing. The OSMF has just added a new featured tile layer on the openstreetmap.org website – and i wanted to quickly point out an interesting quirk in that regard here.

First – the tile layer added is an interesting mashup of OSM-Carto and OpenTopoMap – and individual styling ideas integrated from other map styles and from map development discussion on public channels. If i had to specify a unique and interesting aspect of the style in terms of the guidelines for new tile layers that would be the mashup aspect. It is not the first style that combines design elements developed independently into a distinctly and deliberately heterogeneous design, but it is definitely a noteworthy example of that.

However, the style is not open source and as you might know (i mentioned it in January) OpenTopoMap is licensed under CC-BY-SA. So unless Tracestrack has obtained an explicit license for non-open use from the OpenTopoMap developers we are having a clear license violation here.

The freedom to recombine different design elements and to create mashups is an important and attractive aspect of Open Source map design. But you still have to respect the licenses.

September 16, 2023
by chris
8 Comments

What rule based map design requires from the tools it uses

I have in the past explained on several occasions that the free and open-source software (FOSS) tools available to map designers developing rules for automatically rendered maps are severely lacking and that it would be of high importance for the OSM community to invest strategically in that field because commercial map providers develop tools for fairly narrow short term needs and focus very strongly on technical aspects and have little interest in substantial innovation in actual map design.

But i have never actually explained more in detail what the concrete needs of state-of-the-art rule based map design are regarding tools to be able to produce innovative and high quality maps. The main reason for that is that this is not something you can put into a simple list of requirements a software developer can work through and then ends up with a product that meets the requirements. Most of the needs i will list below are matters that require an in depth discussion between designers and software engineers with a solid background in computer graphics and an open mind towards the work of map designers. Me trying to provide a concise summary of some of the main requirements below is not a substitute for such discussion – although i will try to provide some hints to examples that can help initiating a more in depth contemplation.

The other reason i have been hesitant in discussing this in public is that the most likely practical beneficiaries of such discussion are proprietary software developers. Professional FOSS development in the field of map rendering seems unfortunately very focused on the already mentioned commercial map providers (understandably, because that is what at the moment largely pays for their work) and is fairly detached from the world of actual map design. The vast majority of actual innovation in FOSS tools for map design of the last 5-10 years has happened not by professional software developers but by map designers trying to address their specific needs on their own. Most of these attempts are fairly awkward, too specialized and of low technical quality because most map designers of course lack the software development skills to produce something more solid.

I like to emphasize that this list of requirements is not meant to be complete in any way. Map design is a highly diverse field and my personal areas of focus in map design work of course influence the priorities i see. Other map designers will have additional needs that i do not discuss here. At the same time i tried to keep this limited to clear needs and left out out stuff that would be just nice to have or things that would be worth exploring as potential technical innovations without there being a clear practical necessity visible so far.

I also left out any specific higher level feature requirements for the renderer. I did that because (a) such a list would be highly specific to the design use cases you have in mind and (b) higher level feature can be built directly into the renderer or they could be implemented by the users based on more fundamental drawing functions and a flexible and efficient framework for defining rendering rules. Bottom line: High level rendering features are not strictly required at all. They can, but they do not have to be, an integrated part of a well usable map rendering framework.

I also like to emphasize that this only covers map design in a strict sense. Many map providers these days regard their business as providing an overall map viewing experience with interactivity and integration with other services in the foreground. This is often the most important selling point, not the quality of the map design used. This whole domain i deliberately do not cover.

So here is the list of what – from my perspective – rule based map design requires from the tools it uses to be innovative and produce high quality maps:

A rendering engine capable of precision rendering. In other words: The rendering engine should faithfully execute the instructions of the map designer on how to draw things. I had previously analyzed a number of rendering engines in that regard in a rudimentary fashion. So far Mapnik (and maybe Mapserver) are the only practically usable map rendering engines that come close to meeting this requirement.
A map design centered language/file format allowing the map designer to implement their map rendering rules in a way that matches their work and thought process and that scales well with design complexity. The interpreters of this language/file format should support the full feature set of the rendering engine. CartoCSS was fairly revolutionary w.r.t. being map design centered. At least in the form supported by Carto it has, however, only very rudimentary options for the parametrization of design rules and only very limited modularization and code reuse options and therefore is not that well suitable for more complex map design work. And Carto does not support the full feature set of Mapnik. Examples exist in the broader field of graphics design that can provide valuable cues – i for example like to point at POV-SDL, which supports both geometric modeling and defining styling together in a design centered language (for the most part – user defined functions, which were a late addition, are not particularly well usable for designers).
Practically usable means to define scale dependent precision geometry processing. Practical use cases for this can be found in the AC-Style (like trees, water barriers, fords). Support for this is excellent in the OSM-Carto toolchain but completely lacking in any of the client side rendered frameworks. Even more: This is practically not feasible to add there because the inevitable lossy geometry data compression makes client side precision geometry processing impossible and scale dependent geometry processing on the server side would lead to massive increases of the data volume.
The ability to define drawing order independent of the way the data is structured. This is currently not possible with Mapnik without unifying all the involved data into a single layer (as done in the AC-Style for the road layers). This is one of the main hard limits of all Mapnik based toolchains.
The design rules should have access to detailed information on what has been drawn in the map so far. Currently this is limited to simple blocking in most map rendering frameworks, meaning that you can draw something independent of what has been drawn before or you only allow it to be drawn if no other blocking feature has been drawn before at the same location.
The ability to define drawing order independent of the prioritization of competing elements. Classic example where this is needed is contour line labeling: Contour line labels typically have the lowest priority of all labels and are not to be placed overlapping most of the other features in a map (including line work like roads, water lines etc.) At the same time contour line labels are traditionally cut out from the contour lines with a buffer for good readability and harmonic integration into the map. And the contour lines are drawn below almost everything else in the map. Doing this (drawing the contour lines early but cutting out the low priority labels) is not possible with most automated map rendering frameworks available these days so you will practically not see contour line labels with cutouts in such maps (with very few exceptions).
Flexible raster compositioning options in the rendering engine. Mapnik has rudimentary support for that (feature and layer based compositioning of the current feature/layer relative to what has been drawn so far). More flexibility here would be very important, in particular regarding rendering performance, because it would reduce the need for expensive geometry processing and for drawing features several times and it would practically allow developing solutions for problems that are currently not solvable in an efficient way (like the road line/polygon layering issue). Cues for this could be taken from common raster processing frameworks like ImageMagick and G’MIC.
A data processing chain that provides access to the full data model of the source data to the map designer. This has so far not been the case for the OSM-Carto tool chain for OSM data because osm2pgsql only supported multipolygon and route relations and only in a rudimentary way (without exposing relation membership information). The osm2pgsql flex backend improved this a lot though.
A text rendering engine that provides information on the would-be geometry of labels in the design rules without actually placing the label in the map.
The practical possibility to do automated integration testing of the full rendering tool chain (from the raw source data to the final raster image).

The list is roughly sorted by importance with the most important requirements on top. The first three points i regard as essential. Without meeting these points at least in a rudimentary fashion the map design work i do is practically impossible. That is the main reason why so far i have very little interest in working with any of the client side map rendering frameworks that are popular these days – because they substantially lack in all three of these points without a clear perspective of improvement in any of them.

September 14, 2023
by chris
0 comments

Flooding in eastern Libya

Supplementing the previous post about the extreme rain and resulting flooding in the eastern Mediterranean i also prepared an image from the other side of the sea at the north African coast in Libya, where the same storm that caused the flooding in Greece also lead to extraordinarily intense rainfall and flooding a few days later.

Most of the damage and fatalities due to the flooding happened at the coast in the north in the steep and narrow valleys, indirectly visible in this image mostly due to the sediment carried by the streams in these valleys into the sea and coloring it.

More impressive in this picture is, however, the extensive flooding in the desert further south. In general the flatland regions of much of northern Africa have been very dry during the past few thousand years compared to the long term average before. The more humid long term history of northern Africa is still well visible in many places in the form of valleys shaped by water flow and extensive lakes which are permanently dry in more recent history. In a way the flooding we can observe now in the desert of northeastern Libya gives us a bit of an idea of how the conditions might have been in the Sahara desert thousands of years ago.

September 12, 2023
by chris
0 comments

Flooding in the Thessalian plain

A few days ago a storm has brought extraordinarily intense rain to some parts of the Eastern Mediterranean, leading to substantial flooding in some regions. I prepared a rendering of a satellite image recorded on September the 10th of the Thessalian plain in Greece, showing the flooding a few days after the event around the course of the Pineios River that i wanted to show here.

You can see both the primary flooding in the western part of the plain around Trikala and the secondary flooding further downstream to the east caused by the huge amounts of water raising the level of the Pineios River, right up to the coast were an impressive sediment plume can be observed in the Mediterranean Sea.

I don’t know how useful that is – but i put up the image in my images for mapping in case it is useful to update OpenStreetMap in the area (like indicating impassable roads). Note, however, that mapping temporary water cover itself in OSM is not a good idea.

September 3, 2023
by chris
3 Comments

A note on OSM-Carto

I like to make another note on OSM-Carto here. Background is that there is another wave of negativity (aiming to find a relatively neutral, yet at least a bit descriptive term here) being expressed by some people from the OSM community on some channels regarding the project.

It is not the first such articulation and it was pretty much to be expected that this would become a recurring pattern. The background i already explained at length back in May 2022. Back around that time i made the decision (and i indicated that in the text) that i would continue trying to do my best to support the idea of OSM-Carto, to be an openly cooperative map design project, governed by consensus of a diverse group of maintainers.

What i do is I provide guidance to contributors how to make changes to the style in a way that complies with the documented goals of the project as i understand them and with the unwritten conventions of the style. I explain the historic context how OSM-Carto came to render things the way it does today, explain how things connect design wise creating a harmonic overall result and provide ideas what possibilities exist to resolve the various technical and design problems the style is faced with. And i listen to arguments that challenge the views i articulate and, when arguments are convincing, re-evaluate these views. At the same time i explicitly avoid standing in the way of consensus forming among the other maintainers even if i disagree with those ideas. And i merge changes when they are suitable for the style in my opinion. What i don’t do is approving changes that i don’t stand behind merely because of pressure from others to have them in the style.

I took this approach despite knowing that the chances of OSM-Carto finding its way back to a sustainable strategic consensus among the maintainers would be very small. The alternative would have been to try essentially staging a coup and to attempt single-handedly reshaping the governance of the project in a way that i would consider more sustainable. Or i could have abandoned the project. This would either have killed the project or it would at least have lead to an end of the diversity in governance. And i was not willing to do either of these things because that would have clearly betrayed the idea of OSM-Carto as described. What i did instead was summarizing some of the lessons learned in terms of governance and social dynamics and some key points that should – from my perspective – be taken into account in case someone would want to bootstrap a new cooperative map design project with a similar paradigm as OSM-Carto. It was unclear if something along these line was going to happen of course (and it still is) or if the OSM-Community is going abandon the idea of consensus based cooperative map design and move back to the more common, more traditional single benevolent dictator paradigm or to something governed by a homogeneous group of like-minded people with common interests. But from today’s perspective i stand firmly by my decision to continue holding up the ideals of OSM-Carto but decidedly not trying to actively shape where OSM community map design is going beyond OSM-Carto while continuing to share my experience and thoughts on the matter with everyone willing to listen – both within OSM-Carto development discussion and on this blog.

The recent wave of negativity towards OSM-Carto is remarkable in the level of resentment that is articulated towards the values of the project, its maintainers and me specifically as the person most outspoken about the project and its problems. I have read quite a bit about and also have seen the effects of people radicalizing themselves in closed groups, reaffirming each other in their preconceptions and simplified friend-foe images. But i had previously never seen this process in a setting that is on public record (mostly – at least one of the most vile comments has been silently removed – not before the author received cheers from his peers though).

The whole thing is also telling in context of the bold claims of the OSMF that the channel in question is moderated with the aim that everyone feels safe and confident to take part. I criticized that idea on ethical grounds in the past but it is visible here that this claim also simply fails to live up to its promise. The level of resentment, bad faith insinuations as well as evidently and ostentatiously false claims (including the title of the topic) is remarkable. Personally i don’t really care – it is not the first time people have articulated this kind of spiteful thoughts about me. But it well demonstrates that the whole top-down moderation idea of the OSMF is an empty promise. And there are plenty of people who – when seeing this kind of conversation happening unchallenged in the OSM community – will think twice before they engage actively in this kind of social context. Note this is not me projecting, other people have articulated that very sentiment to me on quite a few occasions. And the problem here is not the lack of moderators sanctioning people breaking the rules (or weaseling around them as it is also openly done here – mit Ansage as we say in German). The problem is that there is apparently no one left on that channel having the courage to push back and calling out the insults, disrespect and plainly false claims, leading to the mentioned radicalization and reaffirmation of preconceptions in a closed group.

I like to emphasize that the sentiment of people, that when things develop in a bad direction they are trying to identify the reason and who is responsible for it, is understandable and often helpful – but also misguided here as far as assigning blame is concerned. And i think i made it clear back in May 2022 already and i will do so again here: The OSM-Carto maintainers having failed to find back to a consensus direction back in 2018 and the years after – that is our failure and we have to collectively accept responsibility for that. Now you could try to assign individual blame to individual maintainers based on how you assess their efforts to achieve consensus within the group, if you like their ideas and preferences regarding map design work or whether you approve of their style of communication or simply who you can relate to better on a personal level. I would not do that. Keep in mind that the OSM-Carto maintainer team was deliberately put together as a heterogeneous group of people, selected not by like-mindedness and common interests but because of their abilities and the diverse viewpoints and ideas they are able to contribute. This diverse composition of the maintainer team is the very essence of the project that made it unique and was largely responsible for its success. Everyone within this heterogeneous group did what they could within their unique abilities for the success of the project. So we own this collective failure as a group.

What is getting lost – and that is the key point – is that the ultimate problem is not OSM-Carto, it is the lack of anything other than OSM-Carto in the field of openly cooperative map design with an ambition of a balanced global representation. I have been pointing out that this is a problem for many years, practically every time i write about OSM-Carto. I have also pointed out that strategically investing in development of software that facilitates openly cooperative map design on that level is imperative for the OSM community. The most recent plans of the OSMF in the context of map rendering are highly disappointing in that regard. Not a single word, let alone a single cent, is invested there into the question what the needs and requirements of community map design are regarding tools.

September 1, 2023
by chris
0 comments

Map style modularization

I have finally gotten around completing the modularization of my Alternative Colors map style and i want to introduce that here.

I already discussed this briefly in context of the new symbol and label rendering system. Introducing that change depended on splitting project.mml into separate files – one for each layer – which then get combined into a single file for use by carto with a python script. I already sketched the idea that this system could also be used to have different style variants and make some more computationally expensive features of the style optional. But this was still incomplete back then.

What was missing was in particular the option to split and further modularize the lengthy SQL query of the roads layer. I added this now and also implemented the option to turn off

entrance cutouts in line barriers
implicit sidewalks on roads
sophisticated rendering of junctions of roads in ground units
contextualized rendering of fords and mountain passes
preprocesed low zoom waterbody rendering
sophisticated tree rendering
contextualized water barrier rendering

How does it work? layers.yaml defines tags for each of the layers configured and when running assemble_project.py you can select which tags you want to use. For example running

scripts/assemble_project.py -t ac

generates the standard AC-Style setup with all the sophisticated and expensive layers while

scripts/assemble_project.py -t simple

leaves out all the more complex stuff or replaces it with a simplified version.

Within SQL code different variants can be implemented with specific directives. Like:

@@if(ac)
  @@include(roads/sidewalks.sql)
@@else
  @@include(roads/sidewalks-dummy.sql)
@@end

A few notes on the purpose of the AC-Style

Since people are sometimes confused about what the purpose of the AC-Style is and why there is no interactive map based on the style available i wanted to add a few notes on that.

In my last update on the OpenStreetMap-Carto project i explained a bit the history of the AC-Style. After starting as a simple demonstrator of how a more holistic and systematic color design in the style can lead to a more harmonic, better readable and better maintainable map, it changed over time into a much broader test bed for case studies in advanced map design.

What has not changed since the beginning though is that i decidedly do not invest into producing an interactive map based on the style. I would support anyone who is interested in deploying a map based on the AC-Style with advice and if necessary adjustments to the style. But i don’t want to spend my time on doing something that i am not good at and administering a map deployment is not my field of expertise.

Why do i not actively try to get any of the developments from the AC-Style into OSM-Carto? Because for many of the core design changes there is no consensus support among the OSM-Carto maintainers. Joseph Eisenberg invested quite a lot of time and energy to get some of the fundamental color changes i developed into OSM-Carto – but without success. And many of the other changes in the AC-Style design wise depend on these changes. There are still some improvements and feature additions that could be integrated and some actually are – at times with modifications, like the allotments rendering or the ridge/arete patterns.

Do i think it would be good to replace OSM-Carto in deployments with the AC-Style? Hell no. OSM-Carto is foremost a cooperative community project where decisions are made through consensus of a fairly diverse team of maintainers. As i have pointed out in the previously mentioned update on OSM-Carto the project has been struggling with the practical difficulties of this approach for some time (for reasons i discussed there in more depth). But any project run by a single person – no matter how benevolent they are – is not a suitable substitute.

As i have pointed out in the past the OSM-Community could very much use a larger variety of independently developed map styles. Quite a bit of progress has been made on that front in the past years, but unfortunately exclusively on styles targeting the needs and wishes of specific local communities and mostly under control of a single decision maker. Nothing is in sight so far even remotely aiming to create a map style for a broad global audience in a similar fashion as OSM-Carto though.

Would it in my eyes makes sense if someone was to start off a new cooperative map design project with such a scope to base it on the AC-Style rather than OSM-Carto? Design wise a strong argument could be made for that but i would still advise against it because of the technical complexity and the massive hurdle that would pose to potential contributors. What might be worth considering though is to have some of the fundamental design changes from the AC-Style, that – as explained – do not have consensus support in OSM-Carto right now, from the start in such a project because it would make further design work so much easier.

August 27, 2023
by chris
0 comments

Watching the watchers

A bit less than two years ago the OpenStreetMap Foundation (OSMF) started centrally imposing behavior regulation on OSMF provided communication channels – channels which were previously largely individually self governed by their users. Originally this applied only on two mailing lists but since then it was successively extended more and more. I had commented on that quite in detail already back then (and before) and it seems a good time now to look at how this turned out so far.

Back in 2021 i had expected the practical consequence of this to be primarily that

community members less aligned to the Anglo-American-European Leitkultur of the OSMF would be more reluctant to actively participate on the channels in question or would withdraw from participation.
remaining participants would engage in preemptive self-alignment, adjusting their communication to diminish the risk of it being interpreted to clash with what is considered acceptable by the powers that be.

The first point is probably something that indeed happened – though it is hard to reliably measure because other changes in communication culture (like the emergence of various new platforms and channels as means of communication of the OSM community, including – among others – the OSMFs discourse based platform), evidently have an effect on that as well that is hard to separate.

The second point is something where i seem to have been wrong. Since the behavior rules imposed are largely vague and often cryptic and difficult to interpret objectively, it seems that contributors on the channels in question largely ignore them. At the same time we see a remarkable volume of punitive measures being imposed by the OSMF-board appointed moderators based on perceived violations of the rules. To me this is a bit surprising because this is not how behavior regulation typically works in Anglo-American style tech communities.

What i think shines through here is what i have pointed out in the past: that OpenStreetMap is not a tech project and local OSM communities have more recently developed a quite robust self confidence in their specific cultural styles of communication and social interaction. Nudging towards self-alignment does not work as well in OSM as it would in a group that is culturally more homogeneous from the start with more effective peer pressure towards conformity.

The way behavior regulation is enforced now contrasts quite strongly in many ways with what during the discussion in 2021 was communicated how it is supposed to work, namely through moderators also otherwise active on the channel in question defusing conflicts through counseling and being a moderating voice in the true sense of the word. This still happens, but in most cases not through the official OSMF moderators. It is positively remarkable and pleasantly surprising how well quite a few people in the OSM community manage to address even strong expressions of emotions in many cases with empathy and sensitivity. But what you can see quite frequently now is that after that, when the participants of a channel have already made substantial progress to defuse the situation and foster understanding and respect for different views among the participants in a heated discussion, the official moderators swoop in (typically without having been involved in the discussion to that point), identify a single culprit and dispense some sort of punishment on them – either merely through an official reprimand (and this way implicitly exonerating everyone else from their responsibility) or through a ban from participation. I like to emphasize this is not universally the case, there are also situations where the OSMF moderators defuse conflicts through counseling in a sensitive manner. But the described pattern is quite common in those cases where ultimately substantial punitive measures are imposed.

On the positive side – official moderation activities are quite decently documented. I can recommend everyone to look a bit through this documentation to get your own impression on how centrally imposed behavior regulation is implemented on OSMF channels these days. A bit of warning though when you look at the official record of the communications on the discourse platform referenced in these incident reports: These are often incomplete since messages have been removed afterwards and are selectively quoted in the incident reports. If you are subscribed to the mailing list mode the picture you get from some of these conversations is substantially different from that you get on the web interface where messages are not only hidden (with there still being a marker indicating where there formerly was a message) but also fully removed from the view of both the general public and logged in members. And the management of this hiding/removing of messages is done not by the official moderators but by a separate self appointed governance team, without either independent oversight or meaningful public reporting. Using the mailing list mode can help with that and with maintaining a more complete record of communication but there is apparently a built in 20 minutes delay in the mailing list mode which makes this not fully reliable either.

Looking at the incident reports and the moderation activities you can make some in my opinion quite remarkable observations. First: The OSMF-board appointed moderation team consists formally of five people. Practically, however, it seems only two of them are actively pursuing moderation. These two (both of them Americans) act as rapporteurs and the other three merely confirm the decisions prepared by these two. Second: eight out of nine documented cases of moderation activity are about sanctioning non-native English speakers for communication activities in English language.

Now i want to keep this blog post relatively brief so i will not discuss the merits of the individual cases here. As said – everyone is invited to read up on these cases for themselves. If anyone is in need of any of the messages that have been removed from public record related to any of these incidents feel free to contact me directly about them. I also won’t further analyze what the statistical observations made mean in this post. Feel welcome to share your own thoughts on this in the comments below.

Apart from that it is worth noting that the original promise that local communities will be allowed to individually self govern their own channels on the discourse platform is not kept. According to comments made, bans on the discourse platform are always implemented globally. Practically that seems to mean someone who runs into conflict with the OSMF rules on the channels under control of the official OSMF moderators will also be banned from participating in their local community channel, even if what they did was perfectly acceptable under their local community’s social standards and conventions. The interesting question is of course if this will practically also work the other way round – that someone violating the local community standards anywhere in the world also will get banned from all the other channels on the OSMF platform.

One other thing that might be food for thought: At least one of the people that have been officially banned by the OSMF-board appointed moderation team so far has made a public statement in the past indicating that they might have a learning disability.

How are things going to develop in the future? I don’t really know of course. There are quite clearly two trends active here pointing in opposite directions. On the one hand the efforts to induce cultural homogenization on OSMF provided communication channels are quite clearly successful to some extent. On the other hand we can quite clearly also observe a trend towards diversification in terms of communication channels and platforms used outside of OSMF control. And while this cannot be reliably proven of course, it is likely that the latter at least partly happens as a reaction to the former – local communities, which are denied true self governance on the OSMF managed channels, opt to establish and use channels they have more control over.

But as much as i welcome and applaud initiatives for true cultural diversity and self determined bottom-up cooperation and think this is essential for OpenStreetMap to function in the long term, it is important to realize that cultural homogenization is an attractive strategy for many as a means to facilitate and simplify cooperation in larger groups – and not only for those whose culture in being imposed on others, even for quite a few of those who need to adjust and denounce their culture in the process. True cultural diversity is hard, even for those who strongly depend on it.

deutsch The Musaicum EU-plus satellite image mosaic

July 26, 2023
by chris
0 comments

Announcing the Musaicum EU-plus 10m resolution image of Europe

I am pleased to here introduce a new satellite image product that i have been working on for a few months. Development of this has been co-financed by Geofabrik. They are going to offer tile services for web maps based on this image in combination with the Green Marble.

Background

Most of the readers of this blog will be familiar with the Green Marble – my global satellite image product offering the highest quality rendering of the whole earth surface available at a resolution of 250m. The Green Marble is produced with a pixel statistics approach, that means an analysis of all observations available is done independently for every pixel of the image to estimate the surface color at this point. This kind of technique is very popular because of its principal simplicity and because processing can be implemented in a very efficient fashion.

But this method has two main disadvantages:

It requires a significant amount of data to actually get to a point where the final product is in visual quality equal to or better than an individual good quality image. How much depends on the algorithm used and its convergence characteristics and obviously this will differ a lot between different parts of the planet. For the Green Marble version 3 for example about 1PB of raw data was processed – which means more than 100kB of data per pixel.
It does not scale well with increasing spatial resolution. I discussed this matter in more depth already back in 2018. This has multiple underlying reasons, the one that is most straight away to understand is that the higher the spatial resolution is you are looking at the more volatile the content of an image becomes. This mean the higher in terms of spatial resolution you go the less you have – on average – a long term stable state of the Earth surface your pixel statistics can converge to.

Long story short: Pixel statistics work very well at a resolution of around 250m if you have a large data basis to work with. They work poorly at much higher resolutions, even if you have a lot of data (which you typically don’t – but that is a different issue). This has not prevented various companies over the last 5-10 years to invest substantial resources in naively trying pixel statistics approaches on Landsat and Sentinel-2 data – with the expected mediocre results.

The alternative to pixel statistics for aggregating satellite imagery into a homogeneous larger area visualization is using classical mosaicing techniques where individual images are essentially put together in the form of a patchwork or mosaic. If you want a concise definition: You have a classical mosaicing techniques when the color at any point in the image is in most cases (a) primarily derived from a single source image and (b) the surrounding pixels are primarily derived from the same image. This is evidently not the case for a pixel statistics process where the processing of a pixel is not correlated to that of its neighbor.

Classical mosaicing techniques are the dominant method for aggregating very high resolution satellite and aerial imagery and for high quality images based on Landsat and Sentinel-2 data. The problem here is that achieving good quality with this approach requires fairly complex processing techniques and there are certain key steps that are notoriously difficult to automatize because the quality of the results depends a lot on competent human judgement.

Hence most satellite image based visualizations using classical mosaicing techniques either are relatively poor quality (high cloud incidences, poor consistency in colors across the seams between images) or are based on fairly old data because updates are costly.

I myself have been producing images using classical mosaicing techniques for nearly 20 years now (an early discussion of this can be found on this blog in 2013) and both improved and streamlined the methods i use over the years. But also for me hand work was so far always a key component in production of these images and as a result in many cases updates are very costly to do. Therefore i had been looking for some time at strategies to eliminate the remaining manual processing steps in the production of my higher resolution mosaics without sacrificing too much in terms of quality. With the help from Geofabrik i was now able to practically implement and evaluate some of these ideas and here i am going to present and discuss the results.

The Musaicum EU-plus – click for larger version

The image

At low resolution the image looks very similar to the Green Marble – which is not astonishing since it is assembled with the same aim – to depict the local vegetation maximum and snow minimum. If you look closely you can see the appearance is not quite as uniform as the Green Marble – with some inhomogenities that are clearly non-natural. Part of this is due to the low volume of data used (put simply: Not everywhere was there a source image available in the time period used that exactly represents the vegetation maximum). Another part is that there is still room for improvement in the data processing – after all this is a first attempt.

Swiss Alps

Western Greece

What you will quickly notice if you look through the sample images is that there are no clouds (or almost none – with very keen eyes you might spot a few, and yes, if you systematically screen the whole image you will find some more). This is a point where the results substantially exceeded my expectations. I was hoping to accomplish a cloud incidence substantially better than what is available on the market otherwise but i was expecting it to be significantly worse than in my manually produced local mosaics. The ultimate result is pretty much at par with the manually produced local mosaics with less than about one in 100k pixels severely affected by clouds. Most of these are small, isolated convective clouds.

Vlieland, Netherlands

The focus of the project was on land area visualization so water surfaces were not a particular concern. Since waterbodies tend to be quite variable in appearance and often not in a strict seasonal pattern, the results in that domain are not always ideal, in particular rivers often change in coloring along their course quite erratically. Sun glint is also a problem at lower latitudes, in particular on smaller lakes.

Gdansk, Poland

Istanbul, Turkey

A few words on the data used for producing the image. In analogy to the numbers i presented above for the Green Marble: The volume of original Sentinel-2 data processed for this project was about 20TB, which means it needed less than 250 bytes per pixel. This is extremely sparse considering that the volume of Sentinel-2 data collected for Europe within a single year alone is much higher. A low volume of data to process helps keeping the processing costs low and it also allows using more expensive processing techniques. And in contrast to pixel statistics methods where adding more data always helps, there is no advantage per se with classical mosaicing techniques to use more data, it is more a quality over quantity thing.

What has been more challenging is that i wanted to keep the time-frame from which to source data from relatively short. Preferably just three years (2020-2022), where necessary another year (2019) and only in rare cases also data from 2018 is used. In areas where the earth surface appearance is very volatile – either across the seasonal cycle or between years – this makes it difficult to produce homogeneous results over larger areas.

Paris, France

Cordoba, Spain

What you have to work with

One of the things that made work on this quite substantially harder than you would naively expect it to be is the poor quality of some of the data supplied.

Sentinel-2 data can be obtained in two different forms:

L1C data – which is the original Top-of-Atmosphere reflectance values as recorded by the satellite
L2A data – which is as estimate of the surface reflectance data based on the TOA measurements

Now most experienced users of satellite imagery will understand that the L2A data is – as i characterized it – just an estimate of the surface reflectance. And while this should reduce the variance due to the variable influence of the Earth atmosphere it will also introduce variance in the form of various forms of noise and systematic and random errors in the estimate. What, however, most data users will not realize is that the Sentinel-2 L2A data distributed also attempts to compensates for illumination differences (shading) and that this compensation – to put it bluntly – is incredibly bad. So bad that it is practically not usable for visualization purposes. Here an example – larger version is linked:

S2A L1C from 2021-07-17

S2A L2A from 2021-07-17

For comparison here the new Europe mosaic (which in the default rendering is an atmosphere but not shading compensated version) and my own shading compensated version. For most of the sample area the mosaic is based on the same recording – except for the top right where the July image contains residual snow so the mosaic uses a later season image.

The new Europe mosaic – Tena Valley, Pyrenees

Shading compensated version

The overall color difference is not the main point here (the tone mapping i applied to the L1C/L2A data is somewhat arbitrary). The main point is that the illumination compensation in the L2A data massively over-compensates, leading to the shaded slopes often being brighter than the sun facing slopes. Also it is geometrically fairly inaccurate, leading to a strong emphasis of ridges and the introduction of substantial amounts of high frequency noise.

There seem to be quite a few people using Sentinel-2 L2A data in visualization applications. That is not a good idea. You should only use shading compensated imagery in visualizations if you know exactly what you are doing and in that case you should use a proper method and not the hatchet job offered by ESA here.

I have discussed the matter of shading compensation, in particular for use in 3d rendering, previously. A shading compensated version is available for the Musaicum EU-plus as well – but i have not had the time for a proper evaluation yet. It is available on request.

What remains

As i have mentioned earlier – this is a first attempt at a more highly automated classical mosaicing process. As often with this kind of work many things remain unclear during process development and only become clear once you run the process at scale. This might sound a bit like undue perfectionism considering the results are pretty good. But this is not just about the quality of the results but also about the robustness and efficiency of the process. As a general rule of thumb for a project like this i would say: Putting as much work into process development after the first use at scale as before is reasonable if your aim is to make full use of the potential of the method used.

Svartisen, Norway

Where to go from here

One thing that i am sure quite a few of my readers are going to ask is: Does this also work beyond Europe? In principle the answer is yes, but there is more to it than just running it on another volume of data. Quite a significant part of the development work that went into this project was for tuning and adjusting methods and parameters for the specific settings in terms of climate and surface characteristics that can be found in the processed area. I know from experience with the Green Marble that the variety of different settings on Earth is larger than people usually expect. This needs to be taken into account and that was not part of this project.

There are a number of other additions and extensions of the project that i would like to work on, for example producing a vegetation map similar to the ones available for various local mosaics is is on the list. And of course over time there will be the question of updating the mosaic with newer data. I don’t know if and when i will have the capacity to do any of this. If any of the readers are interested in supporting further work on any of this please feel welcome to get in touch.

You can find the product description and more sample images on the Musaicum EU-plus product page.

Many thanks to Geofabrik for co-financing this project. If you are interested in tile services based on the the Musaicum EU-plus or the Green Marble, people at Geofabrik will be happy to help you. If you are interested in licensing the image for other purposes you are welcome to contact me.

Tödi, Switzerland

Zakynthos, Greece

July 26, 2023
by chris
5 Comments

Satellite image sources for OpenStreetMap mapping

There has been quite a bit of fuzz in the OSM community recently (like here – warning: link goes to patronizing and broken web interface) because one of the image layers with semi-global coverage that had so far been widely used by mappers as a source for remote (armchair-) mapping has been turned off.

Calls for the authorities (in other words: the OSMF) to fix this were quick and the OSMF board seems to try persuading the image provider to restore the status quo ante. Sadly, however, the event seems to have not initiated any larger scale reflection within the OSM community on its dependency on proprietary data providers, which has increased significantly over the past years.

In wealthy parts of the world with active mapper communities local mappers have over the years invested significant work into collecting suitable local image sources (i.e. aerial images) and obtained permission to use those for mapping in OpenStreetMap from their providers. This not only applies to Europe and North America but also other parts of the world like Japan and parts of South America for example. This is an impressive achievement and highly useful for the practical work of mappers. And because all these image sources are independently produced and provided by different local image providers there is no problematic large scale dependency on a single image source because of that.

But this only applies to a rather limited part of the Earth land surface. For the rest OpenStreetMap currently largely depends on a single satellite image provider (Maxar) and image layers based on Maxar imagery provided by a small number of US Corporations (Microsoft/Bing, Esri, Mapbox and – until recently – Maxar itself). This problem is aggravated by the fact that it is in particular those parts of the world where no local aerial image sources are available where OSM currently lacks significantly in mappers with local knowledge and over-proportionally depends on remote mapping. In addition all of the listed image layers to varying extent have a focus on those parts of the world where other local image sources are available as well and elsewhere often have more patchy and lower quality coverage.

I have pointed out in the past that an important avenue for the OSM community to mitigate this dependency is to focus more on using open data satellite imagery. Even if this cannot fully replace commercial images, open data satellite imagery is currently severely underused in OSM, largely because of the lack of convenient practical availability of high quality images from such sources to mappers.

Of course there are other options the OSM community could try to decrease the current dependency on a single imagery provider:

invest in recruiting mappers with local knowledge in larger parts of the globe (which would of course require the English speaking influential parts of the OSM community to open up more to true cultural diversity).
invest in the capabilities of mappers to map in high quality in the absence of high spatial resolution imagery. It is quite remarkable how dependent even mappers with local knowledge mapping using on-the-ground surveying are often on image availability. Doing so would consist both in educating mappers in techniques that do not rely on high spatial resolution imagery and equipment that allows precision mapping independent of imagery.
diversify the supply of commercial satellite imagery. In the resolution class of Maxar (0.5m GSD or better) there is only a single other provider at the moment (Airbus/CNES) but in the slightly lower resolution range (1m GSD or better) there are quite a few more. I am not aware of any initiative from the OSM community to organize access for mappers to imagery from any of these sources on a larger scale.
lobbying for opening aerial imagery sources in parts of the world where this exists but is not available for mapping in OSM yet.
invest in production of open data aerial imagery, in particular recorded by UAV.
better availability of alternatives to optical imagery for mapping in OSM. There are various parts of the world where no very high resolution optical imagery is available for OSM but other open data sources are – like elevation data in polar regions.

Making open data imagery more accessible for mappers

I have tried with my OSM images for mapping over the last years to demonstrate how competent selection and high quality processing of open data satellite imagery can be useful for mapping in OpenStreetMap. I have added some more images of the Antarctic now, substantially reducing the gaps in coverage of ice free parts of the Antarctic.

I have chosen the Antarctic for this in particular because the proprietary higher resolution image layers tend to not have coverage there (or are very patchy). And also because of the high contrasts between ice and ice free areas the tone mapping used by many of the global image layers works poorly in these regions. Unfortunately, mappers in OSM seem to have a tendency to almost universally pick higher spatial resolution images over lower spatial resolution images, even if those are in all other aspects substantially worse (like more than ten years old with seasonal snow cover or poor processing). In other words: Larger scale use of open data satellite imagery in OSM is not only hampered by difficult access to such imagery in high quality, it is also made difficult by the lack of knowledge on the side of many armchair mappers for competent assessment and selection of the best image source for a specific mapping task.

Change in terms of use

Finally i have also changed my terms of use for the image layers for mapping i provide. This is owed to the increasing use of self adjusting algorithms (a.k.a. artificial intelligence) – in general and for satellite image interpretation in particular. The terms i have chosen mean that you can use my images with such algorithms only if the algorithms are fully open source – including the training data and training algorithms. I consider this a prudent choice considering the widespread trends towards neo-feudalism in the world of digital services in general and AI methods in particular. OpenStreetMap can only maintain its paradigm of a self determined cooperation of people with local knowledge sharing this knowledge with one another and the rest of the world (a map by the people for the people) if mappers retain full control over the methods of mapping. And that includes the algorithms used for image analysis.

July 5, 2023
by chris
1 Comment

Drawing the line (3) – beyond simple line patterns

In this series of blog posts (see part 1 and part 2) i have so far discussed the limitations of current digital map design frameworks regarding the use of line signatures compared to what was state of the art in the pre-digital age. And i provided some examples of what design issues result from the technical constraints we are faced with and showed various methods that can be used to work around these technical constraints to some extent.

All of this was so far about either constant or periodically repeating line signatures. Here i want to show that this is not the end of options to use line signatures in maps. Like the previously discussed designs this is also available in the AC-Style for everyone to use and study.

The feature i am going to look at here is crevasses. Crevasses are so far not a very widely mapped feature in OpenStreetMap, they are kind of a niche interest only relatively few mappers work on. This is partly because mappers sometimes think that, since individual crevasses are very volatile, it is pointless to map them. They often don’t realize that, while individual crevasses move and change fast, the location where and the patterns in which they occur in are typically very stable, often much more stable than the extent of the glacier itself. Mappers who map crevasses in OpenStreetMap use both polygons and linear ways to map these and both can be reasonable choices depending on circumstances. For providing constructive feedback it is therefore prudent that a rendering solution ensures a decent depiction of both variants.

Crevasses in the Antarctic

Crevasses are an interesting and challenging element in map design because they occur in a very broad range of sizes – the width of a crevasse can be from less than a meter to dozens of meters and the length from a few meters to many kilometers. And – as already hinted at – they form complex and in most cases fairly stable patterns that are often highly characteristic and significant for navigation on the glacier.

Because of this, rendering of crevasses is an important element in many traditional maps of glaciated regions and many techniques have been developed to depict crevasses at different map scales. Here are a few examples:

Crevasses in Swiss 1:25k topographic map

Crevasses in French 1:25k topographic map

Crevasses in Soviet 1:50k topographic map

For the design concept i am going to present here i took cues also from early modern maps of the Antarctic, in particular:

Australian 1:100k Aker Peaks from 1966 – source

New Zealand 1:250k provisional edition Cape Hallett from 1967 – source

The basic high zoom level design for single crevasses represented with polygons and linear ways is shown here (together with the other linear features common in this context – see the previous post)

Crevasses and other glacier details at z20

Crevasses and other glacier details at z19

Crevasses and other glacier details at z18

Crevasses and other glacier details at z17

Crevasses and other glacier details at z16

Crevasses and other glacier details at z15

Crevasses and other glacier details at z14

Crevasses and other glacier details at z13

It uses a relatively simple line pattern as the main component – but not on the linear way representing the crevasse, but on a constructed geometry in the form of an open crack along the line of the crevasse, tapered towards the ends. The width in which this geometry is drawn depends on the zoom level, the length of the crevasse and potentially the tagged width. The level of detail of the drawing (with line pattern on the outline or simple constant width line, with center-line or not – or even just a simple center-line) depends on the drawing width.

Crevasse drawing width depending on length at z20

Crevasse drawing width depending on length at z19

Crevasse drawing width depending on length at z18

Crevasse drawing width depending on length at z17

Crevasse drawing width depending on length at z16

Crevasse drawing width depending on length at z15

Crevasse drawing width depending on length at z14

Crevasse ground unit rendering based on tagged width at z18

The real key to a harmonic design, however, is how crossing crevasses and combinations of polygon and linear way mapping are drawn. This is shown here:

Rendering of intersecting crevasses at z18 (link shows double resolution version)

So far, most of the crevasse mapping in OpenStreetMap makes relatively limited use of these options. Here are a few examples:

crevasses mapped with polygons – Séracs du Géant at z15

crevasses mapped with polygons – Séracs du Géant at z16

crevasses mapped with linear ways – Ghiacciaio del Dôme at z16

Conclusions

What i have shown here with the crevasses is an intermediate design approach between the use of higher level features discussed in the previous post and the fully manual drawing of line signatures as i have shown it for tree rows in the past. It uses manually constructed geometries but visualizes them with the help of line patterns. Practical usefulness of this based on OpenStreetMap data is so far rather limited due to the very patchy mapping of crevasses in OpenStreetMap.