Imagico.de | blog

September 29, 2024
by chris
2 Comments

OSMF board elections 2024 – delving into the post-truth era?

The elections for the board of the OpenStreetMap Foundations are upcoming and the most interesting part of this for the OSMF members has traditionally started with the publications of the self-presentations of the candidates, which happened during the last days.

My impression after the first reading of these is that the OSMF seems to have arrived in the post-truth era. Some might claim this has already happened years ago, but to me – and i have been a keen observer of the OSMF for many years now – it became obvious with this year’s election.

The concept of post-truth does not mean the communication of lies or untrue statements, it refers to the trend of defactualization, where the distinction between false and true becomes meaningless in communication and discourse in a certain social context.

There is an euphemism for this style of defactualized presentation commonly used in corporate PR and marketing these days, that is storytelling. The aim here is to create a certain impression or emotional reaction from the recipient of the communication largely sidestepping the facts surrounding the topic. Or in a nutshell: If the story is compelling it does not matter if it is convincing.

The irony is that originally, the term storytelling referred to something rather different: The development and communication of fictional stories either for the purpose of transporting abstract philosophical ideas and moral values in a practically robust fashion in a culture without writing or for the purpose of communicating ideas and thoughts that are subject to social taboos or otherwise not socially accepted in a non-fictionalized form.

My suggestion to the OSMF members for this year’s board election is therefore: Read the self-presentations of the candidates as fictional stories – like fairytales, mythology etc. And do not regard what you read there as propositions in the philosophical sense. If you do so i think you will be able to derive much more of value from your reading. If that is then helpful for your voting decision is a different matter of course. But you will definitely spare yourself some of the pain i had in my first reading of these.

I had considered analyzing this year’s board candidates’ self-presentations for examples of post-truth era communication techniques. But that does not seem right to me. After all it is not the candidates that are to primarily blame here. It is the OSMF members who are – collectively – too passive to set a higher bar for the candidates they elect for the board. But more than that: I also have the impression that quite a few OSMF members have fully accepted the end of history and simply can’t imagine a different kind of candidate. To use an old metaphor: They drink the sand because they don’t know the difference.

For those i will try here – in the tradition of storytelling in the original sense – to tell the story of a fictional board candidate who stands in contrast with the largely defactualized self-presentations we are seeing in this year’s election.

Manifesto

Dear OSMF members,

i am not actually a candidate in this year’s OSMF board election. Nor am i actually a real person that exists outside the mind of my creator, the author of this blog post, and the minds of you, esteemed readers of these lines. None the less i hope that my fictional candidacy in this year’s election inspires you to imagine the idea that some actual candidates might articulate and pursue some of the thoughts i present here in my fictional candidacy with the same clarity and determination as i do.

Feel free to customize the image that my creator has produced in these lines to your cultural or personal preferences and expectations.

Motivation and Objectives

What i hope to accomplish during my time on the OSMF board i am going to outline in the answers to the questions provided. The reason why i make these things my objectives is because, after years of observation and contemplation, i have come to the conclusion that these are crucial for a sustainable development of the OSMF under the goal of supporting the OpenStreetMap project. I am aware that the practical pursuit of many of these objectives is likely going to see opposition from substantial economic interests around the OpenStreetMap project. If i succeed in pursuit of these will depend on your support and that of the rest of the OSM community. I am going to do my best to accomplish these objectives within the OSMF board, but especially in the long term i will only be successful in that if i have active support for that from the OSMF members and the OSM community.

Conflict of Interest Management

First of all i reject the premise of the question that Conflicts of Interest are something that can be universally managed or mitigated without actually addressing the Conflict of Interest itself.

My personal situation allows me more than many others to pursue the interests of the OSM community even if they are in conflict with weighty economic interests around the project. I am not rich by any measure but my livelihood, my professional career and my personal goals are out of reach for even the most influential financiers of the OSMF or of economic interests around OpenStreetMap. So, even if those would want to put pressure on me personally to put their interests above those of the OSM community, the possibilities for that are very limited.

That being said there are, of course, going to be situations where i am going to have Conflicts of Interest in my work as a board member. I hope that i am going to be able to identify such situations myself in most cases. But i am aware that i cannot expect this to work reliably in all cases, so i will need to resort to other mechanisms as backups, which i will discuss in a second. Before that i am going to outline the three measures i expect to consider when i have become aware of a Conflict of Interest as a board member:

Selectively removing myself from the part of my work as a board member where i have the conflict. This only works in cases when the work in question, as well as the Conflict of Interest, are tightly limited in scope. In case of any kind of decision making, removing myself would mean both from the actual formal decision and the complete deliberation process leading up to the decision.
Temporarily suspending my work as a board member altogether. This only works in cases where the Conflict of Interest in question is inherently limited in duration.
Resigning my position as board member. This is always the ultimate fallback option in cases of Conflicts of Interest where no other measure can reliably ensure that the secondary interest does not affect my work as a board member.

All of these three options are going to be on the table whenever i become aware of a Conflict of Interest as a board member. I am going to transparently and publicly document the reasoning behind my decision of what measure to choose in any such case, so that the OSMF members have the opportunity to check on that reasoning and – if necessary – take the appropriate steps to insist on a different option.

Now back to the problem of reliably identifying Conflicts of Interest. The history of the OSMF board has shown beyond doubt that self-identification does not reliably work and even cross-checking among the board members does not, because the other board members are either equally unable to see the conflict or feel socially inhibited to openly point out the conflict of another board member (see here, here and here.

Therefore, other mandatory mechanisms will need to be introduced and this is going to be one of the things i am going to pursue as a board member. Transparency in decision making processes is going to be a key element of that (see next question). Beyond that, i concretely envision to suggest the following measures to the OSMF membership as binding to the board in their work:

Creating a mechanism by which decisions in which a Conflict of Interest has been overlooked are automatically nullified.
Introducing mandatory formal reporting of any Conflict of Interest identified as such by a board member to the OSMF membership.
Creating a mechanism by which Conflicts of Interest can be reported anonymously by any OSMF member.
Creating an oversight mechanism (ethics council) formed by the local chapters that oversees the board’s work with regards to Conflicts of Interest and that reports their findings to the OSMF membership as an additional control mechanism.

Transparency and Accountability

Transparency and Auditability of the work of the OSMF have seen a massive decline during the past years relative to the state after the transparency initiative of the 2015 board:

While originally all of the board meetings were public, the meetings are now split into private mid month meetings were actual deliberation of decisions seems to take place and the public meetings where formal decisions are still made but in depth discussion does not typically take place any more.
Policy document drafts are now routinely published only in the last minute before the formal decision (either during the meeting or in the day before the meeting) depriving the OSMF members from the possibility to review them and provide input before the decision. This communicates substantial disrespect from the board for the OSMF membership.
While some years ago the OSMF board routinely discussed policy development with the OSMF membership at an early stage (both on the board and on members’ initiative) this has almost completely stopped. And if the membership is consulted at all, it is usually about approval of a final draft as a done deal after the main deliberation has already happened in private.
The board does not seem to be bound by their own policy – for example the commitment to open channels is rather loosely interpreted – see here.
The board’s main method of internal communication and deliberation seems to have moved from channels with a permanent record (email) to volatile channels (IRC/Matrix) preventing both independent auditing of the board’s decision making and the board’s own ability to look up the genesis of past decisions.

This is a highly problematic trend on multiple fronts – in terms of oversight and dealing with Conflicts of Interest (see previous question), in terms of recruitment of competent volunteers (if no information is visible from the outside that massively steepens the learning curve for everyone who wants to get involved because they can only access pertinent information after they have become part of the inner circle) and in terms of public communication (transparency by default spares a lot of work in explicitly and selectively communicating only what is meant to be known to the public). Also the compartmentalized information management in the current OSMF stands in sharp contrast with the way the OSM community communicates otherwise, which creates a substantial culture gap between the two.

As a board member i am going to

make sure the OSMF board moves back to channels with a permanent immutable record for internal deliberation by refusing to participate in non-public inner-board communication on channels without such record. The need to be able to look up and refer to past communication in deliberation of the board is so obvious that i feel forcing the hand of the rest of the board this way is warranted.
start publicly reporting on developments in the board as soon as they happen even if they would otherwise not be known to the public at that time. I am not going to disclose content of communication without agreement in the board of doing so, but i believe that the OSMF membership and the OSM community as a whole – who the board is meant to serve – have the right to know for example when the board starts to draft policy affecting the OSM community or starts discussing to contract someone to do certain work for the OSMF and i don’t think, as an elected board member, i would be infringing on anyone’s moral or legal rights by reporting in a timely fashion about such things happening as they do.
present and argue for the advantages of a public-by-default approach to board work with the other board members and attempt to roll back the roll back of the transparency of board work of recent years. I am aware that this might be an uphill battle against an existing organizational culture of compartmentalization. But the benefits of a more open work culture are difficult to ignore.

Strategic Vision and Sustainability

The strategic planning initiated by Allan Mustard in the OSMF was a significant step and it could have been a good starting point for further work in that direction despite some shortcomings. But what the board then made out of this ambitious start, by essentially removing all the clarity and stringency in the original document and replacing it with soft and vague expressions of wishes, amounted to nothing less than moving back to the muddling through the plan originally aimed to overcome.

As a board member i would aim to re-initiate the strategic planning process, starting with the original work of Allan Mustard. I would aim to

Fill the thematic gaps and omissions in the original plan.
Move the discussion of further development to a public venue, giving all interested community members the opportunity to read and provide feedback and inviting people with experience in specific fields to provide advice (in public to provide checks and balances against lobbying for special interests).

But i would in particular aim to introduce a strict subsidiarity principle into the strategic planning of the Foundation. The OSMF should not pursue tasks that can be equally or better handled on a local level or by thematically more specialized organizations. The OSMF needs to focus itself on its core functions as outlined by the OSMF mission. This is more than enough of a challenge with the continued growth of the project.

I also consider a move of the OSMF to the EU a strategic necessity, as well as a chance to restructure the organization in a more robust and more scalable way. The necessity stems from the legal protection of the OSM database under the ODbL, which is based on European database rights. And the move would provide the chance to design a new organization with proper formalized checks and balances, because it would not be subject to the tight constraints of UK corporate law. I envision a structure where the operational management of the organization and policy development are separated and where the local chapters have a formal role of oversight over key aspects of the organization (like decisions about the rights to the OSM database, trademarks and the mapper accounts).

The steps taken by the OSMF board so far regarding a move to the EU seem to suffer from a lack of transparency in decision making. In particular the choice of countries to consider moving to seems to be based on behind the scenes lobbying of special interests, rather than an open and transparent selection process. And it also suffers from the lack of ambition with regards to restructuring the organization. As is it seems likely that the board would try to copy the highly disadvantageous structure of the existing OSMF as required by UK corporate law into the EU and this way miss a huge opportunity that is likely never to come again. Bottom line: I think the whole process needs a restart with a more transparent process and a broader discussion how the new organization should be structured.

Decision-Making and Collaboration

I do not plan to collaborate with anyone during my time on the board, but I hope to cooperate successfully with my fellow board members, with the working groups, with the OSMF membership and with the OSM community as a whole. The key to successful cooperation is a basis of shared goals and values to start from. In the case of OpenStreetMap, this cannot be shared cultural values but has to be the basic ideas and values of cooperative collection of local geographic knowledge. Everything else (like the traditional OSMF mission and its interpretation) needs to derive from that. If there are disagreements in the OSMF about how to practically pursue the OSMF mission, i would try to solve that through arguments and reasoning with regards to the basic ideas and values of OpenStreetMap. In other words: I would ask others to try to convince me that what they suggest is in support of these and i would try to convince them that what i suggest is beneficial in that regard.

As far as the organizational structures within the OSMF are concerned – i think the original idea of largely independent working groups is very good. But the board has, in recent years, substantially crippled their development by frequently interfering ad hoc with their independence and by creating board committees with non-board members as a competing, less independent but much more favorably treated organizational structure. This whole concept needs to be re-considered together with reducing the scope of the OSMF’s own activities in favor of the local chapters.

What would definitely end with me on the board is the current practice of keeping inner-board conflicts and disagreements under cover and the attempts to hide such from the membership. This is not a good way of dealing with disagreement and such pretense damages the credibility of the board as a whole to the outside.

Fundraising and Resource Development

According to the self-presentation of the current OSMF in public, fundraising seems to increasingly turn into a goal on its own. That is not a good development. The expenses of the OSMF have quite significantly increased over the past years but the level of actual internal organization of the Foundation has not kept up with that. And that is not a problem caused by the lack of money, it is caused by the lack of determination in ending the muddling through that Allan Mustard has rightfully criticized in his sketch for a strategic plan. If the OSMF would try to address this by raising and spending additional money (or in other words: try to outgrow the problems through financial expansion) the result would be an organizational automaton the de-facto primary goal of which would be to sustain itself and grow as an organization, meaning raising funds for the main purpose of continuing to raise funds.

As a board member i would aim to follow a different approach. I would try to consolidate or even reduce (based on the subsidiarity principle, by ending the OSMFs involvement in activities that are outside its core functions) the current operational scope of the organization and focus on putting that scope on a more solid and robust basis. That involves in particular

developing, consolidating and publishing internal policy that is binding for everyone regarding the routine operational activities of the organization. In other words: Writing a comprehensive OSMF handbook covering the current operational scope of the organization.
detailing the strategic planning to the level that it provides clear guidance to all the operational activities.
opening up to the wider OSM community in operational activities and strategic planning and making supporting the OSMF through volunteer work attractive again for intrinsically motivated grassroots volunteers without a professional motive.

Of course, even with this approach followed with determination, the OSMF will need continuous fundraising to sustain itself. The goal here needs to be IMO to better diversify the funding. I would try to establish the following goals:

No more than 10% of the OSMF’s income should at any point of time come from a single source (and indirect funding via proxy would need to be considered here of course)
No more than 30% of the OSMF’s income should at any point of time come from a single country (again with indirect funding through dependent organizations in other coutries considered)
No more than 30% of the OSMF’s income should at any point of time come from a single thematic class of financiers (currently for example digital technology companies could be a class where this is potentially an issue)

Accomplishing these will likely require cooperation with local chapters around the world. The OSMF therefore needs to find a sustainable way to handle the competitive situation with the local chapters with regards to funding. The idea currently pursued to incentivize a funding dependency of the local chapters on the OSMF by trying to take a proxy and gatekeeper role between big corporate financiers and the local chapters is a very bad one in my opinion. The opposite approach would, in my opinion, be better with the local chapters being the direct contact with local financiers of OpenStreetMap and handling the local financial bureaucracy. This would also conform with the subsidiarity principle i dicussed above, while the current approach does not.

Handling Legal and Political Challenges

A thorough assessment of legal risks faced by the OSMF is quite clearly long overdue. This should be combined with the development of mitigation strategies that avoid the possibility of core assets of the OSMF (the database rights, trademarks and mapper database) to be at risk in case of serious legal challenges. Ideas for that can be implemented in the course of a move to the EU as sketched above.

On the political front, the OSMF still has a lot to do to repair the damage incurred by the infamous Crimea decision in 2018 (where the board overruled standing OSMF policy, community consensus and the data working group in an attempt to pacify some loud voices articulated in the OSM community and political pressure). The board needs to actively reaffirm standing behind the core principles of OpenStreetMap in documenting the observable reality on the ground – even in cases where this is politically not fashionable. A consistent stance based on clear and broadly supported principles in the long term is a much better defensible position than opportunistic adjustment to the wind direction of the day.

State of the Map

The operational planning of the State of the Map conference should remain at the discretion of the SotM working group and i would, as board member, oppose any attempt of ad hoc interference with that as it happened in the past. Strategically, i would encourage the SotM working group to develop the concept of the State of the Map conference in the following directions:

supporting predominantly regional conferences in a rotating fashion rather than moving an international conference around the world. This would IMO be more suitable to celebrate the cultural diversity of the OSM community world wide.
developing concepts of a decentralized conference with different local meetings of people connecting digitally to the event.
developing asynchronous formats where people can watch presentations at a daytime of their chosing and asynchronous discussion afterwards takes place between presenter and audience.

And to be clear: The OSMF cannot and should not aim to ensure that the conference is safe and accessible for all members of the global community. That idea would be completely impractical if global community really means global community – and it would be morally highly questionable if global community means only the wealthy international OSM jet set. The OSMF should be open to support an international conference in any part of the world where there is an active local OSM community willing and able to organize such an event. If, in some years, that happens to be in places which most people from Europe and North America practically won’t want to or are not able to visit, that is to be accepted.

Your Community Contributions

I have contributed to OpenStreetMap on various levels – in mapping, in tagging discussions, in software development, in map design and in public communication to name the most important. But i have no strong focus on any of these fields in particular that would put me at risk of aiming to cater the specific interests of that field.

I have also followed the development of the OpenStreetMap Foundation for many years as far as this is possible from the outside as a normal member who is not part of the inner circles. And i have frequently talked to board members during that time about the OSMF. But i have never been involved in the OSMF myself beyond being a member – hence it is safe to say i have never been socialized in the organizational culture of the OSMF. This gives me a relatively well informed outside perspective.

I am not a native English speaker and besides my native language and English i also have basic knowledge of another language with wider use across different countries (like French, Spanish, Russian, Arabic, Chinese – pick whatever you like most). This helps me realize that by limiting your horizon to English language only (like the OSMF mostly does at the moment) you miss out on significant cultural diversity and loose immense opportunities. This is why the OSMF urgently needs to start embracing more diversity in languages in its organization. That is a challenge practically, but IMO also a strategic necessity.

Promoting Community and Attracting Volunteers

Recruitment of qualified and intrinsically motivated volunteers has become a big problem for the OSMF during the past years. The OSMF has alienated a lot of highly qualified long term volunteers from the OSM community over the past years through the principle of people whose work we know and enjoy as a recruitment paradigm for both paid and unpaid work in the OSMF assigned by the board. This not only urgently needs to end, the OSMF board needs to take substantial steps to win back credibility with the larger OSM community where the do-ocratic principle is highly valued.

The OSMF also has a lot of homework to do with regards to inclusivity. Getting involved in the OSMF without substantial English language communication ability and without embracing the OSMF organizational work culture is next to impossible. And people usually don’t even have the chance to get a good idea of these requirements before actually taking the step to get formally involved because most of the OSMFs activities happen behind closed doors and are only accessible to those who have decided to get formally involved. That is literally the opposite of being welcoming.

Due to the independence of the Working Groups (which is important to hold up) the board has only limited ability to directly top-down change this situation. But we can lead by example and provide helpful recommendations and organizational support for the working groups to reduce the barrier to get involved as volunteers.

There are a number of concrete steps i would like to pursue in that regard as board member.

Set up a place where work for the OSMF that can be independently pursued by volunteers is openly advertised for community members to take on without a formal barrier. This of course will require more real time transparency in the work of the OSMF in general (so people understand the context of these tasks). Initially that would be for tasks created by the board but the idea would be that this approach could also be adopted by the working groups.
Move more active work from the OSMF website (with top-down control of editing rights) and elsewhere to the OSM wiki (where all OSM community members can get actively involved)
Being more transparent in all practical board work as this reduces the barrier and increases the incentive to get involved.
When consulting the larger community in a formal or informal way specifically inviting and valuing critical and inconvenient comments and reactions.

The real challenge is going to be breaking the English language dominance. I have no definitive approach how to tackle this at this time. But not accepting the currently dominant paradigm that the English language dominance is inevitable is a start. Revising the so called diversity statement in that matter would be a symbolic first step.

Technology and Innovation

OpenStreetMap relies heavily on technology but is not a technology project itself. That is important to keep in mind for the OSMF. So technology for the OSMF always needs to be a means to the end of supporting the cooperative mapping efforts of the OSM community, not a goal on its own.

The OSMF needs to keep a close eye on the technologies OpenStreetMap relies on in its core functions and their development. But in doing so we also need to keep in mind the basic principle to support, but not to control the project. It is highly doubtful if the OSMF taking over the development of some core tool in operation of OpenStreetMap by paying a professional developer would be of long term benefit for the project. OpenStreetMap has, over its history, so far been able to attract and motivate the developers of the technology it relies on on the operational level and the OSMF’s role here should be to ensure it continues to do so, not to become a substitute motivator.

Beyond these fundamental operational needs lies the field of strategic investment into technologies needed in the long term by the OSM community. The focus of the OSMF board here should lie in supervising and guiding the strategic planning in that regard to provide the necessary guidance for the Engineering Working Group in practical implementation of these strategic investments.

Since strategic investments in technology typically have a significant size, there is going to be a necessity for the OSMF to pursue cooperation with other organizations – like OSGeo, or academic institutions as far as fundamental technological innovations are concerned, or other potential users with overlap in strategic needs.

My own first focus as a board member in this domain would be to keep a careful eye on how technological decisions affect the OSM community on the social level. The important thing here is that technology should have the function to support the OSM community in their needs and their consensus decisions. The technology should not be used to steer the OSM community in a certain direction and it is the board’s task to ensure that this does not happen under the aegis of the OSMF.

My second focus would be to ensure that the decision of where to strategically invest the limited means of the OSMF is not primarily guided by lobbying interests of parties who want to profit from the investment, but by a careful and neutral look at the actual strategic needs of the OSM community for its work. Doing the strategic planning in that regard publicly is going to be key to accomplish that.

Data Quality and Protection

We have here the two fairly separate fields of vandalism (i.e. malicious actions) and quality assurance (i.e. the concern about the quality of well intended edits of the data). The first field is handled on the operational level by the working groups, the board should not get involved here. It would be important to give this point substantial consideration in the strategic planning and to involve the concerned working groups in strategy development there.

Regarding quality assurance – this is primarily a matter for the mapper community and the OSMF should not get involved here beyond this potentially being part of the strategic investment in technology (see previous section). One exception: If quality issues are directly or indirectly caused by organized mapping activities. Here adjustments of existing regulation or new regulation could be required in the future.

I see no acute need for adjustments in policy here, but the current organized editing rules have been in place for quite some time now without changes. So one thing i would do as a board member is to initiate a comprehensive review of the organized editing policy and evaluate the need for adjustments. Quite a bit of research has been published during the past years on organized editing in OSM so there is a substantial data basis for such a review. This data should be supplemented by direct input from local mappers regarding the effects of organized activities on their work.

Perspective on Open Source

The fundamental idea of OpenStreetMap is directly tied to the idea and the philosophy of open data, which in turn is closely related to the idea of free open source software and open technology in general. So, as far as the technology the OpenStreetMap Community relies on in its basic work is concerned, the need for this technology to be open is pretty much a given (although, to be accurate, that principle currently largely ends when it comes to hardware technology).

How the OSMF manages its membership database is fairly unrelated to that so you could argue the same principle does not apply here. But the OSMF has a FOSS policy and that derives not only from the philosophical connection between open data and free open source software, but from practical considerations as well. It is practically beneficial for the OSMF to use FOSS rather than proprietary software for multiple reasons

it is less costly because you avoid license costs.
it avoids creating a dependency on a proprietary software provider.
it has security benefits since you don’t need to run code that you can’t inspect (in particular relevant for use cases involving sensitive personal data).
it massively lowers the barrier for volunteer involvement because volunteers can use the same software independently without a license cost hurdle.
it creates an additional incentive for volunteer involvement because volunteers can get practice in using software that they can then also use otherwise without additional license costs.

In the specific case of the task of managing the OSMF membership database the following additional argument applies:

As far as i understand it the incident that triggered the recent discussion was caused by a change in the membership signup process being deployed without proper testing – which would have revealed the problem. Such testing is much easier to organize in a volunteer based project if no proprietary software is involved because any volunteer could perform the testing on their own infrastructure independently while with proprietary software licenses the testing would either need to happen on OSMF infrastructure or the tester would need to be contracted by the OSMF (depending on the terms of the license). In other words: FOSS solutions are much better compatible to the openly cooperative and decentralized work culture of the OSM community than proprietary solutions.

The bottom line is: I see plenty of arguments that speak for continuing to use FOSS solutions in managing the OSMF membership database and none that speak for a proprietary solution.

Perspective on Overture Maps

Overture Maps is essentially an OSM data user like anyone else. What distinguishes them from other data users is that their aim is to re-distribute OSM data in a modified form on a large scale – something that is rare among commercial OSM data users so far.

For the OSMF it is important to treat them like any other data users. That means watching over their compliance with the ODbL. Since they distribute combinations of OSM data with other data this in particular concerns the share-alike rules of the ODbL. These have been neglected in the guidelines regarding the ODbL developed by the OSMF so far and it might be advisable to fill that gap.

And we should communicate to them clearly and openly that we expect them to support the project the data of which they use extensively by the Linux foundation becoming a corporate member of the OSMF. Similarly, we should of course look into contacting large scale data users who use OSM data through Overture and equally suggest them to support OpenStreetMap as the original source of the data they use.

Conclusion

That ends the presentation of my fictional candidacy for the OSMF board election. If you like what you have read then don’t vote for me – because you can’t. Instead you should get out of your mapping armchair and start working towards what you like in what i presented above. The truth is that the OSMF will not change for the benefit of the OSM community unless the OSMF members push it with determination to do so. So if you find my analysis of the situation of the OSMF and the suggestions i derive from it convincing on any of the topics discussed, then it is up to you to pursue these ideas. Because no one else will.

September 17, 2024
by chris
1 Comment

Weaving roads #3 – discussion

Since the previous two blog posts are already quite lengthy as is, i am going to do the overall discussion in a separate post here.

I discussed the situation of road rendering in OSM based maps in general already in earlier texts so here as a brief summary only:

OSM-Carto uses – among dynamically rendered OSM data based maps in practical operation – the most sophisticated system of road display and still substantially falls short currently of consistently displaying the road layering.
What i introduced 3.5 years ago goes significantly beyond that, but neither this concept nor anything else implementing a comparable functionality has made it into operational maps.
What i showed in the previous post here takes a step further by offering options to do selective styling and drawing order adjustment, based on geometric context rather than only feature attributes.

Rendering with the changes shown here

Rendering the AC-Style so far

Rendering in OSM-Carto

The bottom line is: While i further expand the design options available in road rendering with this demonstration, we are certainly pretty far away from such sophistication making it into mainstream maps. And, as hinted at already in the previous post, the lack of meaningful visual feedback on the mapping practice in layered road networks, including polygon features, has a significant impact on mapping. It is clear from the data that mappers have a strong interest in recording relevant information in this domain, but the complexity of this in multi-layered structures, like subway stations, makes this very hard to do well without good visual feedback. And QA tools and specialized indoor map apps allowing per-layer display of such structures (which exist) cannot fully replace this.

Or in other words: The lack of progress in improving road rendering in mainstream maps during the past 5 years seems to be significantly holding back also improvements in mapping in that field.

One important question is, of course, if there are other viable approaches for implementing the kind of rendering i show here. And the answer is yes. I already hinted at that in the first part. You can split the linear road geometries into those parts within and those outside of road polygons and render them differently based on this differentiation. You will then, however, have a serious problem at the edges to ensure the roads stay continuous where they are artificially split despite both parts being treated differently in terms of drawing order. Ultimately, you would be substantially constrained in your design options for drawing the roads – which you are not with the approach i showed.

The other important thing i want to mention is that the additional sophistication in styling, modifying the rules in areas of road polygons, can also make the map more difficult to read. The mere fact that underground structures are shown that are otherwise hidden, the deviation from the physical drawing order and additional styling variations – like i show for road polygons overlapping road polygons – can all contribute to that. The challenge for map design here is to avoid this leading to a negative net benefit for the map reader.

Outlook

As already mentioned, what i showed here is essentially just a proof-of-concept. So far it only deals with surface level road polygons. I also have not looked much at the styling of underground roads – this is definitely a field that requires a closer look since many of the tunnel road signatures are too heavy for a well readable display in areas with a dense underground network. Especially also ground unit rendering needs to be considered again in that context. Not to forget the labeling – which is working quite badly at the moment.

And of course the Mapnik extension of per-layer post processing with GMIC has a lot of potential applications beyond what i showed here. It implements point 7 on my list of features rule based map design requires from the tools it uses on the layer level. Offering the same functionality on the feature level would be very nice and useful (for example to extend what i have shown here to non-surface level road polygons). But it is much harder to implement, because it requires delving into the details of AGG. And it also would likely have much stronger performance implications.

September 15, 2024
by chris
1 Comment

Weaving roads #2 – creative compositioning

In the previous post i explained one of the main design issues of road depiction in OpenStreetMap based maps and identified the most viable approach to address this. Here i want to demonstrate and explain how this can be practically done.

Just like in the case of viewpoint rendering i discussed earlier, this requires extending the functionality of Mapnik. If you want to re-produce what i show here you will, therefore, need to build Mapnik yourself using the modifications i have published.

To recapitulate – the problem i intend to address here is that in normal road rendering, the roads (including road polygons, like pedestrian areas) are rendered above linear features like waterways. Therefore, those line features do not render properly, even if they are diligently cut out in mapping from the road polygon (which – according to broadly established mapping conventions – is not strictly necessary). Also, road tunnels are fully covered underneath larger road polygons, which hides important information on underground road and path connections from the map user.

Current layering of pedestrian road lines and polygons relative to other road and line features

We want to address these issues without actually changing the road drawing order overall. Linear road tunnels are still supposed to be rendered below linear surface roads and linear road bridges above them. And surface roads, as well as tunnels and bridges of the same layer, should remain visually connected to one another to correctly communicate the connectivity in the roads system.

Impossible waterfalls

Working this problem is a bit like drawing an impossible waterfall like the one in the famous drawing by M.C. Escher.

M.C. Escher: Waterfall, 1961

This kind of drawing is consistent in the 2d depiction sketched and locally consistent also in three dimensions. It, however, becomes self contradicting and physically absurd in total. Therefore you cannot create this kind of drawing strictly from physical principles alone, you have to explicitly break with these principles. The challenge is to do so in a way that remains locally consistent.

The map rendering task at hand is similar, what we need to explicitly break with is the strict adherence to the vertical order of features in drawing and the rule that the styling of a feature is only defined by its own attributes. And we want to – as much as possible – maintain local consistency in drawing, giving the correct impression about connectivity and vertical order of features in reality.

This is a fairly complex task so i limited my proof-of-concept here to just the surface level road polygons. For the bridges and tunnels the setup presented here would essentially need to be duplicated once more for every layer (because for each of them a separate set of linear road features will need to be treated specially). Since bridge and tunnel road polygons are rare and large ones substantially covering other roadwork are even less common this limitation is practically not very big.

What we will need to do is rendering some road features (specifically road tunnels and linear features otherwise drawn before the road layers) differently when and for the area where they would normally be covered by a surface level road polygon. The procedure to do that is:

Render the road layer as usual with the drawing order chosen according to the vertical oder of elements in reality as tagged, including the surface level road polygons.
Render separately a second version of the road layers without the surface level road polygons.
the linear features that are visible in this version because they are not hidden by the road polygons need to be drawn here in the design in which they should ultimately show up above the road polygons. The layer implementing this is called roads_noareas.
Render a mask of the line features that are to be shown like in roads_noareas in the area where they are normally covered by the surface level road polygons. The layer implementing this is called roads-line-mask.
Render a mask of the surface level road polygons visible in the standard road display. That is essentially the surface level road polygon footprints minus the bridge road polygons. The layer implementing this is called roads-area-mask.
Combine these four renderings by composing roads_noareas over roads only in areas covered by both roads-area-mask and roads-line-mask.

Advanced compositioning with GMIC

This is currently not possible to do with the rather limited compositioning capabilities of Mapnik. The only thing Mapnik can do out-of-the-box is composing what is newly drawn (either on the feature or on the layer level) with what has been previously drawn in a two component compositioning operation. The kind of multi-component compositioning operation described here is not available.

To overcome this i added support for raster post processing of the rendering using the GMIC image processing framework to Mapnik. GMIC is doing image processing using a dedicated scripting language. This framework can be invoked for every layer of the map style in addition to Mapnik’s internal per-style compositioning operation, either before or after. GMIC perfoms its processing based on an image buffer stack. The way i integrated this framework into Mapnik this stack is retained between layers with the current layer (either before or after Mapnik’s internal compositioning) being added to that stack for further processing.

To understand how to make use of that feature it is useful to understand how Mapnik does its own per-style compositioning. If you don’t use either style level opacity or comp-op (meaning no opacity or comp-op being set for the whole layer in Carto-CSS) Mapnik plainly renders everything into a single rendering buffer one element after the other. If opacity or comp-op is used Mapnik renders the layer into an empty (i.e. originally fully transparent) buffer and then composes this into the buffer retained from the previous layers using the chosen opacity and comp-op. If you use the new gmic style property the compositioning is automatically activated as well, the layer is rendered into a new, empty buffer and that buffer is then added to the GMIC image buffer stack. Afterwards the script specified in the gmic property is run.

But by default nothing is composed into the main rendering buffer. For that to happen the script has to assign the name use to the last image buffer in the stack. Then Mapnik uses this buffer as the source for drawing the layer (potentially using the specified comp-op/opacity) and removes it from the GMIC image buffer stack for the next layer. In other words: the no-op GMIC command is (in CartoCSS syntax)

gmic: '-name. use';

To store the rendering of this layer for use in processing one of the next layers in addition to using it as is you can use

gmic: '+to_rgba. -name. use';

which duplicates the last image on the stack (the current layer rendering) into another rgba buffer and then sets that up to be used. Since only the buffer used will get removed from the stack by Mapnik, the second version of the current buffer will remain for later re-use. Finally, if you just want to store what has been rendered for future use and not directly use the current layer at all, use something like:

gmic: '-to_rgba.';

This converts the last buffer in the stack to rgba – which it already is, hence: it essentially does nothing. Since the name of the buffer is unchanged, Mapnik will neither remove not use it and will simply move on to the next layer.

The other important thing to know is the naming of the buffers. All buffers added by Mapnik are named after the layer they are generated from – with any dash (-) in the layer name being replaced by an underscore (_) since dashes are special characters in the GMIC scripting language.

This whole principle of operation is just designed as i saw it fit for the moment – it is highly probable this is not ideal and needs adjustment in the future.

Back to the actual problem: The normal road layer (roads) and the second variant without the surface level road polygons (roads_noareas) are rendered from the same SQL code and share most of the CartoCSS code. The roads layer is rendered normally, for reference: here is how the rendering looks like after that layer:

Starting base before the roads-area-mask layer – after rendering the roads in classical ordering

After that comes the roads_noareas in addition has the GMIC parameter

#roads-noareas { gmic: '-to_rgba.'; }

The roads-line-mask uses the same, i.e. stores the rendering rather than using it. The actual compositioning happens in roads-area-mask where we use the following GMIC script, written here – for clarity – with one command per line:

+channels[roads_noareas] 3 -name. roads_noareas_mask -channels[roads_line_mask] 3 -channels[roads_area_mask] 3 -min[roads_area_mask] [roads_noareas_mask] -min[roads_area_mask] [roads_line_mask] -to_rgb[roads_noareas] -append[roads_noareas] [roads_area_mask],c -keep[roads_noareas] -name. use

This is explained step by step in the following. I link to the resulting images of the different processing steps. These you can generate yourself by adding suitable -output commands in the script. When you invoke the rendering via Nik4 or similar means, this will generate snapshots of the processing – which is useful for debugging purposes.

duplicate the alpha channel of roads_noareas into a new buffer (result)
name that buffer roads_noareas_mask
reduce roads_line_mask to its alpha channel (result)
reduce roads_area_mask to its alpha channel (result)
calculate the minimum of roads_area_mask and roads_noareas_mask and store it into roads_area_mask (result)
calculate the minimum of roads_area_mask and roads_line_mask and store it into roads_area_mask (result)
strip the alpha channel from roads_noareas
append roads_area_mask as new alpha channel to roads_noareas (result – with transparent parts rendered as dark gray)
remove all buffers except for roads_noareas
use the remaining buffer for composing this layer

The final results on the sample is here:

Final results of rendering the sample setup – link goes to double resolution version

This, of course, not only works for pedestrian areas, but also for all the other polygon features rendered in the road layers:

highway=pedestrian/service/platform

highway=residential/aeroway=taxiway/highway=track

Note that highway=steps lines are rendered above highway polygons, just like barriers and tunnels, in disregard of the z-order because open air steps starting from within a pedestrian area, like at a subway entrance, are often just drawn onto the polygons. This is not a very descriptive mapping practice though – despite the convention that linear features supersede polygons (see previous post). Better options for mapping are sketched below.

Subway entrance on a pedestrian area: (1) plainly mapped onto the pedestrian polygon, (2) cut out and tagged with width, (3) in addition mapping of barrier=wall around the edge

That’s it essentially. This probably sounds much easier than it was though. The design of the roads_line_mask and roads_area_mask is pretty delicate to ensure the results are free of artefacts.

The whole exercise was about the display of line features overlapping with surface level road polygons. One remaining question in light of this is how to treat tunnel road polygons. They could reasonably stay hidden by the surface level rendering in line with the general principle of rendering according to the vertical ordering in reality. I decided instead to render them with outline only. My thought is in particular about subway platforms mapped with polygons – which are useful to be shown underneath pedestrian areas. A bit problematic is the strong difference between the outline rendering of the polygon version and the filled rendering of the linear version, making recognition as the same type of feature unlikely.

Outline rendering of tunnel road polygons overlapping surface level road polygons

Real world examples

A few practical examples of how this looks like with real world data. First a case featuring barrier lines:

Prague at z18 – double resolution version, previous rendering, in OSM-Carto

Note that the new approach not only shows barriers within the pedestrian area, the fence in this case, but also better shows the walls and retaining walls at the edge of the pedestrian area. The next example features waterways in Freiburg – the famous Bächle:

Freiburg at z19 – double resolution version, previous rendering, in OSM-Carto

This well demonstrates that the drawing order change only applies to the road polygons and not to the linear roads – which here leads to partially visible waterway line signature within the pedestrian gray. That is somewhat confusing, but ultimately is caused by the inconsistent mixing of polygon and linear mapping of the pedestrian roads in this case. Another example from Lyon shows a partially mapped subway station:

Lyon at z19 – double resolution version, previous rendering, in OSM-Carto

You can see the underground platform and the subway lines as well as the entrances to the subway. Not mapped so far are the connecting footways in between. A similar case in Marseille:

Marseille at z19 – double resolution version, previous rendering, in OSM-Carto

Mapping is, likewise, incomplete here. And the underground platforms are mapped with linear ways so they are rendered with a fill color. Finally the example from Prague i already showed in the first post of this series:

Prague at z19 – double resolution version, previous rendering, in OSM-Carto

As far as the tunnel rendering is concerned – more elaborate examples of subway stations or other underground structures underneath pedestrian areas are a bit difficult, because mapping consistency is often not that good. While mappers tend to try to apply tagging with layer diligently, the lack of good visual feedback combined with the counterproductive incentive, that existing mainstream map rendering often has, tends to lead to issues in the data. Common examples are:

location=underground/tunnel=yes/covered=yes not being applied consistently to underground platforms.
roads underneath pedestrian areas being deliberately not tagged tunnel=yes/covered=yes to make them show up.

An overall discussion of the rendering technique demonstrated here will follow in the next and last part of this series.

September 13, 2024
by chris
1 Comment

Weaving roads #1 – polygons and lines

When discussing the rendering of roads in detail 3.5 years ago i showed how OpenStreetMap-Carto uses the different options offered by CartoCSS and Mapnik to draw the road network in a consistent fashion but, ultimately, still falls short of this in some aspects. I also showed the new system of rendering the roads in a single layer, used now in the AC-Style, and how this can help overcome some of these limitations.

Road layering in OSM-Carto with inconsistencies

Road layering with tunnel, ground level and bridge elements represented by linear ways and polygons – as shown in OSM-Carto with inconsistencies – link goes to double resolution version

Improved road rendering as used in the AC-Style so far – link goes to double resolution version

All of this was done by defining a consistent drawing order of the different features, in many cases in several instances – casing, background, fill, centerline etc. This fully relies on every aspect of the physical vertical ordering being explicitly represented in the data (splitting the geometries as required for that during mapping and, where necessary, using the well known layer=* tag). That works relatively well since mappers in OpenStreetMap often quite diligently record the relevant information on the roads network explicitly. As an additional bonus, it provides clear direct feedback to mappers on how things have been mapped – which helps maintain this high quality mapping.

There are, however, still fundamental issues with this approach that can be pretty severe in the concrete cartographic reality. Most notably regarding the road polygons, where we have two big (and related) problems that i am going to explain below.

Contradicting mapping conventions

Both problems are related to the fact that road polygons (that is for example highway=pedestrian or highway=service polygons – which are used to map pedestrian or vehicle use areas where no single direction of navigation is dominant) are treated exactly like linear roads in terms of drawing order and this is strictly based on the physical vertical order in reality. Although this logic is easy to understand and leads to a clear and easy to interpret rendering (see above), it is also somewhat unexpected in some cases for mappers – which creates the first problem:

It is common that mappers expect features intersecting a highway=pedestrian polygon to be shown above the pedestrian area – like for example a small pond around a fountain or a kiosk building. They would not have the same expectation for a linear pedestrian road intersecting such features, but for highway=pedestrian polygons this is quite common. Sometimes, mappers add layer tags to such features to explicitly indicate the kiosk/pond is located on the pedestrian area. Semantically, this is questionable though, since a surface water area or a ground level building overlapping a non-bridge, non-tunnel highway=pedestrian polygon is incompatible with the established meanings of these tags – independent of the presence of a layer tag. Or in other words: These things are not actually overlapping. In short: It is quite clear consensus among mappers that road polygons should be limited to the area of actual navigation with no implicit (or explicit – through layer=*) resolution of ambiguous overlaps with other polygons.

Pedestrian areas with non-walkable parts (like flowerbeds, fountain ponds) being excluded from mapping as highway=pedestrian. Not universally though – see here

The situation is different though for overlaps of road polygons with other linear features like waterways or barriers. The overarching convention in OpenStreetMap for linear and polygon elements in general is that overlaps are acceptable in mapping, at least in most cases, even if they are semantically contradicting. A highway=track crossing a landuse=orchard or a landuse=farmland polygon is considered correct mapping. Same for a waterway crossing a natural=scrub or natural=bare_rock. Although no crops are grown on the highway=track and no scrub grows within the river, the mapping convention here is that the linear mapping of the track/waterway implicitly supersedes the polygon landcover/landuse mapping. Data users have to interpret the data accordingly – which, in map rendering, is typically accomplished by drawing the line signatures of the linear map elements on top of the polygon elements.

Variants of mapping pedestrian polygons: (1) gross area including non-walkable polygons ‘within’ (2) only effective pedestrian area but covering linear features and (3) cutting out both polygon and linear features within

And here you probably notice the dilemma in map rendering since we have two conventions clashing:

That linear mapping (of features like barriers, waterways, roads and paths) supersede polygon mapping (like landcover polygons) in cases of overlaps between the two for the area implicitly belonging to the linear feature for where there would be semantic contradictions.
That in road layering both polygon and linear mapping are together ordered based on physical vertical location relative to each other according to the explicit tagging with bridge=*, tunnel=* and layer=*.

So far, the AC-Style for the roads has been following exclusively the latter principle, which is – as mentioned – frequently irritating for mappers with regard to the road polygons because of the former principle. And giving up on the former principle and cutting out even linear features from the road polygons in mapping does not help practically either, because the gap in the road polygon mapping based on actual ground width of the linear feature does not provide enough space for the line signature of the linear feature to be rendered. You will see something on the map (like in the sample 3 above) but not the recognizable line signature of the feature in question.

Rendering so far of overlaps between ground level highway=pedestrian features and line features (barrier=wall on top, waterway=stream on bottom) – (1) linear road with plain intersections, (2) linear road with explicited intersections (barrier=entrance, ford=yes), (3) road polygon

You could now get the idea of rendering all line features of the style after rendering the road layers. That would, however, not work well, because the road network would become difficult to read since it gets frequently overlapped by other line features, especially at the lower zoom levels. The other commonly mentioned idea is to sort everything in rendering according to the layer=* tagged – essentially giving mappers control over the drawing order. That is also not a good idea, because it would effectively sabotage OpenStreetMap as a collection of semantically meaningful information about the geography. Right now layer=* has a fairly well defined meaning regarding the relative physical ordering of close-by or overlapping elements. If mappers instead would start using the layer tag instead to define the drawing order of a map or to define override rules in semantically contradicting mapping that would create a lot of damage to OpenStreetMap as a whole.

Hidden tunnels

The other unresolved aspect of the road layering stems from the fact that defining the drawing order strictly after the vertical ordering in the geographic reality does not universally produce the best readable map – even when you only look at the road features. A drawing order following strictly the vertical ordering in reality means that tunnels are universally covered by surface features. For the linear road features this is rarely an issue (at least for the roads themselves – the road labels and oneway arrows are a different matter). For the road polygons, however, if you have a larger pedestrian (or other road type) polygon, that essentially covers all the underground road infrastructure underneath.

Underground infrastructure hidden by pedestrian area – links to same area in OSM-Carto.

Bottom line: For a well readable map you ideally do not stick to strictly defining the drawing order by vertical ordering for the roads. For elements that would otherwise be fully hidden by what is drawn above them the better approach is to try visualizing them out of order in some form. This avoids creating substantial gaps in the depiction of the road system.

Solutions

Tackling this second issue is not possible with the methods so far deployed in road rendering in either OSM-Carto or the AC-Style. As mentioned above, so far these styles base the drawing order and styling decisions fully on information explicitly stored in the data per feature. What is necessary here is interpreting the context – a tunnel feature needs to be drawn differently (in either order of drawing or styling or both) depending on if they are underneath a road polygon or not. This can be done either by

Intersecting the geometries in SQL. That is computationally expensive and would require adding substantial code complexity to the already complex road layers.
Using raster compositioning. This seems the more elegant approach here. But it is beyond the limited compositioning abilities of current Mapnik.

The good news is that both approaches will equally allow addressing the first problem of properly displaying non-road line signatures on road polygons – dealing with those differently within road polygons is not much different from the dedicated display of tunnel features within road polygons.

In the next part i am going to demonstrate and explain an implementation of the second method offering a partial solution to the problem discussed.

September 10, 2024
by chris
1 Comment

A short State of the Map 2024 comment

State of the Map 2024 took place in Nairobi last weekend. I did not attend, although i tried to get an idea by watching some of the video streams. With tried being the operative word here – the streaming infrastructure did not seem to be quite up to the task. That was clearly not the fault of the local team though (with the exception of the microphoning and audio levels – which was severely challenging for the viewer).

Not going to write an in depth commentary here. To recapitulate the background: The year before (2023) State of the Map was meant to take place in Cameroon but the OSMF board threatened to intervene for political reasons so the SotM working group scuttled their plan – meaning no SotM in 2023. Bottom line: A lot of people were pissed, but for very different reasons.

When, for 2024, the SotM working group selected Nairobi, the opposition was less severe and the OSMF board kept quiet – so the conference went ahead. It was nice to see the African communities got some broader recognition. And given the severe lack of more substantial communication of the OSMF with the larger OSM community otherwise, the program items from the OSMF gave valuable insights into the inner-OSMF mindset that are otherwise not available to the ordinary OSM community member.

Beyond that i want to point interested readers to the commentary and impressions from Severin and Ilya from the conference. If you compare the official public communication from the OSMF to Ilya’s comments, that is truly a night-and-day difference – predominantly PR phrases and random pointers to communications of others vs. thoughtful personal impressions and critical commentary based on concrete individual experiences.

The OSMF has now for quite a few years tried to organize some streamlined synthetic corporate PR style communication (with fairly limited results even by corporate standards). At the same time they are essentially ignoring the substantial grassroots efforts and talents the OSM community has developed in the field of public communication because they do not fit into the OSMF corporate culture. The people who try to help the OSMF writing mastodon and twitter posts or entries on the official OSMF blog are surely well meaning and enthusiastic. But the idea to compete through engineered PR style communication with people freely and independently writing about the topics they are involved and experienced in is simply not a winning proposition. And, as a side effect, it alienates everyone in the wider OSM community who publicly shares their thoughts on OpenStreetMap topics with a wider audience and who has the ambition to do so independently and in a reflected way without being affected by organizational interests.

Ironically, this year’s SotM also showed how much talent and eagerness there is also in the African OSM communities in public communication, which is sadly not going to find the support from the OSMF to thrive and to develop the experience and self confidence to write and talk independently and critically about matters of interest for the global OSM community with a distinctly African perspective. My hope is that such support will at least to some extent come from the broader OSM community outside the OSMF.

Next year’s conference is announced to be in Manila, Philippines – from former British colony to former US colony i might critically add. But i am happy for the Philippine OSM community. They are a very active local community in OpenStreetMap with a very dynamic development of the map in the area. It is unlikely i am going to be there though. I might reconsider that if someone was interested enough in me talking there about map design, generalization, open data satellite imagery or any other topic of interest to finance me visiting the conference. But as is, there is neither enough use for my business in such a visit nor am i interested in becoming part of the international OSM jet set.

A short final remark regarding diversity of locations of SotM conferences. It has occurred to me that up to and including Manila 2025 all SotM conferences with the exception of Japan (2012 and 2017) will always have taken place in predominantly Christian countries. Coincidence?

September 6, 2024
by chris
1 Comment

More dynamic symbols: Viewpoints

Two years ago i showed a new design concept for rendering trees in maps. This used a combination of hand designed components with automatic geometry processing to depict different types of trees of different sizes in a way the symbols naturally overlap in a well readable way when the trees are close to each other.

Non-blocking dynamic tree symbols – click for the blog post discussing these in more depth

Technically, this was done by rendering the symbols from a polygon representation in the rendering database. This facilitates both the construction of the symbols from their hand drawn design elements, the dynamic scaling according to the tree diameter, and the cutting of intersecting tree symbols for the natural display of overlapping symbols.

One thing you might have noticed with the tree symbols is that they are non-blocking. Other symbols and labels overlap with the tree symbols freely. This matches the tree display in OpenStreetMap-Carto and is a prudent choice for tree display. But it also was a practical necessity, since polygons rendered from the database in Mapnik are inherently non-blocking. Collision detection and blocking in Mapnik is traditionally only available for point, marker and text symbolizers. The technique i demonstrated for the trees was therefore limited in its application to situations where non-blocking symbols are the goal.

I have since then extended the capabilities of Mapnik to customize dependencies between symbols drawn using the concept of anchors. I here want to show how that feature can be further extended to allow the use of symbols drawn from the database for blocking applications.

Displaying viewpoints

Viewpoints are locations that offer an extraordinary view of the environment and are therefore often a popular destination to visit. Such are tagged in OpenStreetMap with tourism=viewpoint. A significant percentage of those (>12 percent) are also tagged with a direction tag, indicating the direction in which the viewpoint offers a good view.

OSM-Carto has rendered viewpoints with a static point symbol for a long time, but is not taking into account the direction tag. Depiction of viewpoint directions has been pioneered by the OpenTopoMap style.

I am using SQL functions to calculate azimuth direction and viewing angle from the direction tag. Then i use those to pick and suitably rotate the best fitting symbol from a pre-generated table of symbols in the rendering database. Here is how this looks like for different variants of the direction tag:

Viewpoint display with direction visualization – click to see the corresponding direction values as label

And here are the pre-generated symbols from the symbols table these are rendered from:

Pre-generated symbols stored in the rendering database

There are a number of aspects about this rendering of the individual viewpoint symbols i want to point out. First: Viewpoints without a direction tag remain shown with the generic traditional symbol to distinguish them clearly from the direction=0-360 viewpoints. Second: The size of the symbols is slightly increased as you zoom in – to not take too much space at the lower zoom levels and be more clearly readable at the higher ones.

Symbol size change with zoom level – click for double resolution version

But also note that the size of the symbol varies depending on the angle depicted – for the narrow angles this is increased while for the 360 degree it is decreased. This makes the weight of the symbols of viewpoints with different angles more similar. Finally, also note the positioning of the name labels is adjusted to the symbol bounds so there is no excessive gap between symbol and label for the northward looking viewpoints.

In contrast to most other point symbols in the style, the viewpoints are rendered with a relatively subtle bright halo. This ensures a good readability of the fairly fine grained symbol design above strongly structured backgrounds – which are common near viewpoints with elements like natural=cliff, barrier=retaining_wall or natural=bare_rock polygons.

Cutting and blocking

The real innovation, however, comes from the way symbols are cut and interacting with other symbols. Like in case of trees, close-by viewpoints are cut away in their overlaps, ensuring a clean display without fully dropping the display of some of the symbols. At the same time, symbols that have priority over the viewpoints (because they are displayed at earlier zoom levels) are blocking the display of close-by viewpoints. Also viewpoints are blocking the display of lower priority symbols close-by. Mountain peaks (natural=peak) are treated differently, because they are commonly close to viewpoints or mapped in combination on a single node. Here symbols do not block each other, but the peak symbol is cut out from the viewpoint depiction to allow both to be displayed together. More on that later.

Interaction of viewpoints with other symbols: (1) non-overlapping close-by viewpoints, (2) overlapping viewpoint symbols being cut out from each others, (3) with peaks overlapping as well, either mapped separately or both tagged on the same node, (4) viewpoint symbol being blocked by a higher priority symbol and (5) viewpoint symbols blocking other lower priority symbols – click for double resolution version

There are two modifications of Mapnik necessary to accomplish this:

To support blocking between viewpoints and other symbols, support for anchor-cond was added for polygon symbolizers. This allows implementation of simple blocking by rendering a zero opacity marker symbolizer for the viewpoint like you’d do for an SVG symbol and then tie the rendering of the actual symbols to that via anchor-cond. That, however, does not work in combination with the cutting of the viewpoint symbols with one another, because even symbols that are blocked by other non-viewpoint symbols are getting included in the cutting operation – which they should not. To fix that we need to
add a way to access the list of anchors (the identifiers of symbols rendered successfully so far) from within SQL, where we are doing the cutting of the symbols.

While the first of these was fairly straightforward to implement, the second was a bit more tricky. This was mainly because of the way Mapnik handles database queries and rendering. Instead of running the query of each of the layers of the map style and then rendering its results before moving on to the next layer, Mapnik has the ability to asynchronously start all the queries up-front and then start rendering the map as soon as the queries return with the data. But this procedure, of course, does not work when the query of a layer depends on the rendering results of the layers rendered before. So, to use this feature of allowing access to the anchors from within SQL, we have to move to a strictly synchronous query and rendering procedure.

Practically the whole process looks as follows:

The main POI (amenity-points) layer is rendered with the viewpoints being represented by zero opacity placeholder markers, that have a suitable anchor-set parameter to document successful symbol placement.
A second layer (viewpoints) is set up to display the actual viewpoint symbols. It also performs the intersection with the other viewpoints. This layer is rendered after the POI layer and has an additional parameter in the Datasource: anchors_table: carto_anchors. This tells Mapnik to make available the list of anchors previously set in a temporary table in the database with the specified name. That table is then used in the SQL query of the layer in the form of an additional WHERE condition like AND EXISTS (SELECT 1 FROM carto_anchors WHERE "name" = 'viewpoint_' || osm_id::text).

Viewpoints and peaks

The combined rendering of viewpoints and peaks required some further modifications to the point symbol and label rendering system. The peak symbols are cut out from the viewpoint symbols just like the other viewpoints. In addition, the label position is adjusted as necessary to avoid overlap/blocking with either the peak or the viewpoint symbol in all the different direction variants.

Viewpoints and peaks tagged on the same node – click for double resolution version

Real world examples

While there are quite a lot of viewpoints mapped with direction tags, this tagging is usually patchy – not many regions have all or even most viewpoints tagged this way. None the less, here are a few examples of how viewpoints with specified direction look like in real world contexts. A double resolution version is linked from the images.

Reunion, Indian Ocean at z16

Provence, France at z17

Kaiserstuhl, Germany at z18

Conclusions

What i showed here is how dynamic symbol design can be implemented in maps with various forms of collision and overlap handling. In the presented case of viewpoints, this entails in particular the following components:

The different viewpoint symbols are not blocking each other, but instead their symbols are cut out to avoid overlaps in a manner similar to what i have demonstrated for trees in the past. Symbol priorities in cutting are based on elevation (when tagged) and view angle.
The viewpoint symbols are blocking and get blocked by other point symbols based on their priorities. Except for
peak symbols, which do not block viewpoints, but have their symbols cut out from the viewpoint depiction like viewpoints are cut out from each other. This includes cases where a node is tagged both as a viewpoint and a peak.
Labels are shown under the symbol with an offset dynamically adjusted to the geometry of the viewpoint visualization used.

To accomplish these things additional functions were added to Mapnik and Carto. One allows conditional rendering of polygon symbolizers, depending on the successful placement of other symbolizers, which are subject to collision detection. The other provides access to information on what symbolizers were rendered successfully in previous layers from within SQL via temporary tables. Combined with the use of zero opacity placeholder symbolizers this allows the implementation of the design features described.

The Mapnik modifications are available on Github. To build Carto with support for these new features you also need the modified mapnik-reference.

The modifications of the Alternative-colors style implementing the dynamic viewpoint rendering have been published as well.

The map design concepts demonstrated here and the modifications of Mapnik that enable these further contribute to implementing point 5 on my list of critical features map design work requires from the tools it uses. But keep in mind i am not a software developer, i do not aim for these features to be usable more generally, they are just a proof-of-concept.

August 18, 2024
by chris
1 Comment

The Musaicum West Asia – extending 10m resolution image coverage

Last year i introduced the Musaicum EU-plus, a new 10m resolution satellite image mosaic of Europe, which pioneered a new, more automated classical mosaic production technique. I am happy to now introduce a new regional mosaic of West Asia with similar specifications.

The Musaicum West Asia – click for larger version

The new mosaic covers the region that is commonly called West Asia ranging from the Suez Canal to the eastern border of Iran. In the north the image not only includes the whole Caucasus region but also parts of Ukraine, Russia and Kazachstan up to about 50 degrees latitude – Essentially the southern parts of far eastern Europe that are not included in the Musaicum EU-plus.

Suez Canal, Egypt

Rostov-on-Don, Russia

Golestan, Iran

Quite a bit of work went into further refining the production technique to achieve good results in the diverse geographic settings that can be found in West Asia. In the Caucasus region you can largely find climate and vegetation conditions similar to mountains in Europe. Across much of the region, however, you find predominantly winter rain and a vegetation maximum in early spring, somewhat similar to southern Europe, but much more variable from year to year. Hence i used a significantly broader data basis here than for Europe ranging from 2019 to 2024 to be able to approximately depict the multi-year vegetation maximum. Compared to the Green Marble, which shows a multi-year average of the season of maximum vegetation this more strongly emphasizes the presence of vegetation. In a region like this where dried sparse vegetation is not well discernible from bare ground and where many parts will only see significant greening every few years this is highly beneficial to well work out the differences in ground cover and vegetation.

Sakaka, Saudi Arabia

Euphrates Valley, Syria

Towards Central Asia this seasonal characteristic is further constrained by winter frost which leads to a very short growth period in spring. In addition, in the south of the Arabian Peninsula, we have an influence from the Indian Ocean Monsoon leading to summer rain and a late summer/autumn vegetation maximum.

Ibb, Yemen

Most other satellite image mosaic productions ignore all of that and do not regard West Asia as a particularly challenging region. Cloud incidence is generally low so there is plenty of data available, overall, that can be used to assemble a cloud free coverage without much effort. The result is a dry season image which displays most of the region in fairly uniform and structureless brown-gray colors. This, however, creates a wrong impression of a region where there are significant nuances in natural vegetation even in many drier parts.

So the Musaicum West Asia is not only the highest quality image of the region in that resolution class, it is also, to my knowledge, the only one that consistently shows a vegetation maximum rendering and not predominantly a dry season depiction.

You can find the product description and more sample images on the Musaicum West Asia product page.

Gaza Strip

Bahrain

Caucasus mountains, Georgia/Russia

Dark area in the Caribbean Sea in the Green Marble 4

July 21, 2024
by chris
1 Comment

Color representation and noise in satellite images

In my announcement of the Green Marble 4 global satellite image mosaic i mentioned that i am moving to a 32 bit per channel color representation in processing of the data. I here want to explain the background of this development a bit.

Color representation basics

Color images in computer systems, for example on websites like this, are commonly represented with 8 bit per channel. That allows for 256 different levels of intensity or for 16.7 million different colors in a color image as it is commonly advertised. That is fairly coarse and only works reasonably well because these intensity levels are defined non-linearly in a way that happens to roughly match the physiology of human color perception. I won’t go into the details of that here – it has to do with the physical characteristics of the CRTs which were used as computer displays.

Anyway – for recording images in cameras this has long been insufficient and most digital cameras use 12 or 14 bit color representations (equivalent to 4096 or 16384 levels), older models sometimes also 10 bit (1024 levels). Earth observation satellites roughly match this development.

These raw values are typically cast into a 16 bit representation for further processing – both in digital photography and with earth observation satellites. When you download optical satellite imagery today for analytic applications this is almost always in the form of 16 bit per channel data.

Reflectance representation conventions

The most common form of distributing optical satellite imagery is as reflectance values. Reflectance is a unit-less quantity where a value of 1.0 means at a certain point in the image as much light is recorded as you’d expect to come from a horizontal surface that is a perfect diffuse reflector under the lighting conditions the image is recorded at.

By almost universal convention these values are scaled with a factor of 10000 for representation in 16 bit values. Sometimes, in addition, an offset is applied as well to be able to represent negative reflectances in unsigned 16 bit values – but that is not of much interest here.

Many readers might ask: Why use factor of only 10000 when the full range of 16 bit values (65536 levels) is available. The reason is that reflectance values routinely exceed 1.0. This can be easily understood based on the definition i gave above. If you have a low sun position a mountain slope facing the sun will reflect significantly more light than a horizontal surface. So even if it is not a perfect diffuse reflector it will frequently exceed a reflectance of 1.0. Hence you practically need significant headroom above a reflectance of 1.0 – which is why a scale factor of 10000 makes sense.

Noise

The next question you might ask: Is this representation of reflectance values (integers with a scale factor of 10000) sufficient for an accurate representation of the recorded data?

The answer to that is yes – as long as

we are talking about individual images.
we are talking about data in the visible range of the spectrum.

And most importantly: This is also still going to apply in the future with further improvements in sensor technology.

The reason for that lies in the Earth atmosphere. Whenever a satellite image is recorded it will inevitably contain not only light from the Earth surface but also from the Earth atmosphere. We can try to compensate for the atmosphere part when processing the images – but that compensation is for the bulk effect only. It does not eliminate the noise.

All signal recorded by a satellite image sensor, whether it comes from the Earth surface or from the atmosphere, is subject to noise. And with noise here i do not mean noise from the sensor or from the signal processing in the satellite, i am talking about noise that is already present in the light before it reaches the satellite. This noise is unavoidable and inherently limits the dynamic range of satellite image data. Because of that a dynamic range of 10000 in the data representation is – under the constraints i listed – more than sufficient for an accurate representation of satellite image data.

Aggregation

That is not the end of the story of course. The photon shot noise i discussed follows a well known mathematical characteristic: It is proportional to the square root of the signal. In other words: You can reduce the amount of noise relative to the signal and thereby improve the signal-to-noise ratio and the dynamic range by recording more light. But because of the square root relationship you need a lot more light to have a substantial effect.

There are two potential ways to exploit this possibility:

You can build larger satellites with larger optics. That is rather costly of course.
You can combine multiple images.

The latter is what i do when producing a pixel statistics mosaic like the Green Marble. And when combining thousands of individual images in areas with very low surface reflectance you can reach the limits of the standard integer representation of reflectance values with a scale factor of 10000. Here is a practical example to illustrate that. First the area in standard tone mapping.

Caribbean Sea rendering in the Green Marble 4 with standard tone mapping

This shows several fairly dark reefs in an even darker sea area in the Caribbean Sea between Jamaica and Nicaragua/Honduras. With a brighter tone mapping this becomes better visible.

Caribbean Sea rendering in the Green Marble 4 with brighter tone mapping

The open ocean away from the reefs is of a very dark blue color with an extremely low red color reflectance. And if we further contrast emphasize this area we can actually get to see the residual noise and work out the difference between the Green Marble 3 and 4 here.

Caribbean Sea rendering in the Green Marble 4 with strongly emphasized contrast

Caribbean Sea rendering in the Green Marble 3 with strongly emphasized contrast

In comparison the Green Marble 4 has a lower noise level overall but in particular note it lacks the posterization in contrast to the Green Marble 3. The banding visible in both images is the result of images with different viewing direction being combined with less-than-perfect compensation for the differences in view geometry.

Conclusions

Practically reaching the limit of standard integer surface reflectance representation in visible range satellite image aggregation – as i have demonstrated here – marks a significant milestone in satellite image processing methodology. So far practical relevance for users of the Green Marble is small. Even users of the linear surface reflectance data who do their own custom color processing will in most cases be able to work with the 16 bit version as before without measurable disadvantages.

What this, however, demonstrates is that pixel statistics mosaicing methods – in addition to the main function of assembling a uniform, representative depiction of the Earth surface from individually incomplete and flawed images due to less-than-perfect viewing conditions – are in principle capable to generate images of higher inherent quality than the original individual images that are used as a source.

Green Marble 4 with synthetic relief shading

July 20, 2024
by chris
0 comments

Announcing the Green Marble 4

I am happy to announce another update of my global satellite image mosaic – the Green Marble.

The Tian Shan mountains in the Green Marble 4

Version 4 of the Green Marble does not differ as much visually from its predecessors as version 2 and 3 since it continues to use the same principal data sources as before, that is Sentinel-3 OLCI data and MODIS images. It still represents a significant improvement in quality, in particular with substantially reduced noise levels in the water depiction. It updates and expands the source data for both the water and land depiction until late 2023.

Since the range of dates of source images processed is quite large to ensure high quality results, short term changes on the Earth surface do not usually manifest strongly in the Green Marble. So the data update is primarily visible in the form of long terms trends, which are often relatively subtle. Here an example of irrigation farming in southern Egypt:

Changes in irrigation farming in southern Egypt (large: GM 2, GM 3, GM 4)

In addition to the data update there are also significant changes in data processing, fixing various smaller flaws and improving color uniformity. The biggest change is the move to a 32 bit per channel color representation in assembly of the mosaic. This avoids accumulation of rounding errors that has led to some color distortion in the previous versions on the water depiction and it avoids running into limitations of the standard 16 bit integer reflectance value representation in satellite image processing. I am going to discuss the background of this in a separate blog post.

Improvements of coastal water color depiction between Green Marble version 3 and 4 (large: GM 3, GM 4)

The Green Marble 4 global satellite image mosaic is available for licensing for use in your projects now. Have a look at the Green Marble product page for more information and sample images.

The demo maps with the Green Marble in different rendering variants are updated as well:

Eastern India and Southern China in the Green Marble 4

Patagonia in the Green Marble 4

July 17, 2024
by chris
3 Comments

Twenty years OpenStreetMap – revisiting observations and predictions

Five years ago i wrote about the 15th anniversary of OSM and my outlook into the next fifteen years. We now have the 20th anniversary. This is a bit early to review my predictions (with merely 1/3 of the time span covered having passed) – still i want to newly consider some of the topics i contemplated five years ago.

My big topic for the 15th anniversary was the idea that a separation might occur in the future between the original core idea of OpenStreetMap, the sharing and communication of local geographic knowledge by a cross cultural community of mappers through egalitarian, self determined cooperation and the project with the name of OpenStreetMap.

This scenario consisted of two components:

the (back then increasingly visible) trend for OpenStreetMap to move away from the fundamental ideas and values it was created on.
the possibility that this development might lead people to share and exchange their geographic knowledge increasingly through means outside of OpenStreetMap.

I want to look at both parts here a bit through the somewhat different perspective i have today, five years later.

OpenStreetMap changing

Previously my hypothesis that OpenStreetMap is – at large – moving clearly away from its original ideas and values was primarily backed by the observable trends in mapping, that the data in OpenStreetMap as well as the edits made were increasingly not founded in personal local knowledge of the people making the edits and their own, intrinsic desire to share that knowledge. It seems quite clear to me today, five years later, that this trend has not reverted, although i also have the distinct impression that is has not accelerated either. OpenStreetMap continues to attract mappers sharing their local geographic knowledge and it is good to see that this happens with an increasingly diverse geographic and cultural distribution. Bulk data additions and systematic volume edits not backed by local geographic knowledge have expanded significantly as well – hence no reversal of the trend – but it can’t be said that they are substantially further displacing local knowledge based mapping at this time. Or in other words: You could say the situation has stabilized a bit.

So much for the actual development of mapping in OpenStreetMap. Things look very different when you look at how OpenStreetMap is communicated in public and hence how the wider public perceives OpenStreetMap these days. This means public communication by individual community members, by the OpenStreetMap Foundation as well as – and this is the largest part probably – by third party organizations, be that businesses of all sizes, public institutions or other organizations of all kinds.

If you study this body of public communication about OpenStreetMap these days then the traditional ideals and values of OpenStreetMap are almost completely absent. You can find these being discussed in communications of individual community members with no organizational affiliations as well as some academic studies from the social sciences sector. But the vast majority of organizational communication about OpenStreetMap (and that is what tends to have the largest reach) presents OpenStreetMap these days as a collection of useful geodata that is largely produced by volunteers (with volunteers meaning unpaid workers and not implying either local knowledge or self-determination in what they map and how they map it).

Recent public communication of the OpenStreetMap Foundation is a good example for that – the page created for the 20th anniversary is framing OpenStreetMap exactly along these lines.

Normally you’d expect such a clear and almost universal trend in public communication to lead to mappers increasingly internalizing this framing of OpenStreetMap. Interestingly this does not seem to be the case though. As mentioned OpenStreetMap these days continues to recruit new mappers who map in the traditional way sharing their local knowledge of the areas around them in a self determined fashion and in egalitarian cooperation with the other mappers around them. They must have learned this either

from legacy documentation that still exists
from other mappers by imitation/direct communication
from their local community through decentralized small group communication
through their own bootstrapping process as the way to approach mapping in a way that seems natural to them.

That this is possible and practically happening at scale despite organized public communication in most cases drawing a very different picture of OpenStreetMap is a very positive trend.

The separation

The second element of my five years old prediction was the idea that the people who value the original idea of OpenStreetMap of cooperative collection and sharing of local knowledge might move to doing this increasingly outside of OpenStreetMap. So far there is not much concrete tendency in that direction visible. And, as discussed already, at the moment the traditional mapping values seem astonishingly strong in practical mapping in OpenStreetMap.

What this, however, indicates – and this is different from what i discussed five years ago – is that there is a widening schism within OpenStreetMap between how practical mapping and the social interactions that facilitate it actually happen and the perception of OpenStreetMap and mapping in the organizational world around OpenStreetMap.

It is best not to see this schism as a complete separation of communication bubbles with one being completely disconnected from the other, it is more like a strong cultural divide.

If this schism is the precursor to a true separation as discussed in the past or if it can be a stable way to combine the social necessities of cross cutural cooperation in the form of the traditional values of mapping with the economic necessities of capitalism is not yet decided. It will probably largely depend on if people on the organizational side of the schism develop enough respect and appreciation for the social necessities on the other side. I am sceptical here (as are probably many of my readers). But i also like to point out that a lot of the organizations around OpenStreetMap have a lot of smart people working for them who are, in principle, capable of understanding these fragile inter-relationships and accordingly are able to make responsible decisions. The question is going to be: Will they?

An experiment in social cooperation

The ultimate long term question (and long term here means this is not a matter that can be determined reliably in a 15 or 20 years time frame) is if the cooperative collection of local geographic knowledge across cultures in an egalitarian and self determined fashion is something that can work at scale. As hinted at in the comments of my 15th anniversary post this is not clear, even if it has worked amazingly well during much of the past 20 years. The null hypothesis on this would be: It can’t, OpenStreetMap will necessarily either vanish or turn into a classical organization with a social hierarchy (i.e. become non-egalitarian) – at best with a framework of representative democracy of some sort.

I would very much like to see the thoughts of others on this question, especially if they differ from the null hypothesis.