Systematic testing in map design

December 19, 2022 by chris | 0 comments

This post’s subject is a matter i have been working on for quite some time – the problem of systematic testing in map design. To make clear what this is about i will have to explain a bit the wider context.

Traditional testing in map design

The traditional way of designing rule based map styles, in particular in the OpenStreetMap context, is to do so based on a test database containing a sample of actual map data. Most map designers have a test database, typically containing either an extract (like from Geofabrik or created with the Overpass API) of their area of interest or a number of areas from different parts of the world. You can get an idea of the content of my test database through the ac-style sample gallery.

When you work on the design of a certain type of feature you find one or more occurrences of this in your test database and study the effects of your design work on these occurrences. Commonly used design tools – like TileMill in the early days, or Kosmtik (in case of CartoCSS styles) and other tools, like Maputnik, in case of JSON based styling languages – are specifically designed for this workflow.

Many examples of use of this approach in style development can be found in OSM-Carto – like here.

This design work strategy has several advantages:

It is fairly straight away and does not require significant work from the designer to get started.
It tests the design in exactly the context it is meant for, that is rendering of actual map data.

But it also inevitably comes with disadvantages:

You only test the design in those contexts you happen to have picked as sample occurrences. That usually neither covers all the relevant situations the design is going to be used in nor is it representative for the full diversity of locations that feature occurs in.
It is very difficult to do any sort of systematic testing. Even the comparison across different zoom levels is difficult because the geometric context changes in scale as you change zoom levels.

Synthetic tests

Because of these problems OSM-Carto developers have for a long time and in addition to the traditional testing based on real world test data started using synthetic tests. I can’t say for certain when this was done the first time but here are two examples:

Typically such tests are designed in JOSM, run through osmium renumber and then imported into a test database.

This kind of test is particularly useful for somewhat more experienced map designers who can already predict in advance in what contexts a certain change is in particular critical to be evaluated in.

I would go as far as saying that changes of the road rendering system in OSM-Carto, that often need to work in many different design contexts, are almost impossible to develop without the use of synthetic tests. The unpaved road rendering that recently finally made it to OSM-Carto is a good example for that:

As useful as this kind of testing is, it also bears the risk of map designers completely focusing on synthetic tests and neglecting testing with real world data. That is of course not a good idea. In practical map design real world data testing is always necessary and important.

The other issue is that with this kind of hand drawn synthetic tests you still have the problem of limited comparability across zoom levels.

Automated scaling of tests

This limitation has led me a few years back to develop an approach to scale synthetic test data sets in an automated fashion for better cross zoom level comparability. I have shown samples using this approach on several occasions on this blog. By scaling the test data to the zoom level you want to test at you make rendering results much better comparable across zoom levels. This is in particular useful to make sure that design parameter progressions across zoom levels (like line widths or label sizes) are working well.

Road features at z16

Same at z17 with scaled geometries