New server infrastructure for the Community

In Chiang Mai we talked about server infrastructure from time to time, now it’s real.

I’m please to announce some serious improvements on the server infrastructure.

The server behind http://www.osm-tools.org/ (also known as http://thaimap.osm-tools.org/) is no longer a virtual machine, but running on its own dedicated hardware. This gives a huge improvement in rendering speed of the map as i/o operations have no longer to be shared with other users.

This was made possible with the help of a generous donation by Mentz Datenverarbeitung.

The setup process took a while as I migrated all services bit by bit keeping the system operational and without service interruptions all the time. I mostly succeeded. Only small issues popped up. I had to fix some problems with the rendering stack which I already contributed back to OSM. The server even had its first hard disk failure. I was quite pleased by the fast response time of the new hosting company. The defective disk was replaced within 20 Minutes. All caused only minimal downtime.
Things look pretty stable now so it’s time to announce it officially.

For those interested in the server specs: It is an i3770@3.2 GHz, 32GB RAM, 240 GB SSD for the database and 3TB data storage. 1GBit/s uplink (data-center total connectivity 570 GBit/s). The machine is connected to AS24940 https://www.robtex.com/as/as24940.html

All software was upgraded to the latest available versions. Rendering is now based on CartoCSS enabling us to merge in the main site modifications easier. Currently it’s enabled on a separate layer. Previous bilingual modifications will be merged into it.

Enhanced services:
• to better visualize some dense mapped areas the maximum zoom level is now increased to 20. This is double the resolution than what’s provided on osm.org. One pixel on the map is roughly 15cm in reality.
• The server runs an analyzer back-end for osmose. You can see the reports on the main site. Thai translations of the front-end are possible.
• Currently osmose processing is enabled for Myanmar (Burma), Laos, Cambodia, Thailand, Vietnam, Malaysia, Singapore and Brunei.
• Map coverage was extended. It now includes the full territory of Myanmar (Burma), Laos, Cambodia, Thailand, Vietnam, Malaysia, Singapore and Brunei.
• Quality assurance reports on http://reports.osm-tools.org/ have been improved
○ A new report lists areas tagged as landuse=reservoir where the covered area is suspiciously small. Most likely these are other man-made features like water=pond, leisure=swimming_pool or such
○ A new report lists roads with numbers in the name tag. This is usually not the name of the road. Reference numbers belong into the ref tag
○ Mixed Thai/English detection has been improved to report less false-positives
○ A new page shows some osmose graphs about the development of these issues in Thailand
○ A new report provides some statistics about editing activities
• To check out the server speed for your connection you can use an instance of speedtest mini running at http://speedtest.osm-tools.org/. Depending on your ISP you should see some decent rates.

Future improvements:
• They map style is not that beautiful in the high zoom level 20. Some improvements would be needed. If you would like to work on cartography, please let me know as I certainly would need help with modifying the map style.
• Some front-end pages still use an older release of OpenLayers. This will be updated soon.
• I plan to extend the highway/water comparing map OSM vs. Google to a worldwide coverage. You can find it at http://compare.osm-tools.org/
• I plan to update WDYE so it can actually work with up-to-date data
• I have a ton of more ideas what could be implemented. If you have ideas of something to host for Thailand, please contact me.

If you notice any problems with the site please let me know so it can be fixed. Also if you have suggestions for improvements: Letting me know increases the chance that they will be implemented.

This is great news, thanks Stephan for all the hard work!

That’s great news and while I’m lost in the technicalities, it all sounds good !!

I have some comment on the OSM vs. Google comparitor … firstly, Im not sure what one should see, but when I look at Thailand there seems only a few roads that show after zooming has stopped. Perhaps at higher zoom levels the roads show but its not one of your “easier” tools to use… or of course, the other answer is our major road coverage has caught up with Google !

This brings me onto another related issue that maybe you can design a tool to help ~
Since the recommendations changed a few years ago for how to tag tertiary, secondary, & primary roads in Thailand, the process has been a bit hit or miss.
Notably, the common changes are:

  • Many 4 digit National Hwys need upgrading from tertiary to secondary
  • Some 3 digit roads need upgrading to Primary
  • Many 4 digit tertiary roads need a 2 character province code added.

I upgrade these as I come across them, but its random, and there is no easy way to gauge progress, and spot areas that still need changing. I accept that a simple script won’t work, but do you have any ideas on how to provide a tool that makes it easier to spot and/or change these, albeit manually, but more efficiently ?

I guess one logical argument is that “if a 4 digit ref exists, then (by definition?), it should either be tagged tertiary and have a province prefix, OR be tagged as a secondary national highway”.
Would all users agree ? Maybe not ?

Rgds, Russ.

Hi Russ,

you’re right. Some of the tools might need some more instructions on how to use them. I think I did write some notes when fist presenting it. Given that I’m unable to produce this with some minutes of search it’s needed to place it on the main server and link it.

In short: The map at compare.osm-tools.org will hide all streets and rivers/lakes which are on the google map and in OSM. This leaves you with a map only showing roads/water features which are missing in OSM. By using the edit buttons you can load that area into your editor to complete it.
As we rely on aerial imagery it has the option to show Bing imagery bounds for south-east asia as well.
It might be worth checking out a place even when there is no Bing available as we might have it with the recent addition of mapbox imagery.

Check out Vietnam to get the idea: http://compare.osm-tools.org/?zoom=11&lat=20.32732&lon=105.95842&layers=BTT

Does this explanation help?
The numbering issue is best discussed in it’s own thread to keep the topic focused.

Stephan

Thanks, and yes I pretty much had the right idea with the tools.
Maybe you make the place names a separate layer that always displays - it helps with orientation.
Finally, shall I bump the road upgrading to a new topic, or will u like to do ?
Russ

I noticed that the renderer is picking up names from the bus route relations a street is a member of. This doesn’t seem quite right. Take a look at http://thaimap.osm-tools.org/?zoom=20&lat=13.76556&lon=100.53761&layers=B000T.

Hello friends of OpenStreetMap in Thailand,

with the amount of data in OpenStreetMap growing also the load on the server to process the data increased. Since the server is doing the heavy-lifting for Osmose QA checks in large parts of south-east asia, including also China and India, the hardware is quite busy.

I now took the chance to rent a new server with more CPU power and double the RAM and SSD size.

This will ensure we can keep up for a while with the growing data.

Migration to the new server is expected to be completed until September 5th, 2017 when the old server will be retired.
I hope to run the migration without any longer downtime, so in the best case you won’t notice any interruption of the service.

I plan to upgrade parts of the services as well to run on a more recent software stack. This might lead to subtle changes. If something is broken for you, please contact me so we can see how to fix it.

Stephan

You might or might not have noticed some changes as the services are now running on a new hardware.

It turned out that some parts of the software stack should better be retired, others re-setup on more recent versions of the software.

The most-used services are back operational, but I want to announce some upcoming changes and give some status to those interested.

Be prepared that the rendering style of the map served at thaimap.osm-tools.org will soon change. The current style is quite ugly, compared to what is available on the official site and in addition to that has issues with rendering some of the fonts used in Asia.

So this will be replaced by a slightly adjusted openstreetmap-carto style. Main modifications will be bi-lingual rendering and maybe a slight bump of fontsize.

If you are using my tiles in your own application and want to continue using them you can access them at the tile url oldtiles.osm-tools.org. Old tiles on this URL will no longer be updated. Existing tiles cover large parts and will be kept for a few more weeks. Let me know if you are using them so that it will be possible to coordinate this.

Osmose quality checks are fully functional again.

Some downloads had been retired. There might be references to non-existing data on the website. I am cleaning this up bit by bit.
Piwik analytics is completely retired and won’t come back.

Nametool application is currently having issues but should come back. Finding loose ends of highways will be rewritten to better highlight areas of interest. Report creation is offline due to a similar reason than nametool. Will come back soon hopefully. In the meantime please use osmose. Not sure what to do with WDYE. It did not update since the license change. Either I bring it back online as technology demo or wait until I have the mood to rewrite it to be able to handle updates.

Below you can see how the new rendering looks like.

Question: Shall I increase the font size? By clicking at the small images below you can see the sample in original size.

February 2021: The server infrastructure got a recent refresh.

Most visible would be a change in the rendering style. The old one did use a quite old version of the Google Noto fonts. The new rendering uses sans-serif like other parts of the world.
For me as non-native speaker (or better here: reader) it got a little tricky to read sometimes. But for this reason I have the english labels on the map as well.
The sans-serif results in a clearer map display and blends in better to other labels. The map-style is based on the recent Carto v5.3.

See here the same tile to compare the rendering:

new rendering

old rendering

I will roll out the new style in a few days. You can preview the new style at https://new-thaimap.osm-tools.org/
Tiles on preview are rendered fresh, so expect a bit longer loading time.

I plan to update all the infrastructure on the backend over the next weeks. Potentially I will retire some services. Let me know if I definitely should continue a specific one. Map, Osmose and the compare/loose ends will certainly stay.

Already in production is the processing of the quality checks for Osmose which cover besides China and India all the south-east Asian countries, including Thailand.
The new machine is much faster, so the updates are available earlier as well.

The datacenter has a 40 GBit private peering with AS38082, True Internet. I hope this brings better bandwidth for some of you.

Thanks a lot for your efforts, Stephan!
Unfortunately, I learned the Thai alphabet with the traditional fonts, and find the “modern” style hard to read. But it seems to find wider use among Thais. So I ought to get used to it.
Do you plan to add some Thai-specific rendering too? E.g. for paddies, rubber plantations, oil palm plantations, …?

Hello Bernhard,

I share your problems. The “traditional” serif Thai font seems to be more “reader-friendly”.
This font issue is certainly something I would like to have more input. The “main” OSM map uses the same font.

Google changed the font potentially with the “phase 3 update”:
https://github.com/googlefonts/noto-fonts/blob/bc9353da6f625b7a47035de952c2e8fd9e3ae889/NEWS

The version I used reports back “version 1.04”.

There is also a “Noto Looped Thai” font, which is more serif-like. It seems to have problems with tone marks:
https://github.com/googlefonts/noto-fonts/issues/2005

I am uncertain what else they changed/fixed in the process. I could probably revert to an earlier revision of the font. Not certain whether this might re-introduce meanwhile fixed hinting issues or similar.

Rendering with the current “serif” font makes it IMHO worse. Look at the spacing of Chiang Mai.

As the software stack was partially over a decade old, it was quite hard to apply updates. With the latest rework it should be easier to make changes. As long as we can have patches on top of “carto style”, it should be possible to adjust further.
In the near future I will probably have no capacity to do such changes. If you are interested in working on Thailand specific additions, it could make things easier. As map allows a higher zoom level than OSM default, the style could need some serious improvements on the width of roads (and maybe water features) on zoom 20. I hope that such changes could also be merged upstream, making maintenance easier.

The font in your last screen shot is … ugly. It seems fit for some old-style typewritter: constant width per character - but that does not fit east asian alphabets (and I am sure Khmer would look even worse). Can our local community members give better advice?

As for the style updates, I do not know how to do that - a couple of years ago, I tried to set up an OSM webserver, but failed some where in the Linux equivalent of the dll hell known from Windows.

Noto Looped Thai doesn’t appear to have been officially released yet. Why not use Noto Serif Thai, though? Yes, it’s intended to match the serif style, but the sans/serif distinction doesn’t really mean much for Thai script anyway. If there are readability problems, I’d suggest using the looped fonts that are available and workable.

The Noto Serif font is the example in post #12
https://forum.openstreetmap.org/viewtopic.php?pid=819365#p819365

For me, especially the spacing of the font looks very unpleasant. As I am no native speaker I can’t tell whether this is a real issue. The Noto Sans as in #10 has a better spacing. Probably best is to monitor Noto Looped Thai and use once released.

As Thai, I prefer the existing Noto Sans Thai UI.

There must be some error in calling the font. What’s showing up there looks like the old Unifont which was previously used by the default renderer on osm.org from the beginning of time until a few years ago.

Paul, you are absolutely right here. Good catch. When trying to replace the font I only replaced Sans with Serif and did not notice that there is no UI variant of Serif, so it did fall back to a different font.

So here is the rendering again to directly compare the two fonts:
Noto Sans UI vs. Noto Serif

I kindly ask again for opinions which one to choose.

Sans vs. Serif

I also tried Noto Looped, but the bug with tone marks is still visible, so this font is not usable in the current state (see at Khrang).
https://github.com/googlefonts/noto-fonts/issues/2005

Considering legibility for the widest range of users, I’d go with Noto Serif (though Noto Looped does look better with its mono-thickness lines, so maybe consider adopting it once the bugs have been fixed).

(offtopic as regards this server but) if anyone reckons that the fonts suggested in e.g. https://switch2osm.org/serving-tiles/manually-building-a-tile-server-debian-11/ could be improved please raise an issue at https://github.com/switch2osm/switch2osm.github.io.

In the early days with those pages there was quite a bit of experimentation to get e.g. Burmese names to display at all. Hopefully we’re past that stage now but I wouldn’t be surprised if some non-western characters could still be improved.

  • Andy (occasional maintainer of switch2osm guides)