You are not logged in.

#1 2014-08-18 20:35:19

stephankn
Moderator
Registered: 2010-05-04
Posts: 622

New server infrastructure for the Community

In Chiang Mai we talked about server infrastructure from time to time, now it's real.

I'm please to announce some serious improvements on the server infrastructure.

The server behind http://www.osm-tools.org/ (also known as http://thaimap.osm-tools.org/) is no longer a virtual machine, but running on its own dedicated hardware. This gives a huge improvement in rendering speed of the map as i/o operations have no longer to be shared with other users.

This was made possible with the help of a generous donation by Mentz Datenverarbeitung.

The setup process took a while as I migrated all services bit by bit keeping the system operational and without service interruptions all the time. I mostly succeeded. Only small issues popped up. I had to fix some problems with the rendering stack which I already contributed back to OSM. The server even had its first hard disk failure. I was quite pleased by the fast response time of the new hosting company. The defective disk was replaced within 20 Minutes. All caused only minimal downtime.
Things look pretty stable now so it's time to announce it officially.

For those interested in the server specs: It is an i3770@3.2 GHz, 32GB RAM, 240 GB SSD for the database and  3TB data storage.  1GBit/s uplink (data-center total connectivity 570 GBit/s). The machine is connected to AS24940 https://www.robtex.com/as/as24940.html

All software was upgraded to the latest available versions. Rendering is now based on CartoCSS enabling us to merge in the main site modifications easier. Currently it's enabled on a separate layer. Previous bilingual modifications will be merged into it.

Enhanced services:
• to better visualize some dense mapped areas the maximum zoom level is now increased to 20. This is double the resolution than what's provided on osm.org. One pixel on the map is roughly 15cm in reality. 
• The server runs an analyzer back-end for osmose. You can see the reports on the main site. Thai translations of the front-end are possible.
• Currently osmose processing is enabled for Myanmar (Burma), Laos, Cambodia, Thailand, Vietnam, Malaysia, Singapore and Brunei.
• Map coverage was extended. It now includes the full territory of Myanmar (Burma), Laos, Cambodia, Thailand, Vietnam, Malaysia, Singapore and Brunei.
• Quality assurance reports on http://reports.osm-tools.org/ have been improved
    ○ A new report lists areas tagged as landuse=reservoir where the covered area is suspiciously small. Most likely these are other man-made features like water=pond, leisure=swimming_pool or such
    ○ A new report lists roads with numbers in the name tag. This is usually not the name of the road. Reference numbers belong into the ref tag
    ○ Mixed Thai/English detection has been improved to report less false-positives
    ○ A new page shows some osmose graphs about the development of these issues in Thailand
    ○ A new report provides some statistics about editing activities
• To check out the server speed for your connection you can use an instance of speedtest mini running at http://speedtest.osm-tools.org/. Depending on your ISP you should see some decent rates.

Future improvements:
• They map style is not that beautiful in the high zoom level 20. Some improvements would be needed. If you would like to work on cartography, please let me know as I certainly would need help with modifying the map style.
• Some front-end pages still use an older release of OpenLayers. This will be updated soon.
• I plan to extend the highway/water comparing map OSM vs. Google to a worldwide coverage. You can find it at http://compare.osm-tools.org/
• I plan to update WDYE so it can actually work with up-to-date data
• I have a ton of more ideas what could be implemented. If you have ideas of something to host for Thailand, please contact me.

If you notice any problems with the site please let me know so it can be fixed. Also if you have suggestions for improvements: Letting me know increases the chance that they will be implemented.

Offline

#2 2014-08-19 06:14:53

Mishari
Moderator
Registered: 2013-01-14
Posts: 146

Re: New server infrastructure for the Community

This is great news, thanks Stephan for all the hard work!

Offline

#3 2014-08-20 18:01:20

Russ McD
Member
From: Hereford & Chiang Mai.
Registered: 2011-04-17
Posts: 289

Re: New server infrastructure for the Community

That's great news and while I'm lost in the technicalities, it all sounds good !!

I have some comment on the OSM vs. Google comparitor .... firstly, Im not sure what one should see, but when I look at Thailand there seems only a few roads that show after zooming has stopped.  Perhaps at higher zoom levels the roads show but its not one of your "easier" tools to use.... or of course, the other answer is our major road coverage has caught up with Google !

This brings me onto another related issue that maybe you can design a tool to help ~
Since the recommendations changed a few years ago for how to tag tertiary, secondary, & primary roads in Thailand, the process has been a bit hit or miss. 
Notably, the common changes are:
* Many 4 digit National Hwys need upgrading from tertiary to secondary
* Some 3 digit roads need upgrading to Primary
* Many 4 digit tertiary roads need a 2 character province code added.

I upgrade these as I come across them, but its random, and there is no easy way to gauge progress, and spot areas that still need changing.  I accept that a simple script won't work, but do you have any ideas on how to provide a tool that makes it easier to spot and/or change these, albeit manually, but more efficiently ?

I guess one logical argument is that "if a 4 digit ref exists, then (by definition?), it should either be tagged tertiary and have a province prefix, OR be tagged as a secondary national highway".
Would all users agree ?  Maybe not ?

Rgds, Russ.

Last edited by Russ McD (2014-08-20 18:05:58)

Offline

#4 2014-08-21 08:14:10

stephankn
Moderator
Registered: 2010-05-04
Posts: 622

Re: New server infrastructure for the Community

Hi Russ,

you're right. Some of the tools might need some more instructions on how to use them. I think I did write some notes when fist presenting it. Given that I'm unable to produce this with some minutes of search it's needed to place it on the main server and link it.

In short: The map at compare.osm-tools.org will hide all streets and rivers/lakes which are on the google map and in OSM. This leaves you with a map only showing roads/water features which are missing in OSM. By using the edit buttons you can load that area into your editor to complete it.
As we rely on aerial imagery it has the option to show Bing imagery bounds for south-east asia as well.
It might be worth checking out a place even when there is no Bing available as we might have it with the recent addition of mapbox imagery.

Check out Vietnam to get the idea: http://compare.osm-tools.org/?zoom=11&l … layers=BTT

Does this explanation help?
The numbering issue is best discussed in it's own thread to keep the topic focused.

Stephan

Offline

#5 2014-08-21 11:09:54

Russ McD
Member
From: Hereford & Chiang Mai.
Registered: 2011-04-17
Posts: 289

Re: New server infrastructure for the Community

Thanks, and yes I pretty much had the right idea with the tools.
Maybe you make the place names a separate layer that always displays - it helps with orientation.
Finally, shall I bump the road upgrading to a new topic, or will u like to do ?
Russ

Offline

#6 2014-08-28 12:09:40

Paul_012
Member
Registered: 2011-08-05
Posts: 211

Re: New server infrastructure for the Community

I noticed that the renderer is picking up names from the bus route relations a street is a member of. This doesn't seem quite right. Take a look at http://thaimap.osm-tools.org/?zoom=20&l … yers=B000T.

Offline

#7 2017-08-29 14:51:35

stephankn
Moderator
Registered: 2010-05-04
Posts: 622

Re: New server infrastructure for the Community

Hello friends of OpenStreetMap in Thailand,

with the amount of data in OpenStreetMap growing also the load on the server to process the data increased. Since the server is doing the heavy-lifting for Osmose QA checks in large parts of south-east asia, including also China and India, the hardware is quite busy.

I now took the chance to rent a new server with more CPU power and double the RAM and SSD size.

This will ensure we can keep up for a while with the growing data.

Migration to the new server is expected to be completed until September 5th, 2017 when the old server will be retired.
I hope to run the migration without any longer downtime, so in the best case you won't notice any interruption of the service.

I plan to upgrade parts of the services as well to run on a more recent software stack.  This might lead to subtle changes. If something is broken for you, please contact me so we can see how to fix it.

Stephan

Offline

#8 2017-09-08 17:40:46

stephankn
Moderator
Registered: 2010-05-04
Posts: 622

Re: New server infrastructure for the Community

You might or might not have noticed some changes as the services are now running on a new hardware.

It turned out that some parts of the software stack should better be retired, others re-setup on more recent versions of the software.

The most-used services are back operational, but I want to announce some upcoming changes and give some status to those interested.

Be prepared that the rendering style of the map served at thaimap.osm-tools.org will soon change. The current style is quite ugly, compared to what is available on the official site and in addition to that has issues with rendering some of the fonts used in Asia.

So this will be replaced by a slightly adjusted openstreetmap-carto style. Main modifications will be bi-lingual rendering and maybe a slight bump of fontsize.

If you are using my tiles in your own application and want to continue using them you can access them at the tile url oldtiles.osm-tools.org. Old tiles on this URL will no longer be updated. Existing tiles cover large parts and will be kept for a few more weeks. Let me know if you are using them so that it will be possible to coordinate this.

Osmose quality checks are fully functional again.

Some downloads had been retired. There might be references to non-existing data on the website. I am cleaning this up bit by bit.
Piwik analytics is completely retired and won't come back.

Nametool application is currently having issues but should come back. Finding loose ends of highways will be rewritten to better highlight areas of interest. Report creation is offline due to a similar reason than nametool. Will come back soon hopefully. In the meantime please use osmose. Not sure what to do with WDYE. It did not update since the license change. Either I bring it back online as technology demo or wait until I have the mood to rewrite it to be able to handle updates.

Offline

#9 2017-09-11 00:14:24

stephankn
Moderator
Registered: 2010-05-04
Posts: 622

Re: New server infrastructure for the Community

Below you can see how the new rendering looks like.

Question: Shall I increase the font size? By clicking at the small images below you can see the sample in original size.

cm-new.png

cm-old.png

Offline

#10 2021-02-18 13:27:46

stephankn
Moderator
Registered: 2010-05-04
Posts: 622

Re: New server infrastructure for the Community

February 2021: The server infrastructure got a recent refresh.

Most visible would be a change in the rendering style. The old one did use a quite old version of the Google Noto fonts. The new rendering uses sans-serif like other parts of the world.
For me as non-native speaker (or better here: reader) it got a little tricky to read sometimes. But for this reason I have the english labels on the map as well.
The sans-serif results in a clearer map display and blends in better to other labels. The map-style is based on the recent Carto v5.3.

See here the same tile to compare the rendering:

1830-new.png
new rendering

1830-old.png
old rendering

I will roll out the new style in a few days. You can preview the new style at https://new-thaimap.osm-tools.org/
Tiles on preview are rendered fresh, so expect a bit longer loading time.

I plan to update all the infrastructure on the backend over the next weeks. Potentially I will retire some services. Let me know if I definitely should continue a specific one. Map, Osmose and the compare/loose ends will certainly stay.

Already in production is the processing of the quality checks for Osmose which cover besides China and India all the south-east Asian countries, including Thailand.
The new machine is much faster, so the updates are available earlier as well.

The datacenter has a 40 GBit private peering with AS38082, True Internet. I hope this brings better bandwidth for some of you.

Offline

#11 2021-02-18 18:22:36

Bernhard Hiller
Member
Registered: 2011-05-10
Posts: 1,055

Re: New server infrastructure for the Community

Thanks a lot for your efforts, Stephan!
Unfortunately, I learned the Thai alphabet with the traditional fonts, and find the "modern" style hard to read. But it seems to find wider use among Thais. So I ought to get used to it.
Do you plan to add some Thai-specific rendering too? E.g. for paddies, rubber plantations, oil palm plantations, ...?

Offline

#12 2021-02-18 19:08:55

stephankn
Moderator
Registered: 2010-05-04
Posts: 622

Re: New server infrastructure for the Community

Hello Bernhard,

I share your problems. The "traditional" serif Thai font seems to be more "reader-friendly".
This font issue is certainly something I would like to have more input. The "main" OSM map uses the same font.

Google changed the font potentially with the "phase 3 update":
https://github.com/googlefonts/noto-fon … ae889/NEWS

The version I used reports back "version 1.04".

There is also a "Noto Looped Thai" font, which is more serif-like. It seems to have problems with tone marks:
https://github.com/googlefonts/noto-fonts/issues/2005

I am uncertain what else they changed/fixed in the process. I could probably revert to an earlier revision of the font. Not certain whether this might re-introduce meanwhile fixed hinting issues or similar.

Rendering with the current "serif" font makes it IMHO worse. Look at the spacing of Chiang Mai.
1830-new-serif.png

As the software stack was partially over a decade old, it was quite hard to apply updates. With the latest rework it should be easier to make changes. As long as we can have patches on top of "carto style", it should be possible to adjust further.
In the near future I will probably have no capacity to do such changes. If you are interested in working on Thailand specific additions, it could make things easier. As map allows a higher zoom level than OSM default, the style could need some serious improvements on the width of roads (and maybe water features) on zoom 20. I hope that such changes could also be merged upstream, making maintenance easier.

Offline

#13 2021-02-19 17:43:27

Bernhard Hiller
Member
Registered: 2011-05-10
Posts: 1,055

Re: New server infrastructure for the Community

The font in your last screen shot is ... ugly. It seems fit for some old-style typewritter: constant width per character - but that does not fit east asian alphabets (and I am sure Khmer would look even worse). Can our local community members give better advice?

As for the style updates, I do not know how to do that - a couple of years ago, I tried to set up an OSM webserver, but failed some where in the Linux equivalent of the dll hell known from Windows.

Offline

#14 2021-02-26 05:00:02

Paul_012
Member
Registered: 2011-08-05
Posts: 211

Re: New server infrastructure for the Community

Noto Looped Thai doesn't appear to have been officially released yet. Why not use Noto Serif Thai, though? Yes, it's intended to match the serif style, but the sans/serif distinction doesn't really mean much for Thai script anyway. If there are readability problems, I'd suggest using the looped fonts that are available and workable.

Offline

#15 2021-02-26 09:09:39

stephankn
Moderator
Registered: 2010-05-04
Posts: 622

Re: New server infrastructure for the Community

The Noto Serif font is the example in post #12
https://forum.openstreetmap.org/viewtop … 65#p819365

For me, especially the spacing of the font looks very unpleasant. As I am no native speaker I can't tell whether this is a real issue. The Noto Sans as in #10 has a better spacing. Probably best is to monitor Noto Looped Thai and use once released.

Offline

#16 2021-02-26 13:26:01

nitinatsangsit
Member
From: Bangkok
Registered: 2020-04-03
Posts: 58

Re: New server infrastructure for the Community

As Thai, I prefer the existing Noto Sans Thai UI.

Last edited by nitinatsangsit (2021-02-26 13:26:25)

Offline

#17 2021-02-26 13:26:17

Paul_012
Member
Registered: 2011-08-05
Posts: 211

Re: New server infrastructure for the Community

stephankn wrote:

The Noto Serif font is the example in post #12
https://forum.openstreetmap.org/viewtop … 65#p819365

For me, especially the spacing of the font looks very unpleasant. As I am no native speaker I can't tell whether this is a real issue. The Noto Sans as in #10 has a better spacing. Probably best is to monitor Noto Looped Thai and use once released.

There must be some error in calling the font. What's showing up there looks like the old Unifont which was previously used by the default renderer on osm.org from the beginning of time until a few years ago.

Offline

#18 2021-03-07 08:37:56

stephankn
Moderator
Registered: 2010-05-04
Posts: 622

Re: New server infrastructure for the Community

Paul, you are absolutely right here. Good catch. When trying to replace the font I only replaced Sans with Serif and did not notice that there is no UI variant of Serif, so it did fall back to a different font.

So here is the rendering again to directly compare the two fonts:
Noto Sans UI vs. Noto Serif

I kindly ask again for opinions which one to choose.

1830-new.png Sans vs. Serif 1830-new-noto-serif.png


I also tried Noto Looped, but the bug with tone marks is still visible, so this font is not usable in the current state (see at Khrang).
https://github.com/googlefonts/noto-fonts/issues/2005

1830-new-noto-looped.png

Offline

#19 2021-03-08 15:18:20

Paul_012
Member
Registered: 2011-08-05
Posts: 211

Re: New server infrastructure for the Community

Considering legibility for the widest range of users, I'd go with Noto Serif (though Noto Looped does look better with its mono-thickness lines, so maybe consider adopting it once the bugs have been fixed).

Offline

#20 2021-03-08 17:23:49

SomeoneElse
Member
Registered: 2010-10-13
Posts: 1,496

Re: New server infrastructure for the Community

(offtopic as regards this server but) if anyone reckons that the fonts suggested in e.g. https://switch2osm.org/serving-tiles/ma … debian-11/ could be improved please raise an issue at https://github.com/switch2osm/switch2osm.github.io.

In the early days with those pages there was quite a bit of experimentation to get e.g. Burmese names to display at all.  Hopefully we're past that stage now but I wouldn't be surprised if some non-western characters could still be improved.

- Andy (occasional maintainer of switch2osm guides)

Offline

#21 2021-03-16 07:42:08

stephankn
Moderator
Registered: 2010-05-04
Posts: 622

Re: New server infrastructure for the Community

Hello Andy,

thank you for mentioning this. I Hope to get some more feedback regarding readability of the font. I switched over to the serif variant, as this is simpler to read. I consider filing a PR at carto, as the "main" rendering would benefit from it as well.

I built the new stack again on Ubuntu. In this case on top of 20.04 LTS. For all interested in the technology: it runs inside a docker container, so the required libraries do not conflict with the other contents on the server. It uses tirex to power the rendering engine. I don't understand why others stick so much with renderd, given the nice queue management of tirex. Maybe it simply is a matter of "it works, why change it". Data update and tile expiry is also handled in a special way. These seem to be areas where the requirements differ.
I cross-checked some times with switch2osm, which is quite valuable to give a good starting point. Thanks for sharing insights there.

I did stick with the "stock" mod_tile. I remember I checked your repository, but it is 36 commits ahead, 41 commits behind, So I was uncertain which of the 36 commits contained the gist of the change. I saw a recent discussion regarding this, so will probably jump in there. Seems there are chances to merge back what is required into mod_tile.

PS: I fully switch over the production URL in a day or two.

Stephan

Offline

#22 2021-03-22 07:41:12

stephankn
Moderator
Registered: 2010-05-04
Posts: 622

Re: New server infrastructure for the Community

After some soft-migration during the last days (osmose and tiles had been the major services running on the new server), I finally switched over the remaining things.
Please let me know if something is no longer working as expected. Please note that all cached tiled have been fully re-rendered with the new style up to zoom level 15. If you zoom in further you might currently face some longer loading times as tiles are not yet pre-rendered. There is still a queue of roughly 20 million tiles the server is recreating in the background.

I have retired some of the older services. As getting them back up running with recent software up is certainly work, I am interested in whether someone is actually using them. For privacy reasons I have no user tracking. This makes it harder now to get some usage numbers.
Please let me know whether you are interested in seeing one of these currently disabled services back up:

nametool: It provided a map which highlighted features where either the the script name was missing or the English one. It also offered a helper interface to ease in editing these, like suggesting main street names for sois. Bringing this back in a proper way is a larger effort. Osmose reports missing names. I believe that Johnny was the only one using it besides myself.
taginfo (th): This was similar to the official taginfo instance, but was indexing only Thailand. Getting it back should be lower effort. Is there interest?
wdye: This started very long ago and after the license change I no longer worked on it. It displayed on a heat map where you have edited. A simplified version is available by Pascal Neis (yosmhm).
reports: This listed various statistics and potential tagging issues in Thailand, focusing on bilingual name tags. Any interest in it? It had also a prototype of the "editing trophy", highlighting tagging work done "on the ground".

I mentioned it before, the server is now in a different data-center. It should behave well routing to True due to a direct peering. You can measure speed to the server at https://speedtest.osm-tools.org/

There are still some minor open points for me to work on. This might result in short interruptions, usually shorter than 1min. The server uptime in the last three months was 99.99%. Largest contributor to downtime was work by the data-center on replacing network switches. You can always see server status at http://status.osm-tools.org/

Offline

Board footer

Powered by FluxBB