You are not logged in.

#1 2019-11-09 13:21:18

Bernhard Hiller
Member
Registered: 2011-05-10
Posts: 886

OsmAnd Map for South East Asia

Since I am not content with the default style of the map, I tried to create my own style. Some time ago, I reported about the Garmin style, but this time, it is OsmAnd.
Due to the utter lack of documentation and support, it is not as good as the Garmin style, but:
- "facebook" roads were changed to "highway=road" (except when they were highway=track)
- some dual carriage highways were upgraded to "trunk"
- road surface is differentiated for paved/unpaved only, but tracktype>=2 is also deemed unpaved
- elevation contour lines are included

Some screen shots are available in a zip file at:
https://drive.google.com/file/d/16ZTGCA … sp=sharing
This file also contains the rendering_types.xml used for map creation, and BERNIE.render.xml which is necessary for showing some of the features above (you have to place it into
Computer\Your Phone\Internal Memory\Android\data\net.osmand\files\rendering )

Since the OsmAndMapCreator just crashes with an OutOfMemoryException when I start it with the complete file of the region (tried even on a machine with 16 GB RAM), I had to split the area into 18 areas. Take a look at the splitter.kml file from above zip file to find out which areas you may need (you can view the kml file in Google Earth).

Some files are available now (more to follow):
- 47120001.obf: https://drive.google.com/file/d/15y-lN0 … sp=sharing
- 47120002.obf: https://drive.google.com/file/d/1hNmT44 … sp=sharing
- 47120003.obf: https://drive.google.com/file/d/1MyFnld … sp=sharing
- 47120004.obf: https://drive.google.com/file/d/1O9LmAT … sp=sharing
- 47120005.obf: https://drive.google.com/file/d/11nkT0K … sp=sharing
- 47120006.obf: https://drive.google.com/file/d/1rw_5IO … sp=sharing
- 47120007.obf: https://drive.google.com/file/d/1jYj0wT … sp=sharing
- 47120008.obf: https://drive.google.com/file/d/1Y5JKxJ … sp=sharing
- 47120009.obf: https://drive.google.com/file/d/1NvGPGS … sp=sharing
- 47120010.obf: https://drive.google.com/file/d/1NYALaR … sp=sharing
- 47120011.obf: https://drive.google.com/file/d/172gPBD … sp=sharing
- 47120012.obf: https://drive.google.com/file/d/1HKuaNe … sp=sharing
- 47120013.obf: https://drive.google.com/file/d/11x0hMn … sp=sharing
- 47120014.obf: https://drive.google.com/file/d/1JjCNv7 … sp=sharing
- 47120015.obf: https://drive.google.com/file/d/1JMFWvM … sp=sharing
- 47120016.obf: https://drive.google.com/file/d/1s7iB_v … sp=sharing
- 47120017.obf: https://drive.google.com/file/d/1DK5ajl … sp=sharing (note: the OsmAndMapCreator threw a NullPointerException during the creation process - the map may have some bugs!)
- 47120018.obf: https://drive.google.com/file/d/1n3BAXL … sp=sharing

Countries included:
- Thailand (complete)
- Malaysia (peninsular only, Borneo is missing)
- Singapore (complete)
- Cambodia (complete)
- Laos (complete)
- Vietnam (up to 23 N, i.e. a small area in the far north is missing)
- Myanmar / Burma (south of 23 N, east of 94; e.g. Yangoon and Mandalay are on the map)

Last edited by Bernhard Hiller (2019-11-10 12:48:30)

Offline

#2 2019-11-09 20:31:43

stephankn
Moderator
Registered: 2010-05-04
Posts: 472

Re: OsmAnd Map for South East Asia

Sounds interesting. Could you embed a few screenshots?

What -Xmx setting did you use for the full area? My server has significantly more memory available. Would it help if I run some conversion steps for you?

Offline

#3 2019-11-09 21:00:00

Bernhard Hiller
Member
Registered: 2011-05-10
Posts: 886

Re: OsmAnd Map for South East Asia

Thanks for your comments, Stephan.
2 screen shots - for those "facebook" roads - are included in the zip file.
Here are 2 screen shots for Chiang Mai:
49038898783_a4472926ec_z.jpg
49038897223_5e7e28d955_z.jpg
That pinkish line near Ban San Klang Nuea is a "road" - I did not check here if that is a facebook import or was already tagged as "road".

I used -Xmx15G on the "big" machine.
Adding the step with "splitter" (of the mkgmap suite) is not complicated - the worst thing is that OsmAndMapCreator cannot be scripted easily, it is a GUI tool (I will look for some tools used in software testing, that should do the trick). There are non-GUI tools for scripting, but it's not possible to use one's own style there (at least I did nowhere find documentation on that topic). So I have to start that tool for every file produced by splitter. With splitter setting --maxnodes of 15million, my machine copes still well with the memory requirements of OsmAndMapCreator. I guess I could increase that to some 25million.

Last edited by Bernhard Hiller (2019-11-09 21:01:02)

Offline

#4 2019-11-09 22:07:53

stephankn
Moderator
Registered: 2010-05-04
Posts: 472

Re: OsmAnd Map for South East Asia

I did run it with Xmx=16G on my server with the full Thailand PBF. It took 13 minutes. Maybe just a little bit extra memory and it would work for you.

java -Djava.util.logging.config.file=logging.properties -Xms1G -Xmx16G -cp "./OsmAndMapCreator.jar:lib/OsmAnd-core.jar:./lib/*.jar" net.osmand.util.IndexBatchCreator batch.xml

According to the documentation that batch.xml is there to do the batch mode without GUI. I never tried the GUI. Just added the directories as mentioned in the README and let it run.
It results in 360M. You can have it here: https://downloads.osm-tools.org/Thailand.obf

What are your requirements for creating updates?

If have not checked any details.
Is the file contents filtered by your style? How frequent would you need updates?

I just let it run in parallel to regular server activity with whatever comes with it by default using the full Thailand file. It took only 13 minutes, so running it from time to time in a batch job sounds well possible in case you are interested.

Is there any additional external dependency? SRTM data? Anything for geocoding?

The transliteration of OsmAnd is horrible. As you create the files yourself, how about properly RTGS the missing ones? I recommend this library. The author responded quite fast to a bug report I sent and the results look quite good.
https://pypi.org/project/tltk/

Offline

#5 2019-11-10 03:09:39

Beddhist
Member
From: Doembang Nangbuat, TH
Registered: 2009-07-28
Posts: 426
Website

Re: OsmAnd Map for South East Asia

Bernhard Hiller wrote:

- "facebook" roads were changed to "highway=road" (except when they were highway=track)

Bernhard, thank you very much for doing that. For this highlight alone you deserve a medal.

Stephan, does your extract include the FB highlights? I can't see any.

Offline

#6 2019-11-10 07:58:56

stephankn
Moderator
Registered: 2010-05-04
Posts: 472

Re: OsmAnd Map for South East Asia

my extract was just a demo to see what the server can handle and for Bernhard to check whether it looks reasonable.
It does NOT contain anything special that Bernhard did.

In case the blocking point for not having a single file for Thailand is the constraint of a machine with 16GB of memory to do the processing, then I can help out.

Offline

#7 2019-11-10 08:00:09

Beddhist
Member
From: Doembang Nangbuat, TH
Registered: 2009-07-28
Posts: 426
Website

Re: OsmAnd Map for South East Asia

Thanks for that, all clear now.

Offline

#8 2019-11-10 13:32:09

Bernhard Hiller
Member
Registered: 2011-05-10
Posts: 886

Re: OsmAnd Map for South East Asia

Stephan, thanks for your efforts.
Did you use the rendering_types.xml file contained in the zip file linked in my first post? Without that, the "facebook" roads won't be converted to "highway=road". By the way, where did you find a "readme" file for the batch tool? I do not have it...

Next, the obf files are between 100 and 200 MB each, in sum 2.3 GB - the map contains far more than Thailand only, and it contains contourlines which I created with a patched version of Srtm2Osm (the "normal" rendering of OsmAnd would interfere, and OsmAnd does not cope with closed lines; you must use the "BERNIE.render.xml" on your device to see them).
With OsmAnd, it is not at all necessary to have a single file for the whole region. They do not offer a single file for Germany either. Routing over file boundaries still works.

Transliteration: it is done automatically by OsmAnd, and looks similarly bad to the automatic transliteration used by mkgmap (for Garmin). But with OsmAnd, you can select "local names" or your preferred language.
There's also a menu item for switching transliteration off - but it does not work. I opened a ticket at
https://github.com/osmandapp/Osmand/issues/7891
I do not expect a solution for quite a lot of time.

I'd also like to show "facebook tracks", i.e. keep them as tracks, but add an overlay during rendering which indicates that they are facebooks imports. But I do not know how to get a tag for that purpose into the obf file (there are tools to create an osm file from the obf file, so I can inspect the result with JOSM).
In case you find out how that can be done, please post your answer at https://forum.openstreetmap.org/viewtopic.php?id=67908

Offline

#9 2019-11-10 17:53:56

stephankn
Moderator
Registered: 2010-05-04
Posts: 472

Re: OsmAnd Map for South East Asia

As said, I just wanted to have a rough estimate of what load is to expect on the server, so I followed the steps outlined here:
https://wiki.openstreetmap.org/wiki/Osm … nd_Line.29

All in all I probably only invested 30 minutes in the whole topic, so you are certainly way more experienced with OsmAnd that I am currently.

I assume the converter drops all tags which OsmAnd does not need later. Quickly searching the source points to:
https://github.com/osmandapp/OsmAnd-too … s.java#L47

Have a look around that file. All these HashSet TAGS_TO_SAVE, TAGS_TO_ACCEPT and so on might be the structure you are looking for. Check how they are filled. It is either hard-coded or is parsing some style file somewhere to determine what tags to keep.

I have seen that some stuff is hard-coded there as well. like converting the *_link ways or normalizing units.

Offline

#10 2019-11-12 13:07:13

Bernhard Hiller
Member
Registered: 2011-05-10
Posts: 886

Re: OsmAnd Map for South East Asia

Thanks, Stephan: the parameter could hide in plain sight in the batch file:

process_attributes ... renderingTypesFile="E:\Maps\Development\Thai.Osmand.Test\rendering_types.xml"

Of course, without an underline between "rendering" and "types" here...

I'd better start my holidays now, but have to wait another 3 weeks.

Offline

#11 2019-11-12 14:33:56

Bernhard Hiller
Member
Registered: 2011-05-10
Posts: 886

Re: OsmAnd Map for South East Asia

I'll put this project away for the next couple of months. And then, I'll work on a cycle map of Germany and Czechia instead of a map of Thailand.
In case anyone wants to continue with this map of Thailand (what about Stephan: you seem to have some computing background and know how to use OSM data), I'd like to give a short summary.

"Facebook roads" are the major PITA for many people here. Changing them to "highway=road" except when they are tracks is actually a trivial task: two lines near the top of my rendering_types.xml (included in the zip package) are required:

<entity_convert pattern="tag_transform" from_tag="import" from_value="yes" if_tag1="highway" if_not_value1="track" to_tag1="highway" to_value1="road" to_tag2="facebook_import" to_value2="yes" routing="yes"/>
<entity_convert pattern="tag_transform" from_tag="import" from_value="yes" if_tag1="highway" if_value1="track" to_tag1="highway" to_value1="track" to_tag2="facebook_import" to_value2="yes" routing="yes"/>

That's it already. They will be treated as if they were tagged "highway=road" originally.
Since "road" is rendered in a very pale style, I changed that in "BERNIE.render.xml", but that's just extra eye-candy.

If you would like to add an overlay to the facebook imports, you'll have to get an extra tag into the obf file:

<category name="facebook">
	<type tag="facebook_import" value="yes" minzoom="11" notosm="true" />
</category>

The "minzoom" attribute is required, otherwise the tag will be ignored. Next, add some instructions in the "line" section of "BERNIE.render.xml" (I haven't done that yet).

Converting dual carriage highways to trunk is done with

<entity_convert pattern="tag_transform" from_tag="dual_carriage" from_value="yes" if_tag1="highway" to_tag1="highway" to_value1="trunk" routing="yes"/>

I think most of you will rather oppose that.

Adding a "surface" tag to raods which have a "tracktype" tag is done with

<entity_convert pattern="tag_transform" from_tag="tracktype" if_value1="grade1" to_tag1="tracktype" to_tag2="surface" to_value2="paved" />
<entity_convert pattern="tag_transform" from_tag="tracktype" if_not_value1="grade1" to_tag1="tracktype" to_tag2="surface" to_value2="unpaved" />

I am not sure if it overwrites existing surface tags, so additional changes (something like if_not_tag2="surface") might be necessary.

Somewhere farther down in the rendering_types.xml is a section on "surface". I changed there a lot to reduce all the different tags to "paved" and "unpaved" only, and to get it shown at less zoom.

Originally, I thought adding elevation contour lines could be an easy starting point. But that turned out to be a major WTF. In OsmAnd, you can do an "in app purchase" and buy their elevation contour lines. But since I have all the data and tools to create them for Garmin, that should be feasable with OsmAnd too.
I did not get it working with the "normal" tagging, and also with a totally different tagging it did not work well, see
https://github.com/osmandapp/OsmAnd-tools/issues/328 and
https://github.com/osmandapp/Osmand/issues/7615
The cause is a bug in the OsmAnd rendering engine (running on the phone) which does not render closed lines (also when start and end node of a line are next to each other, the line is deemed closed). I had to patch Srtm2Osm which I use for creating the contours.
In rendering_types.xml, following lines are required:

<category name="elevation">
	<type tag="rsw_type" value="rsw_max" minzoom="11" nameTags="hmz" notosm="true" />
	<type tag="rsw_type" value="rsw_int" minzoom="14" nameTags="hmz" notosm="true" />
	<type tag="rsw_type" value="rsw_min" minzoom="17" nameTags="hmz" notosm="true" />
</category>

and some additional paragraphs in the "order", "text", and "line" sections of BERNIE.render.xml where such oddly named tags are mentioned.

I'll upload my contourline file soon, and add a download link here. You can use "common" tools for cutting the Thai part from the file (Geofabrik also offers a .poly file) and combine it with a fresh download of a Geofabrik extract, and run the obf creation.

Edit:
The contour lines are uploaded, you can get them from
https://drive.google.com/file/d/1wEIj3U … sp=sharing
Note that they have that special odd tagging mentioned above.

Last edited by Bernhard Hiller (2019-11-22 19:38:48)

Offline

Board footer

Powered by FluxBB