osm2mp.pl: OSM -> Polish format converter

nyem · January 17, 2009, 2:47am

Yes, the new nodes should be right at the edge of the tile.

Lambertus · January 17, 2009, 9:18am

Ok, should the node have the same id in both tiles?

nyem · January 17, 2009, 3:32pm

Doesn’t have to be the same id. Here’s an example (in polish format) of a way that extends to another tile. The most important thing is that the coordinates should match up, and it is exactly at the edge of the tile.

Tile 1
[POLYLINE]
Type=0x01
EndLevel=4
Label=~[0x04]E2 NSE
DirIndicator=1
Data0=(3.0391936,101.7062182), … (2.6274582,102)
RoadID=1299
RouteParams=6,4,1,0,0,0,0,0,0,0,0,0
Nod0=0,1046,0
Nod1=
…
Nod16=224,2649,1
[END]

Tile 2
[POLYLINE]
Type=0x01
EndLevel=4
Label=~[0x04]E2 NSE
DirIndicator=1
Data0=(2.6274582,102), … (2.1308003,102.7412188)
RoadID=285
RouteParams=6,4,1,0,0,0,0,0,0,0,0,0
Nod0=0,1143,1
Nod1=56,462,0
Nod2=97,754,0
Nod3=215,375,0
Nod4=223,769,0
[END]

Some info from Rottweiller Auto-Routing guide (via http://cgpsmapper.com/route.htm))

**Routing across tiles. **If you are connecting several maps, or have split a larger map into smaller pieces (tiles), make sure that the nodes where the connections are to take place ACTUALLY will connect (by right clicking each connecting node, select Node Properties, and ensure that Â“This node is routing graph nodeÂ” is checked. ThatÂ’s all (except if you intent to provide turn restrictions at a certain intersection, in which case you can add those to each connecting node as well). The rest will be completed automatically.
The current (most recent versions of GPSMapEdit and cGPSMapper do a splendid job of linking maps Â– BUTÂ… there is one very important caution Â– irregular-shaped map tiles (individual maps) MAY not link correctly. While I am not certain of the reason, this may be a function of the Â‘backgroundsÂ’ that cGPSMapper automatically adds to each tile during processing.

All of the above are if we are using the gpsmapedit method. It seems that maproute can detect and join the matching nodes automatically, which I think is the best way to work with irregular-shaped maps.

Lambertus · January 17, 2009, 7:44pm

Saved by the bell for me

http://www.mkgmap.org.uk/page/tile-splitter

Now only Mkgmap needs to be adapted to set the correct edge routing information.

Lambertus · January 17, 2009, 7:51pm

The following is a copy from the IRC talk about this, contains some additional info:

#osm:

[12:44] Lambertus: ping
[13:31] stever: pong
[13:31] hi
[13:31] i saw your low-key splitter announcement great!
[13:32] ha!
[13:32] have you tried it
[13:32] i was just started to work on adapting osmcut to do something similar, but i’m glad you’ve made something like this
[13:32] it seems like a difficult job
[13:34] anyway, i was wondering if the edge node that you add to the ways that crosses tiles is added to both ways at the same coordinates?
[13:34] no, i did not try it yet, but I certainly will try to do so this weekend
[13:35] i have split the world in half using osmosis and will try to use your splitter to divde that further into small tiles
[13:35] its probably in the same place, but the splitter doesn’t actually do that. It down to mkgmap to cut the line according to the bounding box
[13:36] ah, ok, i misunderstood that apparently
[13:36] half the planet might be a bit ambitious
[13:37] second question: does mkgmap set the special ‘route’ bit on those nodes?
[13:37] well I have a machine with 3 or 4gb ram and 64bit linux to try it on
[13:37] no, but that is my plan.
[13:37] cool!
[13:37] oh cool, well I’d be interested to know if it works
[13:38] do you have any idea how splitter will work when it tries to write to thousands of files ?
[13:38] i.e. do you open a file handle to all those files and keep them open all the time?
[13:39] i mean, usually there is a limit on the number of files a process may have open at any given time
[13:39] yes it keeps a file open. I wouldn’t expect thousands of files though
[13:40] the europe file I was working with is about 1/4 of the planet and its only about 70 files
[13:40] well, currently I create tiles measuring 0.25 x 0.25 degrees, if i continue to do that, then it will be certainly thousands
[13:41] ah, but those tiles will fill up a garmin unit without sd card very quickly
[13:41] is the topology kept between slices?
[13:41] I was looking for a .osm cutter the other day myself
[13:42] i guess you’ve solved the problem of the 3.5 mb tile size at one point? Because in the early days of mkgmap I could not put the whole of rotterdam in a single tile
[13:42] osmcut could only handle 1 degree tile sizes for my area
[13:42] yes that 3.5mb was a bug
[13:43] the splitter creates variable sized tiles so you don’t get very many
[13:43] yes i read that, that’s amazing
[13:44] smart little piece of software already
[13:44] chippy: what do you mean by ‘is the topology kept’
[13:45] so, huge shapefile → osm. split the .osm file so we can upload in chunks, and it keeps things connected
[13:45] or many .osm files → routing, and the routing is unbroken over tile edges
[13:46] chippy: right. Nothing is lost that is required for Garmin maps, but if you want to use the OSM files for other purposes then there could be problems. I’ve not thought about that
[13:46] hmm, actually with creation, and negative ids, I don’t think the first would be possible
[13:47] stever, yep
[13:48] chippy: I’m hoping that routing will work across tiles in garmin maps, but again for routing in other apps, it may or may not be be ok
[13:49] stever, ok cool stuff
[13:57] stever: the input osm file, can it be a gz file as well?
[13:57] yes
[13:57] k
[15:15] Lambertus: it just puts its files in the directory where it was started. You’ve probably noticed that by now though
[16:00] stever: yes, it took a while, but i see the files now
[16:00] <ted_> Lambertus: my machine has 4gb ram and osm2mp choked trying to convert all of Pennsylvania
[16:00] * ted_ goes looking for stever’s announcment
[16:00] how long? it is pretty slow
[16:01] <ted_> well, i shouldn’t say choked, it finished
[16:01] it’s still running, but so it the gosmore update, so they’re fighting for resources
[16:01] <ted_> it just ate up all my ram and swapped like crazy first
[16:01] ted_: i’m seeing that too
[16:02] heh, they will probably both fail then…
[16:03] <ted_> stever: where’s this announcement of yours?
[16:04] http://www.mkgmap.org.uk/page/17
[16:04] and http://www.mkgmap.org.uk/page/tile-splitter
[16:05] <ted_> stever: awesome!
[16:05] <ted_> i was just bemoaning this the other day when i tried to create a routable map of pennsylvania
[16:05] <ted_> and realized that it was too big
[16:06] ok, but note that we don’t have routing between tiles just yet
[16:06] <ted_> ah, bummer
[16:06] <ted_> but you think it should be possible?
[16:09] yes, it should be possible
[16:09] <ted_> cool
[16:14] <ted_> stever: glad to hear you’ve run this on all of europe, so PA should be no problem
[16:16] how big is the PA osm file?
[16:16] <ted_> 2.7Gb uncompressed
[16:23] <ted_> Total time 250s
[16:23] <ted_> not bad at all
[16:23] heh, how many files did it split it into
[16:24] <ted_> 8
[16:24] <ted_> clearly i need to convert areas.list into something i can view graphically
[16:24] that would be cool
[16:25] * ted_ ponders what format would work best
[17:06] ted_: I’m thinking about reading it into OpenLayers and show the tiles as features, something like this: http://wiki.openstreetmap.org/wiki/Image:Garminmaps.jpg
[17:09] <ted_> i just wrote a really quick script to generate a .kml file from it
[17:09] <ted_> can OL handle KML?
[17:09] Yes
[17:10] (Wow, I have good timing)
[17:10] <ted_> hah
[17:10] But the same liitations apply as with all vectors
[17:10] <ted_> these are just rectangles
[17:11] But how many of them?
[17:11] <ted_> http://maps.google.com/maps?f=q&source=s_q&hl=en&geocode=&q=http:%2F%2Fmavra.perilith.com%2F~luser%2Fareas.kml&g=2365+Dewey+Ave,+Northampton,+PA+18067&ie=UTF8&z=7
[17:11] <ted_> depends on how big the input data is
[17:12] <ted_> stever: ^^
[17:12] ted_: I’m just saying, more than 500 is asking for trouble.
[17:12] <ted_> i think stever said all of europe split down into ~70
[17:12] <ted_> Lambertus, stever: http://mavra.perilith.com/~luser/convertarealist.txt
[17:12] <ted_> is the script
[17:13] <ted_> pretty simple
[17:13] ted_: cool
[17:16] <ted_> also, thanks to google maps, you got my home address for free in that first link
[17:20] ted_: Did you see my comments last night re: browser data editor?
[17:21] <ted_> yeah, why do you say that?
[17:22] I think that even the data browser goes too far, really; I think that the OSM website is really designed to be about getting the data out, not in.
[17:22] Also, I think that a real data editor is a full fledged application in its own right – the same way potlatch is
[17:22] (I also think that Potlatch belongs outside of the OSM homepage)
[17:23] <ted_> ok
[17:23] <ted_> makes sense
[17:23] <ted_> but would be nice to have an openstreetbugs type of interface on the map, at least
[17:23] * ted_ wanders off
[17:23] Also, I think that tying an editor into the website is just limiting
[17:23] * crschmidt guesses this is really more appropriate for #osm-dev
[17:24] +1 for browser based editor also
[17:24] does JOSM have a way to view the history of an object?
[17:32] europe tiles shown using ted’s script: http://maps.google.com/maps?f=q&source=s_q&hl=en&geocode=&q=http:%2F%2Fwww.mkgmap.org.uk%2Ftmp%2Fareas.kml&ie=UTF8&z=3
[18:09] <ted_> stever: man, germany really gets tiled
[18:11] yes its interesting seeing the best mapped places

nyem · January 18, 2009, 11:51am

Great stuff indeed

japa-fi · January 27, 2009, 7:51pm

Could someone point me to wiki article or similar describing the tiles you are referring to?
What defines the tile edges? I create map from southern half of finland. Looking at the link http://maps.google.com/maps?f=q&source=s_q&hl=en&geocode=&q=http:%2F%2Fwww.mkgmap.org.uk%2Ftmp%2Fareas.kml&ie=UTF8&z=3
seems that my map contains 3 tiles. Is that the case?

Lambertus · January 28, 2009, 8:37am

The tilesplitter has some kind of tile optimisation algorithm which aparently produces these tile bboxes. Luckily you can define your own bboxes as well, placing the while of Finland in a single tile and have the algorithm determine the other bboxes.

This is my result when working with half the planet (Europe, Asia, Africa and Oceania)

emj · January 28, 2009, 1:25pm

That’s nice, prolly a picture of the month… And also fun seeing Google maps being slow.

liosha · February 3, 2009, 7:03pm

Hi all!

I’ve added basic cropping functionality to osm2mp. It also creates boundary road nodes.
This time available in svn only.
Use --bbox <minlon,minlat,maxlon,maxlat> to specify boundary box.

weolvi · February 4, 2009, 7:27am

Does this mean that I can take one large file (say country.osm), split it into smaller tiles via multiple runs of osm2mp with several matching --bbox’es and routing will work across tile borders?

liosha · February 4, 2009, 8:03am

weolvi,
yes, you can. And routing will work
You can also use tile splitter first, and then osm2mp with bboxes from areas.list
I’ve successfully built map of Russia such way: http://gpsmapsearch.com/osm/mp/

Note that turn restrictions disabled this time

Lambertus · February 4, 2009, 8:51am

Awesome! It looks like we now have a full chain of software capable of generating routable Garmin maps without human intervention for the first time.

Very cool and good work liosha!

/me is going to implement an automated chain this weekend

mtrax · February 4, 2009, 9:24am

great work , I’m sure we are all waiting for this
thanks

cdavila · February 5, 2009, 5:48pm

Always a convert a map from an osm file to mp with osm2mp I get some messages saying: Warning! Turn restriction RelID=XXX is not properly processed, but I check relations and seem to be correct (at least I don’t find the error). Could someone check relations in this area [1] and tell me if there’s something wrong or if it’s a problem of osm2mp?
[1] http://www.openstreetmap.org/?lat=39.481821&lon=-6.363901&zoom=18&layers=B000FTF

liosha · February 5, 2009, 6:06pm

cdavila,
it’s a problem of osm2mp - this time it can’t process restrictions on splitted roads.
hope i’ll fix it soon

cdavila · February 5, 2009, 6:38pm

OK, let’s wait
Thanks

weolvi · February 6, 2009, 2:36pm

Some experiences
Ok, so I also tried to put up a chain to produce some maps for the first time. But it was neither easy nor successful – most likely depending on the tools used. Here’s what I experienced; maybe somebody has some good ideas for improvement:

Using “osm2mp --bbox …” didn’t work for me on europe.osm as osm2mp first loads all nodes into memory and fails after about 8 million because I have only 1GB. Ok, so I have to first make tiles that are slightly larger and then let osm2mp do its job.
Tried “osmosis --bb completeWays=yes” to include every polygon that has at least one node inside the bbox but this is so slow that it’s just unusable. Then I tried just some ordinary but 0.1Â° larger bbox’es and that worked OK. But who knows what that 0.1Â° should really be?
Then I wanted to compile and test my new maps with mkgmap. Compilation worked and I can see all tiles on my Garmin. Routing works but not across tile borders. Is it possible that mkgmap is not yet able to use the links provided by osm2mp?

weolvi · February 6, 2009, 2:42pm

Just in case you succeed – could you please make a complete set of maps of Europe? I am still using Radomir’s, but they are getting outdated and France is broken (at least on my machine).

Lambertus · February 6, 2009, 3:12pm

I planned to use the tilesplitter application.

Osmosis’ completeways function is borked

That is very possible. Perhaps the author of Mkgmap just needs to be informed of the mechanism osm2mp is using.