Progress report:
Made a version which can handle a big data.osm file ( < 2GB or < 4Gb still don’t know)
Use: Put transliterationtable files and matching boundary.gpx files in an "Areas’ directory and drop the file.
As I had a 445MB file from data of Ukrain and places just outside its border not only I put in an Area Ukrain but also from the neighbouring countries. So I could test if all nodes would be handled properly i.e. with the transliterationtable of their country.
It all works.
These were the Areas:
bulgarije
hungary
kazachstan
moldavie
poland
romania
russia
serbia
slowakia
transnistria
ukraina
witrusland
I don’t recommend uploading transliterated names to OSM database.
The reasons are:
Wikipedia lists about a dozen of transliteration schemes for Russian language. Which one should be used? Why?
Bing Maps, Google Maps, Yahoo Maps use 3 different transliteration schemes.
How do you keep transliterated name in sync when cyrillic name is changed?
Some places may have established transliterated name, how do you differentiate those manually edited names vs. automatic?
Some place names in Russia have been imported from vmap0 which is English only, and reverse English->Russian transliteration have been used, as a result there are many errors in the cyrillic names.
There is ongoing project to match all place names from vmap0 against Russian address databases and other sources to spot all errors in cyrillic names and fix those.
IMHO it’s better to populate OSM database with translated names or at least the names matching vmap0, GNS, Geonames(?) databases.
Transliteration depends on the source language. Russian->English (for example Kiev) is different from Ukrainian->English (Kyiv). Transliterating regions around the border can be quite tricky.
Transliteration also depends on the target language, Russian->German is different from Russian->English.
Belarus now has Russian names in the ‘name’ tag and Belarusian names in the ‘name:be’ tag. I don’t know the reason for that (it might be due to import from data sources available only in Russian), but I can imagine they can switch the tags one day to have Belarusian language in the ‘name’ tag and Russian in the ‘name:ru’ tag. Persisted transliterated names will become incorrect…
There are many requests for Russian maps suitable for foreign visitors on many GPS/sat-nav related forums. This problem is well understood and one day OSM will improve on that to include many tourist attractions, site seeing places, etc…, at least in English.
All of the above suggest it’s better to do transliteration on the fly rather than persist one possible version out of many.
This way you can easily change transliteration schemes, change target language, etc…
According to this Russian written in latin script would be ‘name:ru-Latn’.
If you don’t want to specify the language but still want to specify the script, then may be ‘name:’ tag, i.e. name:Latn?
All script codes are 4-letter (the list of all possible values is available at http://unicode.org/iso15924/iso15924-codes.html) so there should be no confusion with language codes which are 2- or 3-letters.
Yuri, all valid arguments I’m sure. But afaik greencaps has no intention of uploading the results of his script. His script can be used by Garmin mapmakers like me (but also others) to help in transliteration. I’m sure that some transliteration in the maps is better then having questionmarks (?) all over the place (I know, it’s especially a problem with my maps and Mkgmap).
Thank you for summarising considerations for adding transliterations to osm database. I understand your concerns. At the moment my goal is an on the fly transliterater which can be put in Lambertus’ build chain. Thank you for the links also. I will study them.
But meanwhile I will alo further develop my persistent updator.
That would be an option. But not enough as still the language had to be specified.
Agreed. I’m not familiar with vmap0, GNS and Geonames but will investigate when time comes… (Anybody who can give me a clue how to use them over the internet is kindly invited to give me a link or hint.)
I made a run for Belarus. There are 27956 nodes with place tags. 169 have name:be, 52 name:en and 11 int_name. It looks as if not to many people in Belarus are adding to osm. Why would not they write belarussian if they added by hand? By the way: looking for someone who can switch them? ;-).
Progress report.
Made a console version which reads from stdin and writes to stdout. As I have no program which would grab that output it was displayed in the console window. As I had no program also which would read the source and write it to stdout where I could read it from stdin I tested by reading from the file self.
Where my not console version takes ten minutes for 500MB the console needed an hour. But I think this is because cmd.exe displays it all in its console window.
What is your build time and how much would you give me?
Ah, I see that you are developing under Windows… I hope you are writing portable code, because my map toolchain is running entirely under Linux…
I know from experience that displaying a lot of stuff takes a lot of extra time, so your 10 minutes-1 hour difference might have been caused by that. Don’t worrry about that too much then. Stdin and Stdout are filepointers just like a regular file under Linux.
One, two, three hours extra for processing all of Europe, Africa, Asia and Oceania is not a problem for weekly updates.
Just a word of warning for anyone about possibly using Geonames or GNS for adding to OSM …
Make sure the data for the country is up to date. I know for a fact that the names for Korea are out of date since the Korean government changed the romanization/transliteration scheme in 2000.
We are doing pretty well, though, in adding names to the Korean data.
Ehhh. In 3:20 the windows console version can handle a half GB file. (only copiing takes 2 minutes and twenty seconds). So for 80 GB that would be more than nine hours. Or is this a wrong calculation? Maybe it has to be done in this way: Handling time for 0.5 GB 1 min (3:20 - 2:20). So 80GB 160 min == more than 2 and the half hour. Reading/writing 7.3 GB one hour. Together three and a half hour. Add some time for uncompressing/compressing.
Ha… not two bytes. UTF-8 is multibyte (from one to six) just read the rfc 2279. http://www.ietf.org/rfc/rfc2279.txt
From multibyte the conversion will -usually- first go to two byte unicode followed by a conversion to one byte ansi.
The Europe, Asia, Africa and Oceania split is about half the planet size. It would be interesting to see how long the conversion actually takes. Depending on the outcome I can decide to run the conversion every time or to do it once in a while. There might be room for optimization here and there as well, so an initial test run that takes a day is no problem. But your numbers are encouraging, 2.5 hours is not a problem for weekly updates.
Notepad does also. Wordpad not. Firefox also.(no charset definition needed). Here on XP.
Tile 63240031 -covering part of Thailand- which I downloaded from Lambertus’ site contains also a lot of questionmarks. I suppose from unconverted Thai characters.
Is it possible or is the result readable/(usable in a navigator) if the characters are transliterated? If so then you could help me with defining a transliteration table. Please first confine if you want to do so. After that you will be presented a table to fill in.
Just wanted to say thank you sooo much for these maps. They make the GPS a whole lot better.
I am doing my part in helping update the map. I am a UPS driver, so I drive down multitudes of streets and see that a lot of them are named wrong in openstreetmap. But anyway, I keep my GPS in the truck with me and upload my traces to check for accuracy also.
Just wanted to say that I appreciate it and I want to do more. Is there a way I could help you update the routable maps so maybe we can get a steady weekly update?
Thanks for offer but the newer Mkgmap versions have some serious changes in output which I need to accommodate. Just did not have the time to do so yet (busy integrating Nominatim in www.yournavigation.org). Probably this weekend.
Having a delivery job really enables you to do some serious tracklogging indeed. Isn’t every truck equipped with a company gps as well? If so, would it be possible to get an anonymized copy of available tracklogs? That would add an awful lot of traces…
Cool, I hope to be able to do some tests this weekend. Do you have a source or 64bit executable somewhere for download?
Every DIAD board has a GPS unit in it, but I do not believe that they use tracks. They are just in there so the supervisors can see where you are at the moment, not where you have been. Even so, I don’t have the authority to grab all the track logs, if they did track them.
