Frontend transliterator: translit. A battle against the ??????'s

csdf · 2009-12-03 18:19:44

This tool would be useful to the wider community - do you intend to share it on the mkgmap-dev mailing list?

Walter Schlögl · 2009-12-03 19:08:05

Can this tool also solve problems with greek, arabic and chinese charakters?

greencaps · 2009-12-03 23:34:13

csdf wrote:

This tool would be useful to the wider community - do you intend to share it on the mkgmap-dev mailing list?

Please see my reply on this question: http://forum.openstreetmap.org/viewtopi … 770#p47770

Walter Schlögl wrote:

Can this tool also solve problems with greek, arabic and chinese charakters?

With greek yes. See: http://forum.openstreetmap.org/viewtopi … 500#p47500

With arabic? I don't know. Is arabic transliteratable? If yes please provide information(links).

With chinese? Maybe. There are some results. See: http://forum.openstreetmap.org/viewtopi … 496#p47496

Walter Schlögl · 2009-12-04 18:08:54

In Wikipedia gibt es eine Tabelle für arabische Zeichen.
Ich kann aber nicht beurteilen, ob die für unsere Zwecke brauchbar ist.

http://en.wikipedia.org/wiki/Romanization_of_Arabic

greencaps · 2009-12-05 12:07:09

Thank you Walter.

For Georgia there was info in http://en.wikipedia.org/wiki/Georgian_alphabet

For <node> place name there is not much to do as mostly all have also name:en or int_name. This is the list of names that were transliterated. The same file is displayed in three difefrent programs (on XP): wordpad, notepad and firefox.

display in different programs

Wordpad does not understand utf8 so it displays all bytes as if they were characters, notepad understand that it is utf8 and converts to the right character but then misses the font. Firefox understands everything (without telling it that it is utf8).

From all programs I copypasted the first three lines to this post. (you never how which conversions are applied using the clipboard).

áƒ•áƒáƒ áƒ«áƒ˜áƒ=vardzia
áƒ¥áƒ•áƒáƒ‘áƒ˜áƒ¡áƒ®áƒ”áƒ•áƒ˜=kvabiskhevi
áƒáƒ¬áƒ§áƒ£áƒ áƒ˜=atsnuri

ვარძია=vardzia
ქვაბისხევი=kvabiskhevi
აწყური=atsnuri

I would like to see comments on the transliteration results. Someone from Georgia out here?

Lambertus · 2009-12-05 15:30:30

As reported on the IRC:

For Turkish you would transliterate ö and ü to o and u - not to oe, ue as in German

greencaps · 2009-12-05 15:44:43

Lambertus wrote:

For Turkish you would transliterate ö and ü to o and u - not to oe, ue as in German

Yes I do already for Turkish. I did not yet bother for German. But then the 60Cx can display Ü and Ö. Or not rotated?

At second thougth: If in utf8 the Turkish ö is coded with the same byte values as the German ö (and i think that will be the case) then if translit keeps the tables per area then the transliteration could be different. But i think that in short i will do away all areas (the .gpx files) and only use one transliterationtable for the whole world as i do now already for the <way>'s.

Last edited by greencaps (2009-12-05 15:52:46)

greencaps · 2009-12-08 20:41:32

About transliterating greek.

In Greece many places have already an int_name or en:name. Whats left are 652 nodes with a place and name tag that need transliteration. Well translit thinks so and it does a transliteration of the utf8 encoded string. If the transliteration result is the same as the utf8 encoded string than the name was already in ansi. Translit counted 559 name's already in ansi. So only 90 tags were added.

For ways these counts were 7097, 5644 and 1439.

On the maps for greece you can see that very often the name is both spelled in greece and internalional.
Here some transliteration results:

name in utf8 = name transliterated
Άγιος Νικόλαος (Agios Nikolaos)=Agios Nikolaos (Agios Nikolaos)
Ευηνοχώρι (Evinochori)=Euienochori (Evinochori)
Πυλαία - Pylea=Pulaia - Pylea
Μικρόκαστρο (Mikrokastro)=Mikrokastro (Mikrokastro)
Σκάλα Συκαμνιάς=Skala Sukamnias
Σίγρι=Sigri
Ζωνιανών=Zonianon
Λιμένας Χερσονήσου=Limenas Chersonisou
Τσούτσουρος=Tsoytsouros
Λιβάδιον=Livadion
Μαρίες (Maries)=Maries (Maries)
Αναφωνήτρια (Anafonitria)=Anafonitria (Anafonitria)
Καταστάρι=Katastari
Βολίμες (Volimes)=Volimes (Volimes)
Μονή=Moni
Άγιος Γεώργιος (Agios Georgios)=Agios Georgios (Agios Georgios)
Αγία Μαρίνα (Agia Marina)=Agia Marina (Agia Marina)
Μονόσπιτα (Monospita)=Monospita (Monospita)
Κορομηλέα (Koromilea)=Koromielea (Koromilea)
Αλίαρτος (Aliartos)=Aliartos (Aliartos)
Βάγια (Vagia)=Vagia (Vagia)
Μαυρομμάτι (Mavrommati)=Maurommati (Mavrommati)
Άσκρη (Askri)=Askrie (Askri)

greencaps · 2009-12-14 14:03:20

The weekly updated img files on http://garmin.na1400.info/routable.php now contain transliterations for 21 Areas:
albania, belarus, bulgarije, cyprus-turkish, czech-republic, estonia, georgia, greece, hungary, kaliningrad, latvia, lithuania, macedonia, moldavie, poland, romania, russia, serbia, slowakia, turkey and ukraina. Attentioen only <node>'s with a valid place AND a name tag are transliterated when needed. (valid place |town|city|suburb|village|hamlet|).

So for instance in greece
http://api.openstreetmap.org/api/0.6/node/83945071
<node id="83945071" lat="37.9043479" lon="20.7090816" version="3" changeset="1768078" user="AiNikolas" uid="14235" visible="true" timestamp="2009-07-08T05:15:31Z">
<tag k="place" v="hamlet"/>
<tag k="name" v="Άγιος Νικόλαος (Agios Nikolaos)"/>
....
will show up in your garmin as " Agios Nikolaos (Agios Nikolaos). While
http://api.openstreetmap.org/api/0.6/node/83943762
<node id="83943762" lat="37.9058311" lon="20.7120171" version="4" changeset="1768078" user="AiNikolas" uid="14235" visible="true" timestamp="2009-07-08T05:15:51Z">
<tag k="place" v="island"/>
<tag k="name" v="Άγιος Ανδρέας (Agios Andreas)"/>
will show up as "????? ??????? (Agios Andreas)" because it is an island.

I now understand that such a restriction is unwanted. In the next update translit will handle all nodes with a name tag.

extremecarver · 2009-12-15 21:20:09

greencaps, do you feel confident by now about puglishing your translit program?
Or maybe even adding it into mkgmap?

greencaps · 2009-12-16 11:56:54

The program at the moment is restricted to do <node>'s. When <way>'s and <relation>'s are also transliterated publication would make sense. Lambertus is the only one who uses (and tests) it. If translit does what I want it to do I will offer it to the community. In which way I still don't know.

Adding to mkgmap is a different story. I never looked into the source of mkgmap so I do not know in which way the code could be integrated. But of course integration in mkgmap is preferred over functioning as a frontend. An extra option --translit which forces mkgmap to trow away all codepages and just take one transliterationtable (the one from translit for instance) would be nice.

skela · 2009-12-16 12:37:53

Would it make sense to introduce a name:translit tag for transliterated names and let mkgmap choose one of them, e.g., mkgmap --name-tag-list=name:en,int_name,name:translit,name or (for native users) just mkgmap --name-tag-list=name:translit,name? The transliteration could be implemented as a separate preprocessing step or integrated in mkgmap so that it can be executed on the fly, when a substitution of name:translit is requested.

ISO 15924 and IANA define some language subcodes, such as sr-Cyrl and sr-Latn for Serbian written in Cyrillic or Latin, but I did not find any subcode for transliterated Russian or Greek, for instance. We could of course invent such codes ourselves, but it could be controversial to add, say, name:gr-Latn and name:ru-Latn to each Greek or Russian location in the OpenStreetMap database. (It could provoke edit wars, because not everyone is following the current version of ISO 9.)

greencaps · 2009-12-17 10:02:43

skela wrote:

Would it make sense to introduce a name:translit tag for transliterated names and let mkgmap choose one of them,..

Thats exactly what happens in Lambertus' toolchain. The tag is added by the frontend.

.. The transliteration could be implemented as a separate preprocessing step

Yes. Transliteration and adding the tag. But all depends on the structure of mkgmap.

ISO 15924 and IANA

I will not look at that to much.

chris66 · 2009-12-23 23:15:05

This pictures shows, that the Nuvi 250 is not able to display utf-8 characters,
but to show lower-case characters.

Map has been build with --charset=utf8 and --lower-case

On MapSource I see the same characters.

The funny thing is, this actually don't looks like utf-8 at all.

But when I use some unknown charset like --charset=xyz, it looks more like utf-8
(but the device is still not able to display them).

Chris

Last edited by chris66 (2009-12-23 23:45:26)

chris66 · 2009-12-23 23:54:03

And this is the build in transliteration in action (only --lower-case option was set).

(MapSource screenshot)

Last edited by chris66 (2009-12-23 23:55:18)

greencaps · 2009-12-24 01:13:20

chris66 wrote:

This pictures shows, that the Nuvi 250 is not able to display utf-8 characters,

My first thougth was : who says that there are utf8 chars in the map?

Map has been build with --charset=utf8

Ahhh. You said that --utf8 was no option. But I had to read it as : use --charset=utf8
So mkgmap can make those .img's. Nice, I hope it just transfers the text untouched.

The funny thing is, this actually don't looks like utf-8 at all.

Well actually it does. That looks exactly like the chars in the transliteration table for greece that I use. See here some characters (copied from wordpad): look at tke leftmost column.

Î›=Ë=L
Î=å=e
Î½=í=n
Ï„=ô=t
Î±=á=a
Ï‚=ò=s
Ã©=e=e
Î”=Ä=D
Î¼=ì=m
Î¤=Ô=T

And the same displayed by firefox and copied from it.
Λ=�=L
έ=�=e
ν=�=n
τ=�=t
α=�=a
ς=�=s
é=e=e
Δ=�=D

The Nuvi displays utf8 with a normal ansi charset (for one byte chars). On the nuvi screen there are 28 character. Those are 14 utf8 characters. Using my transliterationtable i come to Mikala... And looking at openstreetmap http://www.openstreetmap.org/?mlat=37.9 … rs=B000FTF that is right.

chris66 · 2009-12-24 10:34:14

Additional Info: For testing I used the name_tag = name, name:en sequence
in the options file.

When building real maps of couse you would first check for the
english names.

name_tag = name:en, name

chris

chris66 · 2009-12-24 10:44:03

greencaps wrote:

chris66 wrote:
The funny thing is, this actually don't looks like utf-8 at all.
Well actually it does. That looks exactly like the chars in the transliteration table for greece that I use.

Ok, but what does mkgmap do when using a unknown charset like --charset=xyz ?

I would have expected that in this case no charset translation takes place and because the input
data is utf-8 the output also should be utf-8.

Chris

greencaps · 2009-12-24 11:21:00

chris66 wrote:

Ok, but what does mkgmap do when using a unknown charset like --charset=xyz ?

Don't know. Until now I never used mkgmap so I cannot test. But it should bring an errormesage and stop.

I would have expected that in this case no charset translation takes place and because the input
data is utf-8 the output also should be utf-8.

Or nothing put out or only ?????'s. Because thats in line whith what mkgmap does now.

Further my opinion is that if a parameter --charset=.... is omitted the output should be utf8 (and only then). Exactly the same content as in the osm tags. (--charset=utf8 is confusing as no charset will be used. So this should bring an errormessage too and the program should stop, not even start.)

And this is the build in transliteration in action (only --lower-case option was set).

That looks ok. So your claim is that the cyrillic from Bulgaria is correctly transliterated too (in that same .img)?

chris66 · 2009-12-26 14:05:40

Ok, here we are. Sofia compiled with only --lower-case and name-sequence again
is name, name:en, so first taking the lokal name.

Chris

greencaps · 2010-01-02 11:39:09

Thank you for the effort Chris. I see the transliteration. Ok. But you left me unsure as you did not utter a direct "yes" or "no" on my question if both cyrillic and greek are transliterated within one map. (or run of mkgmap).

I would like to know also if georgian, arabic, chinese and so are transliterated. Dou you know what all is transliterated?

greencaps · 2010-01-13 23:26:23

The battle is in a decisive stage. Look at the cyrillic in the center of Irkoetsk, Siberia:
http://www.openstreetmap.org/?mlat=52.2 … rs=B000FTF

Yesterday tile 63340555 on http://garmin.na1400.info/routable.php from Baikal region and Mongolia looked like this in GPSmapedit:

Today (after a special update) it looks like:

Gone are the questionmarks.

Translit now transliterates all items that have a name tag. So nodes, ways and relations. (But only if there are not already tags like name:en or int_name. It adds an extra tag with the transliteration. It only uses one transliteration table with less then 400 entries. It does not use regions or .gpx files anymore.

When I drive to Siberia again this summer my 60Cx will show all placenames and in towns all streets. I'm happy.

Lambertus · 2010-01-14 13:45:03

So very cool

Thanks for you work on this!

acousticnewt · 2010-01-26 11:38:27

greencaps wrote:

Adding to mkgmap is a different story. I never looked into the source of mkgmap so I do not know in which way the code could be integrated. But of course integration in mkgmap is preferred over functioning as a frontend. An extra option --translit which forces mkgmap to trow away all codepages and just take one transliterationtable (the one from translit for instance) would be nice.

If you decide to release this it would be great to add it to mkgmap. Of course I don't know your code either, so I don't know how easy or in what
manner it could be done. Contact me directly or via the mkgmap list if you want to discuss it.

..Steve

greencaps · 2010-01-26 21:30:14

You can find binaries for windows and linux here: http://tinyurl.com/ydcjhpa

I would like an integration in mkgmap. As I never did something with it my plan was to first install mkgmap (on a new system that shortly will arrive). Then play a bit with it and investigate the source. I could start looking where transliteration code could be added. I'm not familiar with java so that would take some time.

When time comes I will post a message in your list.

Announcement

#26 2009-12-03 18:19:44

Re: Frontend transliterator: translit. A battle against the ??????'s

#27 2009-12-03 19:08:05

Re: Frontend transliterator: translit. A battle against the ??????'s

#28 2009-12-03 23:34:13

Re: Frontend transliterator: translit. A battle against the ??????'s

#29 2009-12-04 18:08:54

Re: Frontend transliterator: translit. A battle against the ??????'s

#30 2009-12-05 12:07:09

Re: Frontend transliterator: translit. A battle against the ??????'s

#31 2009-12-05 15:30:30

Re: Frontend transliterator: translit. A battle against the ??????'s

#32 2009-12-05 15:44:43

Re: Frontend transliterator: translit. A battle against the ??????'s

#33 2009-12-08 20:41:32

Re: Frontend transliterator: translit. A battle against the ??????'s

#34 2009-12-14 14:03:20

Re: Frontend transliterator: translit. A battle against the ??????'s

#35 2009-12-15 21:20:09

Re: Frontend transliterator: translit. A battle against the ??????'s

#36 2009-12-16 11:56:54

Re: Frontend transliterator: translit. A battle against the ??????'s

#37 2009-12-16 12:37:53

Re: Frontend transliterator: translit. A battle against the ??????'s

#38 2009-12-17 10:02:43

Re: Frontend transliterator: translit. A battle against the ??????'s

#39 2009-12-23 23:15:05

Re: Frontend transliterator: translit. A battle against the ??????'s

#40 2009-12-23 23:54:03

Re: Frontend transliterator: translit. A battle against the ??????'s

#41 2009-12-24 01:13:20

Re: Frontend transliterator: translit. A battle against the ??????'s

#42 2009-12-24 10:34:14

Re: Frontend transliterator: translit. A battle against the ??????'s

#43 2009-12-24 10:44:03

Re: Frontend transliterator: translit. A battle against the ??????'s

#44 2009-12-24 11:21:00

Re: Frontend transliterator: translit. A battle against the ??????'s

#45 2009-12-26 14:05:40

Re: Frontend transliterator: translit. A battle against the ??????'s

#46 2010-01-02 11:39:09

Re: Frontend transliterator: translit. A battle against the ??????'s

#47 2010-01-13 23:26:23

Re: Frontend transliterator: translit. A battle against the ??????'s

#48 2010-01-14 13:45:03

Re: Frontend transliterator: translit. A battle against the ??????'s

#49 2010-01-26 11:38:27

Re: Frontend transliterator: translit. A battle against the ??????'s

#50 2010-01-26 21:30:14

Re: Frontend transliterator: translit. A battle against the ??????'s

Board footer