You are not logged in.

#26 2011-08-14 12:47:14

Willi2006
Member
From: Khon Kaen, Thailand
Registered: 2009-12-12
Posts: 421
Website

Re: ODbL in Thailand

Northeast is done. There are licence change issues left which I couldn't resolve. May be others who have touched the objects can do that.

|                  Total   Loss    Loss possible   Reduction possible
Relations:           201      1                0                   10        
Ways:             37,739     16               19               9 (121) (*)
Ways (km):        65,375    428              207           597 (5,832) km
Nodes untagged:  493,185    609              261                  216    
Nodes tagged:      8,724      1              282                    5    

(*) 9 ways still have nodes of non agreeing users. From the other 112 ways the nodes have been deleted but the non agreeing users are still in the histories. Thus these ways still get flagged by the plug-in.
The figures are inaccurate due to the use of inaccurate boundaries and ways crossing the boundaries.

Last edited by Willi2006 (2011-08-14 12:50:32)

Offline

#27 2011-08-21 13:09:15

Willi2006
Member
From: Khon Kaen, Thailand
Registered: 2009-12-12
Posts: 421
Website

Re: ODbL in Thailand

Thanks for the new table. It's great that the old tables are still available. Comparing the tables from July 23rd and August, 19th shows that the definitive loss is down from 85,000 (4.77%) to 70,000 (3.67%) nodes. Major decliner down from 65,000 (3.66%) to 51,000 (2.71%) nodes. And anonymous edits down from 19,000 (1.06%) to 17,000 (0.92%) nodes. Recently I recorded GPS tracks and characteristics of the highways 1009 and 1192 (Doi Inthanon) which where created anonymously and replaced them. As far as I've seen the most anonymous edits are in the Northern Region. Unfortunately the link W doesn't work for anonymous.

Would it be possible to count the untagged nodes (just nodes of ways) and the tagged nodes (assume mainly POI and some other) separately?

Last edited by Willi2006 (2011-08-21 13:12:53)

Offline

#28 2011-11-22 22:00:36

stephankn
Moderator
Registered: 2010-05-04
Posts: 622

Re: ODbL in Thailand

Hi,

I uploaded another update of the data.

http://downloads.osm-tools.org/check-od … 11121.html

If you followed the OSM announcements you might already know that the date of the license switch might be the 1st April next year.

So up to that date we should try to replace anonymous creations and data of disagreeing or non-responding users with more accurate data.
With the availability of bing imagery in large parts of Thailand it should be possible to also have accurate road geometry even in situations where GPS has problems. The curvy road to Doi Suthep is one example where Bing is a lot better than my GPS tracks.
Likely it's the same on other parts.

A quite useful tool is the license check plugin in josm.

For anonymous edits this is not working.

Would it help if I provide updated extracts containing the untouched anonymous data? Like the ones i published before?

Stephan

PS: Willi, what is the purpose of the different counting? Would require some additional parsing of the data but not too complicated.

Offline

#29 2011-11-26 10:37:56

Willi2006
Member
From: Khon Kaen, Thailand
Registered: 2009-12-12
Posts: 421
Website

Re: ODbL in Thailand

Untagged nodes are mainly nodes of highways or other features which can be traced easily and quickly from Bing satellite images. Even drawing an average line from several GPS traces is faster at least by a factor of 10 than adding tags to POI's especially in a foreign language.  As I've written earlier in this thread in August I replaced non ODbL data in the Northeast, mainly data which one decliner had just traced from Bing. It took me about 2 weeks to replace about 14,000 untagged nodes without going somewhere. But replacing the tagged non ODbL nodes would require travelling and more time for editing.

Thus I think counting tagged and untagged nodes separately would tell more about the effort needed.

Offline

#30 2011-12-15 00:33:59

stephankn
Moderator
Registered: 2010-05-04
Posts: 622

Re: ODbL in Thailand

Today I got a list of changesets where the user accepted the new contributor terms but wanted to stay anonymous.
Good news: all nodes, ways and relations listed as "anonymous" belong to a user who agreed to the contributor terms. That's about 17.000 nodes that are safe.

I updated my statistic script to reflect this change.

Frederik also put an interactive map online that uses his odbl history service to trace back the history of objects. All data belonging to users who not agreed to the ct is highlighted on the map. We have the next three months to do a survey of these areas. Let's make the data even better than before.

http://tools.geofabrik.de/osmi/?view=wt … ne_created

I did a graph to show the development of the safe nodes in Thailand:
check-odbl-stats-20111214.png

Over 70 percent of the data is ready for the transition to the new license. A huge amount of nodes and ways might be ready but a simple analysis will not tell, so history needs to be evaluated for accurate numbers. We also have nearly 80 people who once edited in Thailand but never answered to the request to relicense data.

Do you know one of these? Please try to contact them. Might be great if they decide to accept the new contributor terms.

Stephan

Offline

#31 2012-03-07 11:26:23

stephankn
Moderator
Registered: 2010-05-04
Posts: 622

Re: ODbL in Thailand

To keep you updated:

As of the quick lookup based on data of this morning there are 34 users who did not respond and 2 who actively declined new CTs.
http://downloads.osm-tools.org/check-od … 120307.htm

I did run a detailed counting of the ODbL status and currently we will have a loss of less than two percent. Still some of these elements carry valuable tags.

Total nodes in DB: 2387515
- safe for ODbL: 2375273 (99.49%)
- tainted: 12242 (0.51%)

Total ways in DB: 176494
- safe for ODbL: 174192 (98.70%)
- tainted: 2302 (1.30%)

Total relations in DB: 1015
- safe for ODbL: 1008 (99.31%)
- tainted: 7 (0.69%)

Stephan

edit: inserted more detailed statistic, corrected wrong percentage calculation

Last edited by stephankn (2012-03-11 09:11:01)

Offline

#32 2012-03-11 09:18:39

stephankn
Moderator
Registered: 2010-05-04
Posts: 622

Re: ODbL in Thailand

We have three more weeks, still about 2000 ways left which will get lost.

Here is a diagram to show the countdown. I'll try to update it daily.

odbl-countdown.png

If you extrapolate you'll see we are not ready till the end of the month. So please increase your effort in remapping.
It would be a shame if we lose valuable tags that are edited on top of tainted ways just because we did not replace the way with a better clean one.

Certainly there are areas which do not provide aerial images and where the GPS tracks had not been uploaded. Here you could try to reach the mapper who did not yet respond. They all did receive a message from OSMF already. Could have happened the email is no longer active. Maybe you know them from another forum and contact them that way.

The statistics on http://downloads.osm-tools.org/check-odbl-th/ now only distinguish between clean and unclean, but take into account the full history of the elements.

Stephan

Offline

#33 2012-03-12 18:56:19

Bernhard Hiller
Member
Registered: 2011-05-10
Posts: 1,056

Re: ODbL in Thailand

I do not think that we will lose so many ways.
Look at the typical scenario: a dissenting user traced a few minor roads from Bing. He connected the minor roads to major roads, thus adding nodes to the major road. What will be the consequence? Those additional nodes of the major road will be lost, not the whole road. Correct me, if I am wrong.
Another example: someone created a road and added a name tag with English content. amai corrected that and changed "name" to "name:en". What will happen? The "name:en" tag will be lost. Since also the "deletion" of the original name tag is an edit by amai, also that deletion should be undone, and we will find the English road name again in the name tag instead of the name:en tag. Correct me, if I am wrong.
The only problem I see is with the roads created by dissenting users. When I use the OSM inspector and select "Ways created" only, the problem is not such big.

Offline

#34 2012-03-12 21:12:06

stephankn
Moderator
Registered: 2010-05-04
Posts: 622

Re: ODbL in Thailand

Bernhard Hiller wrote:

I do not think that we will lose so many ways.
Look at the typical scenario: a dissenting user traced a few minor roads from Bing. He connected the minor roads to major roads, thus adding nodes to the major road. What will be the consequence? Those additional nodes of the major road will be lost, not the whole road. Correct me, if I am wrong.

It depends on how clever the switch bot will be programmed. Internally a way has nodes and tags. Both in union form a version of a way. As they are not treated independent in respect to object version it all depends on the logic of the bot.
In the most simple case the last "clean" version would be used. So all subsequent edits on nodes and tags reverted.

The better approach would be to treat tag and node edits independent. So in your example only the extra nodes would be lost. I hope it will be implemented this way but it is still work in progress.

Bernhard Hiller wrote:

Another example: someone created a road and added a name tag with English content. amai corrected that and changed "name" to "name:en". What will happen? The "name:en" tag will be lost. Since also the "deletion" of the original name tag is an edit by amai, also that deletion should be undone, and we will find the English road name again in the name tag instead of the name:en tag. Correct me, if I am wrong.

In this easy case yes. But what happens to edits built on top of amai? A tag can be changed back and forth. And again it depends on how the algorithm is implemented. Personally I would favor to keep the latest clean version of a tag.

Even assuming that would be done, it still is causing trouble. We'll end up with lots of tags that need to be fixed. Currently we see the problem highlighted by the plugin in JOSM. Easy to spot and fix.
In the future it will be hidden in all the data.

So doing it right before the switch saves time later.

The rebuild working group is trying to figure out the implementation details.
http://lists.openstreetmap.org/pipermai … 00099.html

Bernhard Hiller wrote:

The only problem I see is with the roads created by dissenting users. When I use the OSM inspector and select "Ways created" only, the problem is not such big.

Yes. I'm personally working on these problematic ways for many hours already, others do as well. So the number of affected ways is down already.

I tend to be on the safe side and have a rather pessimistic view regarding the loss of data. So all we could do now to make the data ready for the switch is a plus. I wish your optimistic assumptions come true, but I certainly feel better if we do not have to rely on it.
Quite often it is also possible to really improve the data. I had remapped data from old yahoo images where meanwhile the layout of streets and buildings have changes. So Remapping is also quality improvement.

Stephan

Offline

#35 2012-03-21 08:04:17

stephankn
Moderator
Registered: 2010-05-04
Posts: 622

Re: ODbL in Thailand

The OSMF announced a schedule for the database cleanup.
http://www.osmfoundation.org/wiki/License/Rebuild_Plan


On the 27th the database will go into read-only mode until cleanup is done.

In Thailand we're quite lucky compared to other countries. Here measured today only 0.8% of the ways will be degraded.

Still the majority of tainted nodes/ways is related to the armchair mapping of amai. We have another five days to reduce the effects of the database cleanup. I won't get this easy again to spot the problems.
A good time to start the sprint...

Stephan

Offline

#36 2012-03-26 07:52:26

stephankn
Moderator
Registered: 2010-05-04
Posts: 622

Re: ODbL in Thailand

It's not yet clear whether it's the 27th or later, but expect the deadline for remapping approaching fast.

We have 2467 tainted nodes and 758 tainted ways left.

Having a closer look at the ways:
in north-east only harmless edits left.
On Samui will be more damage. As neither GPS tracks nor imagery is available no more remapping possible.
Similar on Samui.
Phuket will also lose a few ways.

Most severe damage still in Bangkok. It still has tagged ways that build on top of tainted ways.

The latest editions of the statistics provide a link to overpass to fetch tainted ways. Copy URL and paste in JOSM "Open location". You could also save the OSM file to hard disk and open in any other editor.

http://downloads.osm-tools.org/check-odbl-th/

Let's do the finish...

Stephan

Offline

Board footer

Powered by FluxBB