Test Drive - AI-assisted road import by Facebook

Thank you for your response, DrishT and colleagues. I still welcome your project and your efforts, and hope to be able to provide some hints leading to an improved quality of the data.

I’ll collect some more examples when I’ll work on my data from the holidays.

Meanwhile, you can take a look at
https://www.openstreetmap.org/#map=14/8.5383/99.0690

The 5 “circles” there look like small settlements in a plantation.
I came from North to the settlement in the north-east, via bad tracks, some of them are not yet on the map.
Some roads in that settlement are paved, some are not which can be well detected in the imagery (Digital Globe Premium Imagery).
The “roads” to the west and east should be tracks.
The road leading out of it towards the south was somewhen paved, but most of the asphalt cover has been lost (which cannot be detected in the imagery).
The road in the center of the area leading east towards the major road #4037 is a wide asphalt road (unclassified, or perhaps even tertiary; I did not see kilometer stones or DRR signs there). With the imagery, you can see that (assuming you come from the east) it turns north/right just before you connect it to a track, and at the next junction west/left.
Almost all other “roads” in that area should be tracks.
And it is easy to see that you missed some parts of the circular roads in the settlements.

It will take a few days till I’ll edit that region. My last updates were with the GPS data and waypoints from Dec 7, and I visited the region described here on Dec 23.

Hello Bernhard,

This is a great example and we are in complete agreement on this. Look forward to more examples :slight_smile:

Appreciate your feedback.

Thanks,
Drishtie

By the way, when I wrote the message above, I opened that region in a web browser and started iD. In order to see the roads in the imagery, I had to move some nodes of the ways, because the ways are shown with very thick lines in iD by default (don’t know if or how to change that).
Maybe that could be a problem for your team also? In JOSM, the lines are thinner and the imagery can easily be seen without moving any nodes away.

During the last days I mapped along the coast south of Hua Hin. A very touristic region, hence many mappers have been there already and contributed many data. Consequently, the imports did not contribute so much and the amount of problems was less, while north of Prachuap Khiri Khan.
Well, south of Phrachuap Khiri Khan it was … terrible. But not due the imports. A user did an enormous amount of work in 2013/14, and got many things wrong, apart from highway classification also waterways (generally layer=-1, i.e. below the surface of the earth), and also connected them to roads and railways.
When you learn from such data, you can but learn it the wrong way.

I’d like to add a couple of examples with import issues which you asked for:
way id(s) original classification → correct classification
514017526 unclassified → track
513728209 residential → track
514040248+514040250 joined
514037787+514037788 joined
514332683 service → track
514345749 residential → unclassified
513730178 too many nodes

Individual mappers have to be ultimately responsible for their own individual edits. It’s great that there’s also a direct contact within the team that’s directing these mappers, but one of the strengths of OSM is the ability of individual mappers to communicate among themselves.

W makes the lines thin in the ID editor. Would it be possible to collect Mapillary or OpenStreetCam data to help the remote mappers?

More careless work from the FB import team…

Not sure if a one-off, but good evidence of the imports destroying our core Highway data through either carelessness of lack of vetting. I have corrected it.

Actually, we can’t see from this picture what kind of road it is. We can only see it isn’t paved. I agree that residential roads should have houses alongside, at least from time to time (access to rural houses will not always have dense buildings along).

If the track is legally accessible, why would you suggest to interrupt the road network at this spot, is this about surface or ownership or access restrictions?

+1, although what is a track depends a lot on the local legislation and context. In rural settings, most roads, also “normal public roads” are typically serving agricultural and forestry purposes. The decision for track is if they only serve for these purposes.

So for today’s examples of the AI mess, recently I was in Ban Pong near Phayao. A couple of times my GPS showed me the main 1193 hwy seemes off from the true position. Back on the Computer, and with the benefit of multiple GPS traces and newer imagery, we can see that the Bing image it was traced from is “off”, but the newer DG-Std image is correctly aligned.
However, I also see mass of residential roads have been added by Micheal (VLD011) without checking for image alignment, so now in addition to the Hwy, now we have to correct all those.

Furthermore, in just that one screenshot above, working downwards, I see …

1/. The road below the fuel station is actually a track leading to fields. Thats a market above it, not a house.
2/. The school on the left has its access road draw as residential, and for some reason, does not continue in a loop back to the road. Local mappers would know this and draw it as a service road, with some even adding a permissive tag.
3/. Further down on the left, we see a residential loop going around the back of the village Health Centre… of course, this would also be a service road. The health centre is not, and does not look like a house, and besides, the access road passes across the front of the building too … guess the AI missed that one.
4/. And opposite the health centre, we have the District Office … also marked as residential and with only one section of the access road drawn… in reality its a loop, and of course should be service road.

So my point yet again is … if in one small area, we find so much bad mapping, just how much damage has been done to the whole of the Thailand Map. Corrections that local mappers will never have the time to rectify.

I will also send this post to the team osm@fb.com, but please, I have not got time to comment to individuals, or post in every changeset affected as some OSM purists might suggest.

I can spend an hour a day flagging these issues, or get on with the more worthwhile job of inputting data. I know the DWG sees these posts so please, how do we go about reversing all the “imported roads” so as we local mappers, at least have a chance to enter the data in correctly. Can someone help me write a script that deletes all roads marked as import=yes ?

Is the problem really “all import=yes” roads? Ultimately it’s a human that presses the button to add the data, and I’d expect that different people will have different quality thresholds. To take an example from elsewhere, in an African semi-import by several mappers it was clear that there was one whose quality threshold was lower than the others (and in that case adding lots of duplicates). After engaging with everyone involved there we ended up reverted only that one contributor’s additions and letting everyone else continue - it got rid of most of the problems and left most of the valid additions.

If you have not got time to comment on changesets, then the DWG (just like most OSM mappers all volunteers also) are unlikely to have be able to analyse the type and scale of the problems, and other mappers in Thailand who may not frequent this forum won’t know either. It’s great to see the examples posted here (and a link to the actual data would make those much more useful), but without specific details we simply don’t know who the main offenders are, and without any comment to reply to we don’t know their side of the story.

With regard to problem editors, what we’d like to be able to do is:

  1. See comments made about a particular mapper’s work

  2. See that mapper either not respond or do the same sort of thing again.

  3. If they continue creating the same problems despite being told about them we can then take action against that mapper.

With regard to problem data, the wiki page at https://wiki.openstreetmap.org/wiki/Change_rollback is a summary of the options available (perl revert scripts and JOSM). The two main problems with any approach are:

  1. Identifying the data to be reverted.

  2. Deciding what to do with data that has been edited by other mappers afterwards (undo all their changes, keep all their changes, or review after the revert)

The approach with the perl revert scripts is generally “throw all problem data at the script, and then subject to (4) above, handle any remaining problems”. With JOSM you’d go backwards through all changes and interactively handle (4) as you go. Depending on the individual situation either of the two main options (or even some other approach) may be better.

I have been following this thread but have not commented on it because my feelings about the FB team’s work are not as strong as Russ’s. Agreed, their work isn’t perfect, there are many errors and missed connections, but overall I think having the new highways in OSM is more important than whether the ways are service, residential or otherwise. I was bicycling on a perfectly beautiful highway earlier today that was tagged residential. It is a wide, two-lane road, with smooth asphalt paving that, to my mind at least, should be tagged tertiary or at the very least unclassified. There are a few homes situated on it but in actuality, it is a high speed connector deserving of a higher classification.

These sorts of errors happen all the time, especially when new mappers are coming up to speed. I look back occasionally and am surprised to see the tagging I applied to certain objects I created years ago; I would never tag them that way now! But to go back and retag all of my older work using my current understanding would be way too much work for little real improvement. Keep in mind that the tagging is only one part, although an important part, of an OSM object’s value. Just knowing something is there is important as well.

To revert all changesets having an import=yes tag would be overkill, IMHO. I’m not sure how best to address the problems Russ mentions. These are serious issues that only the particular volunteer mapper can properly address either by taking more time to read and educate him- or herself, or by asking questions about every object they add to the database. But didn’t we all begin that way?

Respectfully submitted,

Dave (AlaskaDave)

Wait a moment…
I asked about the quality of the revert tools in the German forum: https://forum.openstreetmap.org/viewtopic.php?id=60955
woodpeck said that “complex_revert.pl” would fail when another mapper worked on that item later on.
Consequently, I reasoned that I can now do many time consuming edits with the data collected during last December - and I don’t like to postpone that, as I can still remember the situation on ground well with the help of my GPS traces, waypoints, photos, and the imagery.
During my last edits, I could correct (or sometimes verify) edits of the FB team, and lots of my editing time was committed to check their data. I do not want that to be lost.
I hope other mappers active in Thailand will agree that such edited ways should not be reverted.

Are you sure it is? I do not know. Traffic law enforcement in Thailand is … ehm, even less effective than in Italy. Hardly ever will you see a road sign blocking access (or a handwritten “ham khao”).

Just look at the message of Russ:

That’s a user experience we should really try to avoid.
Of course, marking a legally accessible - and usable for common car drivers - road to a track could prevent people from considering it in their trip planning activities. That’s not good also. Bit I believe the impact is not so bad.

Thailand is a well developing mid-income country. Unpaved roads (below tertiary) are still common, but generally they’ have a tracktype=grade2 quality. Rare exceptions do exist in very underdeveloped areas. The area near Phetburi is well developed.

By the way, I found an example of a track which is really not accessible to the public, there is a barrier preventing entry:
http://www.openstreetmap.org/way/514881100/history
And looking at the history, you’ll see that it was tagged as a residential.

I certainly don’t want that to happen, Bernard. I’ve too have edited or enhanced many FB objects in the months since they started and my position is that any reverts, should we decide they’re necessary, need to be handled carefully.

As I understand from his messages, Russ did so too. And I saw some edits of mappers not active here in the forum. So I guess we agree that a “complete” revert (i.e. including objects changed by other mappers) should **not **be done.

That’s a different type of issues, and I think it is not the fault of the FB team. Remember the imperative of the past: “if you do not know the reference number of the road, you must not tag it higher than unclassified”.
A year ago, another user approached me regarding my work in the South: “There are some roads tagged ‘tertiary’ that don’t have any ref. May I change these to ‘unclassified’?” I could fortunately convince him not to do so.
When I compare a map of Thailand with a map of a well-mapped country, it is clear to me that we are missing many secondary and tertiary roads - they are hidden among our too many unclassified roads. I upgraded several roads during my last edits. We surely need more local people knowing the situation on the ground.

And sometimes more contributions from our rather passive members. E.g. we have many GPS traces for Phetkasem road (#4) west of Chumphon, i.e other mappers have been there. A large section there is already a dual carriage highway on the ground- a “trunk”, but not on our map. I upgraded a minor section of it yesterday, and another large section between Kraburi and Ranong.

As for the other tagging issues of the FB team (mainly the unpaved roads/tracks), I can imagine to change the style file for my map. I could add some overlay to all the roads having an “import=yes” tag, or use a different line type which makes them non-routable or some other “hacks” like making them toll roads (toll road avoidance can be switched on/off on Garmin), or something like that. That would be a workaround for me.
But what about other users? They’ll download a map from somewhere, or use some application, where such hacks are not used (and if they were used, those users would not understand them, because of lacking background knowledge). I do not know what’s best to do…

Indeed - that’s the stage at which the person who was doing the revert would then have to decide what to do next. If it’s obvious that local mappers have been correcting the data then obviously the best thing to do is to leave the corrected data in place.

Dave, Bernhard, Andy (DWG) … you all speak words of reason, and of course I know that mass reversals are also not good. Indeed, I have manually corrected so many areas I know where the AI has got it wrong, so I too, would be affected.
OK, so now I have got the recently discovered issues off my chest, and I think this situation is something we will will have to accept and deal with.
It seems from Drishtie’s comment that the FB team will change as they get feedback from us, but they have already imported masses of data and I can’t see them going back over old ground… it’s just not human nature.
Its a shame that the Wiki was intrepreted as “if you see a roof, then its residential”, rather than the more balanced approach we take here looking at the use of the road with all the adjacent evidence that supports this.
And I know we have had the discussion before … a 3km track through an orchard to a small dwelling ? Track, or residential? I just wish the FB team had have understood our local protocols before adding such huge amounts, rather than inviting change after the fact.
There is probably no practical way now for the FB team to change what they have done, and I accept that deleting all imports won’t be good, so I guess its just down to us local mappers to fix things, one road at a time … we have done it before, we can do it again … just never on this scale !
Russ.

Hello All,

Thank you for the feedback and discussion. As promised we have no issues going back and fixing roads based on your feedback. In fact we have been doing just that over the past few weeks. The following team members are going through each province we have done.

Alexandra - VLD003
Kurt - VLD007
Stefani - RVR006

Russ please note, the tagging is not done by AI but by a very well trained and experienced group of mappers. As remote mappers I hope you can understand that we cannot possibly get the tagging 100% correct but the time it takes to fix tagging is much less than it would take to re-draw all the roads the team has worked very hard to add.

In response to your comment “I just wish the FB team had have understood our local protocols before adding such huge amounts, rather than inviting change after the fact.” In addition to the months and months of research on Thailand, flying down and meeting the local mappers and continuously fixing all the requests from the community can you please advise on what other protocols we are not completing?

Our goal is to help local mappers make a better and more complete map. We have the same goal. So if there are corrections the local Thailand community can agree on and would like us to change, we will be happy to do so, including re-visiting data we have already added. You don’t have to do it alone. :slight_smile:

We truly want to help so if you can send us some guidance on how we can better tag roads looking at satellite imagery, we will be happy to follow. In summary it sounds like we should not be using residential road when we see buildings since we don’t know the use. How do you suggest we make that call looking at satellite imagery since we cannot tell the use of the road or what the building is on the ground. Would you prefer that we don’t add the roads with only a few buildings? This might mean we delete a lot of well drawn roads and break the road network in some areas though. Let us know your thoughts.

Best,
Drishtie

I agree. Long ago I started tracing the Andaman coast from Bing. A touristic region, many data were already present, but still that work contributed most of the mapped roads there. I did it to have some usable map when cycling in that area - trying to avoid busy major highways. And so did I use it.

And had to find out that some things look different on ground.
Fortunately, no big errors in that area. The track https://www.openstreetmap.org/way/105781766/history exists - but I missed it at first, because of its really bad state, which I could add then later on. Often I was surprised how much I could correctly trace from the imagery.

Overall, this use on ground was the most important feedback for me. And I do recommend that your team members do so also. That’s the best experience you can get.

It is often hardly possible to see the difference of a paved road and an unpaved road. Depending on its surface material, it may look like a concrete or asphalt road. With the GPS data and the notes, I can look at the imagery again and improve my knowledge.
Of course, there were some bad errors (in other places). E.g. near Kemer (Turkey), a dry stream looked like a road to me. What a surprise when I was there…

A bad thing is that our local community is hardly existing. Errors stay in the data for many years. Like https://www.openstreetmap.org/way/147858550/history - I am sure that never were these bridges two oneway roads, the new bridge replaced the older bridge. My tracing error in 2011. This means that the wrong tags of the latest import are likely to stay here virtually for ever. Worse, new mappers with less experience will look at the data and learn from it - and consequently produce such tagging problems themselves. And they might also learn from my wrong tagging of 2011 (I used “service” far too often).

I do not know how to handle it best.
I think having a good and simple method of distinguishing between edited/corrected/verified data on the one hand and the remaining imported data on the other hand could be a good step forward. That’s why I created the thread https://forum.openstreetmap.org/viewtopic.php?id=61183

I think that the impact of an error could be an important criterion for deciding what to do in case of doubt.

Drishtie,

Thanks for your comment … its a shame I didn’t get to meet you when in Thailand as maybe some issues could have been ironed out earlier.
What hurts me is that while I contributed thousands of hours to the Thai OSM, I also run motorcycle tours … last weekend we were heading south on the 1148, and an accident below the junction with the 1097, meant a quick re-route decision.

See https://osm.org/go/4W_XvjFk-

I looked at my Garmin, and it seemed we could go along the 1097, then drop back along the unclassified road back onto the 1148. Off we went and as we turned onto this “unclassified road” it became apparent that this was a track leading through fields and used only by 4wd tractors. So, yes, I was a bit embarrassed in front of the following riders !

Not surprisingly you guys had added it, but looking at the imagery it’s so clear that this is a muddy track, not a road. I have now corrected it to a track, so bear that in mind when looking at the image.

And it happened the week before here https://osm.org/go/4Wt9NXRg-. Heading east on the 1342, and wanting the go north on the 1217, the GPS routes me along the unclassified road your team added, that cuts the corner … which is of course, a track through fields. I have added a unpaved tag now to stop this happening again.

Just two examples I stumbled across … how many hundreds more are there ?

So, what I’m trying to say, is your AI is really hurting the users of the map over here. We can no longer rely on the map, because on the GPS, your edits are not differentiated from any others. While the odd residential road mis-tag can be inconvenient, the addition of these unclassified roads across the countryside, is a major problem. It’s so clear from the imagery, that the road is not covered in asphalt so why not mark it as a track, or at the very least, add an unpaved tag.

I suspect the whole AI test drive, as you call it, was done in a rush, and its produced masses of inaccurate data. I welcome your suggestions to review the data, but how long will this take ? Can you at least urgently look at every unclassified road your team added, and ask the simple question of the way … can I see asphalt and white lines ? If not, then change it to a track, or an unpaved road. And if you are unsure of which, make it a track and we can upgrade later if necessary, from local knowledge.

Russ :rage: