Test Drive - AI-assisted road import by Facebook

Wait a moment…
I asked about the quality of the revert tools in the German forum: https://forum.openstreetmap.org/viewtopic.php?id=60955
woodpeck said that “complex_revert.pl” would fail when another mapper worked on that item later on.
Consequently, I reasoned that I can now do many time consuming edits with the data collected during last December - and I don’t like to postpone that, as I can still remember the situation on ground well with the help of my GPS traces, waypoints, photos, and the imagery.
During my last edits, I could correct (or sometimes verify) edits of the FB team, and lots of my editing time was committed to check their data. I do not want that to be lost.
I hope other mappers active in Thailand will agree that such edited ways should not be reverted.

Are you sure it is? I do not know. Traffic law enforcement in Thailand is … ehm, even less effective than in Italy. Hardly ever will you see a road sign blocking access (or a handwritten “ham khao”).

Just look at the message of Russ:

That’s a user experience we should really try to avoid.
Of course, marking a legally accessible - and usable for common car drivers - road to a track could prevent people from considering it in their trip planning activities. That’s not good also. Bit I believe the impact is not so bad.

Thailand is a well developing mid-income country. Unpaved roads (below tertiary) are still common, but generally they’ have a tracktype=grade2 quality. Rare exceptions do exist in very underdeveloped areas. The area near Phetburi is well developed.

By the way, I found an example of a track which is really not accessible to the public, there is a barrier preventing entry:
http://www.openstreetmap.org/way/514881100/history
And looking at the history, you’ll see that it was tagged as a residential.

I certainly don’t want that to happen, Bernard. I’ve too have edited or enhanced many FB objects in the months since they started and my position is that any reverts, should we decide they’re necessary, need to be handled carefully.

As I understand from his messages, Russ did so too. And I saw some edits of mappers not active here in the forum. So I guess we agree that a “complete” revert (i.e. including objects changed by other mappers) should **not **be done.

That’s a different type of issues, and I think it is not the fault of the FB team. Remember the imperative of the past: “if you do not know the reference number of the road, you must not tag it higher than unclassified”.
A year ago, another user approached me regarding my work in the South: “There are some roads tagged ‘tertiary’ that don’t have any ref. May I change these to ‘unclassified’?” I could fortunately convince him not to do so.
When I compare a map of Thailand with a map of a well-mapped country, it is clear to me that we are missing many secondary and tertiary roads - they are hidden among our too many unclassified roads. I upgraded several roads during my last edits. We surely need more local people knowing the situation on the ground.

And sometimes more contributions from our rather passive members. E.g. we have many GPS traces for Phetkasem road (#4) west of Chumphon, i.e other mappers have been there. A large section there is already a dual carriage highway on the ground- a “trunk”, but not on our map. I upgraded a minor section of it yesterday, and another large section between Kraburi and Ranong.

As for the other tagging issues of the FB team (mainly the unpaved roads/tracks), I can imagine to change the style file for my map. I could add some overlay to all the roads having an “import=yes” tag, or use a different line type which makes them non-routable or some other “hacks” like making them toll roads (toll road avoidance can be switched on/off on Garmin), or something like that. That would be a workaround for me.
But what about other users? They’ll download a map from somewhere, or use some application, where such hacks are not used (and if they were used, those users would not understand them, because of lacking background knowledge). I do not know what’s best to do…

Indeed - that’s the stage at which the person who was doing the revert would then have to decide what to do next. If it’s obvious that local mappers have been correcting the data then obviously the best thing to do is to leave the corrected data in place.

Dave, Bernhard, Andy (DWG) … you all speak words of reason, and of course I know that mass reversals are also not good. Indeed, I have manually corrected so many areas I know where the AI has got it wrong, so I too, would be affected.
OK, so now I have got the recently discovered issues off my chest, and I think this situation is something we will will have to accept and deal with.
It seems from Drishtie’s comment that the FB team will change as they get feedback from us, but they have already imported masses of data and I can’t see them going back over old ground… it’s just not human nature.
Its a shame that the Wiki was intrepreted as “if you see a roof, then its residential”, rather than the more balanced approach we take here looking at the use of the road with all the adjacent evidence that supports this.
And I know we have had the discussion before … a 3km track through an orchard to a small dwelling ? Track, or residential? I just wish the FB team had have understood our local protocols before adding such huge amounts, rather than inviting change after the fact.
There is probably no practical way now for the FB team to change what they have done, and I accept that deleting all imports won’t be good, so I guess its just down to us local mappers to fix things, one road at a time … we have done it before, we can do it again … just never on this scale !
Russ.

Hello All,

Thank you for the feedback and discussion. As promised we have no issues going back and fixing roads based on your feedback. In fact we have been doing just that over the past few weeks. The following team members are going through each province we have done.

Alexandra - VLD003
Kurt - VLD007
Stefani - RVR006

Russ please note, the tagging is not done by AI but by a very well trained and experienced group of mappers. As remote mappers I hope you can understand that we cannot possibly get the tagging 100% correct but the time it takes to fix tagging is much less than it would take to re-draw all the roads the team has worked very hard to add.

In response to your comment “I just wish the FB team had have understood our local protocols before adding such huge amounts, rather than inviting change after the fact.” In addition to the months and months of research on Thailand, flying down and meeting the local mappers and continuously fixing all the requests from the community can you please advise on what other protocols we are not completing?

Our goal is to help local mappers make a better and more complete map. We have the same goal. So if there are corrections the local Thailand community can agree on and would like us to change, we will be happy to do so, including re-visiting data we have already added. You don’t have to do it alone. :slight_smile:

We truly want to help so if you can send us some guidance on how we can better tag roads looking at satellite imagery, we will be happy to follow. In summary it sounds like we should not be using residential road when we see buildings since we don’t know the use. How do you suggest we make that call looking at satellite imagery since we cannot tell the use of the road or what the building is on the ground. Would you prefer that we don’t add the roads with only a few buildings? This might mean we delete a lot of well drawn roads and break the road network in some areas though. Let us know your thoughts.

Best,
Drishtie

I agree. Long ago I started tracing the Andaman coast from Bing. A touristic region, many data were already present, but still that work contributed most of the mapped roads there. I did it to have some usable map when cycling in that area - trying to avoid busy major highways. And so did I use it.

And had to find out that some things look different on ground.
Fortunately, no big errors in that area. The track https://www.openstreetmap.org/way/105781766/history exists - but I missed it at first, because of its really bad state, which I could add then later on. Often I was surprised how much I could correctly trace from the imagery.

Overall, this use on ground was the most important feedback for me. And I do recommend that your team members do so also. That’s the best experience you can get.

It is often hardly possible to see the difference of a paved road and an unpaved road. Depending on its surface material, it may look like a concrete or asphalt road. With the GPS data and the notes, I can look at the imagery again and improve my knowledge.
Of course, there were some bad errors (in other places). E.g. near Kemer (Turkey), a dry stream looked like a road to me. What a surprise when I was there…

A bad thing is that our local community is hardly existing. Errors stay in the data for many years. Like https://www.openstreetmap.org/way/147858550/history - I am sure that never were these bridges two oneway roads, the new bridge replaced the older bridge. My tracing error in 2011. This means that the wrong tags of the latest import are likely to stay here virtually for ever. Worse, new mappers with less experience will look at the data and learn from it - and consequently produce such tagging problems themselves. And they might also learn from my wrong tagging of 2011 (I used “service” far too often).

I do not know how to handle it best.
I think having a good and simple method of distinguishing between edited/corrected/verified data on the one hand and the remaining imported data on the other hand could be a good step forward. That’s why I created the thread https://forum.openstreetmap.org/viewtopic.php?id=61183

I think that the impact of an error could be an important criterion for deciding what to do in case of doubt.

Drishtie,

Thanks for your comment … its a shame I didn’t get to meet you when in Thailand as maybe some issues could have been ironed out earlier.
What hurts me is that while I contributed thousands of hours to the Thai OSM, I also run motorcycle tours … last weekend we were heading south on the 1148, and an accident below the junction with the 1097, meant a quick re-route decision.

See https://osm.org/go/4W_XvjFk-

I looked at my Garmin, and it seemed we could go along the 1097, then drop back along the unclassified road back onto the 1148. Off we went and as we turned onto this “unclassified road” it became apparent that this was a track leading through fields and used only by 4wd tractors. So, yes, I was a bit embarrassed in front of the following riders !

Not surprisingly you guys had added it, but looking at the imagery it’s so clear that this is a muddy track, not a road. I have now corrected it to a track, so bear that in mind when looking at the image.

And it happened the week before here https://osm.org/go/4Wt9NXRg-. Heading east on the 1342, and wanting the go north on the 1217, the GPS routes me along the unclassified road your team added, that cuts the corner … which is of course, a track through fields. I have added a unpaved tag now to stop this happening again.

Just two examples I stumbled across … how many hundreds more are there ?

So, what I’m trying to say, is your AI is really hurting the users of the map over here. We can no longer rely on the map, because on the GPS, your edits are not differentiated from any others. While the odd residential road mis-tag can be inconvenient, the addition of these unclassified roads across the countryside, is a major problem. It’s so clear from the imagery, that the road is not covered in asphalt so why not mark it as a track, or at the very least, add an unpaved tag.

I suspect the whole AI test drive, as you call it, was done in a rush, and its produced masses of inaccurate data. I welcome your suggestions to review the data, but how long will this take ? Can you at least urgently look at every unclassified road your team added, and ask the simple question of the way … can I see asphalt and white lines ? If not, then change it to a track, or an unpaved road. And if you are unsure of which, make it a track and we can upgrade later if necessary, from local knowledge.

Russ :rage:

Hello,

Bernhard, thank you for starting the discussion post to chat with the community about good steps forward. Please keep us posted if there is anything we can do.

Russ, as mentioned, the team is going back to all tasks and looking at the tagging based on your on the ground feedback, especially unclassified roads. We are adding surface tag “unpaved” where we can identify they are unpaved.

Thanks,
Drishtie

Don Pin Soi 9 …
A small soi in Chiang Mai, which just happens to be next to one of my customers - Ducati.

https://osm.org/go/4WxmvWMs8?m=

Since 2010, local mappers have known that this is a dead end, and rightly kept a gap in the road … there is a solid steel fence which stops cars driving into the ditch there, but thats beside the point.

Along comes the Facebook team and spots a tree in this gap, as far as I can tell, you must have thought, so what …“surely the road must connect” … ah, we can make a guess, we don’t live there, who cares. Lets join it up ! Way hay, onto the next road.

Well dear FB mappers, Ducati owners are going to get a tad pissed off being routed down there now and having to do a U-turn when they get to the fence … So, dear Sam RVR009 (with the cute little elephant Avatar), please tell me how you reached the decision this road needed joining up ?
If the fence has been removed in the last week, and you have less than seven day old imagery, then I do apologise, but I think not.
You must be aware how the Thai mappers are fed up with your poor quality work, but this was added just a few days ago … clearly our comments are not getting through.

And yet again, Im forced to ask that painful question, just how much untold damage have you done, and how possibly, can it ever be reviewed ? Every day we stumble on your incorrect work, and this is what worries me … our DWG feel I should comment simply on every incorrect issue via a changeset comment. I wud be typing for ever.

You say "Feel free to reach out to me with questions or e-mail the team osm@fb.com. " … I did a few weeks ago, and never got a reply. In fairness, Dristie has asked for a video call with me, which I’m happy to do but I’m afraid it will take a lot to convince me you can undo the bad. I know you can’t get tagging 100% correct, but wish you would work on that cavalier attitude to the quality of the data and the manner in which you integrate it with existing data.

I’ll leave Soi 9 as it is for a few days, then correct it, just so as you can review & discuss your poor judgement calls… and then, why not have a wild guess at how many other mistakes we will find ? Hundreds, thousands ?

It’s high time the OSM DWG took your mass AI inputs seriously, and decide whether they can allow them to continue, before you move onto other Countries.

Indeed, and we (DWG) do understand how much of a PITA it is to have to indicate each individual problem in that way.

The problem is that something like https://www.openstreetmap.org/way/556830183 is actually very difficult to get right from imagery. I’m guessing that 50% of experienced mappers used to mapping from imagery would get it wrong, since the fence is only really noticeable on one of the three imagery sources there.

If it’s not obvious at first mapping, it’s going to be equally difficult to pick up at (remote) review - unless of course you’ve actually been there and know that there really is no road through.

A few posts earlier at https://forum.openstreetmap.org/viewtopic.php?pid=681717#p681717 Bernhard Hiller was arguing against a complete revert (and to my eyes the new mapping is not as bad as most other stuff that has been completely reverted in the past). The question then is how to go forward - and part of that is knowing exactly what data is problematical and what is not.

Actually - one more thing:

Has anyone tried to do anything with the “import=yes” tag that’s been added to e.g. https://www.openstreetmap.org/way/556830183 etc.? I don’t know whether people in Thailand mostly use web-based (OSM Carto or similar) maps, or Garmins, or Osmand or something else, but in all cases it’s possible to change the map to some extent based on tags in the data. It would be possible to create for example an OSM Carto-style map that looked as normal but rendered “import=yes” ways in a different colour, or a different width, or as a lower class of road.

Let me know if there’s any help that anyone needs doing any of this.

Hello Russ,

I did receive your e-mail where you sent me a link to this post inviting me to chat here which I am thankful for. My idea for the call was to get a chance to talk to you in person and have you meet the team of editors that are making these judgements call, and maybe share some of your valuable on the ground feedback and insights with them.

Thanks,
Drishtie

I suggested that previously, and will do so when creating the map for the next holidays in Thailand (don’t know when that will be, likely next December). Some other afficionados among us are capable of doing so, too. But I am sure most map users just use some common downloads or apps which are created in a way best fitting the use in Western countries - e.g. ignoring “unpaved” tags etc. And those maps won’t show the “import” tag either.

For finding such ways, open https://overpass-turbo.eu/
select a (small) area of interest, and run

way
  [import=yes]
  ({{bbox}});
(._;>;);
out;

I haven’t had as many bad experiences with the FB team as Russ. There are errors, sure, but they are still fairly new to this game. Sometimes I come across things I mapped 4 or 5 years ago and am confused or even embarrassed by the tags I chose at the time. We evolve into better mappers over time. There’s no shortcut to gaining experience.

As soon as I read the story about the Ducati owners having to make a U-turn at the fence, I asked myself why, if that fence was known to be there, it wasn’t on the map as barrier=fence, or the end of Don pin soi 9 marked with a noexit=yes tag? With the absence of that critical information, I might well have completed that connection myself based on the aerial imagery. Mapping errors are everywhere if you look closely enough. Even the English name of the soi is not tagged properly; it should more properly be name:en=Don pin soi 9.

As for the import=yes tag, since some earlier posts on this thread pointed out what others are doing, I started removing it if the way looks okay or if I’ve edited or realigned it. If I have edited it because of an actual ground check, I add a date to the source tag, e.g., 2018-02-09 survey.

Cheers,

Dave

PS: What is the reason the FB group has decided to work on OSM I wonder? Are the FB mappers being paid? There has been considerable push-back on their work, understandably, but what drives their efforts?

Hello Dave,

Thank you for your feedback. At last years SOTM conferences we shared our reason for working with OSM. You can watch the talk here, https://youtu.be/a8ckY6QN5jI.

And yes our mappers are paid just like many companies and organization using OSM. We use OSM in our products, and you can see OSM for some countries across the Facebook products in things like check in and places/pages. Since we use it, we would also like to contribute and fill in gaps where data is lacking.

Hope that helps.

Thanks,
Drishtie

That’s a decent summary/overview of the project, and that was SOTM-US, not SOTM. America != the world. And unlike last time, Facebook are allowing the video to be released. Anyone who’s read this thread won’t learn anything new. There’s some nice graphics and videos. However there’s a generic, marketing, non-reason for why you’re doing it (it’s Facebook’s motto). (The real reason any corporation does anything is for shareholder value)

It’s almost a year since you started, and you haven’t released the iD fork. “Move fast” indeed.

Around 16;47 in the video you talk about crowd sourcing street names, (from I presume OSM data?). That’s interesting, do you have any more on that? Have you released the data behind that? It sounds like it could help OSM.

(Interesting question: In the talk you show a screenshot of name matching (of Kamphaeng Phet 2 Road). Has Facebook just Publicly Usedf a Produced Work of a Derived Database? If that image is based on OSM data, doesn’t Facebook now have to release the other database behind it? ?)

That could lead to a major improvement of the map: when you show the map, also inform the facebook user that they can improve the map! They can become active mappers, or report some information via OSM Notes.
And I do not think so much of the bare road network. Our map is so “anonymous” or name-less. What’s the name of that village there? And what’s the number of that road? Here, a school: does it have a name? If they know it, they can add it to the map, in Thai, in English, or both (does not matter if they don’t know it in all languages).
And where’s the next village shop, restaurant, guest house, etc? There’s so much information local people can provide which we cannot trace from the imagery.
When will you do that step?

Hello Bernhard,

We completely agree and would love to work on these. We are starting to crowdsource road names, but have yet to figure out a good way to import them. You are correct that there’s so much information local people can provide which we cannot trace from the imagery and we hope to start out with geometry making it easier for local community to other details. Happy to hear your ideas on this.

Thanks,
Drishtie