Test Drive - AI-assisted road import by Facebook

Hello,

Thank you as always for your feedback. Happy New Year! I hope you’ve all had a lovely holiday season. Sorry for the delay in response we are just getting back into work mode

I’m sad to hear you are seeing issues. I hope you see we have been responsive to all requests and quickly make edits whenever anyone in the community asks. In some cases the requests often conflict each other so it not as easy to made these decisions. I understand your frustration on roads and routing especially having travelled through many rural areas in Africa, South America and Asia including Thailand! So let’s see how we can make this better. Our goals are the same of making OSM better and we are super grateful for your on the ground knowledge.

As far as quality and data integrity, while our roads are generated by AI our team edits each task 3 different times using the Tasking Manager to ensure various team members are editing consistently and at a high level of quality. We also use quality checks created at Facebook and open source ones such as JOSM, OSMCha, OSMOSE and Keep Right. Additionally our Quality Analysts continuously check live data.

There is quite a bit to digest from your e-mails so let me try to summarize so I can work with the team to make sure we can help solve this. The two main issues are.

1. Too Many nodes on straight roads - We are looking into this and will make changes accordingly.
2. Road Tagging - The tagging is all done by the editing team and not automated and we are cognizant of the limitations of remote mappers, so try to follow the best practices we’ve learned from the local community and OSM guidelines.
   1. Tracks - Earlier we were told by some community members not to use this tag, however some have found it useful. After studying local edits, we decided to keep a minimum amount of Tracks for roads that are mostly agricultural or forestry.
   2. Residential - In an effort to make sure we do not neglect homes, we chose to mark any roads which serve as an access to housing, without function of connecting settlements. Sometimes this mean we are making the decisions if we see 1-5 buildings on that road. I understand things change on the ground and maybe those building do no actually exist as we see in satellite imagery. In those cases we are super grateful you can re-tag the roads based on your knowledge. If you have a different suggestions to tackle these roads please do share and we will be happy to take that into account and make changes.
   3. Unclassified - As seen from most of the local mapping in Southern Thailand unclassified is used to link villages and hamlets. This is why we mark minor roads of a lower classification than tertiary, but are not used to access houses unclassified roads.
   4. Paved/Unpaved - As much as we would like to add this tag it is difficult to do so with high accuracy so as a remote mapper. As the local community if you have some group guidance on this it would be great for us to follow. We assume based on OSM guidelines that the use of the road is what determines the Highway tag and the quality is then set in this case as either paved or unpaved. Is this the same way you see roads?

The best part of OSM is the community working together to make the map better. If your rules for tagging are different from the way we are doing so please may you provide some examples (way IDs and how you would tag them differently) and so we can learn with you and make the map better especially with your local knowledge. I think reverting thousands of really good edits due to some tagging concerns would be a loss for OSM.

For holidays and weekends you can always also e-mail my team directly osm@fb.com as a faster way to reach us.

Best,
Drishtie

Actually (and this applies to all sides of the debate here) I’d suggest that changeset discussion comments are a better way forward than a private email address. That way everyone in the community can see what potential issues there are and what is being done to address them.

Thank you for your response, DrishT and colleagues. I still welcome your project and your efforts, and hope to be able to provide some hints leading to an improved quality of the data.

I’ll collect some more examples when I’ll work on my data from the holidays.

Meanwhile, you can take a look at
https://www.openstreetmap.org/#map=14/8.5383/99.0690

The 5 “circles” there look like small settlements in a plantation.
I came from North to the settlement in the north-east, via bad tracks, some of them are not yet on the map.
Some roads in that settlement are paved, some are not which can be well detected in the imagery (Digital Globe Premium Imagery).
The “roads” to the west and east should be tracks.
The road leading out of it towards the south was somewhen paved, but most of the asphalt cover has been lost (which cannot be detected in the imagery).
The road in the center of the area leading east towards the major road #4037 is a wide asphalt road (unclassified, or perhaps even tertiary; I did not see kilometer stones or DRR signs there). With the imagery, you can see that (assuming you come from the east) it turns north/right just before you connect it to a track, and at the next junction west/left.
Almost all other “roads” in that area should be tracks.
And it is easy to see that you missed some parts of the circular roads in the settlements.

It will take a few days till I’ll edit that region. My last updates were with the GPS data and waypoints from Dec 7, and I visited the region described here on Dec 23.

Hello Bernhard,

This is a great example and we are in complete agreement on this. Look forward to more examples

Appreciate your feedback.

Thanks,
Drishtie

By the way, when I wrote the message above, I opened that region in a web browser and started iD. In order to see the roads in the imagery, I had to move some nodes of the ways, because the ways are shown with very thick lines in iD by default (don’t know if or how to change that).
Maybe that could be a problem for your team also? In JOSM, the lines are thinner and the imagery can easily be seen without moving any nodes away.

During the last days I mapped along the coast south of Hua Hin. A very touristic region, hence many mappers have been there already and contributed many data. Consequently, the imports did not contribute so much and the amount of problems was less, while north of Prachuap Khiri Khan.
Well, south of Phrachuap Khiri Khan it was … terrible. But not due the imports. A user did an enormous amount of work in 2013/14, and got many things wrong, apart from highway classification also waterways (generally layer=-1, i.e. below the surface of the earth), and also connected them to roads and railways.
When you learn from such data, you can but learn it the wrong way.

I’d like to add a couple of examples with import issues which you asked for:
way id(s) original classification → correct classification
514017526 unclassified → track
513728209 residential → track
514040248+514040250 joined
514037787+514037788 joined
514332683 service → track
514345749 residential → unclassified
513730178 too many nodes

Individual mappers have to be ultimately responsible for their own individual edits. It’s great that there’s also a direct contact within the team that’s directing these mappers, but one of the strengths of OSM is the ability of individual mappers to communicate among themselves.

W makes the lines thin in the ID editor. Would it be possible to collect Mapillary or OpenStreetCam data to help the remote mappers?

More careless work from the FB import team…

Not sure if a one-off, but good evidence of the imports destroying our core Highway data through either carelessness of lack of vetting. I have corrected it.

Actually, we can’t see from this picture what kind of road it is. We can only see it isn’t paved. I agree that residential roads should have houses alongside, at least from time to time (access to rural houses will not always have dense buildings along).

If the track is legally accessible, why would you suggest to interrupt the road network at this spot, is this about surface or ownership or access restrictions?

+1, although what is a track depends a lot on the local legislation and context. In rural settings, most roads, also “normal public roads” are typically serving agricultural and forestry purposes. The decision for track is if they only serve for these purposes.

So for today’s examples of the AI mess, recently I was in Ban Pong near Phayao. A couple of times my GPS showed me the main 1193 hwy seemes off from the true position. Back on the Computer, and with the benefit of multiple GPS traces and newer imagery, we can see that the Bing image it was traced from is “off”, but the newer DG-Std image is correctly aligned.
However, I also see mass of residential roads have been added by Micheal (VLD011) without checking for image alignment, so now in addition to the Hwy, now we have to correct all those.

Furthermore, in just that one screenshot above, working downwards, I see …

1/. The road below the fuel station is actually a track leading to fields. Thats a market above it, not a house.
2/. The school on the left has its access road draw as residential, and for some reason, does not continue in a loop back to the road. Local mappers would know this and draw it as a service road, with some even adding a permissive tag.
3/. Further down on the left, we see a residential loop going around the back of the village Health Centre… of course, this would also be a service road. The health centre is not, and does not look like a house, and besides, the access road passes across the front of the building too … guess the AI missed that one.
4/. And opposite the health centre, we have the District Office … also marked as residential and with only one section of the access road drawn… in reality its a loop, and of course should be service road.

So my point yet again is … if in one small area, we find so much bad mapping, just how much damage has been done to the whole of the Thailand Map. Corrections that local mappers will never have the time to rectify.

I will also send this post to the team osm@fb.com, but please, I have not got time to comment to individuals, or post in every changeset affected as some OSM purists might suggest.

I can spend an hour a day flagging these issues, or get on with the more worthwhile job of inputting data. I know the DWG sees these posts so please, how do we go about reversing all the “imported roads” so as we local mappers, at least have a chance to enter the data in correctly. Can someone help me write a script that deletes all roads marked as import=yes ?

Is the problem really “all import=yes” roads? Ultimately it’s a human that presses the button to add the data, and I’d expect that different people will have different quality thresholds. To take an example from elsewhere, in an African semi-import by several mappers it was clear that there was one whose quality threshold was lower than the others (and in that case adding lots of duplicates). After engaging with everyone involved there we ended up reverted only that one contributor’s additions and letting everyone else continue - it got rid of most of the problems and left most of the valid additions.

If you have not got time to comment on changesets, then the DWG (just like most OSM mappers all volunteers also) are unlikely to have be able to analyse the type and scale of the problems, and other mappers in Thailand who may not frequent this forum won’t know either. It’s great to see the examples posted here (and a link to the actual data would make those much more useful), but without specific details we simply don’t know who the main offenders are, and without any comment to reply to we don’t know their side of the story.

With regard to problem editors, what we’d like to be able to do is:

1. See comments made about a particular mapper’s work

2. See that mapper either not respond or do the same sort of thing again.

3. If they continue creating the same problems despite being told about them we can then take action against that mapper.

With regard to problem data, the wiki page at https://wiki.openstreetmap.org/wiki/Change_rollback is a summary of the options available (perl revert scripts and JOSM). The two main problems with any approach are:

3. Identifying the data to be reverted.

4. Deciding what to do with data that has been edited by other mappers afterwards (undo all their changes, keep all their changes, or review after the revert)

The approach with the perl revert scripts is generally “throw all problem data at the script, and then subject to (4) above, handle any remaining problems”. With JOSM you’d go backwards through all changes and interactively handle (4) as you go. Depending on the individual situation either of the two main options (or even some other approach) may be better.

I have been following this thread but have not commented on it because my feelings about the FB team’s work are not as strong as Russ’s. Agreed, their work isn’t perfect, there are many errors and missed connections, but overall I think having the new highways in OSM is more important than whether the ways are service, residential or otherwise. I was bicycling on a perfectly beautiful highway earlier today that was tagged residential. It is a wide, two-lane road, with smooth asphalt paving that, to my mind at least, should be tagged tertiary or at the very least unclassified. There are a few homes situated on it but in actuality, it is a high speed connector deserving of a higher classification.

These sorts of errors happen all the time, especially when new mappers are coming up to speed. I look back occasionally and am surprised to see the tagging I applied to certain objects I created years ago; I would never tag them that way now! But to go back and retag all of my older work using my current understanding would be way too much work for little real improvement. Keep in mind that the tagging is only one part, although an important part, of an OSM object’s value. Just knowing something is there is important as well.

To revert all changesets having an import=yes tag would be overkill, IMHO. I’m not sure how best to address the problems Russ mentions. These are serious issues that only the particular volunteer mapper can properly address either by taking more time to read and educate him- or herself, or by asking questions about every object they add to the database. But didn’t we all begin that way?

Respectfully submitted,

Dave (AlaskaDave)

Wait a moment…
I asked about the quality of the revert tools in the German forum: https://forum.openstreetmap.org/viewtopic.php?id=60955
woodpeck said that “complex_revert.pl” would fail when another mapper worked on that item later on.
Consequently, I reasoned that I can now do many time consuming edits with the data collected during last December - and I don’t like to postpone that, as I can still remember the situation on ground well with the help of my GPS traces, waypoints, photos, and the imagery.
During my last edits, I could correct (or sometimes verify) edits of the FB team, and lots of my editing time was committed to check their data. I do not want that to be lost.
I hope other mappers active in Thailand will agree that such edited ways should not be reverted.

Are you sure it is? I do not know. Traffic law enforcement in Thailand is … ehm, even less effective than in Italy. Hardly ever will you see a road sign blocking access (or a handwritten “ham khao”).

Just look at the message of Russ:

That’s a user experience we should really try to avoid.
Of course, marking a legally accessible - and usable for common car drivers - road to a track could prevent people from considering it in their trip planning activities. That’s not good also. Bit I believe the impact is not so bad.

Thailand is a well developing mid-income country. Unpaved roads (below tertiary) are still common, but generally they’ have a tracktype=grade2 quality. Rare exceptions do exist in very underdeveloped areas. The area near Phetburi is well developed.

By the way, I found an example of a track which is really not accessible to the public, there is a barrier preventing entry:
http://www.openstreetmap.org/way/514881100/history
And looking at the history, you’ll see that it was tagged as a residential.

I certainly don’t want that to happen, Bernard. I’ve too have edited or enhanced many FB objects in the months since they started and my position is that any reverts, should we decide they’re necessary, need to be handled carefully.

As I understand from his messages, Russ did so too. And I saw some edits of mappers not active here in the forum. So I guess we agree that a “complete” revert (i.e. including objects changed by other mappers) should **not **be done.

That’s a different type of issues, and I think it is not the fault of the FB team. Remember the imperative of the past: “if you do not know the reference number of the road, you must not tag it higher than unclassified”.
A year ago, another user approached me regarding my work in the South: “There are some roads tagged ‘tertiary’ that don’t have any ref. May I change these to ‘unclassified’?” I could fortunately convince him not to do so.
When I compare a map of Thailand with a map of a well-mapped country, it is clear to me that we are missing many secondary and tertiary roads - they are hidden among our too many unclassified roads. I upgraded several roads during my last edits. We surely need more local people knowing the situation on the ground.

And sometimes more contributions from our rather passive members. E.g. we have many GPS traces for Phetkasem road (#4) west of Chumphon, i.e other mappers have been there. A large section there is already a dual carriage highway on the ground- a “trunk”, but not on our map. I upgraded a minor section of it yesterday, and another large section between Kraburi and Ranong.

As for the other tagging issues of the FB team (mainly the unpaved roads/tracks), I can imagine to change the style file for my map. I could add some overlay to all the roads having an “import=yes” tag, or use a different line type which makes them non-routable or some other “hacks” like making them toll roads (toll road avoidance can be switched on/off on Garmin), or something like that. That would be a workaround for me.
But what about other users? They’ll download a map from somewhere, or use some application, where such hacks are not used (and if they were used, those users would not understand them, because of lacking background knowledge). I do not know what’s best to do…

Indeed - that’s the stage at which the person who was doing the revert would then have to decide what to do next. If it’s obvious that local mappers have been correcting the data then obviously the best thing to do is to leave the corrected data in place.

Dave, Bernhard, Andy (DWG) … you all speak words of reason, and of course I know that mass reversals are also not good. Indeed, I have manually corrected so many areas I know where the AI has got it wrong, so I too, would be affected.
OK, so now I have got the recently discovered issues off my chest, and I think this situation is something we will will have to accept and deal with.
It seems from Drishtie’s comment that the FB team will change as they get feedback from us, but they have already imported masses of data and I can’t see them going back over old ground… it’s just not human nature.
Its a shame that the Wiki was intrepreted as “if you see a roof, then its residential”, rather than the more balanced approach we take here looking at the use of the road with all the adjacent evidence that supports this.
And I know we have had the discussion before … a 3km track through an orchard to a small dwelling ? Track, or residential? I just wish the FB team had have understood our local protocols before adding such huge amounts, rather than inviting change after the fact.
There is probably no practical way now for the FB team to change what they have done, and I accept that deleting all imports won’t be good, so I guess its just down to us local mappers to fix things, one road at a time … we have done it before, we can do it again … just never on this scale !
Russ.

Hello All,

Thank you for the feedback and discussion. As promised we have no issues going back and fixing roads based on your feedback. In fact we have been doing just that over the past few weeks. The following team members are going through each province we have done.

Alexandra - VLD003
Kurt - VLD007
Stefani - RVR006

Russ please note, the tagging is not done by AI but by a very well trained and experienced group of mappers. As remote mappers I hope you can understand that we cannot possibly get the tagging 100% correct but the time it takes to fix tagging is much less than it would take to re-draw all the roads the team has worked very hard to add.

In response to your comment “I just wish the FB team had have understood our local protocols before adding such huge amounts, rather than inviting change after the fact.” In addition to the months and months of research on Thailand, flying down and meeting the local mappers and continuously fixing all the requests from the community can you please advise on what other protocols we are not completing?

Our goal is to help local mappers make a better and more complete map. We have the same goal. So if there are corrections the local Thailand community can agree on and would like us to change, we will be happy to do so, including re-visiting data we have already added. You don’t have to do it alone.

We truly want to help so if you can send us some guidance on how we can better tag roads looking at satellite imagery, we will be happy to follow. In summary it sounds like we should not be using residential road when we see buildings since we don’t know the use. How do you suggest we make that call looking at satellite imagery since we cannot tell the use of the road or what the building is on the ground. Would you prefer that we don’t add the roads with only a few buildings? This might mean we delete a lot of well drawn roads and break the road network in some areas though. Let us know your thoughts.

Best,
Drishtie