"AI-Based Country Scale Road Import"

Hi DrishT,

As you say, OSM is a community. We’d expect mappers working for Facebook to be open and honest about what they’re doing and to engage with the community, and being responsive when other mappers ask them about edits that they have made, just like any other member of the community. An individual mapper can see commented changesets in their history list, and third-party tools such as http://resultmaps.neis-one.org/osm-discussion-comments?uid=5314363 also exist to see what questions have been asked about what changesets.

https://wiki.openstreetmap.org/wiki/AI-Based_Country_Scale_Road_Import makes it clear that you are performing an import. The links from https://wiki.openstreetmap.org/wiki/Import including https://wiki.openstreetmap.org/wiki/Import/Guidelines describe what needs to be done. Since you are planning (at least according to the wiki page that you created) to import internationally, the best place for discussion about this import is the global imports list https://lists.openstreetmap.org/listinfo/imports . You’ll need to discuss with mappers there what you’re planning to do and how you’re planning to do it. Please do review the steps outlined at https://wiki.openstreetmap.org/wiki/Import/Guidelines and follow them, and note that uploading data is the 6th step of 6.

The Data Working Group works on behalf of both the community and the OSMF. Although we may be able to help with procedural points (as here) the primary communication is with the community both internationally (e.g. the “imports” list) and in national forums such as here. The DWG may also of course be involved with removing failed imports (as we did with e.g. some of last year’s Facebook imports in Egypt) and blocking importers who insist on importing without following the rules (which we’ve not had to do yet in this case). Ideally we’d like any imports to be performed according to the guidelines with community approval so that we don’t need to get involved at all, so we look forward both to your “proposal to import” on the imports list and to the replies from your mappers to the questions that they have been asked in changeset discussions.

Best Regards,

Andy Townsend, on behalf of OSM’s Data Working Group.

Hi Andy & DWG,

Thanks for the clarification. We appreciate the guidance, especially with the number of lists and guides all over for OSM. We will tackle the questions including yours :slight_smile: We will also make sure to follow the changes you mentioned and move forward with the e-mail to the import list and country discussion post.

Editors will also add their company affiliation to their profiles first thing Monday :slight_smile:

Thank-you again!

Best,
Drish

I obtained a license key and TMS Url from the Facebook group, installed it into JOSM and went exploring. I was disappointed to learn that while the members of the FB group have access to high quality satellite imagery from Digital Globe (DG), ordinary OSM users do not. The TMS imagery we are allowed to use is what’s been derived after their post-processing of that high quality DG data. See the sample (I obtained the link from their Import page in the Wiki) image below for a look at the layer we get to use:

https://tinyurl.com/jlnqt4t

What this means is that we must use our available satellite imagery, either Bing or Mapbox, and alter its transparency such that their derived layer, placed beneath the Bing or Mapbox layer, is visible through it. In other words, it might be a small help in some areas but mostly you will have to rely on the usual satellite imagery to do your work. There is no way the layer they provide us can help determine if a road is paved or unpaved, 1-lane or multi-lane or even if it is actually a road. Below is a screenshot I made of an area I was investigating as a test of the layer. You can see the thin white strokes don’t quite match the Mapbox imagery beneath it. Is that because their imagery is more recent or is it an error in their post-processing? There’s no way for us to determine that.

https://www.dropbox.com/s/ciy15te9vdn3t15/FB-TMS%20demo%20screenshot.JPG?dl=0

I’m very disappointed but not surprised. My quick reading of the terms of the EULA the other day suggested to me that ordinary folks like us were excluded from deriving information from Digital Globe’s data. That turned out to be true. I doubt the Facebook TMS layer will be of much help to us and furthermore, without access to the same imagery the FB group is using there won’t be any good way, short of a personal field trip, for us to verify or validate their work.

Dave

I just saw one of their changes.

https://www.openstreetmap.org/way/475128676#map=16/18.5432/99.4948&layers=C

The quality is absolutely low. And reading the comments and discussions they are not willing to work together with us, the community. They just follow their own - most likely commercial - interests.

Seriously. OSM Thailand is off better without their “help”.
Dear Facebook: please go away and play somewhere else.

Agreed. It’s merely a “track” in the middle of nowhere and connected to nothing. How they decided to map that way as a track instead of an unclassified and unpaved highway, is unknown. The changeset comment is not very helpful either. It’s the same old stuff again “#nsroadimport #thailand”. Useless.

How do we stop this train before it crashes and creates another big mess?

One answer is to make the DG data available to skilled OSM members. If we had access to their high-resolution data it could make a huge difference in our ability to add details to the area. But I’m afraid that won’t be possible. DigitalGlobe sold those rights to Facebook and they are not likely to give anybody free access. DG isn’t interested in making roads more accurate, only in making money to cover their investment. Can you blame them?

Oh well, after all that it’s just another day in the 'hood. Copyright is a double-edged sword. It protects the owner and impedes the spread of knowledge at the same time.

Hello Dave,

As we explained in e-mail, we know it’s not ideal, but we cannot share our imagery because of our agreement with DG. This said, we are going to share more about our process and quality checks via the import list.

Stephan,

Thank you for your feedback. We based our tagging on a few things;

  1. The commonly used tags in this area are highway=track. For Example https://www.openstreetmap.org/edit?editor=id&way=156615945#map=16/18.5368/99.4921
  2. This area looks like an agricultural area based on our recent high-resolution satellite imagery and Bing satellite imagery.
  3. This road will eventually connect to another road that is currently tagged highway=track https://www.openstreetmap.org/edit?editor=id&way=156618885#map=16/18.5399/99.4854
  4. We are also using the wiki to guide tagging. In this case http://wiki.openstreetmap.org/wiki/Tag:highway%3Dtrack

We stopped editing last week so gaps in roads are due to that using our tasking manager. Once we have been permitted by the DWG to resume, the holes will be filled.

Best,
Drish

All,

Mishari and I had some of the initial communication with Facebook about their import. We have raised our concerns and answered questions to the best of our knowledge. After what happened with their initial set of edits, they really do want to be extra careful not just with the data, but also with the community.

The way I see it, is that we have an extremely technically talented group at Facebook to our disposal, they have access to resources that we do not have (Like High-Res DigitalGlobe imagery). Instead of dismissing them as Stephan is doing, I’d much rather try to find ways to use those resources.

I don’t think Stephan is fair in saying “Go away and play somewhere else”, especially because he doesn’t seem to communicate his real reasons for saying so. If I understand Stephan correct, then even if Facebook get a 100% hit rate on the geometries and tags, he still wouldn’t want their data.

If I understand Stephan correctly, then he is afraid of the effect it will have on community growth. If all roads are correctly mapped, it gets less fun and more difficult for new mappers to get started. Just like we are seeing in the US with the Tiger import.

I can understand his concern, but we must also realized that one day the road network will be complete with or without the help from Facebook. Should we then all slow down a bit to avoid ever completing? I am personally adding data as fast as I can, I want OSM to be the best map out there and I think Facebook can help.

Stephan let me know if I have presented your opinions incorrectly.

I do not know what the correct way forward is, but I’d much rather work with Facebook on a solution than just dismissing them.

Hi All,

I want to second Johnny on this. The guys at FB have been quite nice and enthusiastic and I think we should discuss ways of incorporating the work done by FB’s team. I’m willing to say that what has transpired has been to a degree my fault as I didn’t have time to think through my feedback to them.

It doesn’t even have to be direct imports but some ideas I have are:

  • done in grids and validated by the community similar to hotosm
  • .osm files that can be validated and incorporated gradually.
  • generate an enhanced maplayer that can then be traced manually

We should consider this a good sign that OSM is so awesome now that FB wants to be involved. Now we just need to find out how we can work together.

Best regards
Mishari

Again a sign that you didn’t understand the fundamental principles of OSM as a community.

It’s not just a storage of free (as in beer) geodata.

It is a community gardening the data. This is one reason the data is of so high quality that even companies chose to use it as a data source.

You think there is a tiny group of “leaders” you have to convince then you have “permission” to do your things on OSM. This is not true.
Neither DWG, nor Johnny, Dave or Mishary have the power to grant you permissions of any sort.

OSM is a community. Your edits have to be considered conforming to our community standards.

Part of this process is transparency. You don’t silently develop something in your office and suddenly present your result. You have to include us in all your steps.

You plan to use “secret” data sources not available to others. This makes it harder to verify your edits. Handing out API keys to your edit interface is not a solution.

Yes, I might be hard by saying “go away”. But I believe this is reasonable as long as you chose no to cooperate in an open and transparent manner with the community. This doesn’t say I disagree with having roads mapped or support in completing OSM data. I disagree with the process you have implemented doing so.

Stephan

I have the feeling you understood the problem. It’s not about having data in or not. It’s about how it is done. It is a meta layer to the actual editing.
It is about finding a way on how to best integrate this with the community. To strengthen the community and grow it.

Referring to a previous comment by Johnny. Yes, even a 100% correct import would be something I see skeptical.

OSM had not yet enough time to grow to master the challenges of gardening the data on that completion level.

With the inflow of occasional mapper using maps.me we already see shortcoming in how the community can integrate this. We have to work on these challenges.

AI has the potential to be useful here. It could be a companion to newbies supporting them in getting started and avoiding common mistakes.

So yes: I see a potential here in multiple areas on how Facebook could support us here. But let me stress it again: It’s not about working in secrecy, it’s about openness and acceptance of community feedback. Just pretending to listen to the community and then continuing as before is not.

Another example that Facebook prefers to work in secret:
They gave a talk with a lot of buzzwords at SOTM us. But declined to have a video recording being made available so that the community can follow their ideas.

http://stateofthemap.us/2016/how-can-ai-help-us-make-maps/

@Facebook: Stop hiding. Open up your communication with the community and share the talk.

As others have stated, the Data Working Group doesn’t give you permission. The community will give you feedback on it. And you haven’t even taken the first step by emailing the imports mailing list!

No, of course not! But in that case there is still more things to map. Buildings (heights, names, postcodes), amenities, landuses, better and more accurate road geometries, road attributes (speed limits, surfaces, lanes, sidewalks, signage, stop signs, traffic lights, road widths, streetlights etc). OSM will never be finished!

It sounds like this Facebook team don’t want to contribute to OpenStreetMap. They appear to want a generic geo database web host. This project is open, we work in the open, we share things in the open, we all work together. If we all don’t have access to the digital globe imagery, then maybe it shouldn’t be used. If you don’t want to discuss your import in the open, and want just find the right high level executives to approve your import, then maybe you don’t really want to contribute to OpenStreetMap.

We shouldn’t throw out the open just because someone says they have nice imagery.

Please work in the open. Please share with us. Please share your imagery. Let’s all work on this project. Stop trying to hide yourself away, and refuse to share your toys. Things are much better if we work together.

As someone who has been active in India for the last 10 years, I can safely say that OSM is struggling here because very few see any point drawing millions of kms of roads across this vast country, most of which are already there on other maps. The majority of contributions of roads in India have come from European mappers, and maybe we should treat that as good as an import as well.

That said, having these roads on the map has acted as a seed for some basic data that makes the map be of use and allow the collection of more valuable data that can only come from local knowledge like streetnames and POIs. In India, we are already seeing that by the way of maps.me contributions from the mobile which is the primary way to access the internet, and would not be possible if there were no roads to begin with.

OSM will be a truly open and global project only when it can build sustainable communities on the ground maintaining the map as stephankn said. If we have access to high quality of open road geometries from FB, it seems like a huge opportunity to improve the quality of the map and boost the utility of OSM, which in turn brings in more editors. Its unreasonable to expect OSM communities in developing nations to form like they have in Europe over beers when basic needs like continuous electricity or safe drinking water is not met. Maybe different regions need different approaches, and this seems like a worthwhile effort if endorsed by the local mappers.

Yes!

Interesting commentary and feedback. I’m always angered when I run into copyright issues because they retard the growth of knowledge. Okay, that’s just the way the world works and I must accept that. I also see the vast amount of mapping that still needs to be done in my primary areas of interest, Thailand and Alaska. I am appreciative of any level of assistance with getting data into OSM because that is truly going to be the proof that the open source model works. When OSM can give Google real competition for offering the best map of the world, we’ll have proved that point. FB could be a valuable ally in helping us reach that goal. Like it or not, we depend on some pretty big outfits to be able to do what we do. Where would we be without Bing?

I have a question about licensing and EULAs. How it is that the FB group can extract information from the DG visual data if that information will be used in OSM, where there are virtually no limits to the further, subsequent uses of that data? Do FB and DG realize that once “their” derived data is in OSM, they have no control of it anymore? That someone may decide to create a map of wheelchair accessible bathrooms in Chiang Mai and turn around and sell that map to anybody with a few baht to spend?

I think we should take a step back, allow the group some latitude (pun intended), let them make a few mistakes, have them participate in this discussion, exchange ideas and advice, and see where it takes us. Only by discussing this important work here will we understand enough about it to feel good about giving the go ahead.

Hi all, I wanted to chime in both with a personal perspective, and one that represents the company I work for, DigitalGlobe.

On behalf of DigitalGlobe, I can confirm that Drishtie’s (Drish’s) statements are accurate and reflect the current agreement between Facebook and DigitalGlobe. Speaking for DigitalGlobe alone, we continue to work with both Microsoft and Mapbox to update satellite imagery available within their respective layers available in the OSM editors. In addition, we are looking at options to publish imagery for OSM editing similar to the imagery that is being used by Facebook.

WRT the copyright comment, I spoke about this at last year’s State of the Map US. In short, DigitalGlobe is a data licensing company. Our goal is to continue to develop, launch and operate really nice satellites so the world continues to benefit from detailed and accurate scientific observations from space. This includes the OpenStreetMap community who have been using DigitalGlobe data by way of Bing and Mapbox for many years. We just want to continue to do that for many more years. You can see my presentation here: https://www.youtube.com/watch?v=1dh60heskmo&index=40&list=PLqjPa29lMiE3eR-gK80irr3xdUiRbIMeg

Last, as an OSM contributor for 5 years, being involved with several State of the Maps both presenting and organizing, I welcome and encourage the work that Facebook is doing to support and improve OSM. We, as a community, must work together; and Facebook has clearly stated it’s willingness to work with us. Let’s continue to be respectful, and acknowledge the commitment Facebook has with OpenStreetMap.

Thanks, Kevin Bullock

Hello all,

Thank you again for all of this feedback. We joined this forum to work with you directly and support high-quality data with your expertise. We are always willing to share and answer any questions you have. My sole focus at Facebook is on OSM, so you will reach me and my team directly if you talk to us through these discussion posts or email us at osm@fb.com (please email if you need a quicker response). Our goal is your goal: to efficiently and accurately create maps powered by our community.

We look forward to continuing to share our derived data so the community can work together to create high-quality maps. We will be sharing sample data with the community next week via the import list. Following that, we’ll share the area we hope to start mapping in Thailand through this forum.

We are currently using the HOT Tasking Manager and iD tool and are working to share our editions/changes to these tools as well. Stay tuned.

Have a lovely weekend wherever you are!
Drish

Thanks to Kevin for clarifying the intentions and philosophy of DigitalGlobe. We are indebted to the people who produce the satellite imagery we use every day and without which we would be lost.

Drish, I wonder if the iD editor is the best one to use for this project. I am not all that familiar with it but the few times I have used it, I found it very limiting. The learning curve is steeper but I think using JOSM will save time and prevent problems in the end. It is very customizable. One can create and use “presets” for example, that make data entry more uniform. In an effort like yours, where many people are adding similar data, you don’t want some of them to tag an object one way while some others choose a different tagging scheme. Also, tagging an object with a preset requires only a couple of keystrokes no matter how many tags are involved.

It’s your project and your decision might need to weigh other important factors. Many of the most prolific contributors to OSM use Potlatch or iD so you are in fine company in that regard. Still, I feel that you might as well be using the most powerful and flexible OSM editor available and that is, unquestionably, JOSM.

Best Regards,

Dave

Another feature of JOSM is the inbuild validator which can detect common mistakes, like crossing ways, not connected way, bad tagging, etc. The Find and To-do plugin can help find and fix errors like this. In computer generated imports like this, you want to be able to automatically detect and fix problems to ensure a high quality of data.

The White Spots on the map of Thailand are still enormously big. Last December in Isaan, I was in places with nothing at all mapped within 10 km from my location. And let’s take a look at the contributors to the map of Thailand: most data were contributed by farangs.
We need some organisations helping us to complete some tasks. Roads, rivers, landuse can be seen on images. Place names, road numbers, shops, hotels, etc. can’t be detected there: we need people on site who will do that. And I hope they’ll more likely be willing to contribute when they see that the map is good (which it isn’t - also Google maps actually isn’t good as it does not differentiate between roads and tracks).
Although the examples shown by Stephan and Dave are very terrible, let’s try to integrate the FB team into the OSM community. With a slower approach, they could get feedback on small change sets, without causing too much disturbances with bad data. Also try to map connected items instead of singular segments. As I see from Dave’s comment, it could be helpful if the FB team tried to work in places where good and recent imagery is available also to us (instead of only to them).