New survey tool: OSMAGENT

I have been working on an Android app more or less the past year and I am getting close to a first release. So, let me advocate the project here. I am happy for feedback and to answer any questions.

*Osmagent *is a tool for surveyors that doesn’t require any knowledge about the tagging schemes in OpenStreetMap and also doesn’t require the surveyor to return home after the survey, launch JOSM and spend an equal amount of time inputting the data into the database:

The app searches for incomplete or extendable data in the user’s vicinity and presents the user with a map of “quests” (like i.e. in Mapdust) that each are solvable by answering a simple question while surveying it. Basically, take the concept of Mapdust and Maproulette, add the ability to solve them on-site and you got osmagent. :smiley:
The data input by the user is then **directly **added to the OSM database, making it available for others immediately and also eliminating the need for editing tools to insert the info later when back from the survey.

Some examples of what the app might ask:

  • What is the name of this road / shop / place / bus stop / etc. ?

  • What are the opening hours of this shop / place / etc.?

  • How many lanes does this road have?

  • What is the surface / smoothness of this road? (User selects which photo fits the best)

  • Does this road have a sidewalk?

  • How many storeys does this building have?

  • etc. … many more

The Idea

The idea for the project came to me when I realized how inefficient and time-consuming a GPS survey is:

  • regardless of which mapping technique I used, it always boiled down having to spend several hours to “process” the GPS trace in front of the PC

  • During a survey, I had no clue how well-surveyed the area already was because many details like road surfaces, building heights, etc. are not shown on the default (offline) map I use during a GPS survey. Also, house numbers might generally be defined, but missing for only one street or a few houses. So, often I noted down information that was already there. Also recording any possible information that might be missing takes a lot of time and a survey starts to feel tedious. Wouldn’t it be nice if someone already collected everything that is missing for me?

  • I like to contribute to the OSM, but I realized that many others also like to contribute while only few have the leisure to acquaint themselves with how to survey and even less are able to program tools and apps that help. So I can better contribute to the OSM in helping others to contribute.

The Name

I had the idea for the name when I overheard someone saying something along the lines of that “google has their agents [who survey these things]” when I was impressed on seeing the Google indoor maps of certain shopping centres. Well, with this app, anyone can be an agent for OpenStreetMap!
I realize that “agent” might carry a negative undertone for some people, so I am not set on this name yet. I thought of osmscout, but this name is already taken. Another name that recently crossed my mind is *mapcomplete *(leaned towards “achievement complete”). Let me know what you think.

Features

  • decentralized: The app only communicates with OSM and Overpass, no third server involved. Consequently, you do not need any other account than your OSM account.

  • data economic: The app downloads sparse data from Overpass and only that which it needs to display the quests

  • offline-usage: Sync-only-on-Wifi option. You can solve the quests offline, when you return to some online-place, the OSM changes are automatically uploaded, conflicts are resolved automatically. You can download the quests for an area beforehand.

  • oauth: The answers you give are created as atomic commits with reasonable comments and source tags in your name

  • Mu: For each quest, you have the possibility to leave a public note instead of answering the question, i.e. if app asks for the opening hours of the shop but in reality the shop does not exist anymore. A note at a location blocks any other quests, so other users are not bothered about an unanswerable question at that position as long as it is not solved. Alternatively, they can just hide the quest for themselves.

  • Answering notes: The other way round, users may also answer on other people’s notes

  • 3D Map rendering: I am using Tangram-ES to render the map, which is capable of rendering the buildings in 3D and also apply all kinds of shaders to the map. Other than that the map may look cool, it also means that it would be possible to directly display the answer the user gave on the map (i.e. the building height) :sunglasses:

State of development

See here: https://github.com/westnordost/osmagent

Of the quests, currently implemented are only “road name”, “shop opening hours” and “contribute to note” as proof-of-concepts, I am still working on the base system.
What’s missing on the base system is the logic when the system should download quests and upload the changes the user made. Also, since the app is making direct changes to the DB in the name of the user, I want to write more tests before I make it available.
But that’s it, pretty much, what is missing for a first release.

Tell me your ideas for further quest(ions) the app might ask the user and the user may answer on-site. :slight_smile:
Also, if you want to contribute somehow to the app itself, you are very welcome (also without Android programming knowledge)!

Some impressions






The idea is great, I thought of something similar (albeit integrated into the mapping app). I think you should post it to talk mailing list, because here nobody will read it.

However, direct editing for masses on mobile devices hasn’t exactly been a success. MAPS.ME users still add junk / mistagged POI, no matter how hard Zverik swears that it’s a minority. Please at least add a kill switch / soft ban that you can manage, with respect to uid (nicknames can change) and application version (so old buggy versions can’t wreak havoc). This would be posted to your server before the upload with an allow/disallow decision returned.

Can you use a geolocation API to have the app sit in background and pop up when necessary (coarse/network location/geofencing + detecting when not in a car)?

Another idea is to (with permission) scan user’s geotagged photos to have an idea where they have been and then ask some questions. Google Maps does this.

Thanks for the feedback, much appreciated!

The idea to integrate this functionality into a popular map app is nice, though it is not for me to decide. Also, you already mention the downsides of that, so we will see :slight_smile:

Oh yeah, a blacklist as you suggest will make sense for old buggy versions. A per-user ban that is managed by osmagent sounds the wrong place to do it though.

Yes I use such an API (LOST) but I am not sure if it is possible to detect that the user is on foot and what the app should do in that case. Would you elaborate on that idea? I did not think through the strategy the app should use to download the data. It is not that much data actually in terms of size, but it takes some time waiting for the Overpass response. I had in mind the following: When active, download quests on the current location in a radius of e.g. 1km, stop downloading when e.g. 50 quests have already been found (quests are downloaded ordered by importance, i.e. “road name” comes ~first, “width of cycleway” comes ~last). When user leaves the geofence of, say, 0,5km radius of the original location, the new location is downloaded again. Then, the old downloaded data needs to get stale somehow after a few days, weeks or months and is then downloaded again. Also, it needs to be downloaded again when the quests are solved and less than X quests are around the current location…
Yeah, anyway, still thinking about how it should work at all and that is why I do not quite get what feature you would like to see there.

It’s intended as a kill switch rather than a thing for regular use. In this sense you would never be able to use it if it’s not there (and it won’t harm if you don’t use it).

As per Overpass API, I think it’s not really meant for client side access in such scenario (Or at least it doesn’t scale beyond a handful of users). Recently they had a problem because some reckless app developer made an app (sending requests without User-Agent) that by sheer number of users DDOSed the Overpass API.

Overpass is not even particularly fast. The response time has worsened since a 2 years or so (as I experienced in Dotevo’s OSM24).

Yeah, Overpass. Don’t worry, I put quite some thought into that already. I intend to be very prudent about using Overpass resources. This is why I am trying to limit the requests made to Overpass wherever possible. A reason why the download-logic is not implemented yet.

I actually started the project thinking I could download from the OSM Api directly. However, let’s say I want to get the quests for a whole city. This would be huge data, I am guessing something between several hundred Megabytes to a Gigabyte. Now that’s what I call a DDOS :stuck_out_tongue: - not mentioning the impact that would have on the users data plan, haha.

Well, there are about 800-1000 queries per minute atm. If you set up your own Overpass instance and tune your query, you can easily achieve more than that. This is recommended anyway if you plan to run more than 10’000 queries per day as per Usage policy.

Performance has improved quite significantly with the recent release 0.7.53. At the same time, the amount of queries per day has also increased quite a lot since 2 years back and most importantly, the OSM data itself keeps on growing (just compare some planet files, there’s about 35% increase in data in the meantime). Unfortunately, osm24 does not seem to be around anymore (Edit: it moved to http://osmapa.pl/osm24/)), so if you happen to have some examples, you should raise an issue on Github maybe.

After a bit of analysis it turned out that the queries created by osm24 are not really ideal, as in the following example: http://overpass-turbo.eu/s/jPw (about 15s runtime). The optimized version using regular expressions just takes 2.7s (http://overpass-turbo.eu/s/jPx). So maybe osm24 has some potential for improvement. :sunglasses:

Oh that’s cool, gonna make use of that. Currently I am using it without ^(…)$ because I thought the regex operator would work like String.matches() in Java (only matching the whole string)

That is interesting, I have not seen this syntax before. However, I do not understand how a regex key definition can be faster than or even as fast as node[shop];node[office];…etc.
I am coming from Java and assuming that the tags are stored in something like a (Hash)Map. Acessing all hash map entries by a matching regex would make it necessary to iterate through the whole map in Java.

This is very interesting. I’ve been thinking there should be a tool for this sort of mapping for a while. But I never knew enough Android to do it myself. I’ll be watching this.

How can I install this?

Glad to hear this :slight_smile:

Still working on it download and upload logic and waiting for a feature of a library I use. I’ll probably post a Beta APK (android executable) on this forum in a few weeks time.

Why using Overpass when you may have the data locally?

OSMAND stores data in vector form locally. Why not making a plugin over OSMAND? Or a fully clean and simple UI over OSMAND data model? Reusing already downloaded OSM data of course.
And it’s compact (32 GB for the whole planet), you download data by states in US, counties in European countries, a less well mapped country can de downloaded as a whole. It’s vectorial so if you want 3D effects, it should be possible (I didn’t checked if the height information is preserved).

To avoid rumor about that. The total capacity (combined capacity for everyone, from the one current server) is:

  • 1 million requests per day
  • 100 GB of delivered data

The server is more or less saturated with this load (as in: 1.5 million requests may or may not work or may feel sluggish, with 2 million requests the API would get unreliable).

The app in question had 150.000 users per day that

  • posed 10.5 million requests on the peak day
  • would have downloaded 5000-10000 GB (5-10 TB) of data per day
    That never could have worked.

By contrast, if you expect for an app

  • at most 10’000 requests per day
  • at most 1 GB of data downloads per day
    then this is two orders of magnitude below the total load on both ways of metering, and most likely I will not even notice the load.

It is now up to your strategy whether you say that the app is just small enough (and cannot do harm) or is big, but not disproportionate (let’s say 100’000 requests and 10 GB of data per day). In that case it would surely be possible to get support for more server capacity if the tool gets a good standing in the community. Or whether it is well above that zone, and it is anyway impossible to get it up and running on the public server.

To give a comparison: there are on the whole planet about 2000 different contributors per day. Hence, even if you double that number (which would be an extreme success) then still 5-50 requests per day and user would work.

Last figure: if you compute with a 500 MB monthly data plan then the user has on average 15 MB per day to spend. A response is often around 100 KB, hence 50 requests will consume about one third of the user’s total data plan. This is probably more that the average user would like.

Hence be both bold and prudent. Then it will work.

Same reason why you do not have the data locally when working with JOSM, really.

Regarding “why not as a plugin”?: Personal taste I guess. I am a friend of apps that do serve one use case and that use case well. In the past, or really still, I am using Orux for taking notes on a survey, but since it has so many features and possible usages, just creating a quick note on a survey is less quick than it could be because the app covers so many other use cases and thus has all those extra options and buttons.

Wow cool, thanks for all the information! I’m limiting each request to a bbox around the user’s current location. For me the question is which minimum bbox would make sense from an Overpass Server perspective? From the perspective of the app, an extremely small bbox (a few hundred square meters) would be sufficient, but I imagine that for each query, regardless of the bbox, there is a certain overhead, so that it does not pay off to make many small requests instead of few requests with a bigger bbox. What would you say which bbox size would make sense? Also, would it make sense to choose very specific bboxes because the data is arranged in the DB in a grid? - I am asking because for simplicity I do plan to download the quests in a grid anyway.

Interesting, some comments:

  • naturally your mapping technique was simply broken (relative to your requirements) as there are multiple mobile apps available (since ages) with which you can edit on the fly, either with immediate upload or later.

  • the challenge approach in an app is not new, for example Kort has used it with varying degrees of success.

  • the main issue with your approach is that the number of simple questions that can be answered correctly and the results added without manipulating the underlying geometry is rather limited. Actually one could argue that they do not exist in general but only as special cases. For example: both adding names and lane counts to roads can require splitting the ways and -not- splitting the ways and adding the tags can result in worse data than previously existed.

  • some supposedly simple operations can be quite complex to do properly if you don’t expose the actual tagging to a knowledgable user, for example changing the type of a POI or deleting one.

  • the above points would indicate that lots of stuff will (have to) be answered with notes and will have to be post processed, which kinds of defeats your original goal.

  • as to downloading data. you really need to decide who your target audience is: somebody that just happens to be somewhere and wants to check their immediate vicinity, say radius of 50m which you can easily do with the API proper or Overpass, or are you looking more at users that will walk a km or two to fix a problem, if it is the later you have to ask yourself why would they do that with an app that only allows them to fix/add a very very small subset of what they potentially will see on the tour.

  • in any case you will have to handle objects (for example POIs) in every kind of form they exist (nodes, ways, relations, objects with multiple primary keys and so on)

Simon

PS: essentially all the “quests” you are thinking of are available as Osmose reports and can be accessed and displayed without actually downloading and inspecting the original data, functionality that Vespucci has provided for quite a long time including notifications when one is in the vicinity (on device and on wearables).

Because when using JOSM I do more than just add tags. Adding tags is not at high risk for conflicts, conflict resolution is pretty simple in most cases.
If 2 users added the same tag, then if different, you’ll have to reject the latest submission I guess.

You can use OSMAND but have a completely different UI. Personal taste too: I prefer one app for cartography. But appreciate plugins to focus on a specific thematic. But it was not my point here, my point was more about reusing infrastructure and data.

Simon I knew I should check Vespucci, thanks for the head-up.

While I’m not really in to self marketing, my talk at SOTM 2016 https://www.youtube.com/watch?v=3wg0P32QhO4 might be interesting,

Simon, you gave me some things to think about. Thank you for that. Let me however respond to some of your points:

If you mean Vespucci, certainly, but I find it impractical to use a full blown editor on a survey, also then I’d have to download the area I will go to in advance. My notes usually look like this: “cbl pv l”, (cobblestone surface, sidewalk left) “34 4.2” (housenumber 34, 4 storeys, 2 in roof) etc. Orux is okay for that, the only thing that bugs me is the long click and the clutter in the “add note” dialog and of course that I cannot see which data is exactly missing with the offline map. Can you recommend one that is better for that use case?
Also, it is too bad that I did not know Kort, you are completely right, the approach is similar.

Yes, I do realize that. But well, a small number is okay by me really, there are also not all that many different checks on MapDust either, still, it is an extremely helpful tool. I am not trying to build the survey tool to rule them all here, after all :). I guess what I am personally missing most is a tool to find the “holes” in an already surveyed area - and that is what osmagent will do.
Out of the same reasons you mention, I do not plan to add support for modifying geometry, exchanging or deleting geometry - it’s only about tags for now. I should also mention that leaving a note is not intended as a regular answer, quests that can often only be “answered” by leaving a note will not be implemented.

You absolutely got a point there and this got me thinking. osmagent very much is not suited as a general surveying app.

For me personally it is perhaps not that much of a caveat because I often don’t do dedicated surveys but casually flip out my smartphone and take notes while going somewhere. If I stop and do that every five meters, my girlfriend will frown upon that, and if I am alone, I will realize sooner or later that I am not getting anywhere. osmagent is something I could do on the side.

In short, the current target audience might be the casual mapper, who takes notes here and there on the side and does not want to invest too much time in it. The other target audience I am aiming at are people who might be “playing” this like a treasure hunt, getting “stars” for every solved quest (I am a games programmer; a little self-advertisment: http://www.openclonk.org :P). Yes, I know there would be much more work to be done to make this attractive, graphics, animations, medals, highscores and stats, meta challenges … so that is perhaps something for the future.
Now I understand RicoElectrico’s request that there should be some kind of proximity alarm when the app is in the background and you are near a quest.

What do you mean with that?

I like the idea. I think the use case it is trying to address is a common one and if executed well (minimize brain load) it could be quite popular :slight_smile:

Just one thing. Why do you even need overpass QL queries? Since you use Tangram-ES for rendering map, which means you use mapzen vector tiles. Mapzen vector tiles have most of OSM information inside of them. And you have those data locally since you are showing the map itself. Some can definitely be used. Road layer has names and POI layer for POI info. The only problem is that some magic is done to get from OSM tags to Vector layers. Queries which make Mapzen layers from OSM data are also available here.

I really did not consider it yet. But thinking about it, I think there are some things that speak against it:

  • it adds a level of indirection to the source data
  • I make myself dependent on that vector tile service
  • they are vector *tiles *, so one and the same map feature might be split up into several in MVT
  • I need to work with OSM source data latest when the changes are uploaded. I need the data as it comes out of the OSM API with id, version, ALL other tags and members (of ways and relations), even if I do not want to change them.
  • I will need Overpass anyway (for many other quest types), because MVT data is only a subset of OSM data. Specifically, it is only data where it makes sense to display it directly on the map.
  • I need to “translate” the tagging from MVT format to OSM format because features are not named the same
  • what comes out of the MVT service is aggregated data. Even if it is aggregated from OSM as the only data source, it would be very impractical to work with, i.e.
  • MVT is not in a format that facilitates picking out single elements
  • Implementation using Overpass already works
  • Overpass works really well actually and the response times are quite good. Yesterday I was testing a quest type “building height” and it fetched about 5000 quests via Overpass in something like 30 seconds. Let’s see how long it takes to solve them all.