You are not logged in.

#1 2019-11-12 14:38:09

ChrisPJS
Member
Registered: 2019-11-12
Posts: 10

Overpass/OSM DB missing data

Hi,

I'm not sure to be in the correct subtopic for my question but I didn't found better.

I try to use Overpass API over Europe OSM data. I want to get all European country (+ their islands). Example, for France : Metropole + Corsica + Guadeloupe + Martinique...

To do it, I downloaded europe OSM datafile from geofabrik : https://download.geofabrik.de/europe-latest.osm.bz2 because I saw that in Europe folder I can get all what I wanted. Example : Europe/France/Corsica+Guadeloupe+Martinique... and same things for others countries.

I followed Overpass API installation : https://wiki.openstreetmap.org/wiki/Ove … stallation with success.

I tried to request on different countries with success but can't get data for many islands. Example : I can get Corsica data but CAN'T get for others French islands (Guadeloupe, Martinique...).

Request example :

[out:json]; node(around:50,16.24263,-61.535662); <; out geom;

--> this returned values using https://overpass-turbo.eu test tool, but nothing with my custom overpass API (tried on different request and different GPS loc).

Also, I checked import log files generated during import of .osm.bz2 into db/ folder, I saw many (more than hundred) lines like :

compute_geometry: Way 83225893 used in relation 10106318 not found.

--> is it related to my problem ?

Can the europe-latest.osm.bz2 be corrupted (missing data) ?
Can I import islands on my current /db to not re-download all data ?

Thank a lot !

Offline

#2 2019-11-13 12:09:08

SomeoneElse
Member
Registered: 2010-10-13
Posts: 1,221

Re: Overpass/OSM DB missing data

Some relations for countries mostly in Europe (for example France) have parts that are outside Europe, so it's quite possible that a "European" extract will be missing some parts of some countries.

What problem are you actually trying to solve?

Offline

#3 2019-11-13 13:42:18

ChrisPJS
Member
Registered: 2019-11-12
Posts: 10

Re: Overpass/OSM DB missing data

My initial goal was : to get European countries (geographic) + all islands associated to them (example : France is an European country, so I want also their islands : in Europe like Corsica + outside Europe like Martinique, Guadeloupe...).

Last edited by ChrisPJS (2019-11-13 13:42:59)

Offline

#4 2019-11-13 16:13:04

SomeoneElse
Member
Registered: 2010-10-13
Posts: 1,221

Re: Overpass/OSM DB missing data

ChrisPJS wrote:

France is an European country

Strictly speaking, France is a country that is _mostly_ in Europe (see https://en.wikipedia.org/wiki/Overseas_France for details).  it sounds like you've got the European bit of France, but not the rest.

The actual error message you've got is because you've got part of the Euphrates river https://www.openstreetmap.org/relation/10106318 (the European bit, I guess) but not https://www.openstreetmap.org/way/83225893 , which is presumably outside Geofabrik's definition of Europe.

How to fix this probably depends on what your goal is; that's why I asked that.

Offline

#5 2019-11-13 21:19:31

muralito
Member
Registered: 2012-09-04
Posts: 1,774

Re: Overpass/OSM DB missing data

SomeoneElse wrote:

Strictly speaking, France is a country that is _mostly_ in Europe

In fact, France largest border is with Brazil....

Offline

#6 2019-11-14 09:40:51

ChrisPJS
Member
Registered: 2019-11-12
Posts: 10

Re: Overpass/OSM DB missing data

Thank you for details.

Finally, someone has an idea on how to achieve my goal ? neutral

Offline

#7 2019-11-14 09:42:03

SomeoneElse
Member
Registered: 2010-10-13
Posts: 1,221

Re: Overpass/OSM DB missing data

ChrisPJS wrote:

Finally, someone has an idea on how to achieve my goal ? neutral

Unfortunately, I don't think that we know what that is yet...

Offline

#8 2019-11-14 10:37:58

PHerison
Member
From: Rhein-Main
Registered: 2008-04-04
Posts: 1,512

Re: Overpass/OSM DB missing data

I assume you will need to import data for other relevant parts of the globe (oceania, south-america, ...) as well.

Offline

#9 2019-11-14 11:57:35

ChrisPJS
Member
Registered: 2019-11-12
Posts: 10

Re: Overpass/OSM DB missing data

How can I import two continents ? Example : Europe + America only (not the whole planet)

Offline

#10 2019-11-15 05:22:58

escada
Moderator
Registered: 2011-08-13
Posts: 1,609

Re: Overpass/OSM DB missing data

Do you just need the boundaries of those countries? In that case, you can use https://wambachers-osm.website/boundaries/
If you need other data such as roads, POI, etc. you need another method.

So please explain which data you mean when you write "I want to get all European country (+ their islands)" It might be obvious to you what you need, but it is not for the people that want to help you.

Offline

#11 2019-11-18 12:09:09

ChrisPJS
Member
Registered: 2019-11-12
Posts: 10

Re: Overpass/OSM DB missing data

Hi, I need all data like  roads, POI...

Thank you for taking time for my problem ! big_smile

Offline

#12 2019-11-18 12:14:10

SomeoneElse
Member
Registered: 2010-10-13
Posts: 1,221

Re: Overpass/OSM DB missing data

ChrisPJS wrote:

Hi, I need all data like  roads, POI...

You've downloaded all that - so you've already got it.  What you haven't said is what you are trying to do with it.

Offline

#13 2019-11-19 16:46:35

ChrisPJS
Member
Registered: 2019-11-12
Posts: 10

Re: Overpass/OSM DB missing data

For example, I try to get all node in one island with :

[out:json]; node(around:100,16.24263,-61.535662); <; out geom;

I don't get any data returned.
Same request on overpass-turbo returns me data.

Also, same request with lat,lon in European country (Germany for example, or France) returns me data
=> so I have a problem (missing data) in my DB

Offline

#14 2019-11-19 21:17:15

alphensebezorger
Member
Registered: 2016-02-08
Posts: 157

Re: Overpass/OSM DB missing data

so I have a problem (missing data) in my DB

That is correct. The file you downloaded from geofabrik doesn't contain any objects outside the polygon shown on http://download.geofabrik.de/europe.html. That is why you can't find any data for the overseas islands. You will have to download more OSM data somehow.
The Europe extract intentionally does not include any data on the overseas parts of European countries, which are not inside the polygon.

Offline

#15 2019-11-20 17:08:23

ChrisPJS
Member
Registered: 2019-11-12
Posts: 10

Re: Overpass/OSM DB missing data

Yes I understand, bu do you know how to combine two polygon into one DB ?

Offline

#16 2019-11-20 17:30:28

SomeoneElse
Member
Registered: 2010-10-13
Posts: 1,221

Re: Overpass/OSM DB missing data

ChrisPJS wrote:

Yes I understand, bu do you know how to combine two polygon into one DB ?

Typically, you'd merge them together before loading into a database.  "osmium" is one way of doing that, "osmosis" is another.  One thing that a map server that I run does is splits a downloaded file into several pieces, translates each piece into a different language, and merges them all back together.  The script that does all this (using osmosis) is here: https://github.com/SomeoneElseOSM/Someo … _render.sh .

There's also documentation on osmosis and osmium in the OSM wiki and linked from it.

Last edited by SomeoneElse (2019-11-20 17:30:42)

Offline

#17 2019-11-21 10:55:59

ChrisPJS
Member
Registered: 2019-11-12
Posts: 10

Re: Overpass/OSM DB missing data

Thank you for response, I'll check in this way. big_smile

Offline

#18 2019-11-21 16:13:47

ChrisPJS
Member
Registered: 2019-11-12
Posts: 10

Re: Overpass/OSM DB missing data

I tried, it worked fine with multiples small files but can't merge huge file (europe :33GB) with others small files (memory error). neutral

Is there a solution to add .osm.bz2 files into an existing database ?
The idea is to initial load with Europe and after: import in the existing DB others files.

Thanks smile

Offline

#19 2019-11-21 17:32:50

SomeoneElse
Member
Registered: 2010-10-13
Posts: 1,221

Re: Overpass/OSM DB missing data

ChrisPJS wrote:

I tried, it worked fine with multiples small files but can't merge huge file (europe :33GB) with others small files (memory error)

What was the "it" that you tried?  I made two suggestions...

What actual command line did you use?  How much memory do you have?  Do you have the opportunity to configure more (e.g. via swap)?

Offline

#20 2019-11-22 12:08:49

ChrisPJS
Member
Registered: 2019-11-12
Posts: 10

Re: Overpass/OSM DB missing data

I tried with osmium :

osmium merge little.osm.bz2 europe.osm.bz2 -o merged.osm.bz2 --input-format=osm.bz2 --v

I have 8 GB RAM + 8GB SWAP

=> Do I must have SWAP >= little+europe (xxxMB + 33GB) ?

Last edited by ChrisPJS (2019-11-22 14:06:42)

Offline

Board footer

Powered by FluxBB