Overpass/OSM DB missing data

Hi,

I’m not sure to be in the correct subtopic for my question but I didn’t found better.

I try to use Overpass API over Europe OSM data. I want to get all European country (+ their islands). Example, for France : Metropole + Corsica + Guadeloupe + Martinique…

To do it, I downloaded europe OSM datafile from geofabrik : https://download.geofabrik.de/europe-latest.osm.bz2 because I saw that in Europe folder I can get all what I wanted. Example : Europe/France/Corsica+Guadeloupe+Martinique… and same things for others countries.

I followed Overpass API installation : https://wiki.openstreetmap.org/wiki/Overpass_API/Installation with success.

I tried to request on different countries with success but can’t get data for many islands. Example : I can get Corsica data but CAN’T get for others French islands (Guadeloupe, Martinique…).

Request example :

→ this returned values using https://overpass-turbo.eu test tool, but nothing with my custom overpass API (tried on different request and different GPS loc).

Also, I checked import log files generated during import of .osm.bz2 into db/ folder, I saw many (more than hundred) lines like :

→ is it related to my problem ?

Can the europe-latest.osm.bz2 be corrupted (missing data) ?
Can I import islands on my current /db to not re-download all data ?

Thank a lot !

Some relations for countries mostly in Europe (for example France) have parts that are outside Europe, so it’s quite possible that a “European” extract will be missing some parts of some countries.

What problem are you actually trying to solve?

My initial goal was : to get European countries (geographic) + all islands associated to them (example : France is an European country, so I want also their islands : in Europe like Corsica + outside Europe like Martinique, Guadeloupe…).

Strictly speaking, France is a country that is mostly in Europe (see https://en.wikipedia.org/wiki/Overseas_France for details). it sounds like you’ve got the European bit of France, but not the rest.

The actual error message you’ve got is because you’ve got part of the Euphrates river https://www.openstreetmap.org/relation/10106318 (the European bit, I guess) but not https://www.openstreetmap.org/way/83225893 , which is presumably outside Geofabrik’s definition of Europe.

How to fix this probably depends on what your goal is; that’s why I asked that.

In fact, France largest border is with Brazil…

Thank you for details.

Finally, someone has an idea on how to achieve my goal ? :expressionless:

Unfortunately, I don’t think that we know what that is yet…

I assume you will need to import data for other relevant parts of the globe (oceania, south-america, …) as well.

How can I import two continents ? Example : Europe + America only (not the whole planet)

Do you just need the boundaries of those countries? In that case, you can use https://wambachers-osm.website/boundaries/
If you need other data such as roads, POI, etc. you need another method.

So please explain which data you mean when you write “I want to get all European country (+ their islands)” It might be obvious to you what you need, but it is not for the people that want to help you.

Hi, I need all data like roads, POI…

Thank you for taking time for my problem ! :smiley:

You’ve downloaded all that - so you’ve already got it. What you haven’t said is what you are trying to do with it.

For example, I try to get all node in one island with :

I don’t get any data returned.
Same request on overpass-turbo returns me data.

Also, same request with lat,lon in European country (Germany for example, or France) returns me data
=> so I have a problem (missing data) in my DB

That is correct. The file you downloaded from geofabrik doesn’t contain any objects outside the polygon shown on http://download.geofabrik.de/europe.html. That is why you can’t find any data for the overseas islands. You will have to download more OSM data somehow.
The Europe extract intentionally does not include any data on the overseas parts of European countries, which are not inside the polygon.

Yes I understand, bu do you know how to combine two polygon into one DB ?

Typically, you’d merge them together before loading into a database. “osmium” is one way of doing that, “osmosis” is another. One thing that a map server that I run does is splits a downloaded file into several pieces, translates each piece into a different language, and merges them all back together. The script that does all this (using osmosis) is here: https://github.com/SomeoneElseOSM/SomeoneElse-style/blob/master/update_render.sh .

There’s also documentation on osmosis and osmium in the OSM wiki and linked from it.

Thank you for response, I’ll check in this way. :smiley:

I tried, it worked fine with multiples small files but can’t merge huge file (europe :33GB) with others small files (memory error). :expressionless:

Is there a solution to add .osm.bz2 files into an existing database ?
The idea is to initial load with Europe and after: import in the existing DB others files.

Thanks :slight_smile:

What was the “it” that you tried? I made two suggestions…

What actual command line did you use? How much memory do you have? Do you have the opportunity to configure more (e.g. via swap)?

I tried with osmium :

I have 8 GB RAM + 8GB SWAP

=> Do I must have SWAP >= little+europe (xxxMB + 33GB) ?