Generating shapefiles from OSM data

Hello everyone,

I’m quite new to OpenStreetMap and my purpose is to generate shapefiles for every country of the world. However, I’ve noticed how difficult it is to find good data.

Until today, I was using geofabrik and wambachers web application to retrieve shp. I’ve also tried Overpass Turbo, Osmosis, osm2shp.

Problem: data provided within ESRI Shapefile are not always true. Let me explain: there are regions and departments that no longer exists, administrative boundaries may be outdated or gathering water areas (Netherland for example). Also, regions are not always on admin_level = 4 (same for departments). This is upsetting as I’m willing to make a generic program.

Is there a way to generate myself shp in order to get administrative land boundaries for

• country level?
• region level?
• department level?

Thanks in advance for your help!

I guess you can download boundaries for any area in different formats (including SHP) from here:

https://wambachers-osm.website/boundaries/

Thank you kocio for your quick response.

https://wambachers-osm.website/boundaries/, that’s where I used to download my shp and where i found the problem.
For instance, netherlands shp includes seas and water areas which I want to exclude.

To be more specific: is there a way to generate administrative boundaries shp using osm file?

If you know there are errors in the database or the data is outdated, the best solution would be to fix those in the OSM database.

As for excluding sea areas of administrative boundaries, that’s much more difficult. The boundary of the Netherlands (and any other country bordering the ocean) extends out into the water in reality, so that’s what the OSM database contains. If you don’t want that area included, you’d need to manually build your own boundary following the coastline in your own database. There’s probably a way to do this part programmatically by following the connected natural=coastline ways between the two points where the boundary intersects the coastline.

You can easily do it with QGIS. Install in it QuickOSM plugin (or open data something like that too), so you can import in .osm or .geojson. You can also download all of .OSM objetcts at http://download.geofabrik.de/. I guess there are only boundaries anywhere too. Or select boundaries or anything that you want by Overpass https://wiki.openstreetmap.org/wiki/Overpass_turbo/Wizard and import to QGIS. Then save what you want as SHP.

Just check LAND in the buttom area and you are done.

much better: Use the CLI and set &ExportArea=land

walter

I’m willing to become an active contributor as OSM concerns are also mine

Can’t we correct the admin boundaries directly into OSM?

@smapsr I’ve tried both QuickOSM and overpass, great tool but sadly same problem comes by about water areas. Maybe there’s a way to filter it from overpass but i can’t figure it out

I’ve follow the instructions on your website about the cli command to get shp and i think it’s the most suitable tool! However, the seas within the borders are not excluded. I’ve managed to merge entities so it doesn’t include water area but I’d like to import it into osm data

thanks.

Yes, only seas with international boundaries are clipped. And some realy big ones too.

insert into water_polygons values(100001,100000,(select st_multi(way) from planet_osm_polygon where osm_id=-1156846),'Bodensee 1156846');
insert into water_polygons values(100002,100001,(select st_multi(way) from planet_osm_polygon where osm_id=-3474227),'Rio de la Plata-Mündung 3474227');
insert into water_polygons values(100003,100002,(select st_multi(way) from planet_osm_polygon where osm_id=-1374224),'Tanganjikasee 1374224');
insert into water_polygons values(100004,100003,(select st_multi(way) from planet_osm_polygon where osm_id=-108807),'Mwerusee 108807');
insert into water_polygons values(100005,100004,(select st_multi(way) from planet_osm_polygon where osm_id=-318675),'Karibasee 318675');
insert into water_polygons values(100006,100005,(select st_multi(way) from planet_osm_polygon where osm_id=-1969016),'Turkanasee 1969016');
insert into water_polygons values(100007,100006,(select st_multi(way) from planet_osm_polygon where osm_id=-332617),'Genfer See 332617');
insert into water_polygons values(100008,100007,(select st_multi(way) from planet_osm_polygon where osm_id=-5901404),'Westlicher Aralsee 5901404');
insert into water_polygons values(100009,100008,(select st_multi(way) from planet_osm_polygon where osm_id=-5901405),'Westlicher Aralsee 5901405');
insert into water_polygons values(100010,100009,(select st_multi(way) from planet_osm_polygon where osm_id=-1269323),'Sarykamysch-See 1269323');
insert into water_polygons values(100011,100010,(select st_multi(way) from planet_osm_polygon where osm_id=-6515),'Laguna Merín 6515');
insert into water_polygons values(100012,100011,(select st_multi(way) from planet_osm_polygon where osm_id=-1125603),'Chankasee 1125603');
insert into water_polygons values(100013,100012,(select st_multi(way) from planet_osm_polygon where osm_id=-3000330),'Titikaka-See 3000330');
insert into water_polygons values(100014,100013,(select st_multi(way) from planet_osm_polygon where osm_id=-1206310),'Ontario-See 1206310');
insert into water_polygons values(100015,100014,(select st_multi(way) from planet_osm_polygon where osm_id=-4039900),'Eriesee 4039900');
insert into water_polygons values(100016,100015,(select st_multi(way) from planet_osm_polygon where osm_id=-4228055),'Saint Clair-See 4228055');
insert into water_polygons values(100017,100016,(select st_multi(way) from planet_osm_polygon where osm_id=-1205151),'Huron-See 1205151');
insert into water_polygons values(100018,100017,(select st_multi(way) from planet_osm_polygon where osm_id=-4228055),'Saint Clair-See 4228055');
insert into water_polygons values(100019,100018,(select st_multi(way) from planet_osm_polygon where osm_id=-4039486),'Oberer See 4039486');
insert into water_polygons values(100020,100019,(select st_multi(way) from planet_osm_polygon where osm_id=-1205149),'Michigan-See 1205149');
insert into water_polygons values(100021,100020,(select st_multi(way) from planet_osm_polygon where osm_id=-2606941),'Victoriasee 2606941');
insert into water_polygons values(100022,100021,(select st_multi(way) from planet_osm_polygon where osm_id=-1110965),'Albertsee 1110965');
insert into water_polygons values(100023,100022,(select st_multi(way) from planet_osm_polygon where osm_id=-940893),'Eduardsee 940893');
insert into water_polygons values(100024,100023,(select st_multi(way) from planet_osm_polygon where osm_id=-2631184),'Malawisee 2631184');
insert into water_polygons values(100025,100024,(select st_multi(way) from planet_osm_polygon where osm_id=-4557289),'Dojransee 4557289');
insert into water_polygons values(100026,100025,(select st_multi(way) from planet_osm_polygon where osm_id=-2905510),'Prespasee 2905510');
insert into water_polygons values(100027,100026,(select st_multi(way) from planet_osm_polygon where osm_id=-2905270),'Ohridsee 2905270');
insert into water_polygons values(100028,100027,(select st_multi(way) from planet_osm_polygon where osm_id=-392575),'Skutarisee 392575');
insert into water_polygons values(100029,100028,(select st_multi(way) from planet_osm_polygon where osm_id=-1298088),'Bilećasee 1298088');
insert into water_polygons values(100030,100029,(select st_multi(way) from planet_osm_polygon where osm_id=-36970),'Urmia Lake 36970');
insert into water_polygons values(100031,100030,(select st_multi(way) from planet_osm_polygon where osm_id=-5486417),'Asowsches Meer 5486417');
insert into water_polygons values(100032,100031,(select st_multi(way) from planet_osm_polygon where osm_id=-2791995),'Asowsches Meer Erweiterung 2791995');
insert into water_polygons values(100033,100032,(select st_multi(way) from planet_osm_polygon where osm_id=-1206317),'Großer Salzsee 1206317');
insert into water_polygons values(100034,100033,(select st_multi(way) from planet_osm_polygon where osm_id=-1240556),'Okeechobeesee 1240556');
insert into water_polygons values(100035,100034,(select st_multi(way) from planet_osm_polygon where osm_id=-9072),'Kiwusee 9072');
insert into water_polygons values(100036,100035,(select st_multi(way) from planet_osm_polygon where osm_id=-1348022),'Abbesee 1348022');
insert into water_polygons values(100037,100036,(select st_multi(way) from planet_osm_polygon where osm_id=-280282),'Nassersee 280282');
insert into water_polygons values(100038,100037,(select st_multi(way) from planet_osm_polygon where osm_id=4741633),'Totes Meer 4741633'); -- keine rel sondern way
insert into water_polygons values(100039,100038,(select st_multi(way) from planet_osm_polygon where osm_id=-2905510),'Prespasee 2905510');
insert into water_polygons values(100040,100039,(select st_multi(way) from planet_osm_polygon where osm_id=-2905270),'Ohridsee 2905270');

technically no problem for me to add more - but where is the end? Which should be added and which not?

Regards
walter

Yes, definitely. Keep in mind, though, that a country boundary extending out into the ocean is already correct. Changing it to follow the coastline in the public database would be quickly reverted because it wouldn’t reflect reality.

definitely no - because they are not wrong.

btw: Clipping boundaries by coastline is not an easy job. My programs are running some seconds up to 30 !! hours to clip all boundaries of one single country depending on coastline complexity

And yes, i’m proud of it

Regards
walter

Don’t know if I got it well, but the filters for overpass wizard are for example:

boundary=* and natural!=coastline in AnyCountry

for only boundaries, all of them; and none of coastline.
(I don’t remember if these examples match with reality, just examples)
More in its wiki.

Hello Walter,

Since the beginning of the year, I’m wondering how to regenerate all shp and data of every country using OSM data. It’ll be much more simpler (for users and me) to correct data directly into OpenStreetMap. This is the ranking of websites providing (I believe) accurate shapefiles in most cases:

http://overpass-turbo.eu/
https://wambachers-osm.website/boundaries
https://gadm.org/data.html
http://download.geofabrik.de
https://extract.bbbike.org
http://www.diva-gis.org/gdata
https://osmaxx.hsr.ch/orders/new/existing_excerpt/
http://www.statsilk.com/maps/download-free-shapefile-maps

and I think you can be pretty proud of it. Yours is only second because when the shapefile is generated, attribute table contains less information than overpass’ - alternateNames for instance.

I’ve been studying how to generate my own shapefiles using osmosis but I often encounter difficulties like the one I’ve been describing in my post. How do you manage to export land boundaries on wambachers-osm?

Warm regards

ok i get the issue right here - for this particular case i’ve been editing the shp with qgis to fit my needs. Is there a solution to filter it automatically instead of using qgis?

Maybe you meant “natural=coastline”? I didn’t retrieve any results for Emirates neither Netherlands. I’ll keep looking into the wiki - thanks!

There are some technical restictions for shape files:

see https://desktop.arcgis.com/en/arcmap/latest/manage-data/shapefiles/geoprocessing-considerations-for-shapefile-output.htm

Because of that i have to

• change attributes like “name:en” to “enname” (no colon allowed in dBase)
• change attributes like “official_name” to “off_name” (length)
• limit the number of attributes to a subset (maximun 255 allowed)

Since there are currently 2896 !! different attributes (tags) in use, I had to make a selection. It’s of course possible, to add more attributes (up to 255 of course).

see https://wambachers-osm.website/images/osm/data/count_tags_usage.txt

There are 2 solutions:

• tell me, what you need - and wait some time
• export GeoJson and convert to Shape on your site. GeoJson has none of those restrictions and does contain all tags. (hope so ;))

Regards
walter

btw: which lakes should be clipped additionally?

I think this is the best solution, I’ve tried to convert geoJSON to shp and you were right about restrictions.
Actually my needs are essentially alternate names for foreign languages.

Actually ijsselmeer and Zeeuwse meren:
https://giphy.com/gifs/k5zUV76AnFY56h8JX8