Here I am again, waiting for two weeks while a planet PBF file is imported into a PostGIS database and hoping I don’t fall victim to an osm2pgsql or PostGIS bug. In the past I’ve lost weeks to failed imports and have finally settled on a specific osm2pgsql version (81) that doesn’t SEGFAULT or otherwise fail days into the process on a perfectly normal, mid-range Ubuntu server with gasp less than 40GB of RAM.
Wouldn’t it be easy and incredibly useful for OSM to provide a simple, regular bzipped dump of the complete OSM PostGIS database along with the XML and PBF files on planet.openstreetmap.org? Developers that really need them can still use them. The other 99% who just want the OSM dataset in PostGIS can get that with a quick and simple psql import, even on a low-end machine.
I’m very willing to help implement and document this if needed.
I think the problem here may be a technical one… that is, maybe rather astonishingly to you, there may be no suitable planet wide PostGIS database that could deliver the dump…
As far as I currently understand the infrastructure, there are essentially two databases:
-
One is the true base “edit” database that stores the relational schema of the nodes and ways, and relation information and tags. This is essentially not a PostGIS, but a normal PostGreSQL database, as the primitives are not stored in an OGC compliant way, but in a specific structure designed by the original OSM developers. This is the database all editor applications are designed to use and interact with via the OpenStreetMap API developed for this.
This database is incapable of delivering the data dump you want, as it doesn’t contain the structure of an osm2pgsql created database. In addition, it also contains the full history of the objects, again something you may not desire.
-
The second database is the osm2pgsql created “render” database. This database delivers the tiles used on the main OpenStreetMap website. This database has originally been created (and re-created) using osm2pgsql, and is now continuously updated through diffs to keep up with the main “edit” database.
This database may not be based on the latest osm2pgsql version, depending on when it was last recreated from a full planet import. Since it is continuously being updated, it follows the structure of updatable databases, and may not be the structure you desire, as I understood that there are some (major?) differences between updatable and non-updatable databases. Also, I am not sure what problems the continuous updates to this database would pose for creating a suitable dump.
For more information and better understanding, see this Wiki page:
https://wikitech.wikimedia.org/wiki/Openstreetmap_Databases
Again, this is just based on my current (limited) understanding of it all, so read the Wiki for more details, and maybe someone else with more knowledge could supplement or correct my answer.
You’re right, I see that there is no database that can currently be used for this. I should have my own soon, and it will be updated weekly. So I could easily provide such a dump on a regular basis, provided the files can be hosted somewhere (hosting them on my servers would probably hit my bandwidth pretty hard). Any ideas?
You might consider contacting one of the (technical) working groups associated with OSM:
https://wiki.osmfoundation.org/wiki/Working_Groups
Maybe they can help you out with ideas about the hosting.