osm2pgsql out of memory for sparse node cache, but my PC has 64GB RAM

General talk
1

Hi,

Hopefully somebody can help.

I am using the osm2pgsql cygwin build from https://vanguard.houghtonassociates.com/browse/OSM-OSM2PSQL (#70).

For some reason it won’t let me set cache -C any larger than about 1400MB. If I do I get the response:

“out of memory for sparse node cache”.

My PC has 64GB ram and 61GB available.

My command line options are:

-c -s -v -W -C 1024 luxembourg.osm.pbf

My postgresql.conf file is default.

If I run with -O null for testing then it works.

Any help would be great!

Thankyou

Verfum

2

I should say my cache option was -C 4096, not -C 1024. How does osm2pgsql decide memory availability before it begins processing?

3

Just entering the error text in an internet search engine of your choice gives some hints:

Maybe in https://github.com/openstreetmap/osm2pgsql/issues/23 there is a workaround?

4

Thanks stephan75,

I have seen that link also, and have tried using --cache-strategy sparse and --cache-strategy dense but I get the same problem.

The build of osm2pgsql I’m using has been compiled for 64-bit id space, but it won’t let me specify more than about 1400MB for the cache.

And I’m getting the response of “out of memory for sparse node cache” before it starts attempting any processing, like it’s checking my RAM but not finding enough even though there’s >60GB free.

I’m needing to import using a Windows version of osm2pgsql as this is the nature of the server it’s running on. I’ve not tried using a Linux build, but might install it on an external drive and try it out on the same PC.

Cheers

5

Then I would encourage you to try it on a linux version … maybe you try to install a linux in anything like Virtualbox or similar?

If your attemps still fail then, please ask at the OSM developer mailing list at http://wiki.openstreetmap.org/wiki/Mailing_lists (recommended) or at http://help.openstreetmap.org

6

Hi,

Try with this a bit older version http://latuviitta.org/downloads/osm2pgsql-2012-08-09.zip
I am not sure where I have downloaded it originally and OSM wiki does not mention it.

It works for me on 32-bit Windows and usage is simple, just go to the installation directory and run. The command I use is
osm2pgsql -H localhost -P 5432 -d db_name -p osm -U username -W -s -S c:\osm2pgsql_2012\default.style -k -G -r pbf --create --unlogged --cache-strategy sparse --cache 600 finland-latest.osm.pbf

7

Hi JRA, thanks for your reply. I will try this and report back.
I am able to import small(ish) amounts of data, like Germany and U.K, but not Europe or Planet.

8

Have you solve your problem verfum? And can you tell me you imporint speed of the osm2pgsql?
My server’s RAM is 20GB and the shared_buffer and other parameters were congfigured. So the cache size is enough, but the speed was too slow:(
Can you tell me what I can do to deal with it?
With much thanks!