How big should be the disk?

Hi!

As I said hier https://forum.openstreetmap.org/viewtopic.php?pid=851294 I’m trying to import the data for Europe.
Since with normal HDD after one month the import is already working, I’d try with an SSD.
The question is how big should be the SSD in order to import the data for Europe…

Can someone help me? I have currently an 512GB SSD that I can use. Is it enough or do I need something greater?

Thanks
Luca

The planet pbf currently has 62GB, and the Europe extract 24GB. A full planet import with all tags (–hstore-all) currently needs about 1TB, so half a TB should still be more than enough for just Europe …

(I did all of Europe on a server with 2x240GB SSD, but that was some five years ago. I then switched to whole planet on that machine, using the osm2pgsql feature to put indexes and actual table data on different tablespaces, and only having indexes on SSD, but that was “living on the edge” as that was just barely fast enough to catch up with diffs to keep the DB up to date, taking over 20 hours to import a daily diff back then)

Hi Hartmut!

Thank you very much.

So my 512GB SSD should be enogh…
I’ll build it in and start the import. I’ll let the forum know the result!

Regards
Luca

Hi again,

I imported the data on a Test-Server I have in office with SSD.
Total space: 576GB… :confused:

So my 512GB SSD is not enough…
You said that it is possible to split the data on more disks. Could you say me how? What are the pros? And what the contras?

Thanks
Luca

P.S.: I really can’t believe… Last year I imported the data of Europe and I’m sure 512GB were enough… Did really the data growed so much?

Another question… I import the data so:

sudo -u _renderd osm2pgsql -d gis --create --slim  -G --hstore --tag-transform-script ~/src/osm-icao/osm-icao.lua \
  -C 8500 --number-processes 12 -S ~/src/osm-icao/osm-icao.style ~/data/europe-latest.osm.pbf

Since I found the declaration in an HowTo… Now I read the Man-Pages of osm2pgsql…
–slim stores temporary data in the database. Good if I don’t have many RAM (currently the Server I’ll use has 16GB)
But I see a --drop:

So, do I understand correctly, that if I import with --slim --drop I’ll get a reduced disk space after the import?
Is there a possibility to “drop” the temporary data after the import? Just to test it on the Test-Server I used…

Thanks
Luca

So, I installed the SSD…
Really fast is the import unfortunately not

Processing: Node(2875001k 577.1k/s) Way(197k 1.60k/s) Relation(0 0.00/s)

I see with top:

 1038 _renderd  20   0   12,3g  11,9g   5764 S  30,2  76,6  84:14.32 osm2pgsql
 1043 postgres  20   0  215744 143940 140752 D  30,2   0,9  58:38.90 postgres

and with iotop:

Total DISK READ:        20.71 M/s | Total DISK WRITE:      1136.28 K/s
Current DISK READ:      20.70 M/s | Current DISK WRITE:     488.07 K/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
 1043 be/4 postgres   20.71 M/s    0.00 B/s  0.00 % 34.23 % postgres: 11/main: _renderd gis [local] idle in transaction
  595 be/4 postgres    0.00 B/s    0.00 B/s  0.00 %  0.16 % postgres: 11/main: background writer
 1040 be/4 postgres    0.00 B/s  518.57 K/s  0.00 %  0.00 % postgres: 11/main: _renderd gis [local] COPY
 1044 be/4 postgres    0.00 B/s  617.71 K/s  0.00 %  0.00 % postgres: 11/main: _renderd gis [local] COPY
    1 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % init
    2 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kthreadd]
    3 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcu_gp]
    4 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcu_par_gp]
    5 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/0:0-mm_percpu_wq]
    6 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/0:0H-kblockd]

I started the import with following parameters:

sudo -u _renderd osm2pgsql -d gis --disable-parallel-indexing --drop --unlogged --create --slim  -G \
  --hstore --tag-transform-script ~/src/osm-icao/osm-icao.lua -C 12000 --number-processes 12 -S ~/src/osm-icao/osm-icao.style \
  ~/data/europe-latest.osm.pbf

Any idea how to speed up the import?
On a test PC at office (even with SSD), the import speed was:

Processing: Node(2899215k 1635.2k/s) Way(351236k 13.23k/s) Relation(5950820 330.38/s)  parse time: 46331s

Maybe just pg_dump from this PC and import in my server?

Thank you very much for your suggestion!
Luca

Ciao Luca. You’re using --slim, so maybe you can use --flat-nodes and provide a path in a non-SSD location. But probably 500GiB is getting more and more tight, as loads of data is pouring in from apps like StreetComplete. Check all the options in osm2pgsql -h -v, specially the Middle options. I thought there were some options for optimizing the db, but I can’t find them now; they should use extra space at the end of the import.