You are not logged in.

#1 2022-01-25 08:48:25

lucabert
Member
From: Dresden
Registered: 2020-11-23
Posts: 73

How big should be the disk?

Hi!

As I said hier https://forum.openstreetmap.org/viewtop … pid=851294 I'm trying to import the data for Europe.
Since with normal HDD after one month the import is already working, I'd try with an SSD.
The question is how big should be the SSD in order to import the data for Europe...

Can someone help me? I have currently an 512GB SSD that I can use. Is it enough or do I need something greater?

Thanks
Luca

Offline

#2 2022-01-25 09:17:10

Hartmut Holzgraefe
Member
Registered: 2015-04-13
Posts: 251

Re: How big should be the disk?

The planet pbf currently has 62GB, and the Europe extract 24GB. A full planet import with all tags (--hstore-all) currently needs about 1TB, so half a TB should still be more than enough for just Europe ...

(I did all of Europe on a server with 2x240GB SSD, but that was some five years ago. I then switched to whole planet on that machine, using the osm2pgsql feature to put indexes and actual table data on different tablespaces, and only having indexes on SSD, but that was "living on the edge" as that was just barely fast enough to catch up with diffs to keep the DB up to date, taking over 20 hours to import a daily diff back then)

Offline

#3 2022-01-25 09:22:16

lucabert
Member
From: Dresden
Registered: 2020-11-23
Posts: 73

Re: How big should be the disk?

Hartmut Holzgraefe wrote:

The planet pbf currently has 62GB, and the Europe extract 24GB. A full planet import with all tags (--hstore-all) currently needs about 1TB, so half a TB should still be more than enough for just Europe ...

(I did all of Europe on a server with 2x240GB SSD, but that was some five years ago. I then switched to whole planet on that machine, using the osm2pgsql feature to put indexes and actual table data on different tablespaces, and only having indexes on SSD, but that was "living on the edge" as that was just barely fast enough to catch up with diffs to keep the DB up to date, taking over 20 hours to import a daily diff back then)

Hi Hartmut!

Thank you very much.

So my 512GB SSD should be enogh...
I'll build it in and start the import. I'll let the forum know the result!

Regards
Luca

Offline

#4 2022-01-25 19:02:16

lucabert
Member
From: Dresden
Registered: 2020-11-23
Posts: 73

Re: How big should be the disk?

Hi again,

I imported the data on a Test-Server I have in office with SSD.
Total space: 576GB... hmm

So my 512GB SSD is not enough...
You said that it is possible to split the data on more disks. Could you say me how? What are the pros? And what the contras?

Thanks
Luca

P.S.: I really can't believe... Last year I imported the data of Europe and I'm sure 512GB were enough... Did really the data growed so much?

Offline

#5 2022-01-25 19:10:59

lucabert
Member
From: Dresden
Registered: 2020-11-23
Posts: 73

Re: How big should be the disk?

Another question... I import the data so:

sudo -u _renderd osm2pgsql -d gis --create --slim  -G --hstore --tag-transform-script ~/src/osm-icao/osm-icao.lua \
  -C 8500 --number-processes 12 -S ~/src/osm-icao/osm-icao.style ~/data/europe-latest.osm.pbf

Since I found the declaration in an HowTo... Now I read the Man-Pages of osm2pgsql...
--slim stores temporary data in the database. Good if I don't have many RAM (currently the Server I'll use has 16GB)
But I see a --drop:

--drop
              Drop the slim mode tables from the database once the import is complete. This can greatly reduce the size of the database,
              as the slim mode tables typically  are the  same  size,  if not slightly bigger than the main tables. It does not, however, reduce
              the maximum spike of disk usage during import. It can furthermore increase the import speed, as no indices need to be
              created for the slim mode tables, which (depending on hardware) can nearly halve import time. Slim mode  tables
              however have to be persistent if you want to be able to update your database, as these tables are needed for diff processing.

So, do I understand correctly, that if I import with --slim --drop I'll get a reduced disk space after the import?
Is there a possibility to "drop" the temporary data after the import? Just to test it on the Test-Server I used...

Thanks
Luca

Offline

#6 2022-01-26 21:52:14

lucabert
Member
From: Dresden
Registered: 2020-11-23
Posts: 73

Re: How big should be the disk?

So, I installed the SSD...
Really fast is the import unfortunately not...

Processing: Node(2875001k 577.1k/s) Way(197k 1.60k/s) Relation(0 0.00/s)

I see with top:

 1038 _renderd  20   0   12,3g  11,9g   5764 S  30,2  76,6  84:14.32 osm2pgsql
 1043 postgres  20   0  215744 143940 140752 D  30,2   0,9  58:38.90 postgres

and with iotop:

Total DISK READ:        20.71 M/s | Total DISK WRITE:      1136.28 K/s
Current DISK READ:      20.70 M/s | Current DISK WRITE:     488.07 K/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
 1043 be/4 postgres   20.71 M/s    0.00 B/s  0.00 % 34.23 % postgres: 11/main: _renderd gis [local] idle in transaction
  595 be/4 postgres    0.00 B/s    0.00 B/s  0.00 %  0.16 % postgres: 11/main: background writer
 1040 be/4 postgres    0.00 B/s  518.57 K/s  0.00 %  0.00 % postgres: 11/main: _renderd gis [local] COPY
 1044 be/4 postgres    0.00 B/s  617.71 K/s  0.00 %  0.00 % postgres: 11/main: _renderd gis [local] COPY
    1 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % init
    2 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kthreadd]
    3 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcu_gp]
    4 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [rcu_par_gp]
    5 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/0:0-mm_percpu_wq]
    6 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/0:0H-kblockd]

I started the import with following parameters:

sudo -u _renderd osm2pgsql -d gis --disable-parallel-indexing --drop --unlogged --create --slim  -G \
  --hstore --tag-transform-script ~/src/osm-icao/osm-icao.lua -C 12000 --number-processes 12 -S ~/src/osm-icao/osm-icao.style \
  ~/data/europe-latest.osm.pbf

Any idea how to speed up the import?
On a test PC at office (even with SSD), the import speed was:

Processing: Node(2899215k 1635.2k/s) Way(351236k 13.23k/s) Relation(5950820 330.38/s)  parse time: 46331s

Maybe just pg_dump from this PC and import in my server?

Thank you very much for your suggestion!
Luca

Offline

Board footer

Powered by FluxBB