OpenStreetMap Forum

Hupa

Member

Registered: 2020-08-11

Posts: 5

nominatim search index stalls on build

Hi,
I am building a private nominatim server. I have access to a CentOS 7 server with 128GB RAM and 10 CPU's, using PostgeSQL 11, Nominatim 3.5.1 and currently loading planet.
After a while and finishing the indexing. See below
=====

INFO: Done 104662026 in 36629 @ 2857.292 per second - rank 30 ETA (seconds): 10.08
INFO: Done 104664883 in 36631 @ 2857.269 per second - rank 30 ETA (seconds): 9.08
INFO: Done 104667740 in 36632 @ 2857.248 per second - rank 30 ETA (seconds): 8.08
INFO: Done 104670597 in 36633 @ 2857.229 per second - rank 30 ETA (seconds): 7.08
INFO: Done 104673454 in 36634 @ 2857.229 per second - rank 30 ETA (seconds): 6.08
INFO: Done 104676311 in 36635 @ 2857.224 per second - rank 30 ETA (seconds): 5.08
INFO: Done 104679168 in 36636 @ 2857.216 per second - rank 30 ETA (seconds): 4.08
INFO: Done 104682025 in 36637 @ 2857.205 per second - rank 30 ETA (seconds): 3.08
INFO: Done 104684882 in 36639 @ 2857.183 per second - rank 30 ETA (seconds): 2.08
INFO: Done 104687739 in 36882 @ 2838.424 per second - rank 30 ETA (seconds): 1.09
INFO: Done 104690577 in 36883 @ 2838.410 per second - rank 30 ETA (seconds): 0.09
WARNING: Done 104690846/104690837 in 36883 @ 2838.395 per second - FINISHED rank 30

2020-08-11 17:00:16 == Index postcodes
2020-08-11 17:41:21 == Create Search indices
=======
At that point the machines CPU consumption goes down to 0% to stay there.
Reading the manual I tried
./utils/setup.php --index --create-search-indices --create-country-names --index-noanalyse

This processes a bunch and then says:
=======
WARNING: Starting interpolation lines (location_property_osmline)
WARNING: Done 0/0 in 0 @ 0.000 per second - FINISHED interpolation lines (location_property_osmline)

WARNING: Starting rank 30
WARNING: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 30

2020-08-11 18:20:46 == Index postcodes
2020-08-11 18:21:37 == Create Search indices
ERROR:  relation "idx_word_word_id" already exists
ERROR: pgsql returned with error code (3)
string(34) "pgsql returned with error code (3)"
=======

Another attempt then is:
=======
./utils/setup.php --create-search-indices
2020-08-11 18:32:25 == module path: /srv/nominatim/build/module
2020-08-11 18:32:25 == Create Search indices
ERROR:  relation "idx_word_word_id" already exists
ERROR: pgsql returned with error code (3)
string(34) "pgsql returned with error code (3)"
=======

Not much luck.
1) is there a reason why creating search indexes stalls?
2) how can I get ride of the relation "idx_word_word_id"? so the system might restart (not feeling like dropping the database and reimporting for another 36 hours or so.
3) does anybody know where I should be able to see at logging what is causing the process to stall?

Thanks in advance

Hupa

Member

Registered: 2020-08-11

Posts: 5

Re: nominatim search index stalls on build

So, one small step further.
I have found the way to remove the relationship and was able to run indexes again.
Again the CPU goes to zero and nothing happens. So the question remains:
- What is the reason for the process to come to a halt?
- Is there a logging that might show me what is causing this premature halt?

Thanks in advance

Hupa

Member

Registered: 2020-08-11

Posts: 5

Re: nominatim search index stalls on build

Hi all,
some more testing and I have found the culprit.
The systems seems to run into a deadlock situation on the database.
I have pulled screenshots but I do not know how to paste them in here.
Anyway While the system tries to do "CREATE INDEX idx_place_addressline_address_place... there is also a "autovacuum: VACUUM public.place_addressline running, both referring to the same table. As the Autovacuum doesn't seem to stop, but holds an exclusive look on the table, the create index waits (indefinitely).
When I terminate the Autovacuum the Create index resumes. The Autovacuum (which is why it is auto I assume) is restarted immediately, but is now waiting for the completion of the CREATE INDEX. Once that is done, the next CREATE INDEX will find its table to be locked. So I needed a number of "terminate process" before it all completed.
I suppose one of the developers should have a look how this behavior was introduced.

Thanks