Get error on OSM file

Amilawelikala · November 11, 2019, 12:04pm

hi team,

I have developed my own OSM file with cobine series of .OSC files. So here is my ultimate file. https://drive.google.com/file/d/1S2oY0knRlLDyXarMXI-cJHc4S0MiuHni/view?usp=sharing I have used “OSMIUM” tool to integrate files.https://docs.osmcode.org/osmium/latest/osmium-merge.html. Using osmium tool I have generated my pbf file also. When I try to implement my own OSMNames it will generate error in here https://github.com/OSMNames/OSMNames/issues/190. So please give your help to solve the issue on this file. According to https://github.com/ImreSamu , he says that some nodes are duplicated again and again. If so please give your community help to solve this.

ImreSamu · November 11, 2019, 11:36pm

I have developed my own OSM file with combine series of .OSC files.
I have used “OSMIUM” tool to integrate files. … osmium-merge

please give more context.

Why do you need series of .OSC files? ( OSC == OSM Change files)
The latest OSM PBF is not perfect for you? This is the ideal for importing with imposm3;

The manual of Working with .osc (OSM change files): https://osmcode.org/osmium-tool/manual.html#working-with-change-files

imho: the “osmium merge” is not ok.

imho: you have to understand the osm file formats

https://osmcode.org/osmium-tool/manual.html#osm-file-formats-and-converting-between-them

Amilawelikala · November 13, 2019, 7:58am

I have build the file according to your suggestion. But after I run “Get started” instruction on OSMNames https://github.com/OSMNames/OSMNames it generate bellow error.What should I do to overcome this. I have checked that node ids are duplicated or not. According to “osmium check-refs abc.osm.pbf” it doesnot generate any errors and it prove that there are no duplicates. I cant image whats going on here?

[2019-11-13T08:47:54+01:00] 0:00:14 [step] Starting: Importing OSM data
[2019-11-13T08:47:54+01:00] 0:00:14 [step] Starting: Writing OSM data
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x575a96]

goroutine 8 [running]:
github.com/omniscale/imposm3/cache.(*coordsBunch).GetCoord(0x0, 0xffffffffffffffc7, 0x2, 0xc0001df1d0, 0x2)
	/go/src/github.com/omniscale/imposm3/cache/delta.go:27 +0x26
github.com/omniscale/imposm3/cache.(*DeltaCoordsCache).FillWay(0xc0045a23c0, 0xc006657d60, 0xc0001df180, 0x1)
	/go/src/github.com/omniscale/imposm3/cache/delta.go:187 +0x16d
github.com/omniscale/imposm3/writer.(*WayWriter).loop.func1(0xc006657d60, 0xc006657d60)
	/go/src/github.com/omniscale/imposm3/writer/ways.go:83 +0x56
github.com/omniscale/imposm3/writer.(*WayWriter).loop(0xc0022ee0a0)
	/go/src/github.com/omniscale/imposm3/writer/ways.go:98 +0x4de
created by github.com/omniscale/imposm3/writer.(*OsmElemWriter).Start
	/go/src/github.com/omniscale/imposm3/writer/writer.go:52 +0x85
Traceback (most recent call last):
  File "./run.py", line 30, in <module>
    run()
  File "./run.py", line 23, in run
    import_osm()
  File "/osmnames/osmnames/import_osm/import_osm.py", line 13, in import_osm
    import_pbf_file()
  File "/osmnames/osmnames/import_osm/import_osm.py", line 54, in import_pbf_file
    "-overwritecache",
  File "/osmnames/osmnames/logger.py", line 30, in logged_check_call
    check_call(parameters)
  File "/usr/lib/python3.7/subprocess.py", line 347, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['imposm', 'import', '-connection', 'postgis://osm@postgres/osm', '-mapping', '/osmnames/osmnames/import_osm/mapping.yml', '-dbschema-import', 'public', '-read', '/osmnames/data/import/abc.osm.pbf', '-write', '-overwritecache']' returned non-zero exit status 2.

ImreSamu · November 13, 2019, 10:29am

Remainder: … Please give me more context.

Why do you need series of .OSC files? ( OSC == OSM Change files)
The latest OSM PBF (“xxxxx-latest.osm.pbf”) is not perfect for you? This is the ideal data source for importing with imposm3;

Please help me to understand your problem.

please create minimal - baby steps - for replicating your problem.
( minimal data sizes; with steps - I can replicate in 5 minutes ; the ideal is the simple bash files … just need to run )

context for the other readers:

But after I run “Get started” instruction on OSMNames
https://github.com/OSMNames/OSMNames it generate bellow error.
…

for the OSMNames - just download any “xxxxx-latest.osm.pbf”

http://download.geofabrik.de/asia/bhutan-latest.osm.pbf ( it is a small size data, so testing is fast ) it is working for you?

again: Why do you need OSC file? what is the problem you want to solve with OSC files?
imho: you can skip your OSC processing ; and just use the “xxxxx-latest.osm.pbf” it is simple ; and it should work.

According to “osmium check-refs muve.osm.pbf” it
doesnot generate any errors and it prove that there are no duplicates.

what is the log output? It is just a simple warning;
“Node ID twice in input. Maybe you are using a history or change file?”
see my output: https://github.com/omniscale/imposm3/issues/210#issuecomment-541134725

Amilawelikala · November 14, 2019, 10:42am

hi team,
This is my research project. There are 4 members in my team. So they have developed series of .OSC files. They have developed 100+ .OSC files in different days. Now we want location search just like in google place search. I am going to use https://github.com/klokantech/osmnames-sphinxsearch to get that behavior. According to their documentation I have to implement “data.tsv” file. Therefore I use https://github.com/OSMNames/OSMNames this to implement data.tsv file. There I got above issue.

Here is the sh content I used to integrate

first I have to integrate all .osc file contents into a single file


max=85
finalName="Dilshan"
rm -rf $finalName/final/*
for i in `seq 1 $max`
do
    rm -rf $finalName/$i/$i.osc	
    osmium merge $finalName/$i/*.osc -o $finalName/$i/$i.osc
    cp $finalName/$i/$i.osc $finalName/final/$i.osc	
done
osmium merge $finalName/final/*.osc -o $finalName/final/$finalName.osc

There are 4 members, so each files has integrated using this code. At the end of execution of this code there are 4 .osc files.

Then I have use this code to integrate “abc.pbf” file


# run separate sh files to generate integrated osc and osm files

sh Dilshan/dilshan.sh
sh Niroshini/niroshini.sh
sh Rashini/rashini.sh
sh Tishan/tishan.sh

#remove all the contents inside final maps
rm -rf Final_maps/*

# download latest srilanka osm file
wget "https://download.geofabrik.de/asia/sri-lanka-latest.osm.pbf" -P /home/amila/projects/OSM/map_osms/osm-20190814T084416Z-001/osm/Final_maps

# move all the .osc files to the Final_maps folder
cp Dilshan/final/Dilshan.osc Final_maps/Dilshan.osc
cp Niroshini/final/Niroshini.osc Final_maps/Niroshini.osc
cp Rashini/final/Rashini.osc Final_maps/Rashini.osc
cp Tishan/final/Tishan.osc Final_maps/Tishan.osc

osmium apply-changes sri-lanka-latest.osm.pbf Final_maps/Dilshan.osc Final_maps/Niroshini.osc Final_maps/Rashini.osc Final_maps/Tishan.osc -o Final_maps/abc.osm.pbf

After I run this code I got “abc.pbf” file.

Here is the output I got after I run “osmium check-refs abc.osm.pbf”


amila@amila-viduranga:~/projects/OSM/map_osms/osm-20190814T084416Z-001/osm/Final_maps$ osmium check-refs abc.osm.pbf
[======================================================================] 100% 
There are 14262970 nodes, 1891886 ways, and 3183 relations in this file.
Nodes in ways missing: 286

ImreSamu · November 14, 2019, 12:27pm

thank you for the context …

They have developed 100+ .OSC files in different days.

be careful;

the change file contains only the differences; so the order of applying change files is important!
modifying same nodes/ways … → potential data integration conflict.
creating new nodes; ways … and it has a same node/way id; → data integration conflict …

see https://wiki.openstreetmap.org/wiki/OsmChange

Check if they used any “placeholders”; Because if the “-1” used multiple times; → data integration problems …

first I have to integrate all .osc file contents into a single file
osmium merge $finalName/$i/*.osc -o $finalName/$i/$i.osc

check the “osmium merge-changes --simplify” in the tutorial
https://osmcode.org/osmium-tool/manual.html#working-with-change-files

probably the order of the change files is important!
“The --simplify option usually makes sense: If an object changed multiple times, all intermediate versions will be removed and only the last version remains.”

see: https://docs.osmcode.org/osmium/latest/osmium-merge-changes.html

“Nodes in ways missing: 286”

data integration problem …
any broken ways (with missing nodes) - not will be imported to the database …

check the “–show-ids” options in the https://docs.osmcode.org/osmium/latest/osmium-check-refs.html

Because you are using the latest “sri-lanka-latest.osm.pbf” … maybe some of this nodes has been removed from the current OSM database. You have to debug …

ImreSamu · November 14, 2019, 12:41pm

You can check the duplicated nodes/ways with this command:

osmium cat muve.osm.pbf  -f opl | cut -d" " -f1 | uniq -d

Amilawelikala · November 20, 2019, 4:16am

hi team,

how to fix “Nodes in ways missing: 286” issue after execute

osmium check-refs abc.osm.pbf

command

ImreSamu · November 25, 2019, 6:52pm

how to fix “Nodes in ways missing: 286” issue after execute

Question:

Have you checked the output of

 osmium check-refs  --show-ids muve.osm.pbf

?
“–show-ids : Print all missing IDs to STDOUT. If you don’t specify this option, only a summary is shown.”

My suggestion:

If the missing IDs is a valid Historical OSM nodes … you can import the missing nodes from some older snapshot
- From this directory https://download.geofabrik.de/asia/ → From this snapshot https://download.geofabrik.de/asia/sri-lanka-191001.osm.pbf
- And check the

https://docs.osmcode.org/osmium/latest/osmium-getid.html

command.

If the missing nodes created by your team ; you have to ask them for the fix.

ps:

you can post the missing nodes list … so we can help in the check…

Amilawelikala · November 27, 2019, 1:56pm

Now I have bellow output

amila@amila-HP-ProBook-440-G5:~/Projects/osm/abc-maps/Final_maps$ osmium check-refs abc.osm.pbf 
[======================================================================] 100% 
There are 14563037 nodes, 1934442 ways, and 3202 relations in this file.
Nodes in ways missing: 0

But still I got the same issue.


[2019-11-27T14:44:34+01:00] 0:00:10 [step] Finished: Reading OSM data in 10.114401104s
[2019-11-27T14:44:34+01:00] 0:00:10 [step] Starting: Importing OSM data
[2019-11-27T14:44:34+01:00] 0:00:10 [step] Starting: Writing OSM data
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x575a96]

goroutine 145 [running]:
github.com/omniscale/imposm3/cache.(*coordsBunch).GetCoord(0x0, 0xffffffffffffffc2, 0x2, 0xc00432b540, 0x2)
	/go/src/github.com/omniscale/imposm3/cache/delta.go:27 +0x26
github.com/omniscale/imposm3/cache.(*DeltaCoordsCache).FillWay(0xc000a96240, 0xc0001f5180, 0xc00432b4f0, 0x1)
	/go/src/github.com/omniscale/imposm3/cache/delta.go:187 +0x16d
github.com/omniscale/imposm3/writer.(*WayWriter).loop.func1(0xc0001f5180, 0xc0001f5180)
	/go/src/github.com/omniscale/imposm3/writer/ways.go:83 +0x56
github.com/omniscale/imposm3/writer.(*WayWriter).loop(0xc00a4ce960)
	/go/src/github.com/omniscale/imposm3/writer/ways.go:98 +0x4de
created by github.com/omniscale/imposm3/writer.(*OsmElemWriter).Start
	/go/src/github.com/omniscale/imposm3/writer/writer.go:52 +0x85
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x575a96]

goroutine 143 [running]:
github.com/omniscale/imposm3/cache.(*coordsBunch).GetCoord(0x0, 0xffffffffffffffc8, 0x8, 0xc0000d5680, 0x8)
	/go/src/github.com/omniscale/imposm3/cache/delta.go:27 +0x26
github.com/omniscale/imposm3/cache.(*DeltaCoordsCache).FillWay(0xc000a96240, 0xc0001f52c0, 0xc000dc8910, 0x1)
	/go/src/github.com/omniscale/imposm3/cache/delta.go:187 +0x16d
github.com/omniscale/imposm3/writer.(*WayWriter).loop.func1(0xc0001f52c0, 0xc0001f52c0)
	/go/src/github.com/omniscale/imposm3/writer/ways.go:83 +0x56
github.com/omniscale/imposm3/writer.(*WayWriter).loop(0xc00a4ce960)
	/go/src/github.com/omniscale/imposm3/writer/ways.go:98 +0x4de
created by github.com/omniscale/imposm3/writer.(*OsmElemWriter).Start
	/go/src/github.com/omniscale/imposm3/writer/writer.go:52 +0x85
Traceback (most recent call last):
  File "./run.py", line 30, in <module>
    run()
  File "./run.py", line 23, in run
    import_osm()
  File "/osmnames/osmnames/import_osm/import_osm.py", line 13, in import_osm
    import_pbf_file()
  File "/osmnames/osmnames/import_osm/import_osm.py", line 54, in import_pbf_file
    "-overwritecache",
  File "/osmnames/osmnames/logger.py", line 30, in logged_check_call
    check_call(parameters)
  File "/usr/lib/python3.7/subprocess.py", line 347, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['imposm', 'import', '-connection', 'postgis://osm@postgres/osm', '-mapping', '/osmnames/osmnames/import_osm/mapping.yml', '-dbschema-import', 'public', '-read', '/osmnames/data/import/abc.osm.pbf', '-write', '-overwritecache']' returned non-zero exit status 2.

and here is the postgress log


postgres_1  | 2019-11-27T13:15:13.522331678Z 
postgres_1  | 2019-11-27T13:15:13.532252970Z 2019-11-27 13:15:13.532 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
postgres_1  | 2019-11-27T13:15:13.532269021Z 2019-11-27 13:15:13.532 UTC [1] LOG:  listening on IPv6 address "::", port 5432
postgres_1  | 2019-11-27T13:15:13.535637461Z 2019-11-27 13:15:13.535 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1  | 2019-11-27T13:15:13.547342910Z 2019-11-27 13:15:13.547 UTC [90] LOG:  database system was shut down at 2019-11-27 13:15:13 UTC
postgres_1  | 2019-11-27T13:15:13.551587740Z 2019-11-27 13:15:13.551 UTC [1] LOG:  database system is ready to accept connections
postgres_1  | 2019-11-27T13:44:49.631472230Z 2019-11-27 13:44:49.631 UTC [219] ERROR:  unexpected EOF on client connection with an open transaction
postgres_1  | 2019-11-27T13:44:49.631515069Z 2019-11-27 13:44:49.631 UTC [219] CONTEXT:  COPY osm_polygon, line 12923
postgres_1  | 2019-11-27T13:44:49.631521179Z 2019-11-27 13:44:49.631 UTC [219] STATEMENT:  COPY "public"."osm_polygon" ("osm_id", "geometry", "type", "imported_country_code", "name", "all_tags", "wikipedia", "wikidata", "admin_level") FROM STDIN
postgres_1  | 2019-11-27T13:44:49.631528279Z 2019-11-27 13:44:49.631 UTC [218] ERROR:  unexpected EOF on client connection with an open transaction
postgres_1  | 2019-11-27T13:44:49.631533578Z 2019-11-27 13:44:49.631 UTC [218] CONTEXT:  COPY osm_linestring, line 224754
postgres_1  | 2019-11-27T13:44:49.631538058Z 2019-11-27 13:44:49.631 UTC [218] STATEMENT:  COPY "public"."osm_linestring" ("osm_id", "geometry", "type", "name", "all_tags", "wikipedia", "wikidata", "admin_level") FROM STDIN
postgres_1  | 2019-11-27T13:44:49.631657886Z 2019-11-27 13:44:49.631 UTC [216] ERROR:  unexpected EOF on client connection with an open transaction
postgres_1  | 2019-11-27T13:44:49.631673675Z 2019-11-27 13:44:49.631 UTC [216] CONTEXT:  COPY osm_relation, line 1
postgres_1  | 2019-11-27T13:44:49.631679424Z 2019-11-27 13:44:49.631 UTC [216] STATEMENT:  COPY "public"."osm_relation" ("osm_id", "name", "type", "street") FROM STDIN
postgres_1  | 2019-11-27T13:44:49.631685575Z 2019-11-27 13:44:49.631 UTC [217] ERROR:  unexpected EOF on client connection with an open transaction
postgres_1  | 2019-11-27T13:44:49.631691244Z 2019-11-27 13:44:49.631 UTC [217] CONTEXT:  COPY osm_relation_member, line 4817
postgres_1  | 2019-11-27T13:44:49.631696174Z 2019-11-27 13:44:49.631 UTC [217] STATEMENT:  COPY "public"."osm_relation_member" ("osm_id", "member_id", "role", "member_type", "relation_type", "name") FROM STDIN
postgres_1  | 2019-11-27T13:44:49.631701824Z 2019-11-27 13:44:49.631 UTC [213] ERROR:  unexpected EOF on client connection with an open transaction
postgres_1  | 2019-11-27T13:44:49.631706574Z 2019-11-27 13:44:49.631 UTC [213] CONTEXT:  COPY osm_point, line 1
postgres_1  | 2019-11-27T13:44:49.631710984Z 2019-11-27 13:44:49.631 UTC [213] STATEMENT:  COPY "public"."osm_point" ("osm_id", "geometry", "type", "name", "all_tags", "wikipedia", "wikidata", "admin_level") FROM STDIN
postgres_1  | 2019-11-27T13:44:49.631717894Z 2019-11-27 13:44:49.631 UTC [215] ERROR:  unexpected EOF on client connection with an open transaction
postgres_1  | 2019-11-27T13:44:49.631723443Z 2019-11-27 13:44:49.631 UTC [215] CONTEXT:  COPY osm_housenumber, line 886
postgres_1  | 2019-11-27T13:44:49.631728813Z 2019-11-27 13:44:49.631 UTC [215] STATEMENT:  COPY "public"."osm_housenumber" ("osm_id", "geometry", "name", "street", "housenumber", "place") FROM STDIN
postgres_1  | 2019-11-27T13:44:49.632155382Z 2019-11-27 13:44:49.632 UTC [216] FATAL:  terminating connection because protocol synchronization was lost
postgres_1  | 2019-11-27T13:44:49.632168002Z 2019-11-27 13:44:49.632 UTC [213] FATAL:  terminating connection because protocol synchronization was lost
postgres_1  | 2019-11-27T13:44:49.632182122Z 2019-11-27 13:44:49.632 UTC [216] LOG:  could not send data to client: Broken pipe
postgres_1  | 2019-11-27T13:44:49.632186821Z 2019-11-27 13:44:49.632 UTC [213] LOG:  could not send data to client: Broken pipe
postgres_1  | 2019-11-27T13:44:49.632235410Z 2019-11-27 13:44:49.632 UTC [217] FATAL:  terminating connection because protocol synchronization was lost
postgres_1  | 2019-11-27T13:44:49.632243541Z 2019-11-27 13:44:49.632 UTC [215] FATAL:  terminating connection because protocol synchronization was lost
postgres_1  | 2019-11-27T13:44:49.632270420Z 2019-11-27 13:44:49.632 UTC [217] LOG:  could not send data to client: Broken pipe
postgres_1  | 2019-11-27T13:44:49.632276370Z 2019-11-27 13:44:49.632 UTC [215] LOG:  could not send data to client: Broken pipe
postgres_1  | 2019-11-27T13:44:49.637007478Z 2019-11-27 13:44:49.636 UTC [219] FATAL:  terminating connection because protocol synchronization was lost
postgres_1  | 2019-11-27T13:44:49.637032058Z 2019-11-27 13:44:49.636 UTC [219] LOG:  could not send data to client: Broken pipe
postgres_1  | 2019-11-27T13:44:49.658300413Z 2019-11-27 13:44:49.658 UTC [218] FATAL:  terminating connection because protocol synchronization was lost
postgres_1  | 2019-11-27T13:44:49.658321043Z 2019-11-27 13:44:49.658 UTC [218] LOG:  could not send data to client: Broken pipe

I cant image whats is going on now. Are there any solution to solve this issue (config change at postgress or any other configuration change)

You can get the file by clicking bellow link
https://drive.google.com/file/d/1O13RGt5xzvVYKe_y0nRjCqPOGSL-HxBN/view?usp=sharing

ImreSamu · November 27, 2019, 8:23pm

You have a lot of negative node/way ID (-1,-2,-3,-4)
and IMPOSM3 don’t like it. ( And the OpenStreetMap data contains only positive ID )

test:

osmium cat muve.osm.pbf -f opl | head -n 4
n-1 v0 dV c0 t i0 u Toffice=company,name=Nature%20%Efficient%20%Electronics%20%Lanka,addr:housenumber=792,addr:street=Colombo%20%-%20%Kandy%20%Road,addr:city=Wedamulla x79.903367 y6.9676891
n-2 v0 dV c0 t i0 u Tpublic_transport=platform,bus=yes,highway=bus_stop,name=Jambu%20%Gas%20%Handiya%20%Bus%20%Stop,network=Delgoda%20%-%20%Kirindiwela%20%Rd,operator=dilshan x80.0450168 y6.9695425
n-3 v0 dV c0 t i0 u Tamenity=cinema,name=Studio%20%LK,addr:housenumber=403/7,addr:street=Hokandara-Thalawathugoda%20%Road,addr:city=Hokandara x79.9517593 y6.8818554
n-4 v0 dV c0 t i0 u T x80.1110064 y7.0595614

Probably you have to re-check all osc -files … and convert negative iDs to intervall … avoiding -1,-2,-3 … collisions.

if this change file independent files … you can easily convert to distinct intervall ( by a python script? or a sed ? )

1.osc → ID start from -100 …
2.osc → ID start from -200 …
3.osc → ID start from -200 …

And after just a simple renumbering … and it should work …

osmium renumber muve.osm.pbf -O -o muve_renumbered.osm.pbf

test … node ID-s


root@a5a3d7f2eeb1:# osmium cat muve_renumbered.osm.pbf -f opl | head -n 4
n1 v0 dV c0 t i0 u Toffice=company,name=Nature%20%Efficient%20%Electronics%20%Lanka,addr:housenumber=792,addr:street=Colombo%20%-%20%Kandy%20%Road,addr:city=Wedamulla x79.903367 y6.9676891
n2 v0 dV c0 t i0 u Tpublic_transport=platform,bus=yes,highway=bus_stop,name=Jambu%20%Gas%20%Handiya%20%Bus%20%Stop,network=Delgoda%20%-%20%Kirindiwela%20%Rd,operator=dilshan x80.0450168 y6.9695425
n3 v0 dV c0 t i0 u Tamenity=cinema,name=Studio%20%LK,addr:housenumber=403/7,addr:street=Hokandara-Thalawathugoda%20%Road,addr:city=Hokandara x79.9517593 y6.8818554
n4 v0 dV c0 t i0 u T x80.1110064 y7.0595614

ImreSamu · December 15, 2019, 5:30pm

@Amilawelikala :
Any news?

Amilawelikala · January 3, 2020, 4:54am

hi @imreSamu,

Sorry for late to response. Thank you for your great support. Now I have created valid pbf file without any issue. Thank you very much.