OpenStreetMap Forum

The Free Wiki World Map

You are not logged in.

#1 2015-10-24 21:40:01

wambacher
Member
From: Schlangenbad/Wambach, Germany
Registered: 2009-12-16
Posts: 16,311
Website

mexico import. International and other boundaries mainly USA destroid.

Hi, the team doing the current imports in mexico "killed" the boundary of the USA and Mexico near Rio Grande. They added "their" new boundaries and did not replace/split/change the original State boundaries of USA.

All missings in USA + the AL2 in Mexico and the USA are results of their Import.

See: http://osm.wno-edv-service.de/index.php … 2015-10-24

and see changeset https://www.openstreetmap.org/changeset/34817435

i sent them mails about 14  hours ago to fix that but they

- did not response
- did not fix the errors
- are still mapping and changing other boundaries.

Please get someone and ask him to stop this.

tn_usa_mex.png

regards
walter

Last edited by wambacher (2015-10-24 21:51:45)

Offline

#2 2015-10-25 03:27:14

JeSe-MX
Moderator
From: Santiago de Querétaro
Registered: 2013-09-27
Posts: 179
Website

Re: mexico import. International and other boundaries mainly USA destroid.

Hi @wambacher, Welcome to subforum of México.
[sorry, I know little english; I used google translator]

A few minutes ago, I sent email to the list Talk-MX[1][2], and I gave notice of your post here in the forum.
In Talk-MX I read something on imports, so hopefully soon get in touch with you here in the forum.
Thank you for coming to this forum to present your concerns.
Greetings from Mexico.

[1] https://lists.openstreetmap.org/listinfo/talk-mx
[2] https://lists.openstreetmap.org/piperma … 00340.html

Offline

#3 2015-10-25 06:22:24

Andresuco
Member
From: México
Registered: 2014-10-08
Posts: 30
Website

Re: mexico import. International and other boundaries mainly USA destroid.

Hi Wambacher,

Any relation from the boundaries that is missing will be reinstated once we finish the import, due to some glitches in our importing script some relation segments might be missing during the period of the import but we've backed up everything before proceeding with the import so that any data before the import is re-uploaded after the import. 

Thanks,
Andres

Offline

#4 2015-10-25 11:21:57

wambacher
Member
From: Schlangenbad/Wambach, Germany
Registered: 2009-12-16
Posts: 16,311
Website

Re: mexico import. International and other boundaries mainly USA destroid.

Andresuco wrote:

Hi Wambacher,

Any relation from the boundaries that is missing will be reinstated once we finish the import, due to some glitches in our importing script some relation segments might be missing during the period of the import but we've backed up everything before proceeding with the import so that any data before the import is re-uploaded after the import. 

Thanks,
Andres

Andrews, you and your guys are doing a good job, but the edits at the american boundary are critical!!! There are missing, duplicated and not connectetd Segments. Mainly you did not break the "new" segments and did not connect the american segments to it. This can not be fixed by "restoring saved segments", that has to be done manually and as soon as possible.

If you are doing that in your own country, this may be acceptable (by your local community),  but you killed about a dozend USA-Boundaries which must be fixed as soon as possible.

See list: http://osm.wno-edv-service.de/index.php … 2015-10-23 (scroll down and display 50 of them))

Current situation: https://osm.wno-edv-service.de/boundari … 27_1521463

walter

btw: When are you going to "kill" Guatemala and Belize too?

Offline

#5 2015-10-25 14:28:15

wambacher
Member
From: Schlangenbad/Wambach, Germany
Registered: 2009-12-16
Posts: 16,311
Website

Re: mexico import. International and other boundaries mainly USA destroid.

Hi Andres,

somebody told me you wrote at talk-mx something like. "no problem because you don't see it on the map".
That is totally wrong! You don't see it on openstreetmap.org but the DATA is damaged. Every application using live data or downloading that data will get wrong data and will have big problems. Routing software like navigation software too.

OSM is a database - not a Map. Somehting you should have heard many times before.

Regards
walter

Offline

#6 2015-10-25 17:05:55

Andresuco
Member
From: México
Registered: 2014-10-08
Posts: 30
Website

Re: mexico import. International and other boundaries mainly USA destroid.

Hi Wambacher,

We're aware of the issue, any valid relation that was affected will be fixed. The data is not damaged since we backed it up before and we will reinstate whatever went wrong (even if that needs to be done manually).

Thanks,
Andres

Offline

#7 2015-10-25 21:45:56

JeSe-MX
Moderator
From: Santiago de Querétaro
Registered: 2013-09-27
Posts: 179
Website

Re: mexico import. International and other boundaries mainly USA destroid.

I see that exist a problem with the imports made by Andrés.
In Talk-MX exist also a problem about who authorized the job of import.[1][2].

For the moment, I think more convinient is stop the import, and revert the changes.

[1] https://lists.openstreetmap.org/piperma … 00320.html
[2] https://lists.openstreetmap.org/piperma … 00344.html

Offline

#8 2015-10-26 09:21:02

gormo
Member
Registered: 2013-08-01
Posts: 2,015
Website

Re: mexico import. International and other boundaries mainly USA destroid.

Andresuco wrote:

We're aware of the issue, any valid relation that was affected will be fixed. The data is not damaged since we backed it up before and we will reinstate whatever went wrong (even if that needs to be done manually).

Wambacher meant the live data in the OSM database is damaged. When I download border relations now directly from the OSM API, they will be broken. And that is bad.


OSM hat nicht das Ziel bis Ende des Monats einen vollständigen Datensatz der Welt zu enthalten.
(nach S.W.) - Aber weil die Welt vielfältig ist, weil sie auch im Detail interessant ist, mag ich genaue Karten (nach C.)

Offline

#9 2015-10-26 11:09:50

wambacher
Member
From: Schlangenbad/Wambach, Germany
Registered: 2009-12-16
Posts: 16,311
Website

Re: mexico import. International and other boundaries mainly USA destroid.

USA(2) fixed.  11:09
Mexico(2) fixed: 11:19

Arizona(4) 11:21
Texas(4) 11:25

Starr County, Hildago County, Maverick County, Hildago County (al6) fixed. 11:46

Laredo(8) 12:11
El Cenizo(8) 12:34
Roma(8) 12:54
Escobares(8) 12:57
Mission(8) 13:07
Brownsville(8) 15:50 (hard job)

fixes done.

Last edited by wambacher (2015-10-26 15:50:49)

Offline

#10 2015-10-26 15:56:12

wambacher
Member
From: Schlangenbad/Wambach, Germany
Registered: 2009-12-16
Posts: 16,311
Website

Re: mexico import. International and other boundaries mainly USA destroid.

Hi, i fixed all damaged boundaries of the USA. The mexican site should be fixed by the your guys.

WARNING: Don't re-import "your" boundaries but fix them. Connect your AL4/Al6 and AL8 to the fixed outer boundaries of mexico and the Usa. If you are going to re-import those borders or keep on doing the same stuff at other areas, i'll call DWG to revert your changes ASAP. That is not a joke.

regards
walter

EDIT: Same situation at least in Juárez(6). Old correct AL2 of Mexico and USA overlayed by new imported AL6 of Juárez.
Guadelupe, Ojinaga too. down the whole Rio Grande.

What are you planning to do?

Last edited by wambacher (2015-10-26 16:29:42)

Offline

#11 2015-10-27 08:22:53

Andresuco
Member
From: México
Registered: 2014-10-08
Posts: 30
Website

Re: mexico import. International and other boundaries mainly USA destroid.

Hi Wambacher,

Our team validated all the relations in the MX-US border for admin_level=6, we don't see any issues for that specific level for US counties in the border with Mexico anymore due to most of the fixes you made. (There's an issue with Cochise county*, whose polygon is not closed but it was not caused by our import.)

California:
http://www.openstreetmap.org/relation/396482
http://www.openstreetmap.org/relation/396515 

Arizona:
http://www.openstreetmap.org/relation/1832216
http://www.openstreetmap.org/relation/1832206
http://www.openstreetmap.org/relation/1832210
http://www.openstreetmap.org/relation/1832184*

New Mexico
http://www.openstreetmap.org/relation/1832196
http://www.openstreetmap.org/relation/1832200
http://www.openstreetmap.org/relation/1832188

Texas
http://www.openstreetmap.org/relation/1812640
http://www.openstreetmap.org/relation/1812578
http://www.openstreetmap.org/relation/1812647
http://www.openstreetmap.org/relation/1812650
http://www.openstreetmap.org/relation/1814604
http://www.openstreetmap.org/relation/1814620
http://www.openstreetmap.org/relation/1814625
http://www.openstreetmap.org/relation/1827036
http://www.openstreetmap.org/relation/1827037
http://www.openstreetmap.org/relation/1827177
http://www.openstreetmap.org/relation/1827178
http://www.openstreetmap.org/relation/1828256
http://www.openstreetmap.org/relation/1828367
http://www.openstreetmap.org/relation/1828370

Now for the MX boundary data in the MX-US border we see some issues with the following relations probably after your fixes:
http://www.openstreetmap.org/relation/5605832
http://www.openstreetmap.org/relation/5605610
http://www.openstreetmap.org/relation/5606039
http://www.openstreetmap.org/relation/5606038
http://www.openstreetmap.org/relation/5605513
http://www.openstreetmap.org/relation/5605844
http://www.openstreetmap.org/relation/5605832

We're going to fix these today.

Thanks,
Andres

Offline

#12 2015-10-27 12:54:09

baditaflorin
Member
Registered: 2015-10-27
Posts: 3

Re: mexico import. International and other boundaries mainly USA destroid.

The last 7 MX-US border boundary data are now fixed. I will use your tool to see if there are other parts of mexico that have some issues

Offline

#13 2015-10-27 13:13:58

baditaflorin
Member
Registered: 2015-10-27
Posts: 3

Re: mexico import. International and other boundaries mainly USA destroid.

I hope this post will help explain a little the method that we had chosen and some of the technique details that forced us to move from the initial proposed solution in some parts.
As always, this is a learning experiencing.
For the future we will have a better check mechanism, 4 persons where responsible for the relinking of the al4 al6 , with vary degree of experience. I will personally check what my colleagues had done and see if there are other parts that need cleaning.

There are still some things that will need to be done, to re-add the wikipedia tags to the relations where we had deleted

https://www.openstreetmap.org/user/badi … iary/36188

Offline

#14 2015-10-27 19:03:45

wambacher
Member
From: Schlangenbad/Wambach, Germany
Registered: 2009-12-16
Posts: 16,311
Website

Re: mexico import. International and other boundaries mainly USA destroid.

Hi Andres,

sorry for the late response.

Yes, i removed some border-lines from the new mexican boundary relations but i think, your team has fixed them. It was much easier for me to concentrate on one site instead of fixing all.

Regards
walter

Offline

#15 2015-10-27 19:07:12

wambacher
Member
From: Schlangenbad/Wambach, Germany
Registered: 2009-12-16
Posts: 16,311
Website

Re: mexico import. International and other boundaries mainly USA destroid.

baditaflorin wrote:

The last 7 MX-US border boundary data are now fixed. I will use your tool to see if there are other parts of mexico that have some issues

Great, but be aware: The Job is fetching the new osm-data about midnight german time.  so you'll see the new situation on the next morning.

Offline

#16 2015-10-27 19:23:18

wambacher
Member
From: Schlangenbad/Wambach, Germany
Registered: 2009-12-16
Posts: 16,311
Website

Re: mexico import. International and other boundaries mainly USA destroid.

baditaflorin wrote:

I hope this post will help explain a little the method that we had chosen and some of the technique details that forced us to move from the initial proposed solution in some parts.
As always, this is a learning experiencing.
For the future we will have a better check mechanism, 4 persons where responsible for the relinking of the al4 al6 , with vary degree of experience. I will personally check what my colleagues had done and see if there are other parts that need cleaning.

There are still some things that will need to be done, to re-add the wikipedia tags to the relations where we had deleted

https://www.openstreetmap.org/user/badi … iary/36188


Hi florin, good work - after some trouble.

The next steps should be the north western part of mex. some newly added AL6 and may be AL4 are still overlapping the common al2 of usa/mexico.

Starting from here http://www.openstreetmap.org/#map=16/28.5588/-100.4089 going west there are two overlapping  border lines.And you can see them on the map wink

I would start a "virtual jurney" using Josm, go to the west and fix one problem after another. Removing overlapping mexican al4/al6 segments and connect the remains to the al2.

regards
walter

Offline

#17 2015-10-28 00:50:43

baditaflorin
Member
Registered: 2015-10-27
Posts: 3

Re: mexico import. International and other boundaries mainly USA destroid.

Checked all the mexico admin_level=4 boundaries that we had added, fixed where there was something not connected
the relation analyzer should update soon, the josm relation manager where showing a beautiful circle for the next provinces in mexico

regards
florin

INEGI State ID    State Name    Validation (after import)    validation link
17    Morelos        http://ra.osmsurround.org/analyzeRelati … noCache=on
13    Hidalgo        http://ra.osmsurround.org/analyzeRelati … noCache=on
19    Nuevo León        http://ra.osmsurround.org/analyzeRelati … noCache=on
5    Coahuila de Zaragoza        http://ra.osmsurround.org/analyzeRelati … noCache=on
8    Chihuahua        http://ra.osmsurround.org/analyzeRelati … noCache=on
26    Sonora        http://ra.osmsurround.org/analyzeRelati … noCache=on
16    Michoacán de Ocampo        http://ra.osmsurround.org/analyzeRelati … noCache=on
22    Querétaro        http://ra.osmsurround.org/analyzeRelati … noCache=on
11    Guanajuato        http://ra.osmsurround.org/analyzeRelati … noCache=on
14    Jalisco        http://ra.osmsurround.org/analyzeRelati … noCache=on
6    Colima        http://ra.osmsurround.org/analyzeRelati … noCache=on
32    Zacatecas        http://ra.osmsurround.org/analyzeRelati … noCache=on
10    Durango        http://ra.osmsurround.org/analyzeRelati … noCache=on
28    Tamaulipas        http://ra.osmsurround.org/analyzeRelati … noCache=on
12    Guerrero        http://ra.osmsurround.org/analyzeRelati … noCache=on
25    Sinaloa        http://ra.osmsurround.org/analyzeRelati … noCache=on
18    Nayarit        http://ra.osmsurround.org/analyzeRelati … noCache=on
2    Baja California        http://ra.osmsurround.org/analyzeRelati … noCache=on
3    Baja California Sur        http://ra.osmsurround.org/analyzeRelati … noCache=on
1    Aguascalientes        http://ra.osmsurround.org/analyzeRelati … noCache=on
24    San Luis Potosí        http://ra.osmsurround.org/analyzeRelati … noCache=on

Offline

#18 2015-10-28 01:40:20

muralito
Member
Registered: 2012-09-04
Posts: 1,753

Re: mexico import. International and other boundaries mainly USA destroid.

Creo, al igual que wambacher que los limites son algo crítico, tanto los de nivel 2 para paises, como los de nivel 8 para las poblaciones.

No quiero parecer autoritario sino resaltar la importancia del asunto, pero a mi criterio, debiera ser, como comunidad, prioritario solucionarlos y asimilar los aprendizajes necesarios para tratar de evitar que la situacion se repita, tanto en un mejor QA de los imports, como en una mas rapida atencion/respuesta ante una advertencia de (posibles) problemas en un changeset que modifique elementos criticos.

Este tipo de ediciones debieran manejarse como algo transaccional, el mismo changeset modifica un limite, quizas mejorable pero coherente y sano, por otro actualizado mejor tambien sano, todo entero, en un mismo changeset. De esa forma los limites siempre se mantendrian idealmente sanos.

Tambien hubo algunos problemas con la linea costera en la zona del HOT por Patricia, pero eso creo que no fue por los imports sino por la cantidad de colaboradores editando simultaneamente en la zona.

Saludos,
M.

Last edited by muralito (2015-10-28 01:43:25)

Offline

#19 2015-10-28 03:40:47

Andresuco
Member
From: México
Registered: 2014-10-08
Posts: 30
Website

Re: mexico import. International and other boundaries mainly USA destroid.

Hi Muralito,

You're right in the sense that these matters should be taken with care, the problem was that as we we're doing this import, hurricane Patricia hit and all of our team and resources (including servers) were put right away in the service of the Humanitarian OpenstreetMap team. At that moment we thought it was much more prioritary to help the HOT team with the resources we were dedicating to the boundaries import because the risk of this event being catastrophic. We don't regret that decision but we didn't plan for the unexpected, and we might have been distracted during the whole weekend with such activities, hence causing some mishaps during the boundaries import.

We're on the track of fixing any issues that might have arised during the import and welcome any input based on clear evidence of an error in our side as the one expressed on this thread.

Offline

#20 2015-10-28 15:58:59

muralito
Member
Registered: 2012-09-04
Posts: 1,753

Re: mexico import. International and other boundaries mainly USA destroid.

Si Andresuco, de las cosas que valoro de OSM es que cada grupo o individuo tenga la libertad para definir sus prioridades y trabajar en base a ellas, por eso agradezco tu explicacion, y de ninguna forma las cuestiono. Es entendible perfectamente que con la catastrofe natural las prioridades se hayan modificado.

Aunque no se refleje en el modelo de datos de OSM (porque es bien plano), hay elementos que son mas importantes que otros, como los limites o carreteras de mayor jerarquía, y es importante su continuidad para multiples consumidores del dato.
Por suerte lo inesperado no siempre es una catastre climática como la que sufrieron, pueden ser problemas personales de diversa indole, por eso es importante tratar de que los changesets sobre ciertos tipos de elementos cambien los mismos de un estado correcto a otro correcto.

Si fuera posible que el conjunto de cambios necesarios para una importación grande cumplan con esa propiedad de que cada cambio intermedio vaya dejando todo corecto, creo que eso le agrega mucho valor a una importacion, y el proceso total de la importacion puede demorarse mas tiempo que no afectaria demasiado.

Offline

#21 2015-10-30 23:42:23

wambacher
Member
From: Schlangenbad/Wambach, Germany
Registered: 2009-12-16
Posts: 16,311
Website

Re: mexico import. International and other boundaries mainly USA destroid.

Hi, i added "Select next admin level" in the context menu of my "Boundaries Map".

tn_boundaries_37_1.png

which gives:

tn_boundaries_37.png

It was a request by the user mexico_boundaries_import:

mexico_boundaries_import wrote:

Regarding your tool, it there such a posibility to select just the next level for a country.

For example, if i am at admin level 2, select just all admin_level=4 that are inside the polygon

if i select one admin_level=4 , select all admin_level=6 inside of the admin_level=4 boundary

have fun
walter/germany

Offline

#22 2015-11-09 00:47:47

wambacher
Member
From: Schlangenbad/Wambach, Germany
Registered: 2009-12-16
Posts: 16,311
Website

Re: mexico import. International and other boundaries mainly USA destroid.

Hi,

after two weeks it seems the big problems are gone smile

But the boundary at the rio grande is still "not the very best". There are still many AL6 of mexico overlapping the international AL2.

it starts here https://www.openstreetmap.org/node/3796 … -102.87108
and goes up to https://www.openstreetmap.org/way/37622 … /-100.4088

If you tell me "fix it" i'll will fix it using the international boundary. Just one hour of work for me and it's done.

Regards
walter

btw: fixed 7 unclosed AL6 in Tamaulipas(4). see CS https://www.openstreetmap.org/changeset/35181208

Last edited by wambacher (2015-11-09 00:56:39)

Offline

#23 2015-11-09 01:00:08

Andresuco
Member
From: México
Registered: 2014-10-08
Posts: 30
Website

Re: mexico import. International and other boundaries mainly USA destroid.

Hi Walter,
Don't worry we will fix it. Florin, please take a look at this and add it to our to fix list.

Thanks!

Offline

#24 2015-11-09 01:05:24

wambacher
Member
From: Schlangenbad/Wambach, Germany
Registered: 2009-12-16
Posts: 16,311
Website

Re: mexico import. International and other boundaries mainly USA destroid.

Andresuco wrote:

Hi Walter,
Don't worry we will fix it. Florin, please take a look at this and add it to our to fix list.

Thanks!

no problem

regards
walter

Offline

Board footer

Powered by FluxBB