mexico import. International and other boundaries mainly USA destroid.

Hi @wambacher, Welcome to subforum of México.
[sorry, I know little english; I used google translator]

A few minutes ago, I sent email to the list Talk-MX[1][2], and I gave notice of your post here in the forum.
In Talk-MX I read something on imports, so hopefully soon get in touch with you here in the forum.
Thank you for coming to this forum to present your concerns.
Greetings from Mexico.

[1] https://lists.openstreetmap.org/listinfo/talk-mx
[2] https://lists.openstreetmap.org/pipermail/talk-mx/2015-October/000340.html

Hi Wambacher,

Any relation from the boundaries that is missing will be reinstated once we finish the import, due to some glitches in our importing script some relation segments might be missing during the period of the import but we’ve backed up everything before proceeding with the import so that any data before the import is re-uploaded after the import.

Thanks,
Andres

Andrews, you and your guys are doing a good job, but the edits at the american boundary are critical!!! There are missing, duplicated and not connectetd Segments. Mainly you did not break the “new” segments and did not connect the american segments to it. This can not be fixed by “restoring saved segments”, that has to be done manually and as soon as possible.

If you are doing that in your own country, this may be acceptable (by your local community), but you killed about a dozend USA-Boundaries which must be fixed as soon as possible.

See list: http://osm.wno-edv-service.de/index.php/10-osm-reports/356-countries-compare-2015-10-23 (scroll down and display 50 of them))

Current situation: https://osm.wno-edv-service.de/boundaries/?zoom=3&lat=15.42845&lon=-91.51723&layers=0BT&selected=114686_148838_287827_1521463

walter

btw: When are you going to “kill” Guatemala and Belize too?

Hi Andres,

somebody told me you wrote at talk-mx something like. “no problem because you don’t see it on the map”.
That is totally wrong! You don’t see it on openstreetmap.org but the DATA is damaged. Every application using live data or downloading that data will get wrong data and will have big problems. Routing software like navigation software too.

OSM is a database - not a Map. Somehting you should have heard many times before.

Regards
walter

Hi Wambacher,

We’re aware of the issue, any valid relation that was affected will be fixed. The data is not damaged since we backed it up before and we will reinstate whatever went wrong (even if that needs to be done manually).

Thanks,
Andres

I see that exist a problem with the imports made by Andrés.
In Talk-MX exist also a problem about who authorized the job of import.[1][2].

For the moment, I think more convinient is stop the import, and revert the changes.

[1] https://lists.openstreetmap.org/pipermail/talk-mx/2015-October/000320.html
[2] https://lists.openstreetmap.org/pipermail/talk-mx/2015-October/000344.html

Wambacher meant the live data in the OSM database is damaged. When I download border relations now directly from the OSM API, they will be broken. And that is bad.

USA(2) fixed. 11:09
Mexico(2) fixed: 11:19

Arizona(4) 11:21
Texas(4) 11:25

Starr County, Hildago County, Maverick County, Hildago County (al6) fixed. 11:46

Laredo(8) 12:11
El Cenizo(8) 12:34
Roma(8) 12:54
Escobares(8) 12:57
Mission(8) 13:07
Brownsville(8) 15:50 (hard job)

fixes done.

Hi, i fixed all damaged boundaries of the USA. The mexican site should be fixed by the your guys.

WARNING: Don’t re-import “your” boundaries but fix them. Connect your AL4/Al6 and AL8 to the fixed outer boundaries of mexico and the Usa. If you are going to re-import those borders or keep on doing the same stuff at other areas, i’ll call DWG to revert your changes ASAP. That is not a joke.

regards
walter

EDIT: Same situation at least in Juárez(6). Old correct AL2 of Mexico and USA overlayed by new imported AL6 of Juárez.
Guadelupe, Ojinaga too. down the whole Rio Grande.

What are you planning to do?

Hi Wambacher,

Our team validated all the relations in the MX-US border for admin_level=6, we don’t see any issues for that specific level for US counties in the border with Mexico anymore due to most of the fixes you made. (There’s an issue with Cochise county*, whose polygon is not closed but it was not caused by our import.)

California:
http://www.openstreetmap.org/relation/396482
http://www.openstreetmap.org/relation/396515

Arizona:
http://www.openstreetmap.org/relation/1832216
http://www.openstreetmap.org/relation/1832206
http://www.openstreetmap.org/relation/1832210
http://www.openstreetmap.org/relation/1832184*

New Mexico
http://www.openstreetmap.org/relation/1832196
http://www.openstreetmap.org/relation/1832200
http://www.openstreetmap.org/relation/1832188

Texas
http://www.openstreetmap.org/relation/1812640
http://www.openstreetmap.org/relation/1812578
http://www.openstreetmap.org/relation/1812647
http://www.openstreetmap.org/relation/1812650
http://www.openstreetmap.org/relation/1814604
http://www.openstreetmap.org/relation/1814620
http://www.openstreetmap.org/relation/1814625
http://www.openstreetmap.org/relation/1827036
http://www.openstreetmap.org/relation/1827037
http://www.openstreetmap.org/relation/1827177
http://www.openstreetmap.org/relation/1827178
http://www.openstreetmap.org/relation/1828256
http://www.openstreetmap.org/relation/1828367
http://www.openstreetmap.org/relation/1828370

Now for the MX boundary data in the MX-US border we see some issues with the following relations probably after your fixes:
http://www.openstreetmap.org/relation/5605832
http://www.openstreetmap.org/relation/5605610
http://www.openstreetmap.org/relation/5606039
http://www.openstreetmap.org/relation/5606038
http://www.openstreetmap.org/relation/5605513
http://www.openstreetmap.org/relation/5605844
http://www.openstreetmap.org/relation/5605832

We’re going to fix these today.

Thanks,
Andres

The last 7 MX-US border boundary data are now fixed. I will use your tool to see if there are other parts of mexico that have some issues

I hope this post will help explain a little the method that we had chosen and some of the technique details that forced us to move from the initial proposed solution in some parts.
As always, this is a learning experiencing.
For the future we will have a better check mechanism, 4 persons where responsible for the relinking of the al4 al6 , with vary degree of experience. I will personally check what my colleagues had done and see if there are other parts that need cleaning.

There are still some things that will need to be done, to re-add the wikipedia tags to the relations where we had deleted

https://www.openstreetmap.org/user/baditaflorin/diary/36188

Hi Andres,

sorry for the late response.

Yes, i removed some border-lines from the new mexican boundary relations but i think, your team has fixed them. It was much easier for me to concentrate on one site instead of fixing all.

Regards
walter

Great, but be aware: The Job is fetching the new osm-data about midnight german time. so you’ll see the new situation on the next morning.

Hi florin, good work - after some trouble.

The next steps should be the north western part of mex. some newly added AL6 and may be AL4 are still overlapping the common al2 of usa/mexico.

Starting from here http://www.openstreetmap.org/#map=16/28.5588/-100.4089 going west there are two overlapping border lines.And you can see them on the map :wink:

I would start a “virtual jurney” using Josm, go to the west and fix one problem after another. Removing overlapping mexican al4/al6 segments and connect the remains to the al2.

regards
walter

Checked all the mexico admin_level=4 boundaries that we had added, fixed where there was something not connected
the relation analyzer should update soon, the josm relation manager where showing a beautiful circle for the next provinces in mexico

regards
florin

INEGI State ID State Name Validation (after import) validation link
17 Morelos http://ra.osmsurround.org/analyzeRelation?relationId=1376332&_noCache=on
13 Hidalgo http://ra.osmsurround.org/analyzeRelation?relationId=1376490&_noCache=on
19 Nuevo León http://ra.osmsurround.org/analyzeRelation?relationId=1661523&_noCache=on
5 Coahuila de Zaragoza http://ra.osmsurround.org/analyzeRelation?relationId=1661524&_noCache=on
8 Chihuahua http://ra.osmsurround.org/analyzeRelation?relationId=1673425&_noCache=on
26 Sonora http://ra.osmsurround.org/analyzeRelation?relationId=1673426&_noCache=on
16 Michoacán de Ocampo http://ra.osmsurround.org/analyzeRelation?relationId=2340636&_noCache=on
22 Querétaro http://ra.osmsurround.org/analyzeRelation?relationId=2340903&_noCache=on
11 Guanajuato http://ra.osmsurround.org/analyzeRelation?relationId=2340909&_noCache=on
14 Jalisco http://ra.osmsurround.org/analyzeRelation?relationId=2340910&_noCache=on
6 Colima http://ra.osmsurround.org/analyzeRelation?relationId=2340912&_noCache=on
32 Zacatecas http://ra.osmsurround.org/analyzeRelation?relationId=2399704&_noCache=on
10 Durango http://ra.osmsurround.org/analyzeRelation?relationId=2399740&_noCache=on
28 Tamaulipas http://ra.osmsurround.org/analyzeRelation?relationId=2415518&_noCache=on
12 Guerrero http://ra.osmsurround.org/analyzeRelation?relationId=2439316&_noCache=on
25 Sinaloa http://ra.osmsurround.org/analyzeRelation?relationId=2455086&_noCache=on
18 Nayarit http://ra.osmsurround.org/analyzeRelation?relationId=2551642&_noCache=on
2 Baja California http://ra.osmsurround.org/analyzeRelation?relationId=2589601&_noCache=on
3 Baja California Sur http://ra.osmsurround.org/analyzeRelation?relationId=2589611&_noCache=on
1 Aguascalientes http://ra.osmsurround.org/analyzeRelation?relationId=2610002&_noCache=on
24 San Luis Potosí http://ra.osmsurround.org/analyzeRelation?relationId=5606256&_noCache=on

Creo, al igual que wambacher que los limites son algo crítico, tanto los de nivel 2 para paises, como los de nivel 8 para las poblaciones.

No quiero parecer autoritario sino resaltar la importancia del asunto, pero a mi criterio, debiera ser, como comunidad, prioritario solucionarlos y asimilar los aprendizajes necesarios para tratar de evitar que la situacion se repita, tanto en un mejor QA de los imports, como en una mas rapida atencion/respuesta ante una advertencia de (posibles) problemas en un changeset que modifique elementos criticos.

Este tipo de ediciones debieran manejarse como algo transaccional, el mismo changeset modifica un limite, quizas mejorable pero coherente y sano, por otro actualizado mejor tambien sano, todo entero, en un mismo changeset. De esa forma los limites siempre se mantendrian idealmente sanos.

Tambien hubo algunos problemas con la linea costera en la zona del HOT por Patricia, pero eso creo que no fue por los imports sino por la cantidad de colaboradores editando simultaneamente en la zona.

Saludos,
M.

Hi Muralito,

You’re right in the sense that these matters should be taken with care, the problem was that as we we’re doing this import, hurricane Patricia hit and all of our team and resources (including servers) were put right away in the service of the Humanitarian OpenstreetMap team. At that moment we thought it was much more prioritary to help the HOT team with the resources we were dedicating to the boundaries import because the risk of this event being catastrophic. We don’t regret that decision but we didn’t plan for the unexpected, and we might have been distracted during the whole weekend with such activities, hence causing some mishaps during the boundaries import.

We’re on the track of fixing any issues that might have arised during the import and welcome any input based on clear evidence of an error in our side as the one expressed on this thread.

Si Andresuco, de las cosas que valoro de OSM es que cada grupo o individuo tenga la libertad para definir sus prioridades y trabajar en base a ellas, por eso agradezco tu explicacion, y de ninguna forma las cuestiono. Es entendible perfectamente que con la catastrofe natural las prioridades se hayan modificado.

Aunque no se refleje en el modelo de datos de OSM (porque es bien plano), hay elementos que son mas importantes que otros, como los limites o carreteras de mayor jerarquía, y es importante su continuidad para multiples consumidores del dato.
Por suerte lo inesperado no siempre es una catastre climática como la que sufrieron, pueden ser problemas personales de diversa indole, por eso es importante tratar de que los changesets sobre ciertos tipos de elementos cambien los mismos de un estado correcto a otro correcto.

Si fuera posible que el conjunto de cambios necesarios para una importación grande cumplan con esa propiedad de que cada cambio intermedio vaya dejando todo corecto, creo que eso le agrega mucho valor a una importacion, y el proceso total de la importacion puede demorarse mas tiempo que no afectaria demasiado.

Hi, i added “Select next admin level” in the context menu of my “Boundaries Map”.

which gives:

It was a request by the user mexico_boundaries_import:

have fun
walter/germany