You are not logged in.

#1 2012-05-23 11:55:30

eric22
Member
Registered: 2010-07-04
Posts: 114

Israel GTFS release

Hi,

Recently the Ministry of Transport has released a complete set of GTFS data for public transport in Israel.
http://tinyurl.com/chhb6wn

From this data we could:
1) Add (via a script) a complete set of bus stops with names in Israel
2) Based on the stop names, add (manually) street/road names and addresses in less-mapped areas that are currently missing names/addresses
3) Add (manually I believe) bus route relations throughout Israel

There are two possible issues:
A) Legal: The data is obviously intended for public use, but maybe it does not meet exact requirements for OSM. If so, we might have to ask directly for permission to use it here.
B) Technical: somebody would have to write the import script(s).

Does anyone have ideas where to get started on A) or B)?

Offline

#2 2012-05-23 22:20:21

yxejamir
Member
From: Israel
Registered: 2011-12-02
Posts: 6

Re: Israel GTFS release

According to http://code.google.com/p/gtfs-osm-sync/, it in the transport agency’s best interest to do the synchronization, and therefore it should be their job. Since I do not see that happening on its own, it’s our job to get the ball rolling. The project page says that “it's highly recommended that GO_Sync users understand OpenStreetMap well and let other OpenStreetMap users know a big change is coming BEFORE any upload/revert.” If you need help with using GO_Sync, ask here and I will try to do my best (I do not know much more about it than you do, but we can learn together).

Regarding the terms of use, reding them with a non-professional eye, they are basically just a disclaimer; do whatever you want with the data, on your own risk – these things are usually in ALL CAPITAL LETTERS, when written in English. But you should probably consult a professional, especially regarding the “we reserve the right to change the terms at any time” clause.

Thanks for the news, BTW, glad to hear about it!

Offline

#3 2012-05-23 22:47:39

yxejamir
Member
From: Israel
Registered: 2011-12-02
Posts: 6

Re: Israel GTFS release

Also, from http://www.tlv1.co.il/?p=5930&cpage=1#comment-16786; forgive my use of Hebrew:

בנוגע לתנאי שימוש – מותר להעתיק ולשכפל את המידע ולהפיצו כראות עיניכם. ידידי, יהונתן קלינגר מסביר:

ראשית, אתר דאטה.גוב אומר בצורה מפורשת כי 'המידע פתוח לשימוש חופשי ולא נדרש כל אישור מיוחד'; אבל, גם אם הדבר לא היה כך, העקרון הוא פשוט: סעיף 5 לחוק זכויות יוצרים קובע כי על 'עובדה או נתון' לא יחולו זכויות יוצרים. לכן, במספר מקרים בהם הועתקו מאגרי מידע כמו ספרי טלפונים, מודעות דרושים או מדריכים מקצועיים, פסק בית המשפט כי ההעתקה מותרת, וכי אין זכות יוצרים בליקוט (מעריב נ' אול יו ניד, קווי מידע נ' בל תקשורת). הסיבה לכך היא פשוטה: במאגר מידע אין את ה'יצירתיות' הדרושה לצורך מתן הגנה בזכויות יוצרים.

Which means, no special permission required for sync'ing the data to OSM.

Offline

#4 2012-05-26 07:38:18

Mr_Israel
Moderator
From: Israel, Germany
Registered: 2009-03-02
Posts: 513

Re: Israel GTFS release

Wow.  Thats great news. Amazing news.

Before we import we need a confirmation that we are allowed to use that data in openstreetmap.
We are not allowed to just import! But as i know the goverment they will approve it.

Can somebody write a email in hebrew to them and ask for a approval?
No hebrew to help in that point. 

The import should very easy with the gtfs osm sync.

Btw:the link to download the zip is not working at the moment.

Last edited by Mr_Israel (2012-05-26 07:54:08)

Offline

#5 2012-05-30 06:31:16

dimka
Member
Registered: 2010-04-01
Posts: 257

Re: Israel GTFS release

Now that's official, no need for confirmation:

מאת ניר הירשמן, data.gov.il:
29 במאי 2012 בשעה 8:19
הי אמיר,
אין שום צורך באישורים לשימוש במידע מדטה גוב איל. ישנם תנאי שימוש באתר שמתירים שימוש חופשי במידע.

Offline

#6 2012-05-31 18:58:39

eric22
Member
Registered: 2010-07-04
Posts: 114

Re: Israel GTFS release

Cool.

And the download is working again (I think it's actually a significantly smaller file now, not sure what they changed).

Now, anyone want to give GO_Sync a try?

Offline

#7 2012-06-04 16:32:43

Mr_Israel
Moderator
From: Israel, Germany
Registered: 2009-03-02
Posts: 513

Re: Israel GTFS release

I just checked the file from today and have the tool is working and I could import the data if the Java tool GO_Sync would know Hebrew.
But it doesn't.

Maybe we should consider uploading the data manually. We have all we need to upload it, right?

Offline

#8 2012-06-26 22:09:11

eric22
Member
Registered: 2010-07-04
Posts: 114

Re: Israel GTFS release

Hi,

I uploaded the data manually (using a modified version of csv2osm.pl and then JOSM).

I never knew how many stops there were. Blue dots all over the map. It seems that in some cases there are duplicates near each other, perhaps placed by different bus companies. Questionable whether we should edit/delete in that case.

The accuracy appears good - in most places I looked, there was <5m deviation from the roads as we mapped them.

I made a couple partially successful upload attempts before the final successful one. If anyone can delete these without affecting the successful changeset, that wouldn't hurt. Keep in mind that the original dataset includes 241 (IIRC) duplicates and perhaps this was intentional, to represent multiple adjacent stops (opposite directions?) or something similar.

The next task, for those who choose to accept it, is to give names to currently unnamed streets/places based on the name of the stop adjacent to them.

Offline

#9 2012-06-27 09:41:28

dimka
Member
Registered: 2010-04-01
Posts: 257

Re: Israel GTFS release

Great!

Do you mean that the partially successful attempts resulted in duplicated data?
Can it be fixed by just reverting the entire corresponding changesets?

dimka

Offline

#10 2012-06-27 18:44:05

eric22
Member
Registered: 2010-07-04
Posts: 114

Re: Israel GTFS release

After taking a look at the data, it appears it was duplicated, so we simply need to delete the old changesets.

Could somebody please revert changesets 12028202 and 12028208? I'm having plugin problems.

Thanks

Offline

#11 2012-06-29 11:33:55

yrtimiD
Moderator
From: Israel
Registered: 2010-10-08
Posts: 249

Re: Israel GTFS release

eric22 wrote:

Could somebody please revert changesets 12028202 and 12028208? I'm having plugin problems.

Just reverted.

Offline

#12 2012-06-29 11:35:43

yrtimiD
Moderator
From: Israel
Registered: 2010-10-08
Posts: 249

Re: Israel GTFS release

What is the next step? How we going to merge old manually added stops to the new one?
In many places I see new stops appear between ways. How accurate was the imported location data?

Offline

#13 2012-06-29 16:47:03

eric22
Member
Registered: 2010-07-04
Posts: 114

Re: Israel GTFS release

I think very few of the manually added stops have any useful information other than the location. So I would recommend deleting them (manually after being checked, not by a script), and if necessary moving the GTFS stops to the location of the old manual stop. The GTFS stops include information, like the official stop name and reference number, which is potentially very valuable if coordinated with other data (for example, the route database).

As for GTFS location accuracy - the first few stops I looked at appeared very accurate. But since then I have seen less accurate ones, particularly outside urban areas. And the fact that there are often stops with identical names a significant distance away from each other implies that the accuracy may not be so high. Of course, manually entered stops can also inaccurate. So I don't know.

Last edited by eric22 (2012-06-29 16:49:07)

Offline

#14 2012-06-29 17:47:42

yrtimiD
Moderator
From: Israel
Registered: 2010-10-08
Posts: 249

Re: Israel GTFS release

So, we doing merge and giving location priority to the old ones. Works for me.

eric22, can you somehow add fixme tag for all the new bus stops, so we can monitor manual validation status?

Offline

#15 2012-06-30 10:41:16

Mr_Israel
Moderator
From: Israel, Germany
Registered: 2009-03-02
Posts: 513

Re: Israel GTFS release

Hi eric22,

just checked the import and its amazing detailed.
Several street names could be fetched from this import.
Can be very very helpful in the future.
Great you found the time to do the job.

Beside that we should import more value from the data we got:

For example the bus numbers that are stopping on each  stop.
And also the name of the bus company taking care of the stop.
Also the route ID could be from interest. Those kind of information are in the Israeli public data and we should really use them.

Currently I will not be able to create a bus route with the data we imported.
Its simply missing more information.

We should consider getting more data into OSM.
Is that possible somehow?

Offline

#16 2012-07-14 14:46:44

yrtimiD
Moderator
From: Israel
Registered: 2010-10-08
Posts: 249

Re: Israel GTFS release

I just added (upload in progress) fixme tag for all (33051) imported bus stops.
Now, each time we see bus stop with fixme tag, we can check it, move it and/or merge it with existing one... until all stops will be approved.

Offline

#17 2012-07-15 09:39:09

Mr_Israel
Moderator
From: Israel, Germany
Registered: 2009-03-02
Posts: 513

Re: Israel GTFS release

We have a problem if we merge the bus stops. We would loose the "ref" number what will cause a overlapping for the next import.
We should not touch the location at the moment as we will have to import it again in the future.

What do you think?

Offline

#18 2012-07-15 09:44:13

yrtimiD
Moderator
From: Israel
Registered: 2010-10-08
Posts: 249

Re: Israel GTFS release

Mr_Israel, no, we must only merge imported with existing and never two stops with different refs.

By the way, I see many stops which isn't exists at all. It's not a good idea to show them on the map and we need to find any solution to leave them in db but not show on rendering. maybe we can mark them as construction or proposed.

Offline

#19 2012-07-15 09:50:57

Mr_Israel
Moderator
From: Israel, Germany
Registered: 2009-03-02
Posts: 513

Re: Israel GTFS release

We could add the REF of the bus stop to a separated field.

Here some examples:

=> ref:merge=123123;123123;123123;123123
=> ref1=123123
     ref2=234213
     ref3=523423
=> description=123123;123123;123123;123123

I simply don't want to loose any kind of information.
I'm expecting that they will fix the GTFS data in the future as it is currently not usefull for Google I think.
It would also mean for Google to have bus stops all over the street without marking the exact bus stop position.

Offline

#20 2012-07-15 09:53:43

yrtimiD
Moderator
From: Israel
Registered: 2010-10-08
Posts: 249

Re: Israel GTFS release

I'd prefer to leave different stops as separate nodes. maybe slightly move them so the nodes will not have same position.

Offline

#21 2012-07-15 09:54:59

Mr_Israel
Moderator
From: Israel, Germany
Registered: 2009-03-02
Posts: 513

Re: Israel GTFS release

This could also be a option. Simply move them to just one position.
This way we would not loose any information and we could import again and skip all existing stop with existing refs.

Offline

#22 2012-08-11 20:03:00

eric22
Member
Registered: 2010-07-04
Posts: 114

Re: Israel GTFS release

I have been working on the GTFS route data, and now I have a .osm file with a large number of ways, each representing a bus route. Here is a screenshot in JOSM:

gtfs.gif

The question is how to incorporate this data into OSM. A bus route in OSM consists of a relation which contains all the street segments the bus travels on. However, my data is not relations but ways. Furthermore, we presumably want to use our pre-existing ways, rather than the ways I'm uploading, to represent streets. (Though we could update our ways if necessary to reflect information in my ways.)

Right now it looks like I should upload all my data as-is to OSM. It will not show on the rendered map (the only tags in each way are ref,name,operator,source). But when we edit, we will see a number of ways overlaid on our streets. One by one, these can then be manually deleted and relations created to hold the equivalent information.

That is a labor-intensive process. Also, we need to find a reliable way of uploading this 300MB file (35MB when bzipped). My relatively new computer basically grinds to a halt when the file is opened in JOSM. But there must be some smaller script that can do the upload, perhaps by breaking the file into pieces.

Any thoughts, or alternative suggestions?

Offline

#23 2012-08-11 20:09:29

yrtimiD
Moderator
From: Israel
Registered: 2010-10-08
Posts: 249

Re: Israel GTFS release

I would suggest to split it to smaller chunks (by operator/area/city/...).
Also, may be an option to create many small files, such as we can get small piece of data, open in in josm, and create normal relation without uploading temporary data and then deleting it.

eric, can you share  some example? any single route?

Offline

#24 2012-08-11 21:11:32

Mr_Israel
Moderator
From: Israel, Germany
Registered: 2009-03-02
Posts: 513

Re: Israel GTFS release

I agree with yrtimiD to have smaller chunks and to try not to upload everything and then delete it.
It's better to do it slowly but with a focus to finish it together. You are welcome to add all routes to Israel by yourself :-)

You are preparing several OSM files with the data you have and split it to the smallest chunks we can do. Maybe bus route by bus route.
And then make it available to download. So everybody that wants to help needs to download the OSM file and add his name to the list at the route.
This way we would add route by route... will take a long time but better that "spaming" OSM with data that needs to be deleted afterwards.

Adding a bus route to osm is a very annoying process as a lot of streets needing to be cut into very small peaces to add the correct relation.
Doing it very slowly and with a system would really help.

Offline

#25 2012-08-12 19:18:13

eric22
Member
Registered: 2010-07-04
Posts: 114

Re: Israel GTFS release

Good ideas smile

I started separating the data by operator, and when looking more closely in JOSM, I realized a portion of the data is wrong. Not sure yet if the problem is in my script or their data. But I'm glad I noticed before uploading...

Offline

Board footer

Powered by FluxBB