Routing website using Gosmore routing engine

Hello,

I’ve been using yournavigation.org for quite some time. But recently I needed a permalink to a route I’ve made, calculated right and as expected, but upon clicking on the permalink I saw a different way around added to the correct one, like this. If I move starting/ending point or tell it to recalculate the route, it shows the correct one, but permalink always adds that strange route around. Sharing such routes will confuse people.
If this problem was reported before, I beg your pardon, didn’t find any references.

Oeps didn’t notice this before. Update is already running for a few days and should finish this weekend.

Looks like a bug where opening the permalink always generates a route for motorcar AND the requested route type. I’ll see if this can be fixed.

Hey Lambertus,

I’m trying to set up a gosmore-based “routing backend” - I guess quite similar to yours - and I’m running into a lot of frustrating little issues, mostly due to missing documentation.

Currently I’m trying to get routing to work for “outside germany”, but creating a .pak file for say europe is impossible on 32 bit hardware (needs more than the addressable 2G Ram), my local system only runs 32bit, and Amazon EC2 is giving me major headaches.

I guess I’ll get it done at some point, and I’ll document how I got it done, but I wonder - how do you build the .pak files, do you split into limited boxes before, and what about “international routing”?

As I need the routing to be able to cross box borders (i.e. from Hamburg to Stockholm), I think there’s a lot of stuff that needs to be done to make this happen?

I’ve read the gosmore rebuild page on the osm wiki, but somehow I can’t make it fit to the current dev state… scratchinghead

Any advise welcome.

I’ve also noted that headless support was broken by some Android changes, but I was able to edit around them and will discuss that with “the author” (I guess Nic) by mail.

I guess a mailing list could or dedicated message board would be really helpful for the project :confused:

The author of Gosmore is indeed Nic Roets. And he also provides the routing backend for yournavigation.org since about a year or so. I also had lots of problems trying rebuilds. Some of these problems are due to difference in locale settings. E.g Dutch use ‘,’ as decimal separator while English countries use ‘.’. But this particular bug has been fixed long time ago, there may be other bugs lurking around. A 64-bit machine is very useful while working with OSM data (especially if you use large areas). I’ve switched to 64-bit a long time ago (not only for Gosmore but for the Garmin maps as well), I don’t know if Gosmore is capable of rebuilding a PAK file on 32 bit platforms. Windows won’t work anyway for PAK rebuilding.

For problems with Gosmore I will have to redirect questions to Nic Roets, but you can ask here as Nic reads these forums as well and it will probably help others. Just open a new topic in the Development forum for your project. Btw, there is a low traffic generic osm-routing mailinglist.

So the routing backend is run by Nic himself? I’ll ask him about details on how to build the proper PAK files in that case. Or maybe he stumbles over this post. I also have his mail address (it’s in the sources for Gosmore) so I can contact him directly.

Currently I’m only using the data for private use and for some experiments, so until recently, I did not need more than the german pak file I obtained somewhere in the internet; now, I have to plan a trip to Finland and need more than that :wink:

My EC2 instance is now happily crunching on the europe area, it has finished parsing the XML and is stable at 4.9G RAM, I just go to bed now and hope the best.

I’d use my local MacBook which sure is 64bit, but Gosmore does not compile on that for some Linux dependencies and I just don’t have the time to go fix on that. My older Linux boxes still run on 32 bit, and my web server is shared with other people so I can’t block it completely with such a task.

Yes, since about a year. I was having performance problems on the limited VM it was running. He found a server sponsor and volunteered to manage Gosmore there.

It will probably very difficult technically, as there are so many components that need to work together. But do you think it would be possible to get “social driving” to a routing server? The kind of driving that is found in the app Waze.

I was thinking of the following scheme:

  • a client app (on a smartphone) such as OsmAnd sends statistics to a server about the speed and direction over a short distance.
  • the server processes that speed information to see if there is a problem. In short term, the router will know something (like an accident) happened and wont give routes that pass along that point for a certain time (say 2 hours). In the long term, the routing service will learn from it, knowing that it’s not a good idea to send people to certain streets at 5PM.
  • client apps can use the better routing

Next to the problem of having a lot of instances working together, there is also a problem with the nature of OSM. First of all is, if a way is edited, the already processed data might be worthless (as the way can have a completely different form or length). So the raw (maybe privacy sensitive) data should be kept for a long time, and processed each time somthing changes.

Do you think it would be possible to develop this? Or am I just dreaming?

With a lot of kind help from Nic, I was able to successfully build my own .pak file on Amazon EC2. I’ve created a - hopefully complete - documentation, and wanted to share it here; I hope this does not violate any rules.

http://www.retro-coding.de/2012/02/03/how-to-re-build-a-gosmore-pak-file-on-amazon-ec2/

I’m currently using EC2 instances for the routing too, but they are quite expensive for that task if you don’t need extreme scaling (where the benefits should start to pay off), so I’m looking to set up a root server for my routing tests as soon as I have some free time.

For building the .pak file, EC2 is quite usable when you have figured out the necessary stuff; I guess it can be done well below $5 for a whole planet file.

This problem was fixed quite some time ago already. When a new route was planned before the results of the previous plan were received, that gave problems in general. The latest version in SVN contains a few other improvements too, but I think yournavigation.org did not update for about a year.

Cheers,
Steven

There seems to be an issue with the current gosmore.php in the 1.0 api folder, when exporting as GeoJSON.

It is including the traveltime in the properties object, but doesn’t have a comma after the description, which causes the json parsing to fail. I looked at the svn copy of the 1.0 api, and it appears that traveltime doesn’t exist there, so it looks like a minor sync issue between the site and the development code.

Also, for some reason, the destination coordinates appear as the very first entry in the coordinates array when using the dev api version. I haven’t had the time to look at the code to see if I could fix this issue yet, I was just testing to see if it was usable until the 1.0 api was repaired.

Despite these issues, this is a great service, and thanks for all your hard work.

Joshua

Something is wrong with the filesystem of the Gosmore routing backend server which causes problems with the route calculations. I’ve contacted the administrator and hope it’s quickly fixed.

About 80.000 to 200.000 routerequests are handled each day by this server, so a lot of people are affected unfortunately.

Edit: the filesystem problem has been fixed.

This a call to the owner(s) of two servers with IP address 194.152.x.x. located in Kyrgyzstan leased from Nurtelecom to contact me: osm [at] na1400 [dot] info.

These two servers are each responsible for much more then 10k requests per day but the requests do not provide an application name or contact information in their request headers. A warning header message is now sent back with each route result and I will wait a few weeks to allow the author(s) of the application(s) to contact me. If I don’t hear anything during this time then the IP addresses will be added to the ban-list.

The same problem as before has reappeared:

Edit: problem should be fixed permanently with a kernel update.

Hi Lambertus,

first of all: Thanks for your great work and support here within the forum !

I’ve got a question: Route calculation on yournavigation.org seems to be working for most of the routes. Now I tried the following:

  1. Waypoint 1: Known address
  2. Waypoint 2: Known address (but this house is not directly connected to the street)

Any idea why this is not working?

Thanks in advance!

best regards,

Fabian

Where is that exactly? Sankt-Rochus Strasse is quite common in Germany… It’s possible that the footpaths are newly mapped and not available in the route database yet, if so then Gosmore simply cannot find a street within a certain radius around the marker.

The current sponsoring for hosting the routing backend is ending, so…

! Call for sponsoring !

For this project a sponsor is needed to host the Gosmore routing backend. Please contact me if you can help. Thanks!

Minimal requirements: dual core, 16 GB ram, 100 GB disk, 500 MB traffic/month

Edit: updated requirements

The routing backend is down again because the filesystem is mounted read-only as a result of a kernel bug. The admin is looking at it.

Edit: it appears that the routing backend is back up, but I don’t know if this serious problem is permanently solved. I’m still looking for hosting sponsors to make the routing backend more robust, so if you can help please contact me.

I could help with hosting, but I don’t reach the memory requrements. So I was wondering, can the hosting be done in a distributed way? 8GB is a bit on the high side for what most hardware around has, but if it were possible to distribute the backend then lower memory requirements would become possible.

Thanks Rebroad, but I think that memory is critical. I’ll try to explain:

The route database is about 15 GB, of which about 5 GB covers North/South America, the rest covers Eurasia, Africa and Oceania. If you want to route from somewhere in North America to South America you need access to the full 5 GB. The data needs to be available in RAM to get any performance from this (this is straight-forward and I’ve done benchmarks that decisively prove this).

On top of this memory payload you need additional memory for the routing application to store intermediate results (see wikipedia for shooting star or Dijkstra algorithm, I believe the A* (shooting star) algorithm is similar to the algorithm used in Gosmore). For long routes these intermediate results require a fair amount of memory as well, up to 4 GB is not uncommon.

So 8 GB RAM is the minimum for routing in the America’s with a concurrency of 1. A concurrency of 2 would need 12 GB RAM (worst case, when two large requests are handled simultaneously). Etc. An Eurasia-only server would need 16 GB as a minimum if you want to be able to keep the whole route database in RAM and have some RAM available for the intermediate route results.

If the Gosmore processes require more RAM then the system has free, the system will first start unloading parts of the routing database from RAM, reducing performance. The system will start swapping when the Gosmore processes use more RAM then physically available. Swapping is really killing performance, you don’t want that ever to happen. It is the reason why the old route server with 2 GB ram (which now only serves the front-end website) was often completely unresponsive because of heavy swapping, so much so that I sometimes even could not login to attempt to kill these large requests. As long as you quickly process each request then there won’t be concurrency, but when you need a lot of time to handle one request other requests will also arrive and increase concurrency which increases RAM usage which increases swapping which causes slow request handling etc. You end up in a killing feedback loop.

The current route server that provides worldwide routing has 16GB ram and performs pretty well, although parts of the routing database will be removed from RAM from time to time. The number of routing requests is also increasing every month so 32 GB ram would give a nice margin to grow. Below 16 GB there is not much performance to be had, perhaps it would be doable for the America’s only.

So to come back to your question: there is (currently) almost no option for distributing the routing backend that will seriously reduce the 16+ GB memory requirements.

An update:

I’ve received several sponsoring offers ranging from 2nd-hand hardware, virtual machines, rack space and dedicated servers. More than I hoped for and more than I can manage, which is really a nice ‘problem’. Many thanks for everyone who contacted me!

So how’s the routing backend doing? A few weeks ago the backend has moved from one server to two servers, one server handles both America’s and one server handles the rest of the world (hereafter: Eurasia). Besides the doubling of CPU’s and memory the memory use per server also halves so the actual routing requests have more memory to work with. I think it’s safe to say that the doubling of processing capacity has increased the routing requests per second by a factor of four. This has shown to be very useful already as last Friday the service handled the most route requests ever: 360.000 in one day, way more than the usual 80.000.

The plan going forward is:
first, to quickly add a sponsored dedicated machine to share the load with the Eurasia server, which receives the most requests.
Secondly, I have received two 2nd-hand servers and two rack-space hosting options and will install the servers (hopefully) next week which brings the amount of available servers to 5.
Third, to add another sponsored dedicated server to the pool. This sponsoring has been promised/discussed but has not translated into an actual server yet.

Software wise at first a simple round-robin scheduler will be implemented to take advantage of all the servers. Availability checking and monitoring must also be implemented. After that I would like to dedicate one of two servers for providing experimental routing options, like many cycling profiles and nautical (if possible). Perhaps other developers can be given access to test routing engine updates, different routing engines etc.

These upgrades will greatly improve performance and reliability of this service :slight_smile:

Another option would be to provide an additional service to commercial parties who require up-time guarantees or bulk routing or … implementing an API-access authentication and subscription mechanism. Contributions from these API users can be used to strengthen the service further (e.g. rent extra servers).

I would like to hear the input of others, so if you have ideas; please share!