No you are right. Maybe Varnish isn’t that helpfull after all. It’s only helpful for same requests yes.
The advantage you can get from running PHP in FPM mode + NginX is not the speed per request, but on how much resources per request your server uses and therefor how much requests per second the server can handle. Apache with running PHP as mod_php is rather resource inefficient.
Indeed running gosmore as binary each time won’t help either with the speed of the request.
I will try to install the netherlands part, internal at our side, because we use only the netherlands and I noticed we were spamming your servers. (we are the 93.186.184.x). So by that I will also take a look on how it can be optimized, because I think it could be done. Although I’ve already taken a look at the source of gosmore, but that’s a serious mess.
From our point of view, caching is possible. Because we calculate distances for many of the same routes. We use it as autocomplete to calculate distances between many same addresses. So I’ll build a local cache on our part so that your servers won’t get that much of requests from our side. And will use our server as first API. If that doesn’t work yournavigation will than be a fallback.
But I can understand that at your side, caching is not an option. What you can do however is add a robots.txt to disable spiders from crawling your API. That will remove some requests from happening.
I will email you about the info on installing the server with gosmore with optimalisations.
I searched the logs for the last few days (including the day which saw 350.000 requests) for your IP address, but there is no match. You’re welcome to use the API service, just remember to add an email address or app name to the request so that I can contact you when excessive use is detected, see the API documentation in the OSM wiki. Ofcourse you may setup your own server and I offer to assist where possible.
Ok, I understand. Apache/PHP isn’t optimal but, considering that I’m not a trained linux server admin, I’m glad to have Apache/PHP running well. The API frontend server has 2GB ram and 2 cores which are doing nothing mostly (relaying requests to the API backend server(s)). The API backend servers are 16GB ram 4-8 core machines and Gosmore is by far the most memory/cpu hungry application on those servers. So my primary question remains: how much does a more complex chain really help?
Regarding Gosmore, it indeed requires some extra time to startup on each request compared to an already running service, but there is a special feature where an external tool keeps the routing database locked and shared in RAM (I don’t think it’s in SVN already). The gosmore instances immediately have the entire routing database available. So I think that the startup overhead per route request is quite low already.
The remark about a robots.txt is interesting I do see some crawler activity, although not significant. But a robots.txt is simple to add
I must apologize to ict4schools because my assertion was incorrect. Yesterday I implemented a very simple cache using tempfs on the frontend webserver and the logs show that 30-40% of the requests are cache-hits. Apparently some applications and/or users produce multiple identical requests within a relatively short time? I don’t know how it’s possible that there are so many non-unique route requests, but anyway, I’m very happy with the results.
The first version of the cache held about 1000 requests, but the final (current) version simply discards requests that haven’t been used for more then 1 hour. The effects are clearly visible on the route backend server. Cache size usage for 1 hour (about 4000 routes) is less then 50 MB currently and varies slightly during the day.
I guess it’s because of the implementation of the applications requesting the routes. I guess the route won’t be cached in the application and will be requested again if the information is needed again.
Excellent work!
I have compared yournavigation website results with another rooting services. I think it gives the best Foot/pedestrian/Bicycle results in Crimea(Ukraine) region. So I want add this functionality to my Crimean tourist guide website. I have not PHP experience. I prefer Java back-end and Java-script front-end (GWT, ZK, or another Java-Javascript framework). I try to get results by easiest for me way. But I can’t see easy to use Java-script examples like competitors have: MQA Optimized Routing examples google maps Optimized Directions
Can you give me close easy examples how to use yournavigation API easiest way on my site or another recommendations?
Hello Lambertus,
I admired by yournavigation website service!
But I can’t use any API revisions remote access by javascript. I try to use www\api\dev\yours.js I see differences trunk and http://www.yournavigation.org/api/dev/yours.js versions. I have tried different urls and versions in yours.js:
Hello Konstantin, your browser will block cross site scripting calls because of possible security issues. Therefore you’ll need to send the request to yournavigation.org via proxy script on your own server.
I’ve been rebuilding our Gosmore maps every month or so, but the latest updates didn’t work out. I’ve build the program from svn and downloaded a fresh planet. The rebuild doesn’t end in errors and delivers pak’s, but these are much smaller than normal:
11473176 Mar 10 07:09 america.pak
45057416 Mar 10 04:56 eurasia.pak
It may very well be an error in our setup, but before I go into low-level debugging: is anyone else having trouble rebuilding Gosmore .pak’s lately?
@stephan75 Great suggestion. Checked my toolchain and most probably I need to update Osmosis, I’m currently running version 0.40.1. Will do so and report back to let you guys know.
Well, updating Osmosis wasn’t enough. it seems that the option “idTrackerType=Bitset” in the update script is not working. Without it, Osmosis uses dynamic id-trackers and the resulting file sizes seem to explode. As a result the machine was in swapping madness this morning and I had to reboot it.
I will do another run over the weekend allowing for more runtime but if anyone found a solution I am very interested.
BTW: @Lambertus, how’s the YOURS website routing maps update? Can we learn from that?
I just read your first post and wanted to say “I haven’t ran an update because I’m afraid it breaks the routing” but you solved it in your second post. Great! I’m really glad it wasn’t a Gosmore problem.
As an alternative to Osmosis you can use osmupdate to keep a local planet up to date. It’s a bit faster and uses less resources compared to Osmosis.
As an second alternative you can use osmconvert to extract the bounding boxes for Eurasia and America’s. Same as osmupdate, osmconvert is a bit faster and uses less resources compared to Osmosis.
On sarel.openstreetmap.org (which only serves Eurasia) I’m using the following update script:
It’s really quick, the buffers are used to spread the load over multiple cores and ensures that gosmore can fully utilize the core on which it runs (gosmore rebuild is single-threaded)
So there are currently still only two servers available, one for the America’s and one for the rest of the world (Eurasia). And the number of users are increasing. This month I’ve seen days with over 300.000 route requests. What a marvellous feat for a routing application (Gosmore) developed for mobile devices!
But as the load on the Eurasia server is increasing, and the unfortunate habit of Gosmore to run for long periods and consume huge amounts of RAM for long routes, there are unfortunately periods where the server starts to drop requests. To combat this while waiting for extra servers to become available a simple load balancing function is implemented that simply redirects requests to the American server when the Eurasia server is working on two or more route requests. I hope this increases the responsiveness. It will become handy when the new servers come online as well.
I was doing some routing near my parents’ place, and noted something odd. When choosing the ‘cycling-fastest’ option, it provided me with a route that was not only longer but also slower than the ‘cycling-shortest’ gave. Why this behaviour? (if you want to check: It’s in the village of Drouwen, going from the three-way crossing Hoofdstraat-Lemenweg-Stobbenweg in the east of the village to the crossing Lemenweg-Gasselterstraat-Drouwenerstraat, further to the (north)east. The shortest route was 1.86 km, taking 307 seconds, the fastest route 2.40 km, taking 615 seconds).
’ Fastest’ is a different algorithm then ‘shortest’, as it uses time penalties to determine the fastest route instead of distance to find the shortest route.
The speeds on different road types are configured and for cars the higher the road class the higher the speed. But I did not want to do that for bicycle routing, I configured cycleways as the fastest road type, then unclassified, tertiary, secondary etc. Therefore, in bicycle mode, the router prefers cycleways and unclassified ways.
Yes, I understand that. But based on that I find it strange that the shortest route is faster than the fastest one (as well as shorter).
Not really, the shortest route uses a tertiary road, the fastest route a (longer) track (more precise data: The shortest route is 1.1 miles of tertiary, the fastest one is 0.4 miles tertiary, 1.0 miles track, 0.1 miles tertiary). And, as said, the outcome for the fastest route was not only longer, but also slower than the one for the shortest route.
I found another strange behaviour now, by the way: In this case the fastest and the shortest route were the same, but where the shortest route was 0.34 km, taking 57 seconds, the fastest one was 0.34 km as well, but took 233 seconds. Surely when getting the same route, they should give the same data for travel time as well as length, no?
(we are the 93.186.184.x). So by that I will also take a look on how it can be optimized, because I think it could be done. Although I’ve already taken a look at the source of gosmore, but that’s a serious mess. 
I do see some crawler activity, although not significant. But a robots.txt is simple to add
