You are not logged in.

#1 2017-06-07 19:24:27

SwiftFast
Member
Registered: 2017-04-10
Posts: 451
Website

Optimizing this overpass query

I am still learning the Overpass API, and I wonder if this query can be optimized in any way,  and the Wiki doesn't seem to have much info about performance.  Although this query in particular runs fast, I have several different variations that take time, and their structure is similar.


Specific questions:

  • I am duplicating the same filters again and again, Should I avoid that, and how?

  • I want the nodes not last edited by SwiftFast, or SwiftFast_bot. Can this be negated with a unary operator rather than requiring a Difference block statement?

  • Does the order of the filters matter?

    • Should I place the filters that weed out the most nodes first?

    • Are there "heavy" filters that are best placed last, such that the minimal number of nodes reaches them and is evaluated by them?

  • Does using a difference block mean the second union has to be buffered in memory, meaning the query will use far more memory than a single union?

Here is the query:

[out:xml][timeout:500][bbox:{{bbox}}];
(
	area(3601473946);
	area(3603791785);
)->.a;

((
  node["amenity"="bank"](area.a)(changed:"2017-05-26T07:00:00Z");
  way["amenity"="bank"](area.a)(changed:"2017-05-26T07:00:00Z");
  relation["amenity"="bank"](area.a)(changed:"2017-05-26T07:00:00Z");
  node["amenity"="fuel"](area.a)(changed:"2017-05-26T07:00:00Z");
  way["amenity"="fuel"](area.a)(changed:"2017-05-26T07:00:00Z");
  relation["amenity"="fuel"](area.a)(changed:"2017-05-26T07:00:00Z");
);  - (
  node["amenity"="bank"](area.a)(user:"SwiftFast_bot","SwiftFast")(changed:"2017-05-26T07:00:00Z");
  way["amenity"="bank"](area.a)(user:"SwiftFast_bot","SwiftFast")(changed:"2017-05-26T07:00:00Z");
  relation["amenity"="bank"](area.a)(user:"SwiftFast_bot","SwiftFast")(changed:"2017-05-26T07:00:00Z");
  node["amenity"="fuel"](area.a)(user:"SwiftFast_bot","SwiftFast")(changed:"2017-05-26T07:00:00Z");
  way["amenity"="fuel"](area.a)(user:"SwiftFast_bot","SwiftFast")(changed:"2017-05-26T07:00:00Z");
  relation["amenity"="fuel"](area.a)(user:"SwiftFast_bot","SwiftFast")(changed:"2017-05-26T07:00:00Z");
));

out meta;

Last edited by SwiftFast (2017-06-08 06:30:34)

Offline

#2 2017-06-07 20:35:37

mmd
Member
Registered: 2010-11-06
Posts: 1,835

Re: Optimizing this overpass query

SwiftFast wrote:

I am duplicating the same filters again and again, Should I avoid that, and how?

I want the nodes not last edited by SwiftFast, or SwiftFast_bot. Can this be negated with a unary operator rather than requiring a Difference block statement?

Currently, you cannot express this in another way. Howver, a new user/uid based filter for (if: ...) is currently in development, yet not released. This would allow you to simplify the query as follows (I left out areas and only check one tag)

(
  node["amenity"="fuel"](if:user() != "SwiftFast_bot" && user() != "SwiftFast")(changed:"2017-05-26T07:00:00Z");
  way["amenity"="fuel"](if:user() != "SwiftFast_bot" && user() != "SwiftFast")(changed:"2017-05-26T07:00:00Z");
  relation["amenity"="fuel"](if:user() != "SwiftFast_bot" && user() != "SwiftFast")(changed:"2017-05-26T07:00:00Z");

);  
out meta;
SwiftFast wrote:

Although this query in particular runs fast, I have several different variations that take time, and their structure is similar.

Best is to provide some actual queries, otherwise it will be impossible to give you further hints.

Last edited by mmd (2017-06-07 20:44:46)

Offline

#3 2017-06-07 21:09:52

SwiftFast
Member
Registered: 2017-04-10
Posts: 451
Website

Re: Optimizing this overpass query

Best is to provide some actual queries, otherwise it will be impossible to give you further hints.

My main interest is theoretical. I would like to understand how optimizations work, and answer the above questions so that maybe I can write something about performance in the Wiki. This query only serves as a discussion example.

Last edited by SwiftFast (2017-06-07 21:10:19)

Offline

#4 2017-06-07 21:12:01

SwiftFast
Member
Registered: 2017-04-10
Posts: 451
Website

Re: Optimizing this overpass query

mmd wrote:

Currently, you cannot express this in another way...

Thanks!

That covers it all except the filter order question.

Offline

#5 2017-06-07 21:22:11

mmd
Member
Registered: 2010-11-06
Posts: 1,835

Re: Optimizing this overpass query

SwiftFast wrote:

Best is to provide some actual queries, otherwise it will be impossible to give you further hints.

My main interest is theoretical. I would like to understand how optimizations work, and answer the above questions so that maybe I can write something about performance in the Wiki. This query only serves as a discussion example.

That's quite a complex topic, due to the large number of combinations data wise and to some extent also configuration wise.

Your observations on overpass-api.de might be heavily skewed by the current CPU usage, and there's still quite a number of pending performance improvements out there on Github. We run this kind of a analysis on dedicated servers without other traffic disturbing measurements.

So best approach would be to post queries you found expensive, or even create some Github issue for it.

Last edited by mmd (2017-06-07 21:23:24)

Offline

#6 2017-06-07 21:54:32

SwiftFast
Member
Registered: 2017-04-10
Posts: 451
Website

Re: Optimizing this overpass query

No doubt it's complex, yet some specifics should be clear-cut. e.g. either the order of the filters matters or it does not. And these questions are not documented as of now, hence my post.

In my original query, I wonder if the second union is simply buffered in memory in order to be compared with the first. This would mean that that query can use much more memory than your `(if: ...)` variation.

Last edited by SwiftFast (2017-06-07 21:55:47)

Offline

#7 2017-06-07 22:14:16

mmd
Member
Registered: 2010-11-06
Posts: 1,835

Re: Optimizing this overpass query

The query evaluation is described in more detail in one of the presentations given by Roland, I'd have to check which one that was. Iirc it is based on 9 different stages with cheap constraints being checked first. Within a query statement reordering is of limited use. However, sometimes splitting a query statement in two separate ones can be used to influence the evaluation order. There are some examples of this approach on help osm. Also you should check if you could use newer instead of changed as it is much cheaper!

There's a default upper memory limit per query, but I haven't checked what the memory consumption was like for your query. In any case it was still below the default cap. Nevertheless your query probably needs more memory as in the (if: ) query example, but it doesn't matter in this case. If you exceed the limit, the query will be cancelled anyway.

To reuse results in a difference statement you should probably read https://github.com/drolbr/Overpass-API/issues/317 as well.

Last edited by mmd (2017-06-08 07:03:38)

Offline

#8 2017-06-08 09:37:18

SwiftFast
Member
Registered: 2017-04-10
Posts: 451
Website

Re: Optimizing this overpass query

Thank you for your time. I believe you're talking about the State of the Map 2013 presentation (starting at 7:20). Unfortunately, the pipeline explanation was too brief. It wasn't explained how the "Collect ids of potential results" step works exactly, and the last three filtering steps were not discussed at all.

Last edited by SwiftFast (2017-06-08 09:38:42)

Offline

#9 2017-06-08 10:02:14

SwiftFast
Member
Registered: 2017-04-10
Posts: 451
Website

Re: Optimizing this overpass query

Also you should check if you could use newer instead of changed as it is much cheaper!

Are you sure? The Wiki says changed is cheaper, but requires Attic Data support.

The changed filter selects all elements that have been changed between the two given dates. If only one date is given, then the second is assumed to be the front date of the database. If only one date is given and it is run with the current timestamp, then it behaves exactly like "newer" with two exceptions: first, it is faster, second, it can also stand as the only filter.

Offline

#10 2017-06-08 10:36:52

mmd
Member
Registered: 2010-11-06
Posts: 1,835

Re: Optimizing this overpass query

SwiftFast wrote:

Are you sure? The Wiki says changed is cheaper, but requires Attic Data support.

changed currently has issues when it comes to memory consumption, especially if you're looking at a large bbox and/or timespan, see https://github.com/drolbr/Overpass-API/issues/278 - a larger redesign is underway in this area anyway due to https://github.com/drolbr/Overpass-API/issues/346 and https://github.com/drolbr/Overpass-API/issues/322

Last edited by mmd (2017-06-08 10:41:40)

Offline

#11 2017-06-08 11:17:29

SwiftFast
Member
Registered: 2017-04-10
Posts: 451
Website

Re: Optimizing this overpass query

You just sped up the above query by 6x! I will modify that Wiki page asap.

Last edited by SwiftFast (2017-06-08 11:24:15)

Offline

Board footer

Powered by FluxBB