Connection closed/aborted errors for requests for vector tiles that take a long time to generate

[amandasaurus]

I have a tmsource project and I'm using tessera to serve the vector tiles from it. This tmsource project uses a lot of PostgreSQL queries that take a long time to run. And when my http client requests a vector tile from tessera it can fail with a http connection aborted / connection closed error. The tessera output shows: GET /pbfs/live//14/7794/3493.pbf - - ms - -.

Is there anyway to make tessera not timeout, and to wait a long time before "giving up". This way, I'd get my vector tiles, even if they take a long time.

In the long run, we need to improve our SQL queries, but we're curring using tessera to generate vector tiles in a batch manner, so waiting a long time is not a big problem.

[mojodna]

Are you proxying tessera through a different web server (e.g. Apache or Nginx)?

tessera shouldn't time out, so I suspect that any timeouts you're experiencing are either due to the reverse proxy layer or the client. The log line you included is symptomatic of Express (the underlying HTTP server) acknowledging that the client disconnected.

You can confirm this by running time curl -vI http://localhost:8080/pbfs/live/14/7794/3493.pbf (-v for verbose, -I to make a HEAD request, which will still render the tile even if it doesn't return it).

If you're doing batch generation of vector tiles, you may be better off with tilelive-copy or tl.

[amandasaurus]

Are you proxying tessera through a different web server (e.g. Apache or Nginx)?

I think it might be going through Apache. What should I change to make apache work?

due to ... the client. The log line you included is symptomatic of Express (the underlying HTTP server) acknowledging that the client disconnected.

I'm using Hyper HTTP client in Rust. I've set the timeout on the HTTP request to 24 hours, so I'm stumped as to what is wrong.

If you're doing batch generation of vector tiles, you may be better off with tilelive-copy or tl.

I initially looked at them, but they only support a few "output formats", and I wanted to store vector tiles in a stepped directory (tilecache layout). Additionally, I want to "expire" some vectortiles, by downloading a list of URLs and saving them, rather than download everything again.

[amandasaurus]

I have time to look at this again.

I think it might be going through Apache. What should I change to make apache work?

I'm not using apache. I'm going direct to localhost:8080, where tessera is listening on.

You can confirm this by running time curl -vI http://localhost:8080/pbfs/live/14/7794/3493.pbf (-v for verbose, -I to make a HEAD request, which will still render the tile even if it doesn't return it).

Here is some output:

$ time curl -vI http://localhost:8080/pbfs/live/14/7768/3519.pbf
#* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8080 (#0)
> HEAD /pbfs/live/14/7768/3519.pbf HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:8080
> Accept: */*
> 
* Empty reply from server
* Connection #0 to host localhost left intact
curl: (52) Empty reply from server

real    2m0.044s
user    0m0.000s
sys 0m0.044s

And the corresponding line in tessera's stdout:

HEAD /pbfs/live/14/7768/3519.pbf - - ms - -

Any ideas?

[mojodna]

(Sorry for the delay.)

It definitely looks like there's a 2 minute timeout somewhere, the big question being where. It might be curl (in which case you could fiddle with command line options or use nc to initiate an HTTP request manually). Other possibilities that come to mind are: tilelive-bridge (which is what tilelive-tmsource's constructor actually returns), or mapnik itself.

I don't see anything in tilelive-bridge, so it seems more likely that the query is being timed out somewhere. Another thing to check would be to select * from pg_stat_activity while the request is sitting there and to see if the query remains active after the HTTP response has timed out.

I initially looked at them, but they only support a few "output formats", and I wanted to store vector tiles in a stepped directory (tilecache layout). Additionally, I want to "expire" some vectortiles, by downloading a list of URLs and saving them, rather than download everything again.

tilelive-file has a safe option (mapbox/tilelive-file@4550754), which may produce the same filesystem layout that tilecache uses (I implemented it for compatibility w/ tilestache's filesystem cache).

tilelive-copy includes support for --scheme=list w/ --list=<filepath> (https://github.com/mapbox/tilelive/blob/master/bin/tilelive-copy#L90-L92 / tilelive-copy --help), which seems like it would help achieve this as well.

[amandasaurus]

it seems more likely that the query is being timed out somewhere. Another thing to check would be to select * from pg_stat_activity while the request is sitting there and to see if the query remains active after the HTTP response has timed out.

The query is continue to be run after the HTTP connection is closed:

time curl http://localhost:8080/ym/pbfs/live/0/0/0.pbf ; psql -U osm -d gis -c "select pid, query_start, state, query from pg_stat_activity;" | cat
curl: (52) Empty reply from server

real    2m0.048s
user    0m0.000s
sys 0m0.004s
  pid  |          query_start          | state  |                                                                                                                                                                        query                                                                                                                                                                        
-------+-------------------------------+--------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 21180 | 2016-09-27 14:36:14.716647+00 | active | SELECT ST_AsBinary("way") AS geom,"area","type" FROM ( SELECT way, way_area AS area, COALESCE(landuse, leisure, "natural", highway, amenity, tourism) AS type  FROM planet_osm_polygon  WHERE way_area > 100000  ORDER BY way_area DESC) AS data WHERE "way" && ST_SetSRID('BOX3D(-20037508.34 -20037508.34,20037508.34 20037508.34)'::box3d, 3857)
 22295 | 2016-09-27 14:38:12.540329+00 | active | select pid, query_start, state, query from pg_stat_activity;
(2 rows)

[amandasaurus]

tilelive-file has a safe option (mapbox/tilelive-file@4550754), which may produce the same filesystem layout that tilecache uses (I implemented it for compatibility w/ tilestache's filesystem cache).

For the record, I've switched to this approach. It solves my problem. I didn't know tilelive-copy had this feature before. Thanks! 👍

However it would still be good to have a proper fix inside tessera, no?