[mapserver-users] MapServer derived MVTiles Benchmark test vs. T-Rex tile server

Mon Mar 11 17:19:41 PDT 2019

Will have a look... I hadn’t spent much time on optimization at this point. I wouldn’t doubt there might be improvements to be made. The simplification code would be the spot to look.

Extent represents the resolution of the tile and guides the simplification done as part of the conversion from world to mvt coordinates. Edge buffer defines tile overlap.

By chance have you done output comparisons at all? That is, tile to tile in terms of size: feature counts, point counts/feature, etc... That would help tune the format parameters and make sure MapServer is returning reasonable output.

One quick note on the SQL (2) is to be explicit with returned columns, avoid *. The t-rex SQL hints at zoom-related attributes in the layer? Again, will have to find time to checkout the benchmark stuff.
________________________________
From: karsten <karsten at terragis.net>
Sent: Monday, March 11, 2019 5:53:35 PM
To: 'karsten'; mapserver-users at lists.osgeo.org
Cc: 'Pirmin Kalberer'; Lime, Steve D (MNIT); lars at lingner.eu; jmckenna at gatewaygeomatics.com; thomas.bonfort at gmail.com
Subject: RE: MapServer derived MVTiles Benchmark test vs. T-Rex tile server

Corrected map file link:
https://github.com/karstenv/mvt-benchmark/blob/master/MapServer/ne.map<https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkarstenv%2Fmvt-benchmark%2Fblob%2Fmaster%2FMapServer%2Fne.map&data=02%7C01%7Csteve.lime%40state.mn.us%7Ca811ae33be2d46f58eed08d6a67469eb%7Ceb14b04624c445198f26b89c2159828c%7C0%7C0%7C636879416281231683&sdata=tjeEJeuolr3ihw9fj%2B2ejyrDb46FJRR3haXMjhRMO1I%3D&reserved=0>

________________________________
From: karsten [mailto:karsten at terragis.net]
Sent: Monday, March 11, 2019 23:15
To: 'mapserver-users at lists.osgeo.org'
Cc: 'Pirmin Kalberer'; 'Lime, Steve D (MNIT)'; 'lars at lingner.eu'; 'jmckenna at gatewaygeomatics.com'; 'thomas.bonfort at gmail.com'
Subject: MapServer derived MVTiles Benchmark test vs. T-Rex tile server

Dear All,

I  am working on optimizing a MapServer /MapCache configuration of  a benchmark for creating MVT tiles (seeding) of MapServer - versus T-Rex.

This will be featured in a presentation at the German FOSS4g ('FOSSGIS') 2019 in Dresden at the end of this week. The whole thing is based on Pirmin Kalberer's automated vector tile benchmark see - http://blog.sourcepole.ch/2018/09/06/the-tale-of-a-vector-tile-benchmark/<https://gcc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fblog.sourcepole.ch%2F2018%2F09%2F06%2Fthe-tale-of-a-vector-tile-benchmark%2F&data=02%7C01%7Csteve.lime%40state.mn.us%7Ca811ae33be2d46f58eed08d6a67469eb%7Ceb14b04624c445198f26b89c2159828c%7C0%7C0%7C636879416281241691&sdata=Rq7vxQk62vpPNJ4qjeq4fkLSuvSJ46t3Bicj760mTCU%3D&reserved=0> . Thus I created a similar configuration of the Natural Earth Data based world map for MapServer (my favorite rendering engine !) to run it against T-REX.. Now Pirmin has added this into the GitHUB repository as a 'MapServer' branch see - https://github.com/pka/mvt-benchmark/tree/mapserver<https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fpka%2Fmvt-benchmark%2Ftree%2Fmapserver&data=02%7C01%7Csteve.lime%40state.mn.us%7Ca811ae33be2d46f58eed08d6a67469eb%7Ceb14b04624c445198f26b89c2159828c%7C0%7C0%7C636879416281251699&sdata=0j4MMJBWx5IYNszsZPrSY1RRTGzvicRUoPw1BZGGcqk%3D&reserved=0>

Results of one of my first trial benchmarks can be found here https://github.com/karstenv/mvt-benchmark/tree/master/firsttests<https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkarstenv%2Fmvt-benchmark%2Ftree%2Fmaster%2Ffirsttests&data=02%7C01%7Csteve.lime%40state.mn.us%7Ca811ae33be2d46f58eed08d6a67469eb%7Ceb14b04624c445198f26b89c2159828c%7C0%7C0%7C636879416281261701&sdata=cu3VQvVDV2ieGjZs6Lld9kLa7SK0W%2FJe27Ak%2BJqcZSc%3D&reserved=0> .

Unfortunately my favorite software (MapServer /MapCache.fcgi) is falling pretty far behind in seeding speed in these tests
see e.g. in https://github.com/karstenv/mvt-benchmark/blob/master/firsttests/http.jpg<https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkarstenv%2Fmvt-benchmark%2Fblob%2Fmaster%2Ffirsttests%2Fhttp.jpg&data=02%7C01%7Csteve.lime%40state.mn.us%7Ca811ae33be2d46f58eed08d6a67469eb%7Ceb14b04624c445198f26b89c2159828c%7C0%7C0%7C636879416281271709&sdata=K7o4vKUwAbL79z%2BPMElpMwMkPXtRNr%2BQO708LgTJN%2Fg%3D&reserved=0>

Thus, I wanted to call in the experts to find out what configuration tweaks will help to increase the MapServer / Mapcache  performance.

So far my next steps I will be working on
--------------------------------------------------------------------------------------------------
1.) Setting the correct map file parameters to match the T-Rex configuration for the FORMATOPTIONS
"EXTENT" and "EDGE_BUFFER" in the OUTPUTFORMAT  bloc.

OUTPUTFORMAT
     NAME "mvt"
     DRIVER MVT
     FORMATOPTION "EXTENT=512" # default is 4096. # Comment by Pirmin for t-rex this is 'tile_size' (per Layer): Width and height of the tile (Default: 4096. Grid default size is 256)
     FORMATOPTION "EDGE_BUFFER=20"  # Comment by Pirmin -  seems to be the 'buffer_size' in mvtbench.toml , is set to 3 for the Country layer, for all others
END

Any suggestions wthat my best bet settings are in the map file is for MapServer for "EXTENT" and "EDGE_BUFFER" ?

--------------------------------------------------------------------------------------------------
2.) Tweaking the data statement of the map file to match the T-Rex config
So far I have e.g. for the countries layer
DATA "wkb_geometry from (SELECT * FROM ne_10m_admin_0_countries) as temp using unique ogc_fid using SRID=3857"
while T-Rex has
sql = """SELECT wkb_geometry, adm0_a3, mapcolor7 FROM ne_10m_admin_0_countries WHERE min_zoom::integer <= !zoom! AND wkb_geometry && !bbox!"""
see https://github.com/karstenv/mvt-benchmark/blob/master/t-rex/mvtbench.toml<https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkarstenv%2Fmvt-benchmark%2Fblob%2Fmaster%2Ft-rex%2Fmvtbench.toml&data=02%7C01%7Csteve.lime%40state.mn.us%7Ca811ae33be2d46f58eed08d6a67469eb%7Ceb14b04624c445198f26b89c2159828c%7C0%7C0%7C636879416281281722&sdata=nWLjUb0zFYZZuNFq6FGc4aaQsF9bhJJnD5Zlnpoheos%3D&reserved=0>
I am still trying to match the T-Rex statement above but did not come up with a  suitable comparable MapServer data statement so far.

Any suggestions for this anyone ?
--------------------------------------------------------------------------------------------------

3.) Does anyone have any other/ additional suggestions what else might help to increase MapServer /MapCache performance ?

Thanks
Karsten

PS: The map file used is here https://github.com/karstenv/mvt-benchmark/blob/master/MapServer/mapcache_mvt.xml<https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkarstenv%2Fmvt-benchmark%2Fblob%2Fmaster%2FMapServer%2Fmapcache_mvt.xml&data=02%7C01%7Csteve.lime%40state.mn.us%7Ca811ae33be2d46f58eed08d6a67469eb%7Ceb14b04624c445198f26b89c2159828c%7C0%7C0%7C636879416281291730&sdata=TIhHGj9EjFXidv6BvhbAloVb9IYYcM3bnOhzd5xiq18%3D&reserved=0>,
the MapCache config here https://github.com/karstenv/mvt-benchmark/blob/master/MapServer/mapcache_mvt.xml<https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkarstenv%2Fmvt-benchmark%2Fblob%2Fmaster%2FMapServer%2Fmapcache_mvt.xml&data=02%7C01%7Csteve.lime%40state.mn.us%7Ca811ae33be2d46f58eed08d6a67469eb%7Ceb14b04624c445198f26b89c2159828c%7C0%7C0%7C636879416281301738&sdata=VEuw6Lx8Me8jdU16oTOKQv17CCOJJ3mLTrPxh%2FJ9pu4%3D&reserved=0>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-users/attachments/20190312/c314b1e7/attachment.htm>