<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body>
Will have a look... I hadn’t spent much time on optimization at this point. I wouldn’t doubt there might be improvements to be made. The simplification code would be the spot to look.<br>
<br>
Extent represents the resolution of the tile and guides the simplification done as part of the conversion from world to mvt coordinates. Edge buffer defines tile overlap.<br>
<br>
By chance have you done output comparisons at all? That is, tile to tile in terms of size: feature counts, point counts/feature, etc... That would help tune the format parameters and make sure MapServer is returning reasonable output.<br>
<br>
One quick note on the SQL (2) is to be explicit with returned columns, avoid *. The t-rex SQL hints at zoom-related attributes in the layer? Again, will have to find time to checkout the benchmark stuff.<br>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> karsten <karsten@terragis.net><br>
<b>Sent:</b> Monday, March 11, 2019 5:53:35 PM<br>
<b>To:</b> 'karsten'; mapserver-users@lists.osgeo.org<br>
<b>Cc:</b> 'Pirmin Kalberer'; Lime, Steve D (MNIT); lars@lingner.eu; jmckenna@gatewaygeomatics.com; thomas.bonfort@gmail.com<br>
<b>Subject:</b> RE: MapServer derived MVTiles Benchmark test vs. T-Rex tile server</font>
<div> </div>
</div>
<div>
<div dir="ltr" align="left"><font color="#000080" size="2" face="Arial"><span class="x_898095322-11032019">Corrected map file link:</span></font></div>
<div dir="ltr" align="left"><font color="#000080" size="2" face="Arial"><span class="x_898095322-11032019">
<div><font color="#000080" size="2" face="Arial"><span class="x_516235122-11032019"><a href="https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkarstenv%2Fmvt-benchmark%2Fblob%2Fmaster%2FMapServer%2Fne.map&data=02%7C01%7Csteve.lime%40state.mn.us%7Ca811ae33be2d46f58eed08d6a67469eb%7Ceb14b04624c445198f26b89c2159828c%7C0%7C0%7C636879416281231683&sdata=tjeEJeuolr3ihw9fj%2B2ejyrDb46FJRR3haXMjhRMO1I%3D&reserved=0" originalsrc="https://github.com/karstenv/mvt-benchmark/blob/master/MapServer/ne.map" shash="OB56Hh7Z9IpOjmRxJbjgo27kscbQklOf4SunROBB0s0au5Vd5szm9+QXigTH30wuT261GGY5oEcLnhjZDkbNwMLeGSrpV1d4QpFdscOi3lpIPs4fa9LKzGO772dfYCNAs9hIAq42kddLgTXgmbzRrLPWJqqF8ZMpFd8O8rR/jiM=">https://github.com/karstenv/mvt-benchmark/blob/master/MapServer/ne.map</a></span></font></div>
</span></font></div>
<br>
<div lang="en-us" class="x_OutlookMessageHeader" dir="ltr" align="left">
<hr tabindex="-1">
<font size="2" face="Tahoma"><b>From:</b> karsten [mailto:karsten@terragis.net] <br>
<b>Sent:</b> Monday, March 11, 2019 23:15<br>
<b>To:</b> 'mapserver-users@lists.osgeo.org'<br>
<b>Cc:</b> 'Pirmin Kalberer'; 'Lime, Steve D (MNIT)'; 'lars@lingner.eu'; 'jmckenna@gatewaygeomatics.com'; 'thomas.bonfort@gmail.com'<br>
<b>Subject:</b> MapServer derived MVTiles Benchmark test vs. T-Rex tile server<br>
</font><br>
</div>
<div></div>
<div dir="ltr" align="left"><span style="font-size:10pt; font-family:"Arial",sans-serif; color:navy">Dear All,</span></div>
<div lang="EN-US">
<div class="x_x_WordSection1">
<div>
<p class="x_x_MsoNormal"><span style="font-size:10pt; font-family:"Arial",sans-serif; color:navy">I <span class="x_131533721-11032019"> am </span>working <span class="x_131533721-11032019">on optimizing a MapServer /MapCache configuration of </span><span class="x_131533721-11032019">a
benchmark for creating <span style="font-size:10pt; font-family:"Arial",sans-serif; color:navy">MVT tiles (seeding) of MapServer - versus T-Rex.
</span></span></span></p>
<p class="x_x_MsoNormal"><span style="font-size:10pt; font-family:"Arial",sans-serif; color:navy"><span class="x_131533721-11032019"><span style="font-size:10pt; font-family:"Arial",sans-serif; color:navy">This will be featured in a presentation at the German FOSS4g
('FOSSGIS') 2019 in Dresden at the end of this week. The whole thing is based on Pirmin Kalberer's automated vector tile benchmark see -
<a href="https://gcc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fblog.sourcepole.ch%2F2018%2F09%2F06%2Fthe-tale-of-a-vector-tile-benchmark%2F&data=02%7C01%7Csteve.lime%40state.mn.us%7Ca811ae33be2d46f58eed08d6a67469eb%7Ceb14b04624c445198f26b89c2159828c%7C0%7C0%7C636879416281241691&sdata=Rq7vxQk62vpPNJ4qjeq4fkLSuvSJ46t3Bicj760mTCU%3D&reserved=0" originalsrc="http://blog.sourcepole.ch/2018/09/06/the-tale-of-a-vector-tile-benchmark/" shash="t3xSfwg6ApU/0IhOQTjs3JmpR6gguAYOwHR7Ytx5LFpEbz2qIFpp4WtcJFIWSqxvXC+DV8kEHdFSmkZtHfFApua/Scn5NIhVF0ksW2yZeAPRRURDVXQkA+KUw9mxhvQ9H7EfOIqZO0qhih+hqTLuqrXpZvOEq2II3p1F2UWhgV8=">
http://blog.sourcepole.ch/2018/09/06/the-tale-of-a-vector-tile-benchmark/</a> . Thus I created a similar configuration of the Natural Earth Data based world map for MapServer (my favorite rendering engine !) to run it against T-REX.. Now Pirmin has added this
into the GitHUB repository as a 'MapServer' branch see - <a href="https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fpka%2Fmvt-benchmark%2Ftree%2Fmapserver&data=02%7C01%7Csteve.lime%40state.mn.us%7Ca811ae33be2d46f58eed08d6a67469eb%7Ceb14b04624c445198f26b89c2159828c%7C0%7C0%7C636879416281251699&sdata=0j4MMJBWx5IYNszsZPrSY1RRTGzvicRUoPw1BZGGcqk%3D&reserved=0" originalsrc="https://github.com/pka/mvt-benchmark/tree/mapserver" shash="wsIWU3b/MBJW4sDqZNT5CUqZATuJ73OFtXXf4W9ehQ/sVS4Qp0ajwy+aKRus4iTTg/KT3iSqCm+SfsO0bZ+Zsi7luSaNBJY5M+8Zibc/Czr0yJkh3ZMtblB0u1loIMa1gXday62LdWmIT+GXyOrH6tv77jiSrV2GmLhwiTdgOp0=">
https://github.com/pka/mvt-benchmark/tree/mapserver</a></span></span></span></p>
<p class="x_x_MsoNormal"><span style="font-size:10pt; font-family:"Arial",sans-serif; color:navy"><span class="x_131533721-11032019">Results of one of my first trial benchmarks can be found here
<a href="https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkarstenv%2Fmvt-benchmark%2Ftree%2Fmaster%2Ffirsttests&data=02%7C01%7Csteve.lime%40state.mn.us%7Ca811ae33be2d46f58eed08d6a67469eb%7Ceb14b04624c445198f26b89c2159828c%7C0%7C0%7C636879416281261701&sdata=cu3VQvVDV2ieGjZs6Lld9kLa7SK0W%2FJe27Ak%2BJqcZSc%3D&reserved=0" originalsrc="https://github.com/karstenv/mvt-benchmark/tree/master/firsttests" shash="DmVSxWR0mr9S/ySPOC1RYynSZsX2qmoHbCkavqI/I1wS5x1OSnHF3Xo9qRph6JpB+cMfyyDAHALuZe1tOrKMeESXZ/VDzbScfMxy5tiaQAvY49Eepfv3h+6w4bI944iRNDggOU/80Z7kb0PsXPMrIhmg1+9M1lMs5INpE/ov6as=">
https://github.com/karstenv/mvt-benchmark/tree/master/firsttests</a> .</span></span></p>
<p class="x_x_MsoNormal"><span style="font-size:10pt; font-family:"Arial",sans-serif; color:navy"><span class="x_131533721-11032019">Unfortunately my favorite software (MapServer /MapCache.fcgi) is falling pretty far behind in seeding speed in these tests
<br>
</span></span><span style="font-size:10pt; font-family:"Arial",sans-serif; color:navy"><span class="x_131533721-11032019">see e.g. in <span style="font-size:10pt; font-family:"Arial",sans-serif; color:navy"><span class="x_131533721-11032019"><a href="https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkarstenv%2Fmvt-benchmark%2Fblob%2Fmaster%2Ffirsttests%2Fhttp.jpg&data=02%7C01%7Csteve.lime%40state.mn.us%7Ca811ae33be2d46f58eed08d6a67469eb%7Ceb14b04624c445198f26b89c2159828c%7C0%7C0%7C636879416281271709&sdata=K7o4vKUwAbL79z%2BPMElpMwMkPXtRNr%2BQO708LgTJN%2Fg%3D&reserved=0" originalsrc="https://github.com/karstenv/mvt-benchmark/blob/master/firsttests/http.jpg" shash="UPyW06nFV02fEsmEjiwPcTdYQCGZYQ9DrRhZeM7c2YyIHxp0ysWou172otLny0iDi/TBxEgmnECU2EOLKptg4H4KHho0fI2BGXu755ax8+NTZiU9JWWF92b7BFULJi4QeQuKF39oT+yQnU1saJrF0IMDYG5mb0ZOxkIyqH0TjpU=">https://github.com/karstenv/mvt-benchmark/blob/master/firsttests/http.jpg</a></span></span></span></span></p>
<p class="x_x_MsoNormal"><span style="font-size:10pt; font-family:"Arial",sans-serif; color:navy"><span class="x_131533721-11032019">Thus, I wanted to call in the experts to find out what configuration tweaks will help to increase the MapServer / Mapcache performance.</span></span></p>
<p class="x_x_MsoNormal"><span style="font-size:10pt; font-family:"Arial",sans-serif; color:navy"><span class="x_131533721-11032019">So far my next steps I will be
</span></span><span style="font-size:10pt; font-family:"Arial",sans-serif; color:navy"><span class="x_131533721-11032019">working on
<br>
--------------------------------------------------------------------------------------------------<br>
</span></span><span style="font-size:10pt; font-family:"Arial",sans-serif; color:navy"><span class="x_131533721-11032019">1.) Setting the correct map file parameters to match the T-Rex configuration for the FORMATOPTIONS
<br>
"EXTENT" and "EDGE_BUFFER" in the <span style="font-size:10pt; font-family:"Arial",sans-serif; color:navy">
<span class="x_131533721-11032019"><span lang="EN">OUTPUTFORMAT<span class="x_131533721-11032019"> bloc.</span></span></span></span></span></span></p>
<p class="x_x_MsoNormal"><span style="font-size:10pt; font-family:"Arial",sans-serif; color:navy"><span class="x_131533721-11032019"><span lang="EN">OUTPUTFORMAT<span class="x_131533721-11032019"> <br>
</span><span class="x_131533721-11032019"> </span>NAME "mvt"<span class="x_131533721-11032019"> <br>
</span><span class="x_131533721-11032019"> </span>DRIVER MVT<span class="x_131533721-11032019"> <br>
</span><span class="x_131533721-11032019"> </span>FORMATOPTION "EXTENT=512" <span lang="EN"># default is 4096. # Comment by Pirmin for t-rex this is 'tile_size' (per Layer): Width and height of the tile (Default: 4096. Grid default size is 256)<br>
</span><span class="x_131533721-11032019"> </span>FORMATOPTION "EDGE_BUFFER=20"<span class="x_131533721-11032019"> # Comment by Pirmin - seems to be the '<span lang="EN">buffer_size' in mvtbench.toml , is set to 3 for the Country layer, for all others<br>
</span></span>END<br>
<br>
Any suggestions wthat my best bet settings are in the map file is for MapServer for "EXTENT" and "EDGE_BUFFER" ?</span></span></span></p>
<p class="x_x_MsoNormal"><span style="font-size:10pt; font-family:"Arial",sans-serif; color:navy"><span class="x_131533721-11032019"><span lang="EN">--------------------------------------------------------------------------------------------------<br>
</span></span></span><span style="font-size:10pt; font-family:"Arial",sans-serif; color:navy"><span class="x_131533721-11032019"><span lang="EN">2.) Tweaking the data statement of the map file to match the T-Rex config
<br>
</span></span></span><span style="font-size:10pt; font-family:"Arial",sans-serif; color:navy"><span class="x_131533721-11032019"><span lang="EN">So far I have e.g. for the countries layer<br>
<font color="#800000">DATA "wkb_geometry from (SELECT * FROM ne_10m_admin_0_countries) as temp using unique ogc_fid using SRID=3857"</font><br>
</span></span></span><span style="font-size:10pt; font-family:"Arial",sans-serif; color:navy"><span class="x_131533721-11032019"><span lang="EN">while T-Rex has<br>
<font color="#800000"><span class="x_pl-smi">sql</span> = <span class="x_pl-s"><span class="x_pl-pds">"""</span>SELECT wkb_geometry, adm0_a3, mapcolor7 FROM ne_10m_admin_0_countries WHERE min_zoom::integer <= !zoom! AND wkb_geometry && !bbox!<span class="x_pl-pds">"""</span></span></font>
<br>
</span></span></span><span style="font-size:10pt; font-family:"Arial",sans-serif; color:navy"><span class="x_131533721-11032019"><span lang="EN">see
<a href="https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkarstenv%2Fmvt-benchmark%2Fblob%2Fmaster%2Ft-rex%2Fmvtbench.toml&data=02%7C01%7Csteve.lime%40state.mn.us%7Ca811ae33be2d46f58eed08d6a67469eb%7Ceb14b04624c445198f26b89c2159828c%7C0%7C0%7C636879416281281722&sdata=nWLjUb0zFYZZuNFq6FGc4aaQsF9bhJJnD5Zlnpoheos%3D&reserved=0" originalsrc="https://github.com/karstenv/mvt-benchmark/blob/master/t-rex/mvtbench.toml" shash="NsyPqSeDTa14y8ELIGIVZiJ0GCcywfmhH02Pcwhf9U25egcAanHN4LwPwQdk6C6h71zgEeTMqcvPVLFFGs9jwxkwLD2CTLgtXh2uQ7R1XFrBTMrGPSvUc4aQ0lxuGLIwj4dxWv3ouBJ/6b9L+BKEjp7hDR9oIAxyXL+Tm29bMpI=">
https://github.com/karstenv/mvt-benchmark/blob/master/t-rex/mvtbench.toml</a><br>
I am still trying to match the T-Rex statement above but did not come up with a suitable comparable MapServer data statement so far.<br>
</span></span></span><span style="font-size:10pt; font-family:"Arial",sans-serif; color:navy"><span class="x_131533721-11032019"><span lang="EN"><font color="#000000" size="3" face="Times New Roman"><font color="#000080" size="2" face="Arial"><br>
Any suggestions for this anyone ?<br>
--------------------------------------------------------------------------------------------------</font></font></span></span></span></p>
<p class="x_x_MsoNormal"><span style="font-size:10pt; font-family:"Arial",sans-serif; color:navy"><span class="x_131533721-11032019"><span lang="EN"></span></span></span><span style="font-size:10pt; font-family:"Arial",sans-serif; color:navy"><span class="x_131533721-11032019">3.)
Does anyone have any other/ additional suggestions what else might help to increase MapServer /MapCache performance ?</span></span></p>
</div>
<div>
<p class="x_x_MsoNormal"><span style="font-size:10pt; font-family:"Arial",sans-serif; color:navy">Thanks<span class="x_131533721-11032019"> <br>
</span></span><span style="font-size:10pt; font-family:"Arial",sans-serif; color:navy">Karsten<span class="x_131533721-11032019"> </span></span></p>
<p class="x_x_MsoNormal"><span style="font-size:10pt; font-family:"Arial",sans-serif; color:navy"><span class="x_131533721-11032019">PS:
</span></span><span style="font-size:10pt; font-family:"Arial",sans-serif; color:navy"><span class="x_131533721-11032019">The map file used is here
<a href="https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkarstenv%2Fmvt-benchmark%2Fblob%2Fmaster%2FMapServer%2Fmapcache_mvt.xml&data=02%7C01%7Csteve.lime%40state.mn.us%7Ca811ae33be2d46f58eed08d6a67469eb%7Ceb14b04624c445198f26b89c2159828c%7C0%7C0%7C636879416281291730&sdata=TIhHGj9EjFXidv6BvhbAloVb9IYYcM3bnOhzd5xiq18%3D&reserved=0" originalsrc="https://github.com/karstenv/mvt-benchmark/blob/master/MapServer/mapcache_mvt.xml" shash="FHBvqdBjkdOLSH46pR19rBdf/y7PEXnc1TRV+QIdFICmVFzEoZ9oc80KzHQBi6ue731zhn/ZhmjBsqFPN4ZxYZ2Y/xkFe88SK+rH16aWh2sYHSVxJPOUzw5eQge4oRdIt8eFg/HXBbKbEoWA986Kt6ZgGyxjrlIRbHghe4gR5PI=">
https://github.com/karstenv/mvt-benchmark/blob/master/MapServer/mapcache_mvt.xml</a>,<br>
</span></span><span style="font-size:10pt; font-family:"Arial",sans-serif; color:navy"><span class="x_131533721-11032019">the MapCache config here
<a href="https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkarstenv%2Fmvt-benchmark%2Fblob%2Fmaster%2FMapServer%2Fmapcache_mvt.xml&data=02%7C01%7Csteve.lime%40state.mn.us%7Ca811ae33be2d46f58eed08d6a67469eb%7Ceb14b04624c445198f26b89c2159828c%7C0%7C0%7C636879416281301738&sdata=VEuw6Lx8Me8jdU16oTOKQv17CCOJJ3mLTrPxh%2FJ9pu4%3D&reserved=0" originalsrc="https://github.com/karstenv/mvt-benchmark/blob/master/MapServer/mapcache_mvt.xml" shash="ZSKxuS6x3xr1ybSxxNZiFPHQbFdLq1/x4pgCe5YumE0oCSO3u/xTt635rd9HuNZzG1w6S5uiWRnUor9Qna0cmGMsIKzCL1BtnJAtBt0e/khmaDYwnX4lE6DdJIOTfoevs1ijJNoG/qqvZW/tW9YHmAAQKW3HlbAN5kKGdIIGslM=">
https://github.com/karstenv/mvt-benchmark/blob/master/MapServer/mapcache_mvt.xml</a> </span></span></p>
</div>
<div>
<p class="x_x_MsoNormal"><span style="font-size:10pt; font-family:"Arial",sans-serif; color:navy"></span> </p>
</div>
</div>
</div>
</div>
</body>
</html>