<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Segoe UI";
panose-1:2 11 5 2 4 2 4 2 2 3;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
code
{mso-style-priority:99;
font-family:"Courier New";}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.Shkpostityyli20
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.Shkpostityyli22
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="FI" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">Hi,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span lang="EN-US">When you seed tiles from WMS the slowest component is usually the WMS server. And what the WMS server is spending its time is usually rendering. And what makes rendering slow is usually that you render too much, and sometimes
that WMS server gets the vector data too slow because the backend data source is slow, very often because it has poor indexes which do not for suit the queries that WMS server is
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">sending.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">If you have a slow WMS you can’t get more speed out of it by bombing it with very many parallel threads. I would have a try if using 2-4, perhaps 8 threads makes seeding faster but if there is no difference I would concentrate
on the WMS configuration. BTW. why did you <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">select to use 20000 threads?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">I am not sure why you see just 1 core running on 100% load. I guess that you are now running WMS and MapCache on the same computer. Mapserver program is using only 1 core but http server should start more processes for
parallel requests. That CPU load is at 100% is generally a good thing because it tells that CPU is working, instead of waiting data from a slow disk. Another thing is that CPU may be doing something heavy and stupid.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Myself I would start by gathering some information about how Mapserver WMS is working be setting “DEBUG 2” on each layer or for the whole mapfile. By looking at the rendering times of each layer the slowest layers can
be found and then it is time to think about how to make those layers faster.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">You wrote that you use SQLite cache so it would be good to know that SQLite database does not support parallel writes
<a href="https://sqlite.org/lockingv3.html">https://sqlite.org/lockingv3.html</a>. Seeding with a few parallel threads may increase the speed if the WMS is the slowest part in your chain but when you have saturated the SQLite writing speed adding threads will
do no good. If you have optimized your WMS so well that SQLite sets the limits then you should test other cache alternatives. I guess that file cache would be faster to seed from multiple threads.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">When is comes to ticket </span><span lang="EN-GB"><a href="https://github.com/mapserver/mapcache/issues/169">https://github.com/mapserver/mapcache/issues/169</a> the latest comment is from a Mpaserver developer and their
last words “</span><span lang="EN-US" style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#24292E;background:white">I wouldn't mind helping to fix if there were interested parties...” feel rather polite to me.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#24292E;background:white"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#24292E;background:white">-Jukka Rahkonen-</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="mso-fareast-language:FI">Lähettäjä:</span></b><span style="mso-fareast-language:FI"> mapserver-users <mapserver-users-bounces@lists.osgeo.org>
<b>Puolesta </b>Sebastiano Laini<br>
<b>Lähetetty:</b> perjantai 8. marraskuuta 2019 14.35<br>
<b>Vastaanottaja:</b> 'mapserver-users@lists.osgeo.org' <mapserver-users@lists.osgeo.org><br>
<b>Aihe:</b> [mapserver-users] mapcache seed speed optimization<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span lang="EN-GB">Hi all,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">I’m evaluating new stacks but seems that I’m stuck with mapserver and mapcache due to the input source of our maps, so I’m trying to improve the speed of the mapserver/mapcache stack and what I cannot improve is the seed
speed.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">The –n and –p parameters seems to be useless.<o:p></o:p></span></p>
<p style="line-height:18.0pt;background:white"><strong><span lang="EN-GB" style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#3E4349">-n | –nthreads</span></strong><span lang="EN-GB" style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#3E4349">:
number of parallel threads that should be used to request tiles from the WMS source. The default is 1, but can be set higher if the WMS server can withstand parallel requests. (As a rule of thumb, the value chosen here should never be much higher than the
number of CPUs on the WMS server.)<o:p></o:p></span></p>
<p style="line-height:18.0pt;background:white"><strong><span lang="EN-GB" style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#3E4349">-p | –nprocesses</span></strong><span lang="EN-GB" style="font-size:9.5pt;font-family:"Arial",sans-serif;color:#3E4349">:
number of parallel processes that should be used to request tiles from the WMS source.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">I’ve tried to use –n 20000 and –p 60000 but for some reason the CPU load is max 100% in 1 core, no way to make it use more power and speed up the seed process.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">Max memory used is 470MB between Centos 7 (with all the services running) and the seed process.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">I’ve setup apache to use HTT/2 and it make quite a difference when the cache is already seeded but during the seed process doesn’t seems to be affected.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">My MapServer build is:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><code><span lang="EN-GB" style="font-size:10.0pt">./configure \</span></code><code><span style="font-size:10.0pt"><o:p></o:p></span></code></p>
<p class="MsoNormal"><code><span lang="EN-GB" style="font-size:10.0pt">--with-ogr=/usr/local/bin/gdal-config \<o:p></o:p></span></code></p>
<p class="MsoNormal"><code><span lang="EN-GB" style="font-size:10.0pt">--with-gdal=/usr/local/bin/gdal-config \<o:p></o:p></span></code></p>
<p class="MsoNormal"><code><span lang="EN-GB" style="font-size:10.0pt">--with-wfsclient \<o:p></o:p></span></code></p>
<p class="MsoNormal"><code><span lang="EN-GB" style="font-size:10.0pt">--with-wmsclient \<o:p></o:p></span></code></p>
<p class="MsoNormal"><code><span lang="EN-GB" style="font-size:10.0pt">--with-wfs \<o:p></o:p></span></code></p>
<p class="MsoNormal"><code><span lang="EN-GB" style="font-size:10.0pt">--enable-debug \<o:p></o:p></span></code></p>
<p class="MsoNormal"><code><span lang="EN-GB" style="font-size:10.0pt">--with-curl-config=/usr/bin/curl-config \<o:p></o:p></span></code></p>
<p class="MsoNormal"><code><span lang="EN-GB" style="font-size:10.0pt">--with-proj \<o:p></o:p></span></code></p>
<p class="MsoNormal"><code><span lang="EN-GB" style="font-size:10.0pt">--with-jpeg \<o:p></o:p></span></code></p>
<p class="MsoNormal"><code><span lang="EN-GB" style="font-size:10.0pt">--with-freetype \<o:p></o:p></span></code></p>
<p class="MsoNormal"><code><span lang="EN-GB" style="font-size:10.0pt">--with-postgis=/usr/pgsql-9.3/bin/pg_config \<o:p></o:p></span></code></p>
<p class="MsoNormal"><code><span lang="EN-GB" style="font-size:10.0pt">--with-geos=/usr/local/bin/geos-config<o:p></o:p></span></code></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span lang="EN-GB">I know there is mapserver 7 and yet I’m using 6 for the test.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">While the MapCache is: <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">cmake -DCMAKE_PREFIX_PATH="/usr/bin/sqlite3" \<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> -DWITH_SQLITE=1 \<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> -DWITH_BERKELEY_DB=0 \<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> -DWITH_TIFF=0 \<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> -DWITH_GEOTIFF=0 \<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> -DWITH_FCGI=0 \<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> -DWITH_PCRE=0 \<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> -DWITH_PIXMAN=1 \<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> -DWITH_OGR=1 \<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> -DWITH_GEOS=1 \<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> ../<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">MapCache is the latest version and this is my mapcache.xml file, we use sqlite as a cache method<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> <cache name="cache_sqlite" type="sqlite3"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> <dbfile>/home/www/html/maps/cache/{tileset}/{grid}/{z}/{x}-{y}.sqlite3</dbfile><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> <xcount>10000</xcount><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> <ycount>10000</ycount><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> <pragma name="max_page_count">1573741823</pragma><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> </cache><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> <tileset name="bccache"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> <source>bcmaps</source><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> <cache>cache_sqlite</cache><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> <grid>bcgrid</grid><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> <format>PNG</format><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> <metatile>16 16</metatile><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> <metabuffer>0</metabuffer><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> <expires>2628000</expires><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> </tileset><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> <grid name="bcgrid"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> <metadata><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> <title>BC custom grid</title><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> </metadata><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> <srs>EPSG:27700</srs><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> <size>256 256</size><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> <extent>0 0 700000 1250000</extent><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> <resolutions>70 28 14 7 2.8 1.4 0.7</resolutions><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> </grid><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> <format name="mypng" type="PNG"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> <compression>best</compression><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> </format><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> <service type="wms" enabled="true"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> <full_wms>assemble</full_wms><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> <resample_mode>bilinear</resample_mode><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> <format>mypng</format><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> <maxsize>4096</maxsize><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> </service><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">I’ve also tried to use –DWITH_MAPSERVER but seems that the integration is not working and they doen’t care?
<a href="https://github.com/mapserver/mapcache/issues/169">https://github.com/mapserver/mapcache/issues/169</a> this issue was open 2 years ago and still open, not even a reply from the developers.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">How can I improve the seed process?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">Should I increase the maxsize and metatile to requests more tiles and cache faster?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D;mso-fareast-language:EN-GB">Sebastiano Laini<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D;mso-fareast-language:EN-GB">Web Developer<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="color:#1F497D;mso-fareast-language:EN-GB">Buchanan Computing<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
</div>
</body>
</html>