<div dir="ltr"><div>Sebastiano,</div><div><br></div><div>Some though in my mind: <br></div><div>* –n 20000 : do you really have 2'000 CPU on your WMS server? if not, it could be worst than better, flooding your WMS server.<br></div><div>* what is the bottleneck? If WMS server is slow you can't make MapCache faster, you should optimise WMS Server. Multi-domain can help, data improvement, etc. also.</div><div>* Does your WMS is on the same server? same network? Does you network fast enough?</div><div>* metatile can help definitively!<br></div><div>* "we use sqlite as a cache method" => I don't know if sqlite is really fast or not, but this could be another bottleneck :)</div><div><br></div><div>What do you mean exactly by improve seed speed? :) This is matter of personal point of view some time.</div><div><br></div><div>Y.<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Le ven. 8 nov. 2019 à 13:43, Sebastiano Laini <<a href="mailto:Sebastiano.Laini@buchanancomputing.co.uk">Sebastiano.Laini@buchanancomputing.co.uk</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-GB">
<div class="gmail-m_5762177471845286621WordSection1">
<p class="MsoNormal">Hi all,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I’m evaluating new stacks but seems that I’m stuck with mapserver and mapcache due to the input source of our maps, so I’m trying to improve the speed of the mapserver/mapcache stack and what I cannot improve is the seed speed.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">The –n and –p parameters seems to be useless.<u></u><u></u></p>
<p style="line-height:18pt;background:white none repeat scroll 0% 0%"><strong><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:rgb(62,67,73)">-n | –nthreads</span></strong><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:rgb(62,67,73)">: number of parallel threads
that should be used to request tiles from the WMS source. The default is 1, but can be set higher if the WMS server can withstand parallel requests. (As a rule of thumb, the value chosen here should never be much higher than the number of CPUs on the WMS server.)<u></u><u></u></span></p>
<p style="line-height:18pt;background:white none repeat scroll 0% 0%"><strong><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:rgb(62,67,73)">-p | –nprocesses</span></strong><span style="font-size:9.5pt;font-family:"Arial",sans-serif;color:rgb(62,67,73)">: number of parallel
processes that should be used to request tiles from the WMS source.<u></u><u></u></span></p>
<p class="MsoNormal">I’ve tried to use –n 20000 and –p 60000 but for some reason the CPU load is max 100% in 1 core, no way to make it use more power and speed up the seed process.<u></u><u></u></p>
<p class="MsoNormal">Max memory used is 470MB between Centos 7 (with all the services running) and the seed process.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I’ve setup apache to use HTT/2 and it make quite a difference when the cache is already seeded but during the seed process doesn’t seems to be affected.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">My MapServer build is:<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><code><span style="font-size:10pt">./configure \<u></u><u></u></span></code></p>
<p class="MsoNormal"><code><span style="font-size:10pt">--with-ogr=/usr/local/bin/gdal-config \<u></u><u></u></span></code></p>
<p class="MsoNormal"><code><span style="font-size:10pt">--with-gdal=/usr/local/bin/gdal-config \<u></u><u></u></span></code></p>
<p class="MsoNormal"><code><span style="font-size:10pt">--with-wfsclient \<u></u><u></u></span></code></p>
<p class="MsoNormal"><code><span style="font-size:10pt">--with-wmsclient \<u></u><u></u></span></code></p>
<p class="MsoNormal"><code><span style="font-size:10pt">--with-wfs \<u></u><u></u></span></code></p>
<p class="MsoNormal"><code><span style="font-size:10pt">--enable-debug \<u></u><u></u></span></code></p>
<p class="MsoNormal"><code><span style="font-size:10pt">--with-curl-config=/usr/bin/curl-config \<u></u><u></u></span></code></p>
<p class="MsoNormal"><code><span style="font-size:10pt">--with-proj \<u></u><u></u></span></code></p>
<p class="MsoNormal"><code><span style="font-size:10pt">--with-jpeg \<u></u><u></u></span></code></p>
<p class="MsoNormal"><code><span style="font-size:10pt">--with-freetype \<u></u><u></u></span></code></p>
<p class="MsoNormal"><code><span style="font-size:10pt">--with-postgis=/usr/pgsql-9.3/bin/pg_config \<u></u><u></u></span></code></p>
<p class="MsoNormal"><code><span style="font-size:10pt">--with-geos=/usr/local/bin/geos-config<u></u><u></u></span></code></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I know there is mapserver 7 and yet I’m using 6 for the test.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">While the MapCache is: <u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">cmake -DCMAKE_PREFIX_PATH="/usr/bin/sqlite3" \<u></u><u></u></p>
<p class="MsoNormal"> -DWITH_SQLITE=1 \<u></u><u></u></p>
<p class="MsoNormal"> -DWITH_BERKELEY_DB=0 \<u></u><u></u></p>
<p class="MsoNormal"> -DWITH_TIFF=0 \<u></u><u></u></p>
<p class="MsoNormal"> -DWITH_GEOTIFF=0 \<u></u><u></u></p>
<p class="MsoNormal"> -DWITH_FCGI=0 \<u></u><u></u></p>
<p class="MsoNormal"> -DWITH_PCRE=0 \<u></u><u></u></p>
<p class="MsoNormal"> -DWITH_PIXMAN=1 \<u></u><u></u></p>
<p class="MsoNormal"> -DWITH_OGR=1 \<u></u><u></u></p>
<p class="MsoNormal"> -DWITH_GEOS=1 \<u></u><u></u></p>
<p class="MsoNormal"> ../<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">MapCache is the latest version and this is my mapcache.xml file, we use sqlite as a cache method<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"> <cache name="cache_sqlite" type="sqlite3"><u></u><u></u></p>
<p class="MsoNormal"> <dbfile>/home/www/html/maps/cache/{tileset}/{grid}/{z}/{x}-{y}.sqlite3</dbfile><u></u><u></u></p>
<p class="MsoNormal"> <xcount>10000</xcount><u></u><u></u></p>
<p class="MsoNormal"> <ycount>10000</ycount><u></u><u></u></p>
<p class="MsoNormal"> <pragma name="max_page_count">1573741823</pragma><u></u><u></u></p>
<p class="MsoNormal"> </cache><u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"> <tileset name="bccache"><u></u><u></u></p>
<p class="MsoNormal"> <source>bcmaps</source><u></u><u></u></p>
<p class="MsoNormal"> <cache>cache_sqlite</cache><u></u><u></u></p>
<p class="MsoNormal"> <grid>bcgrid</grid><u></u><u></u></p>
<p class="MsoNormal"> <format>PNG</format><u></u><u></u></p>
<p class="MsoNormal"> <metatile>16 16</metatile><u></u><u></u></p>
<p class="MsoNormal"> <metabuffer>0</metabuffer><u></u><u></u></p>
<p class="MsoNormal"> <expires>2628000</expires><u></u><u></u></p>
<p class="MsoNormal"> </tileset><u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"> <grid name="bcgrid"><u></u><u></u></p>
<p class="MsoNormal"> <metadata><u></u><u></u></p>
<p class="MsoNormal"> <title>BC custom grid</title><u></u><u></u></p>
<p class="MsoNormal"> </metadata><u></u><u></u></p>
<p class="MsoNormal"> <srs>EPSG:27700</srs><u></u><u></u></p>
<p class="MsoNormal"> <size>256 256</size><u></u><u></u></p>
<p class="MsoNormal"> <extent>0 0 700000 1250000</extent><u></u><u></u></p>
<p class="MsoNormal"> <resolutions>70 28 14 7 2.8 1.4 0.7</resolutions><u></u><u></u></p>
<p class="MsoNormal"> </grid><u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"> <format name="mypng" type="PNG"><u></u><u></u></p>
<p class="MsoNormal"> <compression>best</compression><u></u><u></u></p>
<p class="MsoNormal"> </format><u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"> <service type="wms" enabled="true"><u></u><u></u></p>
<p class="MsoNormal"> <full_wms>assemble</full_wms><u></u><u></u></p>
<p class="MsoNormal"> <resample_mode>bilinear</resample_mode><u></u><u></u></p>
<p class="MsoNormal"> <format>mypng</format><u></u><u></u></p>
<p class="MsoNormal"> <maxsize>4096</maxsize><u></u><u></u></p>
<p class="MsoNormal"> </service><u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I’ve also tried to use –DWITH_MAPSERVER but seems that the integration is not working and they doen’t care?
<a href="https://github.com/mapserver/mapcache/issues/169" target="_blank">https://github.com/mapserver/mapcache/issues/169</a> this issue was open 2 years ago and still open, not even a reply from the developers.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">How can I improve the seed process?<u></u><u></u></p>
<p class="MsoNormal">Should I increase the maxsize and metatile to requests more tiles and cache faster?<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Regards,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Sebastiano Laini<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Web Developer<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Buchanan Computing<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
_______________________________________________<br>
mapserver-users mailing list<br>
<a href="mailto:mapserver-users@lists.osgeo.org" target="_blank">mapserver-users@lists.osgeo.org</a><br>
<a href="https://lists.osgeo.org/mailman/listinfo/mapserver-users" rel="noreferrer" target="_blank">https://lists.osgeo.org/mailman/listinfo/mapserver-users</a></blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>Yves Jacolin</div><div>Training and support manager - Team Manager<br>Camptocamp<br><br>Tel (France) : +33 4 58 48 20 43<br>Tel (Switzerland) : +41 21 619 10 43<br>Mob. : +33 6 18 75 42 21<br><br>email : <a href="mailto:yves.jacolin@camptocamp.com" target="_blank">yves.jacolin@camptocamp.com</a><br><a href="http://www.camptocamp.com" target="_blank">http://www.camptocamp.com</a></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>