[mapserver-users] mapcache seed speed optimization

[Sebastiano Laini]

Hi all,

I'm evaluating new stacks but seems that I'm stuck with mapserver and mapcache due to the input source of our maps, so I'm trying to improve the speed of the mapserver/mapcache stack and what I cannot improve is the seed speed.

The -n and -p parameters seems to be useless.

-n | -nthreads: number of parallel threads that should be used to request tiles from the WMS source. The default is 1, but can be set higher if the WMS server can withstand parallel requests. (As a rule of thumb, the value chosen here should never be much higher than the number of CPUs on the WMS server.)

-p | -nprocesses: number of parallel processes that should be used to request tiles from the WMS source.
I've tried to use -n 20000 and -p 60000 but for some reason the CPU load is max 100% in 1 core, no way to make it use more power and speed up the seed process.
Max memory used is 470MB between Centos 7 (with all the services running) and the seed process.

I've setup apache to use HTT/2 and it make quite a difference when the cache is already seeded but during the seed process doesn't seems to be affected.

My MapServer build is:

./configure \
--with-ogr=/usr/local/bin/gdal-config \
--with-gdal=/usr/local/bin/gdal-config \
--with-wfsclient \
--with-wmsclient \
--with-wfs \
--enable-debug \
--with-curl-config=/usr/bin/curl-config \
--with-proj \
--with-jpeg \
--with-freetype \
--with-postgis=/usr/pgsql-9.3/bin/pg_config \
--with-geos=/usr/local/bin/geos-config

I know there is mapserver 7 and yet I'm using 6 for the test.

While the MapCache is:

cmake   -DCMAKE_PREFIX_PATH="/usr/bin/sqlite3" \
        -DWITH_SQLITE=1 \
        -DWITH_BERKELEY_DB=0 \
        -DWITH_TIFF=0 \
        -DWITH_GEOTIFF=0 \
        -DWITH_FCGI=0 \
        -DWITH_PCRE=0 \
        -DWITH_PIXMAN=1 \
        -DWITH_OGR=1 \
        -DWITH_GEOS=1 \
         ../

MapCache is the latest version and this is my mapcache.xml file, we use sqlite as a cache method

  <cache name="cache_sqlite" type="sqlite3">
  <dbfile>/home/www/html/maps/cache/{tileset}/{grid}/{z}/{x}-{y}.sqlite3</dbfile>
  <xcount>10000</xcount>
  <ycount>10000</ycount>
  <pragma name="max_page_count">1573741823</pragma>
  </cache>

  <tileset name="bccache">
    <source>bcmaps</source>
    <cache>cache_sqlite</cache>
    <grid>bcgrid</grid>
    <format>PNG</format>
    <metatile>16 16</metatile>
    <metabuffer>0</metabuffer>
    <expires>2628000</expires>
  </tileset>

  <grid name="bcgrid">
               <metadata>
               <title>BC custom grid</title>
               </metadata>
               <srs>EPSG:27700</srs>
               <size>256 256</size>
               <extent>0 0 700000 1250000</extent>
               <resolutions>70 28 14 7 2.8 1.4 0.7</resolutions>
  </grid>

  <format name="mypng" type="PNG">
               <compression>best</compression>
  </format>

  <service type="wms" enabled="true">
    <full_wms>assemble</full_wms>
    <resample_mode>bilinear</resample_mode>
    <format>mypng</format>
    <maxsize>4096</maxsize>
  </service>

I've also tried to use -DWITH_MAPSERVER but seems that the integration is not working and they doen't care? https://github.com/mapserver/mapcache/issues/169 this issue was open 2 years ago and still open, not even a reply from the developers.

How can I improve the seed process?
Should I increase the maxsize and metatile to requests more tiles and cache faster?

Regards,

Sebastiano Laini
Web Developer
Buchanan Computing

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-users/attachments/20191108/2825d09d/attachment.html>