[mapserver-users] mapcache seed speed optimization

[Yves Jacolin]

Sebastiano,

Some though in my mind:
* –n 20000 : do you really have 2'000 CPU on your WMS server? if not, it
could be worst than better, flooding your WMS server.
* what is the bottleneck?  If WMS server is slow you can't make MapCache
faster, you should optimise WMS Server. Multi-domain can help, data
improvement, etc. also.
* Does your WMS is on the same server? same network? Does you network fast
enough?
* metatile can help definitively!
* "we use sqlite as a cache method" => I don't know if sqlite is really
fast or not, but this could be another bottleneck :)

What do you mean exactly by improve seed speed? :) This is matter of
personal point of view some time.

Y.

Le ven. 8 nov. 2019 à 13:43, Sebastiano Laini <
Sebastiano.Laini at buchanancomputing.co.uk> a écrit :

> Hi all,
>
>
>
> I’m evaluating new stacks but seems that I’m stuck with mapserver and
> mapcache due to the input source of our maps, so I’m trying to improve the
> speed of the mapserver/mapcache stack and what I cannot improve is the seed
> speed.
>
>
>
> The –n and –p parameters seems to be useless.
>
> *-n | –nthreads*: number of parallel threads that should be used to
> request tiles from the WMS source. The default is 1, but can be set higher
> if the WMS server can withstand parallel requests. (As a rule of thumb, the
> value chosen here should never be much higher than the number of CPUs on
> the WMS server.)
>
> *-p | –nprocesses*: number of parallel processes that should be used to
> request tiles from the WMS source.
>
> I’ve tried to use –n 20000 and –p 60000 but for some reason the CPU load
> is max 100% in 1 core, no way to make it use more power and speed up the
> seed process.
>
> Max memory used is 470MB between Centos 7 (with all the services running)
> and the seed process.
>
>
>
> I’ve setup apache to use HTT/2 and it make quite a difference when the
> cache is already seeded but during the seed process doesn’t seems to be
> affected.
>
>
>
> My MapServer build is:
>
>
>
> ./configure \
>
> --with-ogr=/usr/local/bin/gdal-config \
>
> --with-gdal=/usr/local/bin/gdal-config \
>
> --with-wfsclient \
>
> --with-wmsclient \
>
> --with-wfs \
>
> --enable-debug \
>
> --with-curl-config=/usr/bin/curl-config \
>
> --with-proj \
>
> --with-jpeg \
>
> --with-freetype \
>
> --with-postgis=/usr/pgsql-9.3/bin/pg_config \
>
> --with-geos=/usr/local/bin/geos-config
>
>
>
> I know there is mapserver 7 and yet I’m using 6 for the test.
>
>
>
> While the MapCache is:
>
>
>
> cmake   -DCMAKE_PREFIX_PATH="/usr/bin/sqlite3" \
>
>         -DWITH_SQLITE=1 \
>
>         -DWITH_BERKELEY_DB=0 \
>
>         -DWITH_TIFF=0 \
>
>         -DWITH_GEOTIFF=0 \
>
>         -DWITH_FCGI=0 \
>
>         -DWITH_PCRE=0 \
>
>         -DWITH_PIXMAN=1 \
>
>         -DWITH_OGR=1 \
>
>         -DWITH_GEOS=1 \
>
>          ../
>
>
>
> MapCache is the latest version and this is my mapcache.xml file, we use
> sqlite as a cache method
>
>
>
>   <cache name="cache_sqlite" type="sqlite3">
>
>
> <dbfile>/home/www/html/maps/cache/{tileset}/{grid}/{z}/{x}-{y}.sqlite3</dbfile>
>
>   <xcount>10000</xcount>
>
>   <ycount>10000</ycount>
>
>   <pragma name="max_page_count">1573741823</pragma>
>
>   </cache>
>
>
>
>   <tileset name="bccache">
>
>     <source>bcmaps</source>
>
>     <cache>cache_sqlite</cache>
>
>     <grid>bcgrid</grid>
>
>     <format>PNG</format>
>
>     <metatile>16 16</metatile>
>
>     <metabuffer>0</metabuffer>
>
>     <expires>2628000</expires>
>
>   </tileset>
>
>
>
>   <grid name="bcgrid">
>
>                <metadata>
>
>                <title>BC custom grid</title>
>
>                </metadata>
>
>                <srs>EPSG:27700</srs>
>
>                <size>256 256</size>
>
>                <extent>0 0 700000 1250000</extent>
>
>                <resolutions>70 28 14 7 2.8 1.4 0.7</resolutions>
>
>   </grid>
>
>
>
>   <format name="mypng" type="PNG">
>
>                <compression>best</compression>
>
>   </format>
>
>
>
>   <service type="wms" enabled="true">
>
>     <full_wms>assemble</full_wms>
>
>     <resample_mode>bilinear</resample_mode>
>
>     <format>mypng</format>
>
>     <maxsize>4096</maxsize>
>
>   </service>
>
>
>
> I’ve also tried to use –DWITH_MAPSERVER but seems that the integration is
> not working and they doen’t care?
> https://github.com/mapserver/mapcache/issues/169 this issue was open 2
> years ago and still open, not even a reply from the developers.
>
>
>
> How can I improve the seed process?
>
> Should I increase the maxsize and metatile to requests more tiles and
> cache faster?
>
>
>
> Regards,
>
>
>
> Sebastiano Laini
>
> Web Developer
>
> Buchanan Computing
>
>
> _______________________________________________
> mapserver-users mailing list
> mapserver-users at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/mapserver-users



-- 
Yves Jacolin
Training and support manager - Team Manager
Camptocamp

Tel (France) : +33 4 58 48 20 43
Tel (Switzerland) : +41 21 619 10 43
Mob. : +33 6 18 75 42 21

email : yves.jacolin at camptocamp.com
http://www.camptocamp.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-users/attachments/20191108/b7cece0a/attachment-0001.html>