[mapserver-users] mapcache seed speed optimization
Rahkonen Jukka (MML)
jukka.rahkonen at maanmittauslaitos.fi
Fri Nov 8 06:01:55 PST 2019
Hi,
When you seed tiles from WMS the slowest component is usually the WMS server. And what the WMS server is spending its time is usually rendering. And what makes rendering slow is usually that you render too much, and sometimes that WMS server gets the vector data too slow because the backend data source is slow, very often because it has poor indexes which do not for suit the queries that WMS server is
sending.
If you have a slow WMS you can't get more speed out of it by bombing it with very many parallel threads. I would have a try if using 2-4, perhaps 8 threads makes seeding faster but if there is no difference I would concentrate on the WMS configuration. BTW. why did you
select to use 20000 threads?
I am not sure why you see just 1 core running on 100% load. I guess that you are now running WMS and MapCache on the same computer. Mapserver program is using only 1 core but http server should start more processes for parallel requests. That CPU load is at 100% is generally a good thing because it tells that CPU is working, instead of waiting data from a slow disk. Another thing is that CPU may be doing something heavy and stupid.
Myself I would start by gathering some information about how Mapserver WMS is working be setting "DEBUG 2" on each layer or for the whole mapfile. By looking at the rendering times of each layer the slowest layers can be found and then it is time to think about how to make those layers faster.
You wrote that you use SQLite cache so it would be good to know that SQLite database does not support parallel writes https://sqlite.org/lockingv3.html. Seeding with a few parallel threads may increase the speed if the WMS is the slowest part in your chain but when you have saturated the SQLite writing speed adding threads will do no good. If you have optimized your WMS so well that SQLite sets the limits then you should test other cache alternatives. I guess that file cache would be faster to seed from multiple threads.
When is comes to ticket https://github.com/mapserver/mapcache/issues/169 the latest comment is from a Mpaserver developer and their last words "I wouldn't mind helping to fix if there were interested parties..." feel rather polite to me.
-Jukka Rahkonen-
Lähettäjä: mapserver-users <mapserver-users-bounces at lists.osgeo.org> Puolesta Sebastiano Laini
Lähetetty: perjantai 8. marraskuuta 2019 14.35
Vastaanottaja: 'mapserver-users at lists.osgeo.org' <mapserver-users at lists.osgeo.org>
Aihe: [mapserver-users] mapcache seed speed optimization
Hi all,
I'm evaluating new stacks but seems that I'm stuck with mapserver and mapcache due to the input source of our maps, so I'm trying to improve the speed of the mapserver/mapcache stack and what I cannot improve is the seed speed.
The -n and -p parameters seems to be useless.
-n | -nthreads: number of parallel threads that should be used to request tiles from the WMS source. The default is 1, but can be set higher if the WMS server can withstand parallel requests. (As a rule of thumb, the value chosen here should never be much higher than the number of CPUs on the WMS server.)
-p | -nprocesses: number of parallel processes that should be used to request tiles from the WMS source.
I've tried to use -n 20000 and -p 60000 but for some reason the CPU load is max 100% in 1 core, no way to make it use more power and speed up the seed process.
Max memory used is 470MB between Centos 7 (with all the services running) and the seed process.
I've setup apache to use HTT/2 and it make quite a difference when the cache is already seeded but during the seed process doesn't seems to be affected.
My MapServer build is:
./configure \
--with-ogr=/usr/local/bin/gdal-config \
--with-gdal=/usr/local/bin/gdal-config \
--with-wfsclient \
--with-wmsclient \
--with-wfs \
--enable-debug \
--with-curl-config=/usr/bin/curl-config \
--with-proj \
--with-jpeg \
--with-freetype \
--with-postgis=/usr/pgsql-9.3/bin/pg_config \
--with-geos=/usr/local/bin/geos-config
I know there is mapserver 7 and yet I'm using 6 for the test.
While the MapCache is:
cmake -DCMAKE_PREFIX_PATH="/usr/bin/sqlite3" \
-DWITH_SQLITE=1 \
-DWITH_BERKELEY_DB=0 \
-DWITH_TIFF=0 \
-DWITH_GEOTIFF=0 \
-DWITH_FCGI=0 \
-DWITH_PCRE=0 \
-DWITH_PIXMAN=1 \
-DWITH_OGR=1 \
-DWITH_GEOS=1 \
../
MapCache is the latest version and this is my mapcache.xml file, we use sqlite as a cache method
<cache name="cache_sqlite" type="sqlite3">
<dbfile>/home/www/html/maps/cache/{tileset}/{grid}/{z}/{x}-{y}.sqlite3</dbfile>
<xcount>10000</xcount>
<ycount>10000</ycount>
<pragma name="max_page_count">1573741823</pragma>
</cache>
<tileset name="bccache">
<source>bcmaps</source>
<cache>cache_sqlite</cache>
<grid>bcgrid</grid>
<format>PNG</format>
<metatile>16 16</metatile>
<metabuffer>0</metabuffer>
<expires>2628000</expires>
</tileset>
<grid name="bcgrid">
<metadata>
<title>BC custom grid</title>
</metadata>
<srs>EPSG:27700</srs>
<size>256 256</size>
<extent>0 0 700000 1250000</extent>
<resolutions>70 28 14 7 2.8 1.4 0.7</resolutions>
</grid>
<format name="mypng" type="PNG">
<compression>best</compression>
</format>
<service type="wms" enabled="true">
<full_wms>assemble</full_wms>
<resample_mode>bilinear</resample_mode>
<format>mypng</format>
<maxsize>4096</maxsize>
</service>
I've also tried to use -DWITH_MAPSERVER but seems that the integration is not working and they doen't care? https://github.com/mapserver/mapcache/issues/169 this issue was open 2 years ago and still open, not even a reply from the developers.
How can I improve the seed process?
Should I increase the maxsize and metatile to requests more tiles and cache faster?
Regards,
Sebastiano Laini
Web Developer
Buchanan Computing
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-users/attachments/20191108/83b857ff/attachment.html>
More information about the mapserver-users
mailing list