[mapserver-users] mapcache seed slow

[Sebastiano Laini]

I’m using this configuration

  <cache name="cache_sqlite" type="sqlite3">
  <dbfile>/var/www/vhosts/buchananmapping.co.uk/httpdocs/maps/cache/{tileset}/{grid}/{z}/{x}-{y}.sqlite3</dbfile>
  <xcount>10000</xcount>
  <ycount>10000</ycount>
  <pragma name="max_page_count">1573741823</pragma>
  </cache>

  <source name="bcmaps" type="wms">
    <getmap>
      <params>
        <FORMAT>image/png</FORMAT>
        <LAYERS>miniscale</LAYERS>
        <MAP>…/maps/bcmaps.map</MAP>
      </params>
    </getmap>
    <http>
      <url>https://.../cgi-bin/mapserv?</url>
    </http>
  </source>

  <tileset name="bccache">
    <source>bcmaps</source>
    <cache>cache_sqlite</cache>
    <grid>bcgrid</grid>
    <format>PNG</format>
    <metatile>4 4</metatile>
    <metabuffer>0</metabuffer>
    <expires>2628000</expires>
  </tileset>

  <grid name="bcgrid">
               <metadata>
                              <title>BC custom grid</title>
               </metadata>
               <srs>EPSG:27700</srs>
               <size>128 128</size>
               <extent>0 0 700000 1250000</extent>
               <resolutions>280 140 70 28 14 7 6.16 2.8 1.4 0.7</resolutions>
  </grid>

  <format name="mypng" type="PNG">
   <compression>best</compression>
  </format>

It supposed to have a limit that it’s the  full EPSG extent

Sebastiano Laini
Web Developer

From: Travis Kirstine [mailto:traviskirstine at gmail.com]
Sent: 22 February 2019 13:57
To: Sebastiano Laini <Sebastiano.Laini at BuchananComputing.co.uk>
Cc: mapserver-users at lists.osgeo.org
Subject: Re: [mapserver-users] mapcache seed slow

The time you have indicated seems excessive to seed the UK. My guess is that the current process it seeding the extents of your grid (world?) but default.  I've seeded the province of ontario 1 million km2 z0-17 in about 2 weeks using a sqlite cache

Ramping up the -n will only bottle neck the process more (same with the -p flag) the limiting factor is SQLite and it limitation on concurrent writes (I use -n 4).  I've noticed that if you try to use other sqlite3 journal modes you can degrade performance (http://osgeo-org.1560.x6.nabble.com/mapcache-sqlite-cache-large-wal-files-and-blocked-checkpoints-when-seeding-td5377571.html).  I would just use most basic cache config as a starting point.  BTW using SSD should increase the SQLite writes as well (if you are lucky enough to have them)

   <cache name="sqlite" type="sqlite3">
      <dbfile>./{tileset}.sqlite3</dbfile>
   </cache>

There are a few things that you could do when seeding to speed up the process:

Use the "restricted_extent" parameter in the tileset grid to limit the number of tiles

Example
   <tileset name="ontario_2018">
     <source>ontario_2018_wms</source>
     <cache>sqlite</cache>
     <grid out-of-zoom-strategy="reassemble" max-cached-zoom="19" restricted_extent="-12829476.786723 7281268.350007 -12618425.953856 7521514.261991">gmaps</grid>
     <format>mixed</format>
     <metatile>5 5</metatile>
   </tileset>

Use a polygon shapefile (or other ogr datasource) to limit the seeding using the -d flag in mapcache_seed command (note the projection of the datasource should be the same as the grid)

Use a larger metatile when seeding (-M flag override in command line).  Mapcache will "bulk" insert the resultant tiles based on the metatile - if <metatile>5 5</metatile> mapcache will insert 25 tiles at a time if <metatile>8 8</metatile> it will do 64 (I think this is the way it works).  The limiting factor here is trying to balance this with your source WMS performance.  I use 8,8 when seeding

Regards















On Fri, 22 Feb 2019 at 06:35, Sebastiano Laini <Sebastiano.Laini at buchanancomputing.co.uk<mailto:Sebastiano.Laini at buchanancomputing.co.uk>> wrote:
Hi all,

I’m trying to seed the whole UK at all zoom lv with these resolutions

<resolutions>280 140 70 28 14 7 6.16 2.8 1.4 0.7</resolutions>

But after 13 days it arrives at the 5th level where before I kill the process it seed just 900MB on that level.

This is my script

su -s /bin/sh apache -c "/usr/local/bin/mapcache_seed -c /usr/local/src/mapcache-1.6.1/build/mapcache.xml -t bccache -g bcgrid -z 5,6 –n 400" 2>&1>  /var/www/vhosts/$SITE/scripts/cache_log.txt

I know that in the documentation say that –n should be less or max the number of cpu in the server but is really slow if I put 2, so I tried with 100 and after 13 days that was the result so now I’m trying 400

Is there something else I can use to speed up the seed? I’m using the SQlite cache method

Should I use the -p (| –nprocesses: number of parallel processes that should be used to request tiles from the WMS source) Instead of the –n parameter?

Sebastiano Laini
Web Developer

_______________________________________________
mapserver-users mailing list
mapserver-users at lists.osgeo.org<mailto:mapserver-users at lists.osgeo.org>
https://lists.osgeo.org/mailman/listinfo/mapserver-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-users/attachments/20190222/2746b376/attachment-0001.html>