[mapserver-users] MapCache: cache some zoom levels, proxy others?

[Paul Madden]

Reading the docs more carefully, it sounds like SQLite databases may have concurrency problems even with only mapcache_seed accessing them, given a sufficiently high number of threads. I'm guessing that 8 is not "a high number", and that a few extra from MapCache itself isn't either. Does anybody know differently?

On 10/28/15 20:08, Paul Madden wrote:
> Travis,
> 
> Thank you -- I hadn't realized before what you could do with the various meta caches. I'll definitely be experimenting with a front-end memcache at some point.
> 
> But, I can't yet see how I can overcome the sqlite3 issue with these methods. It seems like the following would *almost* work:
> 
> <cache name="multitier" type="multitier">
> <cache>db1</cache>
> <cache>db2</cache>
> </cache>
> 
> where both db1 and db2 are sqlite3 caches, and db1 is being actively seeded by mapcache_seed. At first I thought: MapCache will check db1 to see if the requested tile has already been seeded and, if not, will move on to db2, which will make the appropriate WMS requests to MapServer. Only mapcache_seed will write to db1, and only MapCache will write to db2: No concurrency issue. But I see this in the docs: "If a given tile isn’t found in the first child cache, it will be read from the second child cache AND copied into the first child cache for subsequent accesses." Given the "AND" part, it seems that both mapcache_seed and MapCache will write concurrently to db1. If this is right, does such a configuration actually avoid the sqlite3 concurrency issue? Is there a better configuration that I've failed to imagine?
> 
> On a related note: If *only* mapcache_seed writes to a sqlite3, but runs in multi-threaded mode, is it sqlite3-safe?
> 
> (The questions are for anyone kind enough to answer...)
> 
> paul
> 
> On 10/28/15 17:42, Travis Kirstine wrote:
>> Paul,
>>
>> I would have a look at some of the new tiered caching options available in 1.4.  SQLite can have issues locking under load
>>
>> On Wednesday, October 28, 2015, Paul Madden <pmadden at nsidc.org <mailto:pmadden at nsidc.org>> wrote:
>>
>>      Thomas,
>>
>>      Enabling metatiling indeed led to a significant speedup. I can also verify that <threaded_fetching> works for me (I can see the additional threads running), but it did not lead to a noticeable speedup, perhaps due to a bottleneck where my MapServer talks to a remote database. I also increased my tile size from 256x256 to 1024x1024, which seems fine for my application and which obviously reduces the number of tiles helpfully.
>>
>>      If I may ask one more question: Are you aware of any problems running mapcache_seed concurrently with users accessing MapCache via the web application, if one uses the sqlite3 cache? I saw a thread about potential conflicts, but I think that might have been specific to disk caches.
>>
>>      Thanks again,
>>      paul
>>
>>      On 10/28/15 9:59 AM, thomas bonfort wrote:
>>
>>              Out of curiosity, can I assume that my other idea, about proxying some zoom
>>              levels directly to WMS and caching others, is unrealistic?
>>
>>          not unrealistic, but not implemented
>>
>>          --
>>          thomas
>>
>>              paul
>>
>>
>>              On 10/28/15 9:36 AM, thomas bonfort wrote:
>>
>>                  Paul,
>>
>>                  Are you using metatiling? That would substantially reduce the number
>>                  of requests mapcache does to mapserver.
>>
>>                  You also have the
>>                  <mapcache>
>>                  ...
>>                        <threaded_fetching>true</threaded_fetching>
>>                  </mapcache>
>>                  option, that lets mapcache launch mapserver requests in parallel
>>                  instead of sequentially (YMMV, this option isn't very tested)
>>
>>                  --
>>                  thomas
>>
>>                  On 28 October 2015 at 16:20, Paul Madden <pmadden at nsidc.org> wrote:
>>
>>                      I am working with a web application that makes WMS queries to MapCache.
>>                      The application supports zoom levels 0 through 13, and I have used
>>                      mapcache_seed to seed the cache through level 10. The deeper levels take so
>>                      long to seed that I am considering stopping at level 10 and letting MapCache
>>                      / MapServer produce deeper levels on demand, as users visit them. The
>>                      problem I'm facing is that, as far as I can tell, MapCache requests tiles
>>                      from MapServer serially, then stitches together the required image from
>>                      these tiles, when it receives a request for an uncached level. In one test,
>>                      34 separate tiles were requested from MapServer, one at a time, and it took
>>                      ~30 seconds to render the final image.
>>
>>                      I'm wondering whether there is a way to configure MapCache to
>>                      cache/provide a defined set of zoom levels, and to proxy requests at other
>>                      zoom levels directly to MapServer. I tried setting minzoom and maxzoom on my
>>                      <grid> in mapcache.xml to "0 10". I tried adding a proxy for my cached
>>                      layer, hoping that MapCache would redirect requests for unsupported zoom
>>                      levels to MapServer (I have proxies in place for other, completely uncached
>>                      layers, and those work fine). I tried copying/renaming the
>>                      GoogleMapsCompatible grid definition into my mapcache.xml and removing from
>>                      <resolutions> zoom levels I don't want MapCache to cache. So far, these
>>                      attempts apparently just prevent MapCache from serving the deeper layers at
>>                      all.
>>
>>                      Any thoughts on how I might address this issue? It might also help if
>>                      MapCache would make concurrent requests for tiles at uncached levels, but I
>>                      have no idea if this is feasible. And it would be nice to be able to let
>>                      MapServer handle some zoom levels directly in any case.
>>
>>                      Thanks in advance for any ideas.
>>
>>                      Paul Madden
>>                      National Snow and Ice Data Center
>>                      _______________________________________________
>>                      mapserver-users mailing list
>>                      mapserver-users at lists.osgeo.org
>>                      http://lists.osgeo.org/mailman/listinfo/mapserver-users
>>
>>
>>              _______________________________________________
>>              mapserver-users mailing list
>>              mapserver-users at lists.osgeo.org
>>              http://lists.osgeo.org/mailman/listinfo/mapserver-users
>>
>>
>>      _______________________________________________
>>      mapserver-users mailing list
>>      mapserver-users at lists.osgeo.org
>>      http://lists.osgeo.org/mailman/listinfo/mapserver-users
>>