[MapProxy] Mapproxy Cleanups Speed

[Travis Kirstine]

I think your best bet is to rm the folder using an alternative process.  In
my experience it has been faster to use custom scripts to perform the
cleanups especially if you are just looking to remove everything from a
cache, as Johannes indicated a file cache may not be the best for this type
of operation as it takes time for MP to walk the directories.

I was thinking that you could maybe utilize dimensions to generate a new
cache when you update the data however it appears MP does not support that
yet, this is something you could do with MapCache.


On Wed, 31 Mar 2021 at 09:34, Johannes Weskamm <weskamm at terrestris.de>
wrote:

> Hi,
>
>
> Try saving your cache in a database -> e.g. sqlite (
> https://mapproxy.org/docs/1.13.0/caches.html#sqlite) instead of a file
> based cache. That should speed up cache cleanup.
>
>
> Greetings,
>
> Johannes
>
>
>
> Am 31.03.21 um 15:27 schrieb Nuno Silva:
>
>
> Hi everybody.
>
> I'm using Mapproxy to create a cache of a WMS layer on a production server
> to prevent direct access from the users to this server, I used in the past
> MapCache, but problems related to GetFeatureInfo and GetLegendGraphic with
> some WMS Servers made me try this alternative.
>
> The problem:
>
> I have a layer that is generated every 3-4h using as source a netcdf
> file...the backend server has to read that netcdf file, with high cost of
> processing the file, with the cache between the client and the server, only
> the first requests would take longer, but when I'm using the mapproxy-seed
> utility to do a cache cleanup, assuming that the user could use all 18
> levels of that layer, is taking now 4h to complete 25% cleaning of the
> cache....making this process take about 16h to achieve 100% (again, I have
> new information being created every 3-4h)
>
> Is there a way to achieve fast cache cleaning? I'll already set  *remove_all:
> true *, but even so it takes toooooo long even without folder timestamp
> compare...
>
> I'm considering making a manual removal of the folder with rm -fR
> /cache_folder/* and the seeding for *to: 5* just to fill the initial
> cache with some data.
>
> Any suggestions/alternatives would be appreciated , thanks in advance!
>
>
>
> --
> Kind regards,
> nsilva
>
> _______________________________________________
> MapProxy mailing listMapProxy at lists.osgeo.orghttps://lists.osgeo.org/mailman/listinfo/mapproxy
>
> _______________________________________________
> MapProxy mailing list
> MapProxy at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/mapproxy
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapproxy/attachments/20210401/032eebea/attachment.html>