From traviskirstine at gmail.com  Thu Apr  1 05:29:32 2021
From: traviskirstine at gmail.com (Travis Kirstine)
Date: Thu, 1 Apr 2021 08:29:32 -0400
Subject: [MapProxy] Mapproxy Cleanups Speed
In-Reply-To: <7cff2c8b-fcff-9493-4217-49917a8494ce@terrestris.de>
References: <CAJj3HzCBE2eEiLhXM3A0V42Aq1aSviiy4MdkW7id_PmOcJYfqQ@mail.gmail.com>
 <7cff2c8b-fcff-9493-4217-49917a8494ce@terrestris.de>
Message-ID: <CALtm4h2fWM_WTPJM5zXrH+KfaOJDWh5O0AmN0hg2d5AxRXFmXw@mail.gmail.com>

I think your best bet is to rm the folder using an alternative process.  In
my experience it has been faster to use custom scripts to perform the
cleanups especially if you are just looking to remove everything from a
cache, as Johannes indicated a file cache may not be the best for this type
of operation as it takes time for MP to walk the directories.

I was thinking that you could maybe utilize dimensions to generate a new
cache when you update the data however it appears MP does not support that
yet, this is something you could do with MapCache.


On Wed, 31 Mar 2021 at 09:34, Johannes Weskamm <weskamm at terrestris.de>
wrote:

> Hi,
>
>
> Try saving your cache in a database -> e.g. sqlite (
> https://mapproxy.org/docs/1.13.0/caches.html#sqlite) instead of a file
> based cache. That should speed up cache cleanup.
>
>
> Greetings,
>
> Johannes
>
>
>
> Am 31.03.21 um 15:27 schrieb Nuno Silva:
>
>
> Hi everybody.
>
> I'm using Mapproxy to create a cache of a WMS layer on a production server
> to prevent direct access from the users to this server, I used in the past
> MapCache, but problems related to GetFeatureInfo and GetLegendGraphic with
> some WMS Servers made me try this alternative.
>
> The problem:
>
> I have a layer that is generated every 3-4h using as source a netcdf
> file...the backend server has to read that netcdf file, with high cost of
> processing the file, with the cache between the client and the server, only
> the first requests would take longer, but when I'm using the mapproxy-seed
> utility to do a cache cleanup, assuming that the user could use all 18
> levels of that layer, is taking now 4h to complete 25% cleaning of the
> cache....making this process take about 16h to achieve 100% (again, I have
> new information being created every 3-4h)
>
> Is there a way to achieve fast cache cleaning? I'll already set  *remove_all:
> true *, but even so it takes toooooo long even without folder timestamp
> compare...
>
> I'm considering making a manual removal of the folder with rm -fR
> /cache_folder/* and the seeding for *to: 5* just to fill the initial
> cache with some data.
>
> Any suggestions/alternatives would be appreciated , thanks in advance!
>
>
>
> --
> Kind regards,
> nsilva
>
> _______________________________________________
> MapProxy mailing listMapProxy at lists.osgeo.orghttps://lists.osgeo.org/mailman/listinfo/mapproxy
>
> _______________________________________________
> MapProxy mailing list
> MapProxy at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/mapproxy
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapproxy/attachments/20210401/032eebea/attachment.html>

From bconrad at aerialservicesinc.com  Fri Apr 16 12:11:39 2021
From: bconrad at aerialservicesinc.com (Bernard Conrad)
Date: Fri, 16 Apr 2021 14:11:39 -0500
Subject: [MapProxy] Configuring Mapproxy with Mbtiles created by MapTiler
Message-ID: <CAAAdHpM+m6SdAY=AaSyc3BRO+MtQNDC7hH-XqcyJ7Hea=87X0g@mail.gmail.com>

Hello, we've been using MapTiler and MapProxy for several years now and
have had good success streaming imagery using the File cache approach. We
are moving to the cloud and instead want to use the mbtiles cache as a way
of streaming since it is easier to move around than all the individual
images. So far we have been unsuccessful in getting MapProxy to display the
imagery.

I have uploaded mbtile as well as the image, cutline, and the configuration
file for MapTiler Desktop. The mbtiles are in EPSG:3857 and with the hybrid
jpg/png specified by a shapefile. It's the same configuration we use for
the File directory listing and it works fine that way. I also created a
mbtile using only png but that didn't work either.

https://drive.google.com/drive/folders/1vmkzN9X7UBxJmR78EUHZUVI71FnqkEgm?usp=sharing

*MapTiler Configuration*

-work_dir G:\work_dir -o G:\output\sample.mbtiles -srs EPSG:26911 -f
hybridext jpg png8a -jpeg_quality 65 -zoom 0 19 -tms G:\image\sample.tif
-cutline_proj G:\cutline\aoi.shp

I'm following the same configuration for the mbtiles cache per the
documentation (https://mapproxy.org/docs/nightly/caches.html#cache-mbtiles)
and use a grid but it only returns 400 and 500 errors. Tried flipping the
origins but that didn't work either.

*MapProxy Configuration*
caches:
  test_mbtiles_cache:
    sources: []
    grids: [ mbtiles_grid ]
    cache:
      type: mbtiles
      filename: /var/mbtiles/sample.mbtiles

grids:
  mbtiles_grid:
      base: GLOBAL_WEBMERCATOR
      origin: 'sw'
      bbox: [-116.376767,34.066863,-116.179884,34.257388]
      bbox_srs: 'EPSG:4326'

Any help is appreciated. Thanks.

--
*Bernard Conrad*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapproxy/attachments/20210416/1fb40fbe/attachment.html>

From etiengo at gmail.com  Sun Apr 18 08:08:46 2021
From: etiengo at gmail.com (Erik Tiengo)
Date: Sun, 18 Apr 2021 17:08:46 +0200
Subject: [MapProxy] WSGI Filter working example
Message-ID: <CAKhGo2Z5kcRzb99T2tDBRN=n-g5f70A9R+hjN82PceNp2FAPpQ@mail.gmail.com>

Hello to everybody,

someone could share a simple example of customization by WSGI filter.
Documentation is a very clear entry point, but I am not able to see
__call__ method in action :-( I guess that GetCapabilities request is
enough to see __call__ in action, right? __init__ is it but no __call__ for
every request I make

I am using Python 3.9.1, MapProxy 1.13.0 and gunicorn 20.1.0

Thank in advance for support

et

-- 
*Erik Tiengo*
GIS Solution Architect
------------------------------
mobile:    *+39 348 1230364*
Skype:     *erik.tiengo*
Hangouts:  *etiengo at gmail.com <etiengo at gmail.com>*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapproxy/attachments/20210418/7a6ed93f/attachment.html>

From walkermatt at longwayaround.org.uk  Sun Apr 18 10:19:48 2021
From: walkermatt at longwayaround.org.uk (Matt Walker)
Date: Sun, 18 Apr 2021 18:19:48 +0100
Subject: [MapProxy] WSGI Filter working example
In-Reply-To: <CAKhGo2Z5kcRzb99T2tDBRN=n-g5f70A9R+hjN82PceNp2FAPpQ@mail.gmail.com>
References: <CAKhGo2Z5kcRzb99T2tDBRN=n-g5f70A9R+hjN82PceNp2FAPpQ@mail.gmail.com>
Message-ID: <CACcwLY-VPmx1s3A4EJxPiBehGayPHagFFrWbO7q08kv5MHVRRQ@mail.gmail.com>

Hi Erik,

I don't have time to test with Python 3 right now but here is an example
from a few years ago which wraps MapProxy in order to set
the mapproxy.decorate_img callback.

https://gist.github.com/walkermatt/1cae669defce037511ada4e0ed01b3cd

Kind regards,

Matt.

On Sun, 18 Apr 2021, 16:09 Erik Tiengo, <etiengo at gmail.com> wrote:

> Hello to everybody,
>
> someone could share a simple example of customization by WSGI filter.
> Documentation is a very clear entry point, but I am not able to see
> __call__ method in action :-( I guess that GetCapabilities request is
> enough to see __call__ in action, right? __init__ is it but no __call__ for
> every request I make
>
> I am using Python 3.9.1, MapProxy 1.13.0 and gunicorn 20.1.0
>
> Thank in advance for support
>
> et
>
> --
> *Erik Tiengo*
> GIS Solution Architect
> ------------------------------
> mobile:    *+39 348 1230364*
> Skype:     *erik.tiengo*
> Hangouts:  *etiengo at gmail.com <etiengo at gmail.com>*
> _______________________________________________
> MapProxy mailing list
> MapProxy at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/mapproxy
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapproxy/attachments/20210418/927f3590/attachment.html>

From etiengo at gmail.com  Sun Apr 18 21:52:22 2021
From: etiengo at gmail.com (Erik Tiengo)
Date: Mon, 19 Apr 2021 06:52:22 +0200
Subject: [MapProxy] WSGI Filter working example
In-Reply-To: <CACcwLY-VPmx1s3A4EJxPiBehGayPHagFFrWbO7q08kv5MHVRRQ@mail.gmail.com>
References: <CAKhGo2Z5kcRzb99T2tDBRN=n-g5f70A9R+hjN82PceNp2FAPpQ@mail.gmail.com>
 <CACcwLY-VPmx1s3A4EJxPiBehGayPHagFFrWbO7q08kv5MHVRRQ@mail.gmail.com>
Message-ID: <CAKhGo2aEOWTbBmWZTcxkkUbzHA81q+TESEpktOB6U87sjSGBuw@mail.gmail.com>

Thank you so much Matt! Your snippet works with Python 3.9 as well. It was
definitely my error: now with *application = <wsgi_filter>(app=application)
*works!

et

Il giorno dom 18 apr 2021 alle ore 19:19 Matt Walker <
walkermatt at longwayaround.org.uk> ha scritto:

> Hi Erik,
>
> I don't have time to test with Python 3 right now but here is an example
> from a few years ago which wraps MapProxy in order to set
> the mapproxy.decorate_img callback.
>
> https://gist.github.com/walkermatt/1cae669defce037511ada4e0ed01b3cd
>
> Kind regards,
>
> Matt.
>
> On Sun, 18 Apr 2021, 16:09 Erik Tiengo, <etiengo at gmail.com> wrote:
>
>> Hello to everybody,
>>
>> someone could share a simple example of customization by WSGI filter.
>> Documentation is a very clear entry point, but I am not able to see
>> __call__ method in action :-( I guess that GetCapabilities request is
>> enough to see __call__ in action, right? __init__ is it but no __call__ for
>> every request I make
>>
>> I am using Python 3.9.1, MapProxy 1.13.0 and gunicorn 20.1.0
>>
>> Thank in advance for support
>>
>> et
>>
>> --
>> *Erik Tiengo*
>> GIS Solution Architect
>> ------------------------------
>> mobile:    *+39 348 1230364*
>> Skype:     *erik.tiengo*
>> Hangouts:  *etiengo at gmail.com <etiengo at gmail.com>*
>> _______________________________________________
>> MapProxy mailing list
>> MapProxy at lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/mapproxy
>>
>

-- 
*Erik Tiengo*
GIS Solution Architect
------------------------------
mobile:    *+39 348 1230364*
Skype:     *erik.tiengo*
Hangouts:  *etiengo at gmail.com <etiengo at gmail.com>*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapproxy/attachments/20210419/831ce4c0/attachment.html>