[MapProxy] Help with getting s3 working as a cache

[Luke Bellamy]

Hi all,

I'm trying to get s3 working as a back end cache type and am having no end
of issues (almost certainly of my own doing).

Using MapProxy version: 1.13.2 on Ubuntu 21.10 (on a gcs VM if that makes
any difference....).

Here's my mapproxy.yaml:
-------
services:
  tms:
    use_grid_names: true
    origin: 'nw'

layers:
  - name: test_layer
    title: Test Layer
    sources: [combined_cache]

caches:
  combined_cache:
    grids: [webmercator]
    sources: [test-imagery, test-overlay]
    cache:
      type: s3
      #path ripped out of an example - figured it shouldnt matter too much?
I've also tried with straight /test/
      directory: /1.0.0/my_layer/default/20110501/4326/
      profile_name: default
      bucket_name: xxxxx-test


sources:
  test-imagery:
     type: tile
     grid: GLOBAL_WEBMERCATOR
     url: <source 1>
  test-overlay:
     type: tile
     grid: GLOBAL_WEBMERCATOR
     url: <source 2>
     transparent: true

grids:
    webmercator:
        base: GLOBAL_WEBMERCATOR

globals:
-------

This config works fine if I use file system as the cache back end (eg. omit
the cache option in the cache definition)

When I point an xyz client at it (in this case, qgis, though I've also
tested with leaflet) at it, I'm getting the following error output from
serve-develop (from every single tile request from the client):

PermissionError: [Errno 13] Permission denied: '/1.0.0'Traceback (most
recent call last): File
"/usr/local/lib/python3.9/dist-packages/mapproxy/wsgiapp.py", line 141, in
__call__ resp = self.handlers[handler_name].handle(req) File
"/usr/local/lib/python3.9/dist-packages/mapproxy/service/base.py", line 30,
in handle return handler(parsed_req) File
"/usr/local/lib/python3.9/dist-packages/mapproxy/service/tile.py", line 82,
in map tile = layer.render(tile_request,
use_profiles=tile_request.use_profiles, coverage=limit_to,
decorate_img=decorate_img) File
"/usr/local/lib/python3.9/dist-packages/mapproxy/service/tile.py", line
312, in render tile = self.tile_manager.load_tile_coord(tile_coord, File
"/usr/local/lib/python3.9/dist-packages/mapproxy/cache/tile.py", line 115,
in load_tile_coord return self.load_tile_coords( File
"/usr/local/lib/python3.9/dist-packages/mapproxy/cache/tile.py", line 139,
in load_tile_coords tiles = self._load_tile_coords( File
"/usr/local/lib/python3.9/dist-packages/mapproxy/cache/tile.py", line 171,
in _load_tile_coords created_tiles = creator.create_tiles(uncached_tiles)
File "/usr/local/lib/python3.9/dist-packages/mapproxy/cache/tile.py", line
325, in create_tiles created_tiles = self._create_single_tiles(tiles) File
"/usr/local/lib/python3.9/dist-packages/mapproxy/cache/tile.py", line 349,
in _create_single_tiles created_tiles.extend(self._create_single_tile(tile))
File "/usr/local/lib/python3.9/dist-packages/mapproxy/cache/tile.py", line
363, in _create_single_tile with self.tile_mgr.lock(tile): File
"/usr/local/lib/python3.9/dist-packages/mapproxy/util/lock.py", line 46, in
__enter__ self.lock() File
"/usr/local/lib/python3.9/dist-packages/mapproxy/util/lock.py", line 63, in
lock self._make_lockdir() File
"/usr/local/lib/python3.9/dist-packages/mapproxy/util/lock.py", line 57, in
_make_lockdir raise e File
"/usr/local/lib/python3.9/dist-packages/mapproxy/util/lock.py", line 54, in
_make_lockdir os.makedirs(os.path.dirname(self.lock_file)) File
"/usr/lib/python3.9/os.py", line 215, in makedirs makedirs(head,
exist_ok=exist_ok) File "/usr/lib/python3.9/os.py", line 215, in makedirs
makedirs(head, exist_ok=exist_ok) File "/usr/lib/python3.9/os.py", line
215, in makedirs makedirs(head, exist_ok=exist_ok) [Previous line repeated
2 more times] File "/usr/lib/python3.9/os.py", line 225, in makedirs
mkdir(name, mode)PermissionError: [Errno 13] Permission denied: '/1.0.0'[info]
127.0.0.1 - - [25/Oct/2021 08:47:01] "GET
/tms/1.0.0/test_layer/webmercator/1/0/3.png HTTP/1.0" 500 -

I was originally attempting to connect to a Google Cloud Storage bucket,
but was receiving the above error, so thinking it might have been some
weird interop issue, or permissions issue I switched to AWS s3, and the
issue remains.

I'm... fairly confident the back end aws / google cloud storage bucket &
associated permissions are set up correctly (though happy to be wrong if
they aren't).

I've checked to see that the perms file (~/.aws/config) is correct, and if
I supply a known bad key, MapProxy doesn't even start, so that's definitely
correct.

Does anyone have any ideas / has anyone seen this issue before ?

Regards,

Luke
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapproxy/attachments/20211025/a7d42612/attachment.html>