[MapProxy] Cache migration, SQLite and NAS

Christian Siegert christian at christian-siegert.com
Thu Aug 25 08:20:30 PDT 2016


We currently use MapProxy with the default file system cache. As we are running out of inodes, the plan is to migrate to an SQLite cache (cache type `sqlite`). Regarding this we have some questions:

1. Does MapProxy open connections to the <level>.mbtile files once and keep them open as long as it is running, or does MapProxy open and close these connections repeatedly, e.g. with every request?

2. We would use SQLite on network attached storage (NAS). Are there any known problems with MapProxy’s handling of this?

3. Our migration strategy is as follows: The current instance of MapProxy (Instance 1) remains live, we do not change its configuration. A new instance of MapProxy (Instance 2) is created. Instance 2 uses SQLite as cache. The source of Instance 2’s cache is Instance 1’s cache. Then we seed Instance 2’s cache. In a small test, this worked well. The question is: As Instance 1 remains live and potentially adds new files to its file system cache, could there be any issues when Instance 2 uses Instance 1’s cache?

(Instance 2 is configured to not write to Instance 1’s cache, see configuration below. Both instances run via WSGI on the same Apache web server.)

4. Since NAS is slow, we thought of putting some database files into RAM disk and periodically copying these files to persistent storage as backups. I assume copying an SQLite database while it is being written to results in a corruption of the copy. Is it possible to pause MapProxy writes to the cache so the copy will not be corrupted?

Christian


Example migration config used by Instance 2:
`sqlite_cache` uses `file_cache` as source. `file_cache` was created by Instance 1. Here, in the Instance 2 config, we access `file_cache` but do not write to it as we set its sources to an empty list.

caches:
 file_cache:
   grids: [germany]
   sources: []
 sqlite_cache:
   cache:
     directory: /nas/sqlite-cache
     type: sqlite
   grids: [germany]
   sources: [file_cache]

grids:
 germany:
   bbox: [6, 47.3, 15.1, 55]
   bbox_srs: 'EPSG:4326'
   srs: 'EPSG:900913'

layers:
 - name: test_layer
   sources: [sqlite_cache]
   title: Test Layer

services:
 demo:
 wms:
   md:
     title: MapProxy WMS Proxy!


More information about the MapProxy mailing list