[mapserver-commits] r12620 - trunk/docs/en/mapcache
svn at osgeo.org
svn at osgeo.org
Wed Oct 5 13:25:51 EDT 2011
Author: tbonfort
Date: 2011-10-05 10:25:51 -0700 (Wed, 05 Oct 2011)
New Revision: 12620
Added:
trunk/docs/en/mapcache/mapcache_caches.txt
Modified:
trunk/docs/en/mapcache/index.txt
trunk/docs/en/mapcache/mapcache_config.txt
trunk/docs/en/mapcache/mapcache_install.txt
trunk/docs/en/mapcache/mapcache_seed.txt
Log:
add documentation for different mapcache caches, along with the documentation for upcoming TIFF caches
Modified: trunk/docs/en/mapcache/index.txt
===================================================================
--- trunk/docs/en/mapcache/index.txt 2011-10-05 12:55:28 UTC (rev 12619)
+++ trunk/docs/en/mapcache/index.txt 2011-10-05 17:25:51 UTC (rev 12620)
@@ -1,7 +1,7 @@
.. _mapcache:
*****************************************************************************
- Mapserver MapCache
+MapCache
*****************************************************************************
:Author: Thomas Bonfort
@@ -19,6 +19,7 @@
mapcache_install
mapcache_config
mapcache_seed
+ mapcache_caches
Features
-----------------------------------------------------------------------------
Added: trunk/docs/en/mapcache/mapcache_caches.txt
===================================================================
--- trunk/docs/en/mapcache/mapcache_caches.txt (rev 0)
+++ trunk/docs/en/mapcache/mapcache_caches.txt 2011-10-05 17:25:51 UTC (rev 12620)
@@ -0,0 +1,249 @@
+.. _mapcache_caches:
+
+*****************************************************************************
+Cache Types
+*****************************************************************************
+
+:Author: Thomas Bonfort
+:Contact: tbonfort at terriscope.fr
+
+This document details the different cache backends that can be used to store tiles
+
+Disk Caches
+--------------------------------------------------------------------------------
+
+The disk based cache is the simplest cache to configure, and the one with the
+the fastest access to existing tiles. It is ideal for small tile repositories,
+but may cause trouble for sites hosting millions of tiles, as the number of
+files or directory may rapidly overcome the capabilities of the underlying
+filesystem. Additionaly, the block size chosen for the filesystem must closely
+match the mean size of a stored tile: ideally, any given tile should just fit
+inside a filesystem block, so as not to waste storage space inside each block,
+and not have to use up multiple blocks per tile.
+
+The location of the files/directories has to be readable and writable by the
+user running the tile server.
+
+
+There are two types of disk caches, that create a different hierarchy of files:
+
+Default Structure
+================================================================================
+
+The default disk cache stores tiles in a structure nearly identical to the
+file/directory hierarchy used by TileCache. The only change is that a top level
+directory corresponding to the name of the grid is added (as MapCache supports
+multiple grids per tileset)
+
+This cache is capable of detecting blank (i.e. uniform color) tiles and using
+a symbolic link to a single blank tile to gain disk space.
+
+
+.. code-block:: xml
+
+ <cache name="disk" type="disk">
+ <base>/tmp</base>
+ <symlink_blank/>
+ </cache>
+
+The two only configuration keys are the root directory where the tiles will be
+stored, and a key to activate the symbolic linking of blank tiles
+
+
+Template Structure
+================================================================================
+
+The template based disk cache allows you to create (or reuse an existing) tile
+structure that you define in advance. The <template> parameter takes a string
+argument where various template entries will be replaced at runtime by the correct
+value for each tile to store.
+
+.. code-block:: xml
+
+ <cache name="tmpl" type="disk">
+ <!-- template
+
+ string template that will be used to map a tile (by tileset, grid name, dimension,
+ format, x, y, and z) to a filename on the filesystem.
+ the following replacements are performed:
+ - {tileset} : the tileset name
+ - {grid} : the grid name
+ - {dim} : a string that concatenates the tile's dimension
+ - {ext} : the filename extension for the tile's image format
+ - {x},{y},{z} : the tile x,y,z values
+ - {inv_x}, {inv_y}, {inv_z} : inverted x,y,z values (inv_x = level->maxx - x - 1). This
+ is mainly used to support grids where one axis is inverted (e.g. the google schema)
+ and you want to create on offline cache.
+
+ * note that this type of cache does not support blank-tile detection and symlinking.
+
+ * warning: it is up to you to make sure that the template you chose creates a unique
+ filename for your given tilesets. e.g. do not ommit the {grid} parameter if your
+ tilesets reference multiple grids. Failure to do so will result in filename
+ collisions !
+
+ -->
+ <template>/tmp/template-test/{tileset}#{grid}#{dim}/{z}/{x}/{y}.{ext}</template>
+ </cache>
+
+
+Sqlite Caches
+--------------------------------------------------------------------------------
+
+There are three different sqlite caches that vary by the database schema they
+create and query. Sqlite caches have the advantage that they store tiles as
+blobs inside a single database file, and therefore do not have the
+disadvantages of disk caches with regards to the number of files stored. As the
+image blobs are stored contiguously, the block size chosen for the filesystem
+has no influence on the storage capacity of he volume.
+
+The sqlite based caches are a bit slower than the disk based caches, and may
+have write-locking issues at seed time if a high number of threads all try to
+insert new tiles concurrently.
+
+
+Default Schema
+================================================================================
+
+There are two different configuration options for sqlite caches, which drive how
+the database files are named on the filesystem for each tileset. You can either
+supply a base directory, where multiple database files will be created, or
+supply a template that will be populated with the name of the tileset and grid.
+
+Note
+
+.. code-block:: xml
+
+ <cache name="sqlite" type="sqlite3">
+ <base>/tmp/sqlitecachedir</base>
+ </cache>
+
+
+.. code-block:: xml
+
+ <cache name="sqlitetemplate" type="sqlite3">
+ <dbname_template>/tmp/{tileset}-{grid}.db</dbname_template>
+ </cache>
+
+
+The default sqlite schema is created by the call
+
+.. code-block:: sql
+
+ create table tiles(x integer, y integer, z integer, data blob, dim text, ctime datetime, atime datetime, hitcount integer default 0, primary key(x,y,z,dim))
+
+
+
+Custom Schema
+================================================================================
+
+This cache can use any database schema, it is up to you to supply the SQL that
+will be exectuted to select or insert a new tile.
+
+This cache type is not fully implemented yet (there is no way to configure it
+from the mapcache.xml configuration file yet)
+
+
+MBTiles Schema
+================================================================================
+
+This cache type is a shortcut to the previous custom schema sqlite cache, with
+pre-populated SQL queries that correspond to the MBTiles specification.
+
+Note that most mbtiles file you download cannot be populated on demand as the
+spec-mandated table names are actually views and not simple tables.
+
+
+.. code-block:: xml
+
+ <cache name="mbtiles" type="mbtiles">
+ <dbname_template>/Users/XXX/Documents/MapBox/tiles/natural-earth-1.mbtiles</dbname_template>
+ </cache>
+
+
+Memcache Caches
+--------------------------------------------------------------------------------
+
+This cache type stores tile to an external memcached server running on the local
+machine or accessible on the network. This cache type has the advantage that
+memcached takes care of expiring tiles, so the size of the cache will never
+exceed what has been configured in the memcache instance.
+
+Memcache support requires a rather recent version of the apr-util library. Note
+that under very high loads (usually only atainable on synthetic benchmarks on
+localhost), the memcache implementation of apr-util may fail and start dropping
+connections for some intervals of time before coming back online afterwards.
+
+You can add multiple <server> entries.
+
+
+.. code-block:: xml
+
+ <cache name="memcache" type="memcache">
+ <server>
+ <host>localhost</host>
+ <port>11211</port>
+ </server>
+ </cache>
+
+
+TIFF Caches
+--------------------------------------------------------------------------------
+
+TIFF caches are the most recent addition to the family of caches, and use the
+internal tile structure of the TIFF specification to access tile data. Tiles can
+be stored in JPEG only (TIFF does not support PNG tiles).
+
+As a single tiff file may contain many tiles, there is a drastic reduction in
+the number of files that have to be stored on the filesystem, which solves the
+major shortcomings of the disk cache. Another advantage is that the same TIFF
+files can be used by programs or WMS servers that only understand regular GIS
+raster formats, and be served up with high performance for tile access.
+
+The TIFF cache should be considered read-only for the time being. Write access
+is already possible but not ready for production, and is missing basic
+functionality (such as choosing the JPEG compression level, or adding the
+geotiff referencing).
+
+Note that the tiff tile structure must
+exactly match the structure of the grid used by the tileset, and the tif file
+names must follow strict naming rules.
+
+There are three parameters to configure a TIFF cache:
+
+- <xcount> the number of tiles stored along the x (horizontal) direction of
+ the TIFF file.
+- <ycount> the number of tiles stored along the y (vertical) direction of
+ the TIFF file.
+- <template> the template to use when looking up a TIFF file name given the
+ x,y,z of the requested tile
+
+ - {x} is replaced by the x value of the leftmost tile inside the TIFF file
+ containing the requested tile
+ - {inv_x} is replaced by the x value of the rightmost tile.
+ - {y} is replaced by the y value of the bottommost tile.
+ - {inv_y} is replaced by the y value of the topmost tile.
+ - {div_x} is replaced by the index of the TIFF file starting from the left
+ of the grid (i.e. {div_x} = {x}/<countx>)
+ - {inv_div_x} same as {div_x} but starting from the right.
+ - {div_y} is replaced by the index of the TIFF file starting from the bottom
+ of the grid (i.e. {div_y} = {y}/<county>)
+ - {inv_div_y} same as {div_y} but starting from the top.
+
+ Note that {inv_x} and {inv_div_x} will probably be rarely used, whereas
+ {inv_y} and {inv_div_y} will find some usage for people who prefer to index
+ their TIFF files from top to bottom rather than bottom to top.
+
+
+.. code-block:: xml
+
+ <cache name="tiff" type="tiff">
+ <template>/data/tiffs/{tileset}/{grid}/L{z}/R{inv_y}/C{x}.tif</template>
+ <xcount>64</xcount>
+ <ycount>64</ycount>
+ </cache>
+
+In this example, assuming a grid using 256x256 tiles, the files that are read
+to load the tiles are tiled TIFFs with jpeg compression, who's size are
+16384x16384. The number of files to store on disk is thus reduced 4096 times
+compared to the basic disk cache.
Modified: trunk/docs/en/mapcache/mapcache_config.txt
===================================================================
--- trunk/docs/en/mapcache/mapcache_config.txt 2011-10-05 12:55:28 UTC (rev 12619)
+++ trunk/docs/en/mapcache/mapcache_config.txt 2011-10-05 17:25:51 UTC (rev 12620)
@@ -1,8 +1,7 @@
.. _mapcache_config:
+***************************************************************************** Configuration File
*****************************************************************************
- Mapserver MapCache Configuration File
-*****************************************************************************
:Author: Thomas Bonfort
:Contact: tbonfort at terriscope.fr
Modified: trunk/docs/en/mapcache/mapcache_install.txt
===================================================================
--- trunk/docs/en/mapcache/mapcache_install.txt 2011-10-05 12:55:28 UTC (rev 12619)
+++ trunk/docs/en/mapcache/mapcache_install.txt 2011-10-05 17:25:51 UTC (rev 12620)
@@ -1,7 +1,7 @@
.. _mapcache_install:
*****************************************************************************
- Mapserver MapCache Compilation/Installation Instructions
+Compilation/Installation Instructions
*****************************************************************************
:Author: Thomas Bonfort
Modified: trunk/docs/en/mapcache/mapcache_seed.txt
===================================================================
--- trunk/docs/en/mapcache/mapcache_seed.txt 2011-10-05 12:55:28 UTC (rev 12619)
+++ trunk/docs/en/mapcache/mapcache_seed.txt 2011-10-05 17:25:51 UTC (rev 12620)
@@ -1,7 +1,7 @@
.. _mapcache_seed:
*****************************************************************************
- MapCache Seeder
+Seeder
*****************************************************************************
:Author: Thomas Bonfort
@@ -41,6 +41,7 @@
the value chosen here should never be much higher than the number
of cpus on the wms server)
-q: don't print progress messages to the standard output
+ -v: print verbose debugging info (if compiled in)
-D "DIMENSION=VALUE": used to specify which dimension to use if the tileset supports
dimensions. Can be used multiple times to set multiple
dimensions, e.g. -D "DIM1=VAL1" -D "DIM2=VAL2"
More information about the mapserver-commits
mailing list