[Tiling] compact tile format anyone

[thomas bonfort]

On Thu, Dec 23, 2010 at 11:43, Oliver Tonnhofer <olt at bogosoft.com> wrote:
> Some requirements we would have for such format:
>
> - Should be dynamic: add, remove and update single tiles
>  The OSM project stores 64 tiles in a single file, but as far as I now, it is only a file with a list of offsets and then all png tiles concatenated.
> - Support for tile metadata: size, timestamps for creation and access (maybe not on tile level but for groups of tiles)
> - Support for file metadata: bbox, levels, x/y range. (should be possible to split a tileset into multiple files)
> - Support for internal links: single color tiles (like ocean tiles) should be stored outside the normal x-y-z tiles and other tiles should be able to link to these tiles. MapProxy uses symbolic links for that on the file system, but that does not work on Windows.

another requirement that comes to mind would be the support of
dimensions (eg time and elevation for wms, but we should not restrict
ourselves to those two). Having one or more dimensions for a set of
tiles is not an uncommon requirement, for example in the
meteorological field where you want to represent the forecast at a
given time, calculated from the data available at a given (different)
time.

>
> Storing a bunch of tiles in a single file (as the OSM project does) is pretty easy. But designing and implementing a file format that supports the above features could become an ambitious project. Efficient access (e.g. read a minimum of file blocks), cross platform file locking and file fragmentation comes to my mind. So I'm strongly in favor of using something existing that already handles these issues.

just to clarify here, when talking about file locking, I assume you
are talking about concurrent read/write access to the actual tiles,
and not the locking that (should) occur(s) when metatiling and the
first process that creates a metatile puts a lock on the other tiles
of the metatile so that the metatile is not rendered multiple times.
In essence, the chosen format must be accessible by multiple
independant processes (as sqlite permits), but their should not be an
is_locked column for each tile, as that kind of locking is not the
responsability of the actual caching implementation (i.e. sqlite,
files, s3, ...) used.

best regards,
thomas