[Tiling] compact tile format anyone

[Gabriel Roldán]

Hi, thanks all for the interest.
I know it's not such an easy topic specially because there are just so
many storage options and so little time to conduct proper research on
each of them.

If there's an advantage on simple file system storage of plain tile
files though, is its simplicity, which is not to easily disregard.

So it comes with a couple gotchas, specially when volume matters, and we
know about that since our data volume increases exponentially:
- waste of storage space (fs block size as the culprit)
- difficulty for the fs to cope up with with so many files and
directories
- hard to transfer
- needs a separate storage mechanism for meta information

I totally agree with you that any solution should allow for easy storage
and retrieval of single tiles, so the OSM approach does not fit us. And
as we mentioned at the FOSS4G BOF, it would be better to rely on
something existent than inventing a new wheel.

SQLite is tempting cause it's a single file db, low footprint, and
widely supported. I just don't know if/how it will scale to the terabyte
range.

Another option that's tempting to me is Berkeley DB, as it's a key/value
store, very mature, available on most platforms, supposedly having
binding for almost any language you can think of, and claims to scale
really well.

But of course I didn't get to try it for tile storage at all neither.

So, I propose we start by defining a clear set of requirements. Please
help me on this as I'm writing off the top of my head and quite in a
rush, and even if I weren't it would be incomplete/inaccurate.
So, such a tile storage option should:
- be easy to transfer
- be simple
- contain both tile sets and their metadata
- "scale well" (as loose as it sounds)
- support single tile storage/retrieval efficiently
- ideally provide some benefit over bulk operations (truncation?) over
the simple file system storage

anything else?

Or we could also try to define an abstract schema or information model
for it, and then try different technology options?

I personally envision having one such storage package per layer, so
that's as easy as scp <tileset> <someplace> to copy a layer's cache.

Cheers,
Gabriel

On Thu, 2010-12-23 at 11:43 +0100, Oliver Tonnhofer wrote:
> Hi Gabriel,
> 
> On 21.12.2010, at 16:19, Gabriel Roldán wrote:
> > But wouldn't it be nice if we defined our own thing collaboratively
> > using this list as a communication board? Then tile blob stores could be
> > used interchangeably by our projects?
> > 
> > If interested, we could start thinking about this. Sounds like a nice
> > new year project to me.
> 
> We (MapProxy) are interested. Currently we are using the TileCache
>  schema for storing the tiles on disc. But, as Chris said at the Tiling
>  BoF in Barcelona, the schema was designed to work on all file systems
>  even in worst cases (30 zoom levels). I'm not aware of any benchmarks
>  for different tile schemas, but I can imagine that a lower nesting of
>  directories could bring some (small) improvements in tile access and
>  traversal (during seed or rsync operations).
> 
> Of course, a blob store should be more efficient, especially for
>  exchange between different servers. Some requirements we would have
>  for such format:
> 
> - Should be dynamic: add, remove and update single tiles   The OSM
>  project stores 64 tiles in a single file, but as far as I now, it is
>  only a file with a list of offsets and then all png tiles
>  concatenated. - Support for tile metadata: size, timestamps for
>  creation and access (maybe not on tile level but for groups of tiles)
> - Support for file metadata: bbox, levels, x/y range. (should be
>  possible to split a tileset into multiple files) - Support for
>  internal links: single color tiles (like ocean tiles) should be stored
>  outside the normal x-y-z tiles and other tiles should be able to link
>  to these tiles. MapProxy uses symbolic links for that on the file
>  system, but that does not work on Windows.
> 
> Storing a bunch of tiles in a single file (as the OSM project does) is
>  pretty easy. But designing and implementing a file format that
>  supports the above features could become an ambitious project.
>  Efficient access (e.g. read a minimum of file blocks), cross platform
>  file locking and file fragmentation comes to my mind. So I'm strongly
>  in favor of using something existing that already handles these
>  issues.
> 
> I really like the idea of using SQLite as MBTiles does. SQLite solves
>  all the hard issues and I'm expecting that the performance would be
>  good too. I know that SQLite belongs to the "C tribe", but I know of
>  several projects that offer bindings Java bindings for SQLite, though
>  I don't know how they perform. Does someone have experience with Java
>  and SQLite? Maybe there is another library we could use as the base
>  for a new file format?
> 
> If we go the single file SQL DB route but we can't agree on a specific
>  file format, we still could design a SQL schema specification. That
>  way it would be easy to convert between different implementations
>  (e.g. from H2 to SQLite).
> 
> 
> PS: We have a student that we like to sponsor to do some research in
>  that direction in his bachelor theses, but that would start not until
>  next summer.
> 
> Regards,
> Oliver

-- 
Gabriel Roldan
groldan at opengeo.org
Expert service straight from the developers