[Tiling] compact tile format anyone

[Oliver Tonnhofer]

Hi Gabriel,

On 21.12.2010, at 16:19, Gabriel Roldán wrote:
> But wouldn't it be nice if we defined our own thing collaboratively
> using this list as a communication board? Then tile blob stores could be
> used interchangeably by our projects?
> 
> If interested, we could start thinking about this. Sounds like a nice
> new year project to me.

We (MapProxy) are interested. Currently we are using the TileCache schema for storing the tiles on disc. But, as Chris said at the Tiling BoF in Barcelona, the schema was designed to work on all file systems even in worst cases (30 zoom levels). I'm not aware of any benchmarks for different tile schemas, but I can imagine that a lower nesting of directories could bring some (small) improvements in tile access and traversal (during seed or rsync operations).

Of course, a blob store should be more efficient, especially for exchange between different servers. Some requirements we would have for such format:

- Should be dynamic: add, remove and update single tiles
  The OSM project stores 64 tiles in a single file, but as far as I now, it is only a file with a list of offsets and then all png tiles concatenated.
- Support for tile metadata: size, timestamps for creation and access (maybe not on tile level but for groups of tiles)
- Support for file metadata: bbox, levels, x/y range. (should be possible to split a tileset into multiple files) 
- Support for internal links: single color tiles (like ocean tiles) should be stored outside the normal x-y-z tiles and other tiles should be able to link to these tiles. MapProxy uses symbolic links for that on the file system, but that does not work on Windows.

Storing a bunch of tiles in a single file (as the OSM project does) is pretty easy. But designing and implementing a file format that supports the above features could become an ambitious project. Efficient access (e.g. read a minimum of file blocks), cross platform file locking and file fragmentation comes to my mind. So I'm strongly in favor of using something existing that already handles these issues.

I really like the idea of using SQLite as MBTiles does. SQLite solves all the hard issues 
and I'm expecting that the performance would be good too. I know that SQLite belongs to the "C tribe", but I know of several projects that offer bindings Java bindings for SQLite, though I don't know how they perform. Does someone have experience with Java and SQLite? Maybe there is another library we could use as the base for a new file format?

If we go the single file SQL DB route but we can't agree on a specific file format, we still could design a SQL schema specification. That way it would be easy to convert between different implementations (e.g. from H2 to SQLite).


PS: We have a student that we like to sponsor to do some research in that direction in his bachelor theses, but that would start not until next summer.

Regards,
Oliver