[Tiling] compact tile format anyone

[Oliver Tonnhofer]

Hi Gabriel,

On 23.12.2010, at 14:59, Gabriel Roldán wrote:
> SQLite is tempting cause it's a single file db, low footprint, and
> widely supported. I just don't know if/how it will scale to the terabyte
> range.

From http://www.sqlite.org/whentouse.html
"With the default page size of 1024 bytes, an SQLite database is limited in size to 2 terabytes (2^41 bytes)"

Switching from millions of files to a single (multi)terabyte file is just asking for new problems instead of solving all handling headaches. The storage should work with multiple files, e.g. one per level, or even smaller segments when a level contains million tiles.
I think a few gigabytes is a good size, of course there shouldn't be a limitation and it should be possible to create a single file storage.

> Another option that's tempting to me is Berkeley DB, as it's a key/value
> store, very mature, available on most platforms, supposedly having
> binding for almost any language you can think of, and claims to scale
> really well.
> 
> But of course I didn't get to try it for tile storage at all neither.

That's where we should do some little testing.

> So, I propose we start by defining a clear set of requirements. Please
> help me on this as I'm writing off the top of my head and quite in a
> rush, and even if I weren't it would be incomplete/inaccurate.
> So, such a tile storage option should:
> - be easy to transfer
> - be simple
> - contain both tile sets and their metadata
> - "scale well" (as loose as it sounds)
> - support single tile storage/retrieval efficiently
> - ideally provide some benefit over bulk operations (truncation?) over
> the simple file system storage
> 
> anything else?

Sounds loose but it makes a good 'mission statement' IMO :)

We should create a page in the wiki and start to define requirements.


Regards,
Oliver