[Tiling] Steps towards a better filemanagement

[Oliver Tonnhofer]

Hi Andreas,

On 11.04.2011, at 13:38, Andreas Trawoeger wrote:
> I'm not really sure if switching to SQLite is going to make tile-cache-management a lot easier in the long run. SQLite would be perfect for small and somewhat static tilestores or offline caches for mobile applications.
> But once they get bigger and have constant read & write access databases get notorously difficult to backup.

Thanks for raising some concerns. It's too easy to neglect the cons when discussing these things.

> Having thousand of files can be hard to handle. But there are myriad of tools like e.g. rsync that makes handling them a lot easier.

Copying thousands of small files is much slower then a few big files. I have a virtual machine where it takes 2.5 minutes to do a `du -sk cache_data` on 300.000 tiles. And 300.000 tiles is a small number. These are the issues when we are talking about manageability. 

For managing SQLite files, there is a backup API that allows copying databases without locking issues. http://www.sqlite.org/backup.html

My database design provides support splitting a cache into multiple databases (e.g. one DB per resolution, or even multiple DBs for large tile sets (level > 16)). So for large caches you would have multiple files.

Regards,
Oliver

-- 
Oliver Tonnhofer    | Omniscale GmbH & Co KG    | http://omniscale.de
http://mapproxy.org | https://bitbucket.org/olt | @oltonn