[Tiling] Steps towards a better filemanagement

[Andreas Trawoeger]

Hi Oliver!

2011/4/11 Oliver Tonnhofer <olt at omniscale.de
>
>
> On 11.04.2011, at 13:38, Andreas Trawoeger wrote:
> > I'm not really sure if switching to SQLite is going to make
> tile-cache-management a lot easier in the long run. SQLite would be perfect
> for small and somewhat static tilestores or offline caches for mobile
> applications.
> > But once they get bigger and have constant read & write access databases
> get notorously difficult to backup.
>
> Thanks for raising some concerns. It's too easy to neglect the cons when
> discussing these things.
>
> > Having thousand of files can be hard to handle. But there are myriad of
> tools like e.g. rsync that makes handling them a lot easier.
>
> Copying thousands of small files is much slower then a few big files. I
> have a virtual machine where it takes 2.5 minutes to do a `du -sk
> cache_data` on 300.000 tiles. And 300.000 tiles is a small number. These are
> the issues when we are talking about manageability.
>
> For managing SQLite files, there is a backup API that allows copying
> databases without locking issues. http://www.sqlite.org/backup.html
>
> My database design provides support splitting a cache into multiple
> databases (e.g. one DB per resolution, or even multiple DBs for large tile
> sets (level > 16)). So for large caches you would have multiple files.
>

It always depends on where your bottleneck is.

My 1&1 virtual server for 30 Euro/month currently holds 1.339.303 tiles and
doing a `du -sk cache_data` needs 1m 19sek, and it can easily saturate its
100Mbit internet uplink, even when copying small tiles. So having lots of
small files never has been a problem to me.

Things definitely change once you run a server with multiple Gigabit
Ethernet connections or use a backup solution like e.g. TSM [0] that stores
the location of every file into a database.



cu andreas


[0]
https://secure.wikimedia.org/wikipedia/en/wiki/IBM_Tivoli_Storage_Manager
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/tiling/attachments/20110411/6f8403eb/attachment.html