[Tilecache] Tilecache: Large implementation

[Christopher Schmidt]

On Fri, Nov 02, 2007 at 01:08:04PM +0100, Espen Isaksen wrote:
> > When you say 'files', do you mean 'tiles'? I don't know how many actual
> > files I have, but I have about 120Gb of caches on the main labs server.
> > TileCache doesn't scale differently with different numbers of files
> > though, so it seems like perhaps that isn't what you meant.
> 
> 
> I did mean tiles. We are looking at a solution where our storage needs
> will be about 2-3 TB(approximately 150 million tiles). You say that
> TileCache work just as well with TB of data as GB of data? Not
> considering the hardware challenges of course(if anybody have
> suggestions for hardware for taking care of this many tiles, I will of
> course be happy).

There is no difference in TileCache scaling between 1 tile and 10 
million tiles, by design. It is possible that your filesystem will make
storing this many files difficult, but TileCache doesn't do anything 
different.

One thing to be aware of, for exaple, is that TileCache's default
DiskCache expects that directory traversals will be fast: files are 
stored many directories deep in order to ensure that the 
links-per-directory doesn't approach the 32,000 limit in place on ext3 
file systems. This would mean that storing TileCaches on AFS would be 
supremely stupid, since AFS is designed such that directory traversals 
are slow.

But that applies to a single tile as much as it does to 10 
billion. TileCache is designed to support up to 30 zoom levels for a
tile set on an ext3 filesystem, and I think it works okay for that --
and if it doens't, you can always modify the DiskCache to fix it.

Regards,
-- 
Christopher Schmidt
MetaCarta