[Tilecache] Making TileCache seed faster?

[Christopher Schmidt]

On Fri, Apr 04, 2008 at 10:59:58AM -0500, Stephen Woodbridge wrote:
> Christopher Schmidt wrote:
> >On Fri, Apr 04, 2008 at 09:30:55AM +0200, Miguel Eduardo Gil Biraud wrote:
> >>Hi,
> >>I was following this thread and then I was surprised by your statement...
> >>
> >>On Fri, Apr 4, 2008 at 4:23 AM, Christopher Schmidt
> >><crschmidt at metacarta.com> wrote:
> >>>On Thu, Apr 03, 2008 at 02:01:22PM -0600, Gregor Mosheh wrote:
> >>> The pure act of 'precaching' is ~20 tiles/sec via CGI, 300 tiles/sec via
> >>> mod_python, and 10000 tiles/sec (essentially only limited by disk read
> >>> speed) via local Python.
> >>How do you invoke it via local Python to reach the 10000 tiles/sec
> >>performance? I am particularly interested to use it when a seeding
> >>process breaks down and I have to resume. In low levels it is not a
> >>problem as the number of tiles is small, but as soon as you go a
> >>little bit deeper the number of tiles explodes and then it means a
> >>long time to get to the point when you start to get again cache
> >>misses.
> >
> > 1. Use Tilecache > 1.9
> > 2. Get Really Really fast disks.
> >
> >Regards,
> 
> I have not looked at Tilecache yet. But I when I was working with ka-map 
> precache.php script, I modified it to mark completed rows and completed 
> metatiles. I just use touched a marker file on completion and then 
> skipped it if it existed. This made restarts much faster. This also 
> allowed me to have multiple scripts running on separate processors sll 
> feed the same SAN (often restricted to separate geographical areas, some 
> overlapping with others) and later run a cleanup and sweep script over 
> the whole area to pickup and bits and pieces that were not done. I also 
> made it easy to restrict the tiles generated to specific rows of 
> metatiles or bounding boxes, so I could start non-overlapping processes.
> 
> Are you doing something like this in Tilecache > 1.9? Darn, yet another 
> tools I really need to look into.

Nope. No plans to, either. Someone else is welcome to tackle it;
tilecache_seed is a tiny wrapper around TileCache/Client.py, which is
itself pretty easy to understand.

Regards,
-- 
Christopher Schmidt
MetaCarta