[Tilecache] Making TileCache seed faster?

[Stephen Woodbridge]

Christopher Schmidt wrote:
> On Fri, Apr 04, 2008 at 09:30:55AM +0200, Miguel Eduardo Gil Biraud wrote:
>> Hi,
>> I was following this thread and then I was surprised by your statement...
>>
>> On Fri, Apr 4, 2008 at 4:23 AM, Christopher Schmidt
>> <crschmidt at metacarta.com> wrote:
>>> On Thu, Apr 03, 2008 at 02:01:22PM -0600, Gregor Mosheh wrote:
>>>  The pure act of 'precaching' is ~20 tiles/sec via CGI, 300 tiles/sec via
>>>  mod_python, and 10000 tiles/sec (essentially only limited by disk read
>>>  speed) via local Python.
>> How do you invoke it via local Python to reach the 10000 tiles/sec
>> performance? I am particularly interested to use it when a seeding
>> process breaks down and I have to resume. In low levels it is not a
>> problem as the number of tiles is small, but as soon as you go a
>> little bit deeper the number of tiles explodes and then it means a
>> long time to get to the point when you start to get again cache
>> misses.
> 
>  1. Use Tilecache > 1.9
>  2. Get Really Really fast disks.
> 
> Regards,

I have not looked at Tilecache yet. But I when I was working with ka-map 
precache.php script, I modified it to mark completed rows and completed 
metatiles. I just use touched a marker file on completion and then 
skipped it if it existed. This made restarts much faster. This also 
allowed me to have multiple scripts running on separate processors sll 
feed the same SAN (often restricted to separate geographical areas, some 
overlapping with others) and later run a cleanup and sweep script over 
the whole area to pickup and bits and pieces that were not done. I also 
made it easy to restrict the tiles generated to specific rows of 
metatiles or bounding boxes, so I could start non-overlapping processes.

Are you doing something like this in Tilecache > 1.9? Darn, yet another 
tools I really need to look into.

-Steve