[Tilecache] Too Many Layers in tilecache.cfg?

[Christopher Schmidt]

On Wed, Feb 25, 2009 at 07:24:38PM -0600, Brian Walawender wrote:
> Hello,
>  
> Is there a limit on how large tilecache.cfg can be before you start seeing
> performance issues?  We have a very large tilecache.cfg file
> .         Nearly 15,500 layers
> .         154563 lines
> .         4.3 MB in size
>  
> We are using mod_python to run tilecache.  A typical entry in the file looks
> like:
>  
> [mia_tr0_8]
> type=WMS
> url=http://192.168.101.111/cgi-bin/mapserv?map=/usr/local/mapserver_config/t
> dwr/MIA.map&transparent=true
> layers=mia_tr0_8
> extension=png
> bbox=-20037508.3427892,-20037508.3427892,20037508.3427892,20037508.3427892
> maxResolution=156543.0339
> srs=EPSG:900913
> debug=false
>  
> I can validate the Mapserver is functioning properly and I don't prolonged
> heavy loading on that machine.  The performance issues on the tilecache
> machine are  missing tiles and/or heavy system loading (to the point that
> the machine becomes un-useable and has to be rebooted).   The two servers
> (wms and tilecache) are 8 core Xeon blades talking across the chassis.   
>  
> When our tilecache.cfg file was an order of magnitude smaller (in line
> count) we didn't see these issues.  Have we hit the wall?  Any suggestions
> on how to deal with this (less layers really isn't an option)? 

Whenever an apache child restarts -- which happens occasionally,
depending on your configuration -- the 150000 line config file will be
re-parsed. At that point, a number of Python structures will be
insantiated; I would not be surprised to find out that this is a slow
process. (With 20 layers during performance testing, I have seen a
significant slowdown in the past.)

My recommendation to you would be:
 1. Create some non-.cfg based storage for your TileCache layers
    A SQLIte database or similar would probably suffice. 
 2. Create a custom 'handler', mimicking the one in TileCache.Service.
 3. Instead of calling Service.load(**cfgfiles), construct a Layer 
    definition on the fly based on URL parameters, puling the required
    information from your database.

I would not be entirely against including this in TileCache in some way;
perhaps as an alternative to Service.py like DBService.py.

In the TIleCache distribution, in doc/example.py.txt, there is a
demonstration of how to consturct tiles manually rather than using the
config file. (Originally, TileCache didn't *have* config files;
definition was going to be all in Python until Schuyler came up with the
quite-brilliant idea to use a config file.) You could use this as a
starting point for some code to do this.

I must say, I'm amazed you got nearly so far as you did; this seems like
it's going far above and beyond where I would have expected TileCache to
fall over and beg for mercy. This says something for the technology, I
guess!

Regards,
-- 
Christopher Schmidt
MetaCarta