[mapserver-dev] MapCache seeding issues

[thomas bonfort]

On Mon, Jan 23, 2012 at 15:41, Stephen Woodbridge
<woodbri at swoodbridge.com> wrote:
> On 1/23/2012 9:01 AM, Smith, Michael ERDC-CRREL-NH wrote:
>>
>> All,
>>
>> I'm running a cache seeding and it will occasionally fail with
>>
>> unknown error (another thread/process failed to create the tile I was
>> waiting for
>>
>> Unfortunately this stops the seeding and restarting takes about 30 mins
>> while it reads through all the created tiles (I'm using a TIFF Cache) to
>> see where to start (this is a cache for a MapServer WMS Tileindex of
>> about 800,000 1m jp2 images).
>> Is there any way to make the seeding continue even if it encounters an
>> error (perhaps writing the errors to a log) or to restart the seeding
>> quicker by passing the last seed location (something like 54820 157495
>> 18)?
>
>
> Thomas,
>
> I wrote a seeder for another tile cache system in the past and the restart
> issue was an issue for me also. What I did was to create some sentinel files
> that indicated if a row was completed and when a metatile was complete. Then
> the seeder could check and skip completed rows and also check for complete
> metatiles in the row when it restarted. Putting them is a separate directory
> makes it easy to remove them later.

That could be a feasible enhancement. I had refrained from doing this
at the beginning because the looping over the existing tiles in
mapcache is so much faster than on tilecache/python that my
preliminary testing made me think there was little use in doing so.
Apparently for very large caches, and/or tiff caches this is not the
case, so it makes sense to use sentinel files. It won't be as simple
as recording the row numbers, as the seeder does not iterate
sequentially on the tile rows and columns, it recursively drills down
into a given tile in order to take advantage of hot os-level caches on
the source wms server.

>
> Also in my seeder, I could restrict it to only seed some specified row range
> or a bounding box, so that I could run multiple instances of it on separate
> threads or separate computers and have them all write to the same SAN device
> or write to local disks and later copy the tiles to the final cache area.

mapcache_seed already accepts a bounding box as its -e argument.

--
thomas

>
> These might be useful features to consider adding to mapcache.
>
> Thanks,
>  -Steve
> _______________________________________________
> mapserver-dev mailing list
> mapserver-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-dev