[mapserver-users] Lock file limit on many concurrent seeding processes [SEC=UNCLASSIFIED]

[Stephen Woodbridge]

Hi Thomas,

Does the seeder delete the lock files once on startup?
If so then would it be safe to stop apache, start the seeder, then 
restart apache and allow the seeder and apache to generate tiles?

Thanks,
   -Steve W

On 2/26/2013 11:29 AM, thomas bonfort wrote:
> Hi Matthew,
>
> Running the seeder in parallel to incoming user requests is definitely
> something that wasn't planned for, and may potentially end up in
> corrupted tiles being written to the caches. The heart of the problem
> is that the seeder deletes the existing lockfiles at startup as it
> expects it will be the only instance creating tiles; clearly this
> isn't the case if there are other seeding instances running, or if
> live web users are requesting tiles. The reason the locks are deleted
> is to avoid a deadlock when old lockfiles are still present after e.g.
> a server crash.
>
> I'd say the risk of tile corruption is pretty low, however the error
> messages (another thread/process failed to create the tile ...) are to
> be regularly expected (namely each time you launch a seeder process
> when a lockfile created by a web-user request is present). I'd say
> your options are
>
> - to live with the small risk of tile corruption, and the fact that
> your users will be receiving transient error messages, as is the case
> today
> - to trivially patch mapcache to deactivate the lockfile deletion when
> launching a seeder. note that in that case this will end up in a
> deadlock if there where leftover lockfiles, and they would need to be
> manually deleted.
> - to sponsor the implementation of some kind of lockfile deadlock
> detection/recovery mechanism, either by checking that the lockfile
> creator is still alive, and/or that the lockfile creation date is more
> recent than a certain threshold.
>
> cheers,
> thomas
>
> On 18 February 2013 08:09, Matthew Doyle <M.Doyle at bom.gov.au> wrote:
>> Hi Thomas and MapCache users,
>>
>> Is there a way to raise the limit on the number of lockfiles which MapCache
>> uses by default?
>>
>> I am seeding a *lot* of layers twice a day, and when there are many parallel
>> processes at once, i get a steady stream of messages in my apache error_log
>> like:
>>
>>   [Mon Feb 18 06:16:14 2013] [error] [client 134.178.63.147] found old
>> lockfile /tiletmp/scratch/_gc_lock3-4-26-ADFDWEATHER1820130218061203.lck,
>> deleting itfound old lockfile
>> /tiletmp/scratch/_gc_lock3-4-28-ADFDWEATHER1820130218061203.lck, deleting
>> itfound old lockfile
>> /tiletmp/scratch/_gc_lock3-5-29-ADFDWEATHER1820130218061203.lck, deleting
>> itfound old lockfile
>> /tiletmp/scratch/_gc_lock3-4-27-ADFDWEATHER1820130218061203.lck, deleting
>> itfound old lockfile
>> /tiletmp/scratch/_gc_lock3-4-29-ADFDWEATHER1820130218061203.lck, deleting
>> itfound old lockfile
>> /tiletmp/scratch/_gc_lock3-5-26-ADFDWEATHER1820130218061203.lck, deleting
>> itfound old lockfile
>> /tiletmp/scratch/_gc_lock3-6-27-ADFDWEATHER1820130218061203.lck, deleting
>> itfound old lockfile
>> /tiletmp/scratch/_gc_lock0-0-0-ADFDWEATHER1820130218061203.lck, deleting
>> itfound old lockfile
>> /tiletmp/scratch/_gc_lock2-1-6-ADFDWEATHER1820130218061203.lck, deleting
>> itfound old lockfile
>> /tiletmp/scratch/_gc_lock1-0-2-ADFDWEATHER1820130218061203.lck, deleting
>> itfound old lockfile
>> /tiletmp/scratch/_gc_lock3-5-28-ADFDWEATHER1820130218061203.lck, deleting
>> itfound old lockfile
>> /tiletmp/scratch/_gc_lock3-5-27-ADFDWEATHER1820130218061203.lck, deleting
>> itfound old lockfile
>> /tiletmp/scratch/_gc_lock3-5-25-ADFDWEATHER1820130218061203.lck, deleting
>> itfound old lockfile
>> /tiletmp/scratch/_gc_lock3-6-26-ADFDWEATHER1820130218061203.lck, deleting
>> itfound old lockfile
>> /tiletmp/scratch/_gc_lock2-1-7-ADFDWEATHER1820130218061203.lck, deleting it
>>
>> Subsequently, i also see a lot of messages like:
>> [Mon Feb 18 06:17:08 2013] [error] [client 134.178.63.167] tileset
>> ADFDWEATHER: unknown error (another thread/process failed to create the tile
>> I was waiting for), referer: http://www.bom.gov.au/australia/meteye/
>> when a user tries to access a tile which is in the process of being seeded.
>>
>> Is this in any way related to a limit on the lockfiles? Is it possible to
>> avoid these errors, or are they false positives and something to ignore?
>>
>> Regards,
>> Matt
>>
>>
> _______________________________________________
> mapserver-users mailing list
> mapserver-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-users
>