[mapserver-users] Lock file limit on many concurrent seeding processes [SEC=UNCLASSIFIED]

[thomas bonfort]

Hi Steve,

On 26 February 2013 17:34, Stephen Woodbridge <woodbri at swoodbridge.com> wrote:
> Hi Thomas,
>
> Does the seeder delete the lock files once on startup?
> If so then would it be safe to stop apache, start the seeder, then restart
> apache and allow the seeder and apache to generate tiles?
Unfortunately no, as the apache instance also deletes existing
lockfiles for the same reason. In practice this *would* however help a
bit as the seeder is never supposed to lock on a lock it has created
itself, however you might still end up with corruptions in the case
where multiple seeding instances have been launched.
Given Matthew's usecase, I also suspect that restarting apache is not
an acceptable behavior.

cheers,

--
thomas

>
> Thanks,
>   -Steve W
>
>
> On 2/26/2013 11:29 AM, thomas bonfort wrote:
>>
>> Hi Matthew,
>>
>> Running the seeder in parallel to incoming user requests is definitely
>> something that wasn't planned for, and may potentially end up in
>> corrupted tiles being written to the caches. The heart of the problem
>> is that the seeder deletes the existing lockfiles at startup as it
>> expects it will be the only instance creating tiles; clearly this
>> isn't the case if there are other seeding instances running, or if
>> live web users are requesting tiles. The reason the locks are deleted
>> is to avoid a deadlock when old lockfiles are still present after e.g.
>> a server crash.
>>
>> I'd say the risk of tile corruption is pretty low, however the error
>> messages (another thread/process failed to create the tile ...) are to
>> be regularly expected (namely each time you launch a seeder process
>> when a lockfile created by a web-user request is present). I'd say
>> your options are
>>
>> - to live with the small risk of tile corruption, and the fact that
>> your users will be receiving transient error messages, as is the case
>> today
>> - to trivially patch mapcache to deactivate the lockfile deletion when
>> launching a seeder. note that in that case this will end up in a
>> deadlock if there where leftover lockfiles, and they would need to be
>> manually deleted.
>> - to sponsor the implementation of some kind of lockfile deadlock
>> detection/recovery mechanism, either by checking that the lockfile
>> creator is still alive, and/or that the lockfile creation date is more
>> recent than a certain threshold.
>>
>> cheers,
>> thomas
>>
>> On 18 February 2013 08:09, Matthew Doyle <M.Doyle at bom.gov.au> wrote:
>>>
>>> Hi Thomas and MapCache users,
>>>
>>> Is there a way to raise the limit on the number of lockfiles which
>>> MapCache
>>> uses by default?
>>>
>>> I am seeding a *lot* of layers twice a day, and when there are many
>>> parallel
>>> processes at once, i get a steady stream of messages in my apache
>>> error_log
>>> like:
>>>
>>>   [Mon Feb 18 06:16:14 2013] [error] [client 134.178.63.147] found old
>>> lockfile /tiletmp/scratch/_gc_lock3-4-26-ADFDWEATHER1820130218061203.lck,
>>> deleting itfound old lockfile
>>> /tiletmp/scratch/_gc_lock3-4-28-ADFDWEATHER1820130218061203.lck, deleting
>>> itfound old lockfile
>>> /tiletmp/scratch/_gc_lock3-5-29-ADFDWEATHER1820130218061203.lck, deleting
>>> itfound old lockfile
>>> /tiletmp/scratch/_gc_lock3-4-27-ADFDWEATHER1820130218061203.lck, deleting
>>> itfound old lockfile
>>> /tiletmp/scratch/_gc_lock3-4-29-ADFDWEATHER1820130218061203.lck, deleting
>>> itfound old lockfile
>>> /tiletmp/scratch/_gc_lock3-5-26-ADFDWEATHER1820130218061203.lck, deleting
>>> itfound old lockfile
>>> /tiletmp/scratch/_gc_lock3-6-27-ADFDWEATHER1820130218061203.lck, deleting
>>> itfound old lockfile
>>> /tiletmp/scratch/_gc_lock0-0-0-ADFDWEATHER1820130218061203.lck, deleting
>>> itfound old lockfile
>>> /tiletmp/scratch/_gc_lock2-1-6-ADFDWEATHER1820130218061203.lck, deleting
>>> itfound old lockfile
>>> /tiletmp/scratch/_gc_lock1-0-2-ADFDWEATHER1820130218061203.lck, deleting
>>> itfound old lockfile
>>> /tiletmp/scratch/_gc_lock3-5-28-ADFDWEATHER1820130218061203.lck, deleting
>>> itfound old lockfile
>>> /tiletmp/scratch/_gc_lock3-5-27-ADFDWEATHER1820130218061203.lck, deleting
>>> itfound old lockfile
>>> /tiletmp/scratch/_gc_lock3-5-25-ADFDWEATHER1820130218061203.lck, deleting
>>> itfound old lockfile
>>> /tiletmp/scratch/_gc_lock3-6-26-ADFDWEATHER1820130218061203.lck, deleting
>>> itfound old lockfile
>>> /tiletmp/scratch/_gc_lock2-1-7-ADFDWEATHER1820130218061203.lck, deleting
>>> it
>>>
>>> Subsequently, i also see a lot of messages like:
>>> [Mon Feb 18 06:17:08 2013] [error] [client 134.178.63.167] tileset
>>> ADFDWEATHER: unknown error (another thread/process failed to create the
>>> tile
>>> I was waiting for), referer: http://www.bom.gov.au/australia/meteye/
>>> when a user tries to access a tile which is in the process of being
>>> seeded.
>>>
>>> Is this in any way related to a limit on the lockfiles? Is it possible to
>>> avoid these errors, or are they false positives and something to ignore?
>>>
>>> Regards,
>>> Matt
>>>
>>>
>> _______________________________________________
>> mapserver-users mailing list
>> mapserver-users at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/mapserver-users
>>
>
> _______________________________________________
> mapserver-users mailing list
> mapserver-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-users