[mapserver-users] Lock file limit on many concurrent seeding processes [SEC=UNCLASSIFIED]

[thomas bonfort]

Hi Matthew,

Running the seeder in parallel to incoming user requests is definitely
something that wasn't planned for, and may potentially end up in
corrupted tiles being written to the caches. The heart of the problem
is that the seeder deletes the existing lockfiles at startup as it
expects it will be the only instance creating tiles; clearly this
isn't the case if there are other seeding instances running, or if
live web users are requesting tiles. The reason the locks are deleted
is to avoid a deadlock when old lockfiles are still present after e.g.
a server crash.

I'd say the risk of tile corruption is pretty low, however the error
messages (another thread/process failed to create the tile ...) are to
be regularly expected (namely each time you launch a seeder process
when a lockfile created by a web-user request is present). I'd say
your options are

- to live with the small risk of tile corruption, and the fact that
your users will be receiving transient error messages, as is the case
today
- to trivially patch mapcache to deactivate the lockfile deletion when
launching a seeder. note that in that case this will end up in a
deadlock if there where leftover lockfiles, and they would need to be
manually deleted.
- to sponsor the implementation of some kind of lockfile deadlock
detection/recovery mechanism, either by checking that the lockfile
creator is still alive, and/or that the lockfile creation date is more
recent than a certain threshold.

cheers,
thomas

On 18 February 2013 08:09, Matthew Doyle <M.Doyle at bom.gov.au> wrote:
> Hi Thomas and MapCache users,
>
> Is there a way to raise the limit on the number of lockfiles which MapCache
> uses by default?
>
> I am seeding a *lot* of layers twice a day, and when there are many parallel
> processes at once, i get a steady stream of messages in my apache error_log
> like:
>
>  [Mon Feb 18 06:16:14 2013] [error] [client 134.178.63.147] found old
> lockfile /tiletmp/scratch/_gc_lock3-4-26-ADFDWEATHER1820130218061203.lck,
> deleting itfound old lockfile
> /tiletmp/scratch/_gc_lock3-4-28-ADFDWEATHER1820130218061203.lck, deleting
> itfound old lockfile
> /tiletmp/scratch/_gc_lock3-5-29-ADFDWEATHER1820130218061203.lck, deleting
> itfound old lockfile
> /tiletmp/scratch/_gc_lock3-4-27-ADFDWEATHER1820130218061203.lck, deleting
> itfound old lockfile
> /tiletmp/scratch/_gc_lock3-4-29-ADFDWEATHER1820130218061203.lck, deleting
> itfound old lockfile
> /tiletmp/scratch/_gc_lock3-5-26-ADFDWEATHER1820130218061203.lck, deleting
> itfound old lockfile
> /tiletmp/scratch/_gc_lock3-6-27-ADFDWEATHER1820130218061203.lck, deleting
> itfound old lockfile
> /tiletmp/scratch/_gc_lock0-0-0-ADFDWEATHER1820130218061203.lck, deleting
> itfound old lockfile
> /tiletmp/scratch/_gc_lock2-1-6-ADFDWEATHER1820130218061203.lck, deleting
> itfound old lockfile
> /tiletmp/scratch/_gc_lock1-0-2-ADFDWEATHER1820130218061203.lck, deleting
> itfound old lockfile
> /tiletmp/scratch/_gc_lock3-5-28-ADFDWEATHER1820130218061203.lck, deleting
> itfound old lockfile
> /tiletmp/scratch/_gc_lock3-5-27-ADFDWEATHER1820130218061203.lck, deleting
> itfound old lockfile
> /tiletmp/scratch/_gc_lock3-5-25-ADFDWEATHER1820130218061203.lck, deleting
> itfound old lockfile
> /tiletmp/scratch/_gc_lock3-6-26-ADFDWEATHER1820130218061203.lck, deleting
> itfound old lockfile
> /tiletmp/scratch/_gc_lock2-1-7-ADFDWEATHER1820130218061203.lck, deleting it
>
> Subsequently, i also see a lot of messages like:
> [Mon Feb 18 06:17:08 2013] [error] [client 134.178.63.167] tileset
> ADFDWEATHER: unknown error (another thread/process failed to create the tile
> I was waiting for), referer: http://www.bom.gov.au/australia/meteye/
> when a user tries to access a tile which is in the process of being seeded.
>
> Is this in any way related to a limit on the lockfiles? Is it possible to
> avoid these errors, or are they false positives and something to ignore?
>
> Regards,
> Matt
>
>