[GRASS-dev] Re: what if: a new GRASS directory layout?

[Glynn Clements]

Ivan Shmakov wrote:

>  >>> It seems that the reasonable behaviour would be to make a
>  >>> back-reference if possible, and issue a warning if not.
> 
>  >> A warning isn't sufficent. If the GC relies upon back-references,
>  >> but you can't create one, the data could disappear while a process
>  >> is trying to use it.
> 
>  > Still it's an improvement, since currently the data could just as
>  > well be overwritten.  The transition to tmp/ + rename () alone
>  > doesn't help much, as a support file may be renamed while the other
>  > is being read.
> 
> 	Thinking on it further, the change to the
> 	LOCATION/MAPSET/raster/RASTER/ scheme doesn't actually improve
> 	the situation, as the directory may be renamed while the other
> 	party reads the support files

That can be handled by locking, and it can probably be done without
significant changes to the API.

If modules are changed to always read support files before reading the
data, the library functions can acquire the lock on G_open_cell_old()
and release it upon the first call to G_get_raster_row().

For output maps, the situation is slightly more tricky, but it may be
possible to use a similar mechanism.

The module would write any support files after the last call to
G_put_raster_row() but before the G_close_cell() call. When opening
for write any element of a map which is open for write, the call would
open a new file in the output map's temporary directory. The
G_close_cell() call would lock the destination; upon acquiring the
lock, the entire directory would be replace with the new version, and
the lock released.

Also, locking doesn't have any permission issues; if you can read a
file, then you can acquire a lock on it.

>  >> Essentially, if you can't ensure that the back-references can be
>  >> created, you can't design a GC which relies upon them.
> 
>  > Well, yes.  That's why I've suggested some solutions in my previous
>  > post.  To simplify, they are:
> 
>  > * hardlink the entire maps into a place where you can create
>  > back-references, e. g. the current mapset;
> 
>  > Pro's: could be done with `g.copy' (changed); becomes possible as
>  > soon as both the ``in place'' and ``atomic replace'' issues have
>  > been dealt with, no need for an inventory scheme;
> 
> 	Actually no, as long as the raster might be replaced while
> 	`g.copy' is being run.  And since the other mapset is read-only,
> 	there's no way to prevent it.
> 
> 	It thus seems that among the RASTER/ and the inventory scheme,
> 	only the latter provides good support for concurrency, and not
> 	the former.

Even with no changes to the directory structure, you can eliminate the
race conditions with enough locking. A single directory just makes
locking simpler.

The inventory system can prevent inconsistencies without any locking. 
However, there's still the potential for updates to be discarded if
you don't have locking. I.e. if two process attempt to update a module
concurrently, the update which completes first will be lost.

-- 
Glynn Clements <glynn at gclements.plus.com>