[GRASS-dev] Unable to rename cell/null file - NULL file messed up

[Paulo van Breugel]

On 03-12-15 05:15, Glynn Clements wrote:
> Paulo van Breugel wrote:
>
>> Can't answer that, but I see that a related (?) ticket
> #2434 isn't related to #2712. #2712 relates specifically to the null
> file compression changes. #2434 was opened a year before those changes
> were started.
>
>> https://trac.osgeo.org/grass/ticket/2434 was retargeted to 7.03? From the
>> description that issue seems to be very similar to the problem described
>> above?
> The problem described above:
>
>>>> WARNING: Unable to rename cell file
>>>> 'C:\GIS\GrassJvdS/latlong/YLD_NUT/.tmp/unknown/3988.0' to
>>>> 'C:\GIS\GrassJvdS/latlong/YLD_NUT/fcell/Shan3_CA_Renyi_0_0': File exists
>>>> WARNING: Unable to rename null file
>>>> 'C:\GIS\GrassJvdS/latlong_2005/YLD_NUT/.tmp/unknown/4100.1' to
>>>> 'C:\GIS\GrassJvdS/latlong_2005/YLD_NUT/cell_misc/Shan3_CA_Renyi_1_0/null':
>>>> File exists
> doesn't match the symptoms of either #2434 or #2712. In both of those
> cases, the rename() fails with EACCESS ("Permission denied"), which is
> symptomatic of the file being open.
>
> This case fails with EEXIST ("File exists"), which suggests that the
> destination file wasn't deleted first. On Unix, deleting the original
> isn't required (or even desired); rename() will atomically remove any
> existing file. Windows generates an error if the destination exists.
> So we (try to) remove it first (on all platforms).
>
> 	remove(path);
> 	if (rename(fcb->temp_name, path)) {
> 	    G_warning(_("Unable to rename cell file '%s' to '%s': %s"),
> 		      fcb->temp_name, path, strerror(errno));
> 	    stat = -1;
> 	}
>
> (close_new() in lib/raster/close.c).
>
> The warning suggests that the file exists and remove() failed to
> remove it. On Windows, that can happen if the file is open.
>
> And looking at r.biodiversity.py, I notice:
>
>              grass.mapcalc("$outl = -1 * $outl",
>                                outl=out_renyi,
>                                overwrite=True,
>                                quiet=True)
I changed that already, now output is different from any of the input layers

> I.e. it's running r.mapcalc with the same map as both input and
> output. That isn't guaranteed to work. And in fact it doesn't, because
> the close_maps() function (which closes all of the input maps) is
> never actually called.
>
> So when it tries to close the output maps, which involves renaming the
> new cell/fcell/null files into place, the original cell/fcell/null
> files are still open for reading, meaning that the remove() will fail,
> as will the subsequent rename().
>
> Having r.mapcalc close the input maps would avoid that, and should be
> done, but ultimately that's fixing the symptoms. r.biodiversity
> shouldn't be trying to modify out_renyi "in-place". It should create a
> new map then, if necessary, replace the original using g.remove and
> g.rename.

Thanks for the detailed explanation. After I changed the r.mapcalc 
statement the script also works on Windows, but it is good to know 
better the underlying problem.

>
> One problem with in-place modification is that closing an output map
> writes a number of support files (history, range, quant, cats,
> histogram). But after closing the output maps, r.mapcalc will
> sometimes copy some of the metadata (cats, colors, history) from the
> input map. Currently, it only does this if the entire expression is
> just a map with optional modifiers, but it may get more "intelligent"
> in the future. Clearly, that's a problem if the input map has already
> been replaced.
>