[GRASS-dev] r.series with file option - too many files open

[Markus Metz]

On Mon, Jan 15, 2018 at 8:41 PM, Veronica Andreo <veroandreo at gmail.com>
wrote:
>
> Hi all,
>
> 2018-01-15 20:11 GMT+01:00 Markus Neteler <neteler at osgeo.org>:
>>
>> On Mon, Jan 15, 2018 at 5:27 PM, Paulo van Breugel
>> <p.vanbreugel at gmail.com> wrote:
>> > On 1/15/18 4:39 PM, Markus Metz wrote:
>> ...
>> >> the manual is wrong:
>> >> Use the file option to analyze large amount of raster maps without
hitting
>> >> open files limit and the size limit of command line arguments.
>> >>
>> >> must be
>> >> Use the -z flag to analyze large amount of raster maps without
hitting open
>> >> files limit and the file option to avoid hitting the size limit of
command
>> >> line arguments.
>> >
>> > Ah, yes, I totally forgot about that flag. Thanks (also Anna and
Veronica).
>> > Perhaps somebody with write permission can correct his in the manual
page?
>>
>> Done in r72081: please verify. I think it could still be better.
>> https://trac.osgeo.org/grass/changeset/72081
>
>
> AFAIU, the file option does not necessarily make the process slower (it
just replaces a list of comma separated maps in a command by a file), it is
the -z flag that does that, because it will open and close files once and
again, instead of keeping them open (Am I right, @markusM?)
>
> Moreover, I checked r.hants and r.series.lwr, which are based on r.series
and have the same file option and -z flag. They all have different wording.
Maybe we could agree here on the 3 of them. From my understanding, they are
not entirely correct. Here the text snippets:
>
> r.hants:
> "Use the file option to analyze large amount of raster maps without
hitting open files limit and the size limit of command line arguments. The
computation is slower than with the input option method. For every single
row in the output map(s) all input maps are opened and closed. The amount
of RAM will rise linearly with the number of specified input maps. The
input and file options are mutually exclusive. The option input is a comma
separated list of raster map names and the option file is a text file with
a new line separated list of raster map names. Note that the order of maps
in one option or the other is very important."
>
> r.series.lwr:
> "Use the -z flag to analyze large amounts of raster maps without hitting
the open files limit and the size limit of command line arguments. This
will however increase the processing time. For every single row in the
output map(s) all input maps are opened and closed. The amount of RAM used
will rise linearly with the number of specified input maps.The input and
file options are mutually exclusive. Input is a text file with a new line
separated list of raster map names"
>
> I can change them accordingly once we understand correctly the effect of
both file option and -z flag.

The manual of r.series is IMHO now correct. The file option avoids hitting
the size limit of command line arguments, and the -z flag avoids hitting
the limit of the number of open files.

In all three manuals, the sections starting with "The maximum number of
raster maps" (in r.series "Number of raster maps to be processed") should
be identical. Ideally, the manual of r.hants or r.series.lwr would be used
as template, adding the latest change of r.series with regard to the file
option and the -z flag, then sync all manuals.

> Which other modules share this -z flag?

r.series.accumulate

t.rast.series and t.rast.aggregate do not have the -z flag, but should
probably have it because they call r.series.

Markus M

>
> best,
> Vero
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-dev/attachments/20180115/82dcff8c/attachment.html>