[GRASS-dev] how to work around "Arument list too long" error in a GRASS python script ?

[Glynn Clements]

Moritz Lennert wrote:

> >> In a python script I have the following call:
> >>
> >> grass.run_command('r.series',
> >>                      input = rate_maps,
> >>                      output = sum_rates,
> >>                      method = 'sum',
> >>                      overwrite = True,
> >>                      quiet=True)
> >>
> >> rate_maps is a list which in one instance contains 8559 map names,
> >> leading to an "OSError: [Errno 7] Argument list too long".
> >>
> >> I know that in the shell I could use xargs to work around such a
> >> problem. But how to do this in python ?
> >
> > What it the OS limit for it?
> 
> I suppose this is ARG_MAX ?
> 
> getconf ARG_MAX
> 2097152
> 
> A text file with all file names only uses 144551 bytes.
> 
> Or is there another limit I should look at ?

Apparently there's a limit on the maximum length of a single argument,
which appears to be fixed at 128 KiB:

  > subprocess.call(["echo", "x"*131071],stdout=devnull,stderr=devnull)
  0
  > subprocess.call(["echo", "x"*131072],stdout=devnull,stderr=devnull)
  Traceback (most recent call last):
 [snipped]
  OSError: [Errno 7] Argument list too long

This specific issue could be worked around by having
grass.make_command() use multiple arguments when the value is a list,
e.g. "r.series input=map1 input=map2 ...".

But such cases are likely to be problematic on other platforms where
ARG_MAX may be much lower. POSIX only requires that it is at least
4096, and it wasn't so long ago that such a value was commonplace. And
that value includes the environment as well as the command line.

Modules which commonly use many input maps should have the option to
read map names from a file. This mostly affects r.series, which
already does so.

If this issue was more common, we could consider extending G_parser()
to allow transparently reading any option value from a file, using
e.g. input=@filename. But that may just cause similar issues with
--interface-description, G_recreate_command(), etc.

-- 
Glynn Clements <glynn at gclements.plus.com>