[GRASS-dev] how to work around "Arument list too long" error in a GRASS python script ?

[Moritz Lennert]

On 15/06/15 13:41, Glynn Clements wrote:
>
> Moritz Lennert wrote:
>
>>>> In a python script I have the following call:
>>>>
>>>> grass.run_command('r.series',
>>>>                       input = rate_maps,
>>>>                       output = sum_rates,
>>>>                       method = 'sum',
>>>>                       overwrite = True,
>>>>                       quiet=True)
>>>>
>>>> rate_maps is a list which in one instance contains 8559 map names,
>>>> leading to an "OSError: [Errno 7] Argument list too long".
>>>>
>>>> I know that in the shell I could use xargs to work around such a
>>>> problem. But how to do this in python ?
>>>
>>> What it the OS limit for it?
>>
>> I suppose this is ARG_MAX ?
>>
>> getconf ARG_MAX
>> 2097152
>>
>> A text file with all file names only uses 144551 bytes.
>>
>> Or is there another limit I should look at ?
>
> Apparently there's a limit on the maximum length of a single argument,
> which appears to be fixed at 128 KiB:
>
>    > subprocess.call(["echo", "x"*131071],stdout=devnull,stderr=devnull)
>    0
>    > subprocess.call(["echo", "x"*131072],stdout=devnull,stderr=devnull)
>    Traceback (most recent call last):
>   [snipped]
>    OSError: [Errno 7] Argument list too long
>
> This specific issue could be worked around by having
> grass.make_command() use multiple arguments when the value is a list,
> e.g. "r.series input=map1 input=map2 ...".
>
> But such cases are likely to be problematic on other platforms where
> ARG_MAX may be much lower. POSIX only requires that it is at least
> 4096, and it wasn't so long ago that such a value was commonplace. And
> that value includes the environment as well as the command line.
>
> Modules which commonly use many input maps should have the option to
> read map names from a file. This mostly affects r.series, which
> already does so.
>
> If this issue was more common, we could consider extending G_parser()
> to allow transparently reading any option value from a file, using
> e.g. input=@filename. But that may just cause similar issues with
> --interface-description, G_recreate_command(), etc.
>

I think that the file option solution is the most adequate. r./v.patch 
might be potential candidates for that as well.

For the other modules that have such input lists, I doubt that anyone 
will ever use these with such long lists of maps.

Moritz