[GRASS-dev] r.series map names buffer overflow

[Markus Neteler]

On Fri, Nov 14, 2008 at 1:25 PM, Glynn Clements
<glynn at gclements.plus.com> wrote:
>
> Markus Neteler wrote:
>
>> we try to calculate the average of 1460 maps (4 observations per day) from MODIS
>> but r.series seems to overflow somewhere:
>>
>> YEAR=2002
>> LIST=`g.mlist type=rast pat="*_lst1km${YEAR}*"
>> mapset="modisLSTinterpolationPAT" sep=","`
>> r.series -n $LIST out=lst_${YEAR}_avg method=average --o
>> ...
>> Reading raster map <terra_lst1km20021215.LST_Night_1km.rst>...
>> Reading raster map <terra_lst1km20021216.LST_Day_1km.rst>...
>> Reading raster map <terra_lst1km20021216.LST_Night_1km.rst>...
>> Reading raster map <terra_lst1km20021217.LST_Day_1km.rst>...
>> Reading raster map <terra_lst1km20021217.LST_Night_1km.rst>...
>> Reading raster map <terra_lst1km20021218.LST_Day_1km.rst>...
>> Reading raster map <terra_lst1km20021218.LST_Night_1km.rst>...
>> Reading raster map <terra_lst1km20021219.LST_Day_1km.rst>...
>> Reading raster map <terra_lst1km20021219.LST_Night_1km.rst>...
>> Reading raster map <terra_lst1km20021220.LST_Day_1km.rst>...
>> WARNING: Unable to open
>>          '/hardmnt/eden0/castellani/grassdata/patGB1/modisLSTinterpolationPAT/cell_misc/terra_lst1km20021220.LST_Day_1km.rst/f_format'
>> WARNING: quantization file [terra_lst1km20021220.LST_Day_1km.rst] in mapset
>>          [modisLSTinterpolationPAT] missing
>> Reading raster map <terra_lst1km20021220.LST_Night_1km.rst>...
>> WARNING: Unable to open raster map
>>          <terra_lst1km20021220.LST_Night_1km.rst at modisLSTinterpolationPAT>
>> ERROR: Unable to open raster map <terra_lst1km20021220.LST_Night_1km.rst>
>>        in mapset <modisLSTinterpolationPAT>
>>
>> The map is there and ok.
>>
>> I am unsure how to track this down (maybe there is a fixed buffer in libgis?).
>
> What does "ulimit -n" say? That's the OS-imposed limit on the number
> of open file descriptors per process.

Bingo:
ulimit -n
1024


> On my system, both the hard and soft limits are 1024. The soft limit
> can be changed with e.g. "ulimit -n 1500", but only up to the hard
> limit. The hard limit can only be changed by root.

I forgot about this limitation.
This is somewhat dangerous, say, could it be trapped? If r.series
gets more input files than ulimit -n (C equivalent) allows, could
it spit out an error (the manual suggesting than to split into smaller
jobs)?

> Limits are typically set at login by the pam_limits.so module, which
> is configured via /etc/security/limits.conf. Limits are inherited by
> child processes; any changes to limits.conf won't take effect until
> you to log in again.
>
> If you have root access via sudo, you can spawn a root shell, increase
> the hard limit, then spawn a child shell under your normal account
> which will inherit the increased limit. E.g.:
>
>        $ sudo bash
>        # ulimit -Hn 1500
>        # sudo -u markus bash
>        $ ulimit -n 1500
>        $

Thanks for your detailed explanations. In this case, this would do the
trick.

Markus