[GRASS5] r.series, max files open

[Glynn Clements]

H Bowman wrote:

> > We could just increase the MAXFILES value. However, each slot uses 552
> > bytes on x86, so memory consumption could be an issue (bearing in mind
> > that it affects every process which uses libgis). Also, there's no
> > point increasing it beyond the OS limit (so 8760 files may not be
> > possible, even if you can afford an extra 4.6Mb per process).
> 
> Just curious if anyone knows what the operating system open file limits
> are for Linux/Irix/Solaris/MacOSX/Win98/WinNT ?

IIRC, for Linux, it was 256 in 2.0, 1024 in 2.2 and run-time
configurable in 2.4.

However, there is also the issue that select() is limited to the
number of bits in an fd_set (1024 in GNU libc 2.1). Although select()
isn't used on files, if a program uses up the first 1024 descriptors
for files, it won't be able to use select() on any other descriptors
(e.g. sockets, the terminal, devices).

In general, it's best to avoid solutions which rely upon having vast
numbers of files open simultaneously. OTOH, it's also best to avoid
solutions which rely upon storing vast numbers of files in memory
simultaneously.

For processing a time series of 8760 files, the most viable solutions
would be:

a) Open/close each map for each row read, rather than keeping them
open throughout.

b) Generate an intermediate file in BIL order (map0/row0, ...,
mapN/row0, map1/row1, ...) using random access on the result file,
then process the intermediate file.

The second approach would be faster (it avoids opening and closing
each map once per row), but would require a significant amount of disk
space (but it's better than requiring the same amount of virtual
memory).

-- 
Glynn Clements <glynn.clements at virgin.net>