[GRASS5] r.series: WARNING: Too many open raster files

[Glynn Clements]

Markus Neteler wrote:

> At time I am processing time series of MODIS data,
> unfortunately I get:
> 
> wc -l /tmp/lstlist2003
>     362 /tmp/lstlist2003
> r.series  in=`cat /tmp/lstlist2003 | tr '\n' ','` \
>    out=modis.av method=average
> WARNING: Too many open raster files
> 
> I was expecting something similar, is there any risk
> to increase this number?
> grass57/lib/gis/G.h
> G.h:#define MAXFILES    256  

Only memory consumption.

MAXFILES determines the number of "struct fileinfo"s, each of which is
552 bytes (on x86), so increasing it to 1024 would require 552Kb. I
guess that's unlikely to be critical, even on a PDA.

OTOH, the fileinfo structure includes pointers to dynamically
allocated memory:

	Data buffer:		1-8 bytes (one value) per map column
	Row pointers:		4/8 bytes (one long) per row, but only for 
				compressed maps
	Column mapping:		4 bytes per window column
	Null work buffer:	1 bit per map column
	Null buffers:		8 * 1 bit per window column
	Name, Mapset:		strings

Also, some of the fields in the fileinfo structure are themselves
structure which include pointers to dynamically allocated memory:

        struct Reclass reclass	4 bytes per category
	struct Cell_stats	16 bytes per tree node	
        struct Quant quant	?

The reclass table presumably isn't an issue here; I don't know about
the other two.

The G__ variable isn't explicitly initialised, so it will be stored in
the BSS segment, and so increasing MAXFILES won't increase the size of
executables.

The OS will also impose its own limit upon the maximum number of open
files (1024 for Linux 2.4.x on x86).

> The maps are quite small:
> rows:       908
> cols:       1040
> 
> Data Type:    DCELL = xx bytes
> 
> Is memory an issue?
> 908 * 1040 * xx * 362 = 341843840 * xx 

r.series processes one row at a time. For each map, there's a row
buffer (8 bytes per window column), plus whatever is used (directly or
indirectly) by the fileinfo structure, as documented above.

> The machine has 2GB RAM and enough swap space.
> Other side effects to be expected?

I don't see any problems.

-- 
Glynn Clements <glynn.clements at virgin.net>