[GRASS-user] r.patch: number of rasters

[Glynn Clements]

Markus Neteler wrote:

> Here an increase to 1500 files (use * instead of neteler to aply
> it to all users):
> 
> #<domain>      <type>  <item>         <value>
> neteler               hard    nofile          1500
> 
> But myself, I got troubles when having registered the maps with r.external:
> 
> GRASS 6.4.1svn (patUTM32):~ > r.patch in=`g.mlist type=rast
> pattern=dtm00* sep=comma mapset=dtm_1m_external` out=dtm_trentino_2m
> ERROR 1: TIFFOpen:/geodata/base_cartography/pat_DTM_LIDAR_2007_UTM_WGS84/geotiffs/dtm000501.tif:
> Too many open files

This message matches strerror(EMFILE):

	#define	EMFILE		24	/* Too many open files */

It's possible that some library (i.e. GDAL or TIFF) has its own limit,
and returns EMFILE if the limit is reached. But I'd start by assuming
that you're hitting an OS limit.

> WARNING: Unable to open
>        '/grassdata/patUTM32/dtm_1m_external/cell_misc/dtm000501/f_format'
> WARNING: quantization file [dtm000501] in mapset [dtm_1m_external] missing
> WARNING: Unable to open raster map <dtm000502 at dtm_1m_external>
> ERROR: One or more input raster maps not found
> 
> So far I could not find out the reason. Here I am using 500 files registered
> with r.external.

I assume that it's holding multiple open descriptors per map.

Note that 7.0 will typically have at least 2 open descriptors per map:
one for the raster data, the other for the null data. But not for
r.external "linked" files, which don't have a separate null file.

6.x doesn't keep the null file open, which reduces the number of open
descriptors at the expense of performance (open() can be fairly
expensive).

Run r.patch under gdb, set a breakpoint on G_fatal_error(). When the
breakpoint is hit, "ls -l /proc/<pid>/fd" (where <pid> is the PID of
the r.patch process) to see *which* files it has open. Or run r.patch
under strace, and manually examine the output to see which files are
open()ed and which are close()d.

-- 
Glynn Clements <glynn at gclements.plus.com>