[GRASS5] [bug #2599] (grass) r.patch file size limit exceeded

[Glynn Clements]

Request Tracker wrote:

> this bug's URL: http://intevation.de/rt/webrt?serial_num=2599
> -------------------------------------------------------------------------
> 
> Subject: r.patch file size limit exceeded
> 
> Platform: Solaris2.X/Sparc
> grass obtained from: CVS
> grass binary for platform: Compiled from Sources
> GRASS Version: 5.3-cvs
> 
> I am attempting to run r.patch to create a LARGE (57330 rows x 47340
> cols) raster map and at row 45622 it craps out and says
> 
> WARNING: map [Orthophoto] - unable to write row 45622
> File size limit exceeded
> 
> I tried this on a Sparc running solaris and on an Intel Xeon running
> linux, and it craps out on the same line, so I don't think it's
> hardware...
> 
> Is this a soft limit that can be reset in code, or is this deeper?

Raster files are limited by the number of bits in the platform's off_t
type. However, if you are using a version from prior to mid-August,
it's also limited by the number of bits in the platform's int type
(older versions stored offsets using long, but some offset
calculations were performed using int).

On platforms with a 64-bit long type, then the current 5.3 CVS version
should support raster files larger than 2Gb.

On platforms where long is only 32 bits (e.g. x86), you will need to
compile GRASS (or, at least, certain parts of libgis) with
"-D_FILE_OFFSET_BITS=64", so that off_t is defined as "long long int". 
Again, this requires a version more recent than mid-August.

However, ideally only the following files (in src/libes/gis) would be
compiled with that switch:

	opencell.c
	closecell.c
	get_row.c
	put_row.c
	format.c

Those are the files which are involved in reading and writing raster
data; they have all been updated to use off_t for file offsets, and
will work correctly where an off_t is larger than an int or long.

Much of the rest of GRASS uses long for file offsets, and in some
cases performs offset calculations using ints. Such code will
typically fail in unpredictable ways if it tries to handle files
larger than 2Gb.

Developer notes:

Making GRASS reliably handle files larger than 2Gb would involve
analysing all uses of the lseek, fseek and ftell functions, which
amounts to 164 files (56 use lseek, 95 use fseek, and 45 use ftell).

The lseek case is relatively straightforward; offsets must be stored
and calculated as off_t rather than int or long (note: if x and y are
ints then you have to use "(off_t) x * y" and not "(off_t) (x * y)",
as the latter won't promote to off_t until after the value has been
truncated to fit an int).

The use of fseek/ftell is harder, as these are ANSI C (rather than
POSIX) functions, and explicitly use long for offsets. Some platforms
(e.g. GNU libc 2.x) provide fseeko/ftello functions, which use off_t
for offsets instead of long. However, some platforms don't have these
functions, so they can't just be changed unconditionally.

However, a significant number of such files only use fseek to rewind
to the beginning of the file, i.e. fseek(fp, 0, SEEK_SET). Such usage
doesn't need to be changed (although it could be changed to just use
rewind(fp) instead).

-- 
Glynn Clements <glynn.clements at virgin.net>