[GRASS5] Raster lib and CELL files > 2GB

[Glynn Clements]

Markus Neteler wrote:

> thanks for your comments below! I forgot to submit additional info:
> 
> original ERDAS files:
> -rwxr-xr-x  1 dassau users 11184610066 2004-07-09 16:25 040706_rgb_juist.ige*
> -rwxr-xr-x  1 dassau users       18261 2004-07-09 16:25 040706_rgb_juist.img*
> -rwxr-xr-x  1 dassau users   942589967 2004-07-09 16:26 040706_rgb_juist.rrd*
> 
> GRASS (certainly useless) and GDAL recompiled with LFS support:
> CFLAGS="-D_FILE_OFFSET_BITS=64 -D_LARGE_FILES" ./configure ...
> 
> r.in.gdal then imports the ERDAS files without problem:
> grassdata/nnw/juist/cell> l juist_dmcnc.1
> -rw-r--r--   1 dassau users 2317620271 2004-07-09 17:06 juist_dmcnc.1
> -> this file is larger than 2GB, probably a correct file.
> 
> But then d.rast and other reading commands crash:
> row 18793
> ERROR: error reading compressed map [juist_dmcnc.3] in mapset [juist],
> 
> -> does this need the 'long' changes as you suggest?

Yes. For files larger than 2Gb, file offsets won't fit into 32 bits
(i.e. a "long" on x86). You have to use off_t instead. Using
-D_FILE_OFFSET_BITS=64 makes off_t a 64 bit type ("long long"), but
that only helps if you're using off_t, and the raster I/O code in
libgis uses "long".

Also, note that the raster files begin with an array of offsets to the
start of each row. These would have to be changed to use off_t
(format.c, G.h, maybe other places).

One consequence of this is that maps which were created by a version
of GRASS which used 64-bit offsets wouldn't be readable by a version
of GRASS which used 32-bit offsets, and vice-versa. This issue already
exists for systems with a 64-bit "long" type, but supporting 64-bit
offsets on x86 would make the situation a lot more common.

-- 
Glynn Clements <glynn.clements at virgin.net>