[GRASS5] Raster lib and CELL files > 2GB

[Otto Dassau]

On Tue, 13 Jul 2004 11:50:08 +0100
Glynn Clements <glynn.clements at virgin.net> wrote:

> 
> Markus Neteler wrote:
> 
> > thanks for your comments below! I forgot to submit additional info:
> > 
> > original ERDAS files:
> > -rwxr-xr-x  1 dassau users 11184610066 2004-07-09 16:25 040706_rgb_juist.ige*
> > -rwxr-xr-x  1 dassau users       18261 2004-07-09 16:25 040706_rgb_juist.img*
> > -rwxr-xr-x  1 dassau users   942589967 2004-07-09 16:26 040706_rgb_juist.rrd*
> > 
> > GRASS (certainly useless) and GDAL recompiled with LFS support:
> > CFLAGS="-D_FILE_OFFSET_BITS=64 -D_LARGE_FILES" ./configure ...
> > 
> > r.in.gdal then imports the ERDAS files without problem:
> > grassdata/nnw/juist/cell> l juist_dmcnc.1
> > -rw-r--r--   1 dassau users 2317620271 2004-07-09 17:06 juist_dmcnc.1
> > -> this file is larger than 2GB, probably a correct file.
> > 
> > But then d.rast and other reading commands crash:
> > row 18793
> > ERROR: error reading compressed map [juist_dmcnc.3] in mapset [juist],
> > 
> > -> does this need the 'long' changes as you suggest?
> 
> Yes. For files larger than 2Gb, file offsets won't fit into 32 bits
> (i.e. a "long" on x86). You have to use off_t instead. Using
> -D_FILE_OFFSET_BITS=64 makes off_t a 64 bit type ("long long"), but
> that only helps if you're using off_t, and the raster I/O code in
> libgis uses "long".
> 
> Also, note that the raster files begin with an array of offsets to the
> start of each row. These would have to be changed to use off_t
> (format.c, G.h, maybe other places).
> 
> One consequence of this is that maps which were created by a version
> of GRASS which used 64-bit offsets wouldn't be readable by a version
> of GRASS which used 32-bit offsets, and vice-versa. This issue already
> exists for systems with a 64-bit "long" type, but supporting 64-bit
> offsets on x86 would make the situation a lot more common.

Hi Glynn,

Thank you very much for your work! I just tested some maps and AFAICS it work.  

dassau at berlin:~> l grassdata/nnw/juist/cell/
-rw-r--r--   1 dassau users 2317620271 2004-07-09 17:06 juist_dmcnc.1
-rw-r--r--   1 dassau users 2277930187 2004-07-09 17:28 juist_dmcnc.2


    Otto