[GRASS5] Raster lib and CELL files > 2GB

[Markus Neteler]

Glynn,

thanks for your comments below! I forgot to submit additional info:

original ERDAS files:
-rwxr-xr-x  1 dassau users 11184610066 2004-07-09 16:25 040706_rgb_juist.ige*
-rwxr-xr-x  1 dassau users       18261 2004-07-09 16:25 040706_rgb_juist.img*
-rwxr-xr-x  1 dassau users   942589967 2004-07-09 16:26 040706_rgb_juist.rrd*

GRASS (certainly useless) and GDAL recompiled with LFS support:
CFLAGS="-D_FILE_OFFSET_BITS=64 -D_LARGE_FILES" ./configure ...

r.in.gdal then imports the ERDAS files without problem:
grassdata/nnw/juist/cell> l juist_dmcnc.1
-rw-r--r--   1 dassau users 2317620271 2004-07-09 17:06 juist_dmcnc.1
-> this file is larger than 2GB, probably a correct file.

But then d.rast and other reading commands crash:
row 18793
ERROR: error reading compressed map [juist_dmcnc.3] in mapset [juist],

-> does this need the 'long' changes as you suggest?

[ Another more simple option might be to split the file
  with GDAL tools and then to import the pieces ... ]

Thanks

 Markus



On Mon, Jul 12, 2004 at 10:05:28PM +0100, Glynn Clements wrote:
> 
> Markus Neteler wrote:
> 
> > for a remote sensing project we face the problem to arrive
> > at file sizes > 2GB. I see two options:
> > 
> > a) change the CELL file compression from RLE to DEFLATE 
> >   -> how to do that? I just want to change locally for me
> 
> That's far from simple. The existing low-level raster I/O
> implementation is a total mess; learning how it works may well take
> longer than re-writing it from scratch.
> 
> Compare read_data_fp_compressed() with read_data_compressed() in
> get_row.c, and put_data() with put_fp_data() in put_row.c.
> 
> > b) enable large file support (http://www.suse.de/~aj/linux_lfs.html)
> >   -> any functions which need a change?
> 
> Search for all occurrences of "long" in libgis, then figure out which
> ones need to be changed to off_t. AFAICT, most of them are likely to
> be in the following files:
> 
> 	G.h (row_ptr field)
> 	closecell.c
> 	format.c
> 	get_row.c
> 	opencell.c
> 	put_row.c
> 
> > Variant a) sounds somewhat better to me and maybe already solves
> > the problem.
> > 
> > Recommendations are welcome.
> 
> c) obtain a system where "long" is 64 bits.
> 
> -- 
> Glynn Clements <glynn.clements at virgin.net>

-- 
Markus Neteler     <neteler itc it>       http://mpa.itc.it
ITC-irst -  Centro per la Ricerca Scientifica e Tecnologica
MPBA - Predictive Models for Biol. & Environ. Data Analysis
Via Sommarive, 18        -       38050 Povo (Trento), Italy