[GRASSLIST:8008] Re: error reading large rasters in 5.4/6.0
adanner at cs.duke.edu
Sun Aug 21 21:50:06 EDT 2005
After looking into the problem of reading very large rasters, I believe
I have isolated the bug. The problem only happens when reading row 0 and
triggers the following block in flate.c:186
else if (b != G_ZLIB_COMPRESSED_YES)
/* We're not at the start of a row */
I think the problem is a bug in writing rasters, not reading them. When
creating a new raster, it writes a vector of row offsets to the fcell
file in the function G__write_row_ptrs. It attempts to guess the size of
the file offsets.
int nbytes = sizeof(off_t);
if (nbytes > 4 && fcb->row_ptr[nrows] <= 0xffffffff)
nbytes = 4;
len = (nrows + 1) * nbytes + 1;
b = buf = G_malloc(len);
The problem is that when a new raster is created fcb->row_ptr[nrows]=0,
so nbytes is always 4. Later when writing the rows and actually
computing the offsets, if fcb->row_ptr[nrows] > 0xffffffff, it bumps
nbytes upto 8, but row 0 was written to the file assuming an offset of 4
and row 0 becomes corrupted if the raster is larger than 4GB.
Commenting out the "if" block seems to fix the problem.
Should I submit a bug report?
On Thu, 2005-08-18 at 18:01 -0400, Andrew Danner wrote:
> I'm having problems working with some large raster files in GRASS, even
> though I have enabled large file support.
> I built GRASS from source
> I get the following problems:
> 1) "r.info test" displays the wrong number of total cells. e.g.,
> > | Type of Map: cell
> > | Data Type: DCELL
> > | Rows: 50590
> > | Columns: 71840
> > | Total Cells: -660581696
> > | Projection: Lambert Conformal Conic (zone 0)
> > | N: 967200 S: 461300 Res: 10
> > | E: 2638600 W: 1920200 Res: 10
> 2) "d.rast test" complains.
> WARNING: error reading compressed map [test] in mapset [local], row 0
> Even though the image tends to display fine and look correct, the
> problem causes some programs to quit, e.g. r.mapcalc.
> r.mapcalc test2=test
> ERROR: error reading compressed map [Neuse10] in mapset [local], row 0
> The problem seems to be on rasters whose fcell file is bigger than 4GB.
More information about the grass-user