[GRASSLIST:8008] Re: error reading large rasters in 5.4/6.0

[Andrew Danner]

After looking into the problem of reading very large rasters, I believe
I have isolated the bug. The problem only happens when reading row 0 and
triggers the following block in flate.c:186

 else if (b[0] != G_ZLIB_COMPRESSED_YES)
    {
        /* We're not at the start of a row */
        G_free (b);
        return -1;
    }

I think the problem is a bug in writing rasters, not reading them. When
creating a new raster, it writes a vector of row offsets to the fcell
file in the function G__write_row_ptrs. It attempts to guess the size of
the file offsets. 

 int nbytes = sizeof(off_t);
...
 if (nbytes > 4 && fcb->row_ptr[nrows] <= 0xffffffff)
    nbytes = 4;
...
 len = (nrows + 1) * nbytes + 1;
 b = buf = G_malloc(len);

The problem is that when a new raster is created fcb->row_ptr[nrows]=0,
so nbytes is always 4. Later when writing the rows and actually
computing the offsets, if fcb->row_ptr[nrows] > 0xffffffff, it bumps
nbytes upto 8, but row 0 was written to the file assuming an offset of 4
and row 0 becomes corrupted if the raster is larger than 4GB. 

Commenting out the "if" block seems to fix the problem. 

Should I submit a bug report?

-Andy

On Thu, 2005-08-18 at 18:01 -0400, Andrew Danner wrote:
> I'm having problems working with some large raster files in GRASS, even
> though I have enabled large file support. 
> 
> I built GRASS from source

> I get the following problems:
> 
> 1) "r.info test" displays the wrong number of total cells. e.g., 
> 
> >  |   Type of Map:  cell 
> >  |   Data Type:    DCELL         
> >  |   Rows:         50590     
> >  |   Columns:      71840 
> >  |   Total Cells:  -660581696         
> >  |        Projection: Lambert Conformal Conic (zone 0)      
> >  |            N:     967200    S:     461300   Res:    10   
> >  |            E:    2638600    W:    1920200   Res:    10
>  
> 2) "d.rast test" complains. 
> WARNING: error reading compressed map [test] in mapset [local], row 0

> Even though the image tends to display fine and look correct, the
> problem causes some programs to quit, e.g. r.mapcalc. 
> 
> r.mapcalc test2=test 
> ERROR: error reading compressed map [Neuse10] in mapset [local], row 0
> 
> The problem seems to be on rasters whose fcell file is bigger than 4GB. 
>