[GRASSLIST:8009] Re: error reading large rasters in 5.4/6.0

[Gordon.Keith at csiro.au]

While you are looking at that code, could you please keep an eye out for another bug that I have encountered, but haven't yet had a chance to look for, which may be related.

Specifically r.in.bin and r.in.gdal create rasters containing only NaNs for many of my data samples. It is very likely that these data samples do have only NaNs in the first row.

If you happen to find a fix for this problem it will save me a lot of work at some stage in the future (when I think I have enough time to learn how rasters are written from scratch).

Regards
Gordon



-----Original Message-----
From:	owner-GRASSLIST at baylor.edu on behalf of Andrew Danner
Sent:	Mon 8/22/2005 11:50 AM
To:	GrassList
Cc:	
Subject:	[GRASSLIST:8008] Re: error reading large rasters in 5.4/6.0
After looking into the problem of reading very large rasters, I believe
I have isolated the bug. The problem only happens when reading row 0 and
triggers the following block in flate.c:186

 else if (b[0] != G_ZLIB_COMPRESSED_YES)
    {
        /* We're not at the start of a row */
        G_free (b);
        return -1;
    }

I think the problem is a bug in writing rasters, not reading them. When
creating a new raster, it writes a vector of row offsets to the fcell
file in the function G__write_row_ptrs. It attempts to guess the size of
the file offsets. 

 int nbytes = sizeof(off_t);
...
 if (nbytes > 4 && fcb->row_ptr[nrows] <= 0xffffffff)
    nbytes = 4;
...
 len = (nrows + 1) * nbytes + 1;
 b = buf = G_malloc(len);

The problem is that when a new raster is created fcb->row_ptr[nrows]=0,
so nbytes is always 4. Later when writing the rows and actually
computing the offsets, if fcb->row_ptr[nrows] > 0xffffffff, it bumps
nbytes upto 8, but row 0 was written to the file assuming an offset of 4
and row 0 becomes corrupted if the raster is larger than 4GB. 

Commenting out the "if" block seems to fix the problem. 

Should I submit a bug report?

-Andy

On Thu, 2005-08-18 at 18:01 -0400, Andrew Danner wrote:
> I'm having problems working with some large raster files in GRASS, even
> though I have enabled large file support. 
> 
> I built GRASS from source

> I get the following problems:
> 
> 1) "r.info test" displays the wrong number of total cells. e.g., 
> 
> >  |   Type of Map:  cell 
> >  |   Data Type:    DCELL         
> >  |   Rows:         50590     
> >  |   Columns:      71840 
> >  |   Total Cells:  -660581696         
> >  |        Projection: Lambert Conformal Conic (zone 0)      
> >  |            N:     967200    S:     461300   Res:    10   
> >  |            E:    2638600    W:    1920200   Res:    10
>  
> 2) "d.rast test" complains. 
> WARNING: error reading compressed map [test] in mapset [local], row 0

> Even though the image tends to display fine and look correct, the
> problem causes some programs to quit, e.g. r.mapcalc. 
> 
> r.mapcalc test2=test 
> ERROR: error reading compressed map [Neuse10] in mapset [local], row 0
> 
> The problem seems to be on rasters whose fcell file is bigger than 4GB. 
>