[gdal-dev] Large Raw Files

[ldjohn at u...]

On Wed, 12 Dec 2001, Frank Warmerdam wrote:

> Lowell and Others,
> 
> I have altered the RawRasterBand class so it includes an optional
> extra bIsVSIL parameter. If FALSE (the default), then the normal
> VSI API will be used for file IO, and files may only be up to 2GB. If
> TRUE then the large file API (ie. VSIFSeekL(), VSIFReadL() and VSIFWriteL())
> will be used instead of the regular one.
> 
> I have altered the PAux, and HKV (aka MFF2) formats to utilize this.
> 
> To utilize this for LDF please ensure that you use the large file API
> to open and access your file, and pass bIsVSIL=TRUE when in the constructor
> call to RawRasterDataset. Let me know if you have questions about how to
> use the large file API.
> 
> I don't have any convenient way to test this, so if you could do some testing
> with 5GB LDF and PAux files I would appreciate it.
> 
> The large file API changes can be extented to other formats like ESRI labelled
> BIL, Envisat, and SAR_CEOS if needed. Anyone who wants this, and is willing
> to test it let me know and I will make the changes.

I just tested with LDF format and it seems to work fine. Created a
3-band image with:

band 1: 40,000 lines x 60,000 samples, GDT_Byte
band 2: 250 x 250 , GDT_Int16
band 3: 500 x 500 , GDT_Int32

Image file (.img) size: 2,401,125,000

Metadata file (.ldf):

VERSION = 2.0
BANDS = 3
OBJECT = BAND1
DATA_LOCATION = 0
LINES = 40000
SAMPLES_PER_LINE = 60000
DATA_TYPE = UNSIGNED_INTEGER_8
PIXEL_SIZE = (1.000000, 1.000000)
END_OBJECT = BAND1
OBJECT = BAND2
DATA_LOCATION = -1894967296
LINES = 250
SAMPLES_PER_LINE = 250
DATA_TYPE = INTEGER_16
PIXEL_SIZE = (1.000000, 1.000000)
END_OBJECT = BAND2
OBJECT = BAND3
DATA_LOCATION = -1894842296
LINES = 500
SAMPLES_PER_LINE = 500
DATA_TYPE = INTEGER_32
PIXEL_SIZE = (1.000000, 1.000000)
END_OBJECT = BAND3
END

I see that I need to correct the type for the data location counter, but
everything else looks good. (This is a prototype, so the pixel size,
which should be scaled by the resolution difference, is known to be
wrong.)

One thing I've noticed is that the RawRasterBand->nImgOffset member is of
type unsigned int. That will take us up to 4GB, but what after
that? Should this be a long long?

-- 
--------------------------------------------------
Lowell D. Johnson
Software Engineering
Raytheon Systems
EROS Data Center
47914 252nd Street, Mundt Federal Bldg
Sioux Falls, SD 57198
--------------------------------------------------