[gdal-dev] Large Raw Files

ldjohn at u... ldjohn at u...
Fri Dec 14 12:18:11 EST 2001

On Wed, 12 Dec 2001, Frank Warmerdam wrote:

> ldjohn at u... wrote:
> > I just tested with LDF format and it seems to work fine. Created a
> > 3-band image with:
> > 
> > band 1: 40,000 lines x 60,000 samples, GDT_Byte
> > band 2: 250 x 250 , GDT_Int16
> > band 3: 500 x 500 , GDT_Int32
> > 
> > Image file (.img) size: 2,401,125,000
> Lowell,
> Excellent! It would be good to test over 4GB too. What platform was
> this testing on?

This testing was on IRIX.

I just finished running a test case similar to the above, except band 1 is
50,000 lines x 100,000 samples (~5GB). I ran this test on Solaris, using
the GCC compiler. I did have to make the following modifications to get
this to work:

o frmts/raw/rawdataset.cpp
< if( Seek( nImgOffset + iLine * nLineOffset, SEEK_SET ) == -1
> if( Seek( nImgOffset + (vsi_l_offset) iLine * nLineOffset,
SEEK_SET ) == -1 
< if( Seek( nImgOffset + nBlockYOff * nLineOffset, SEEK_SET ) == -1 )
> if( Seek( nImgOffset + (vsi_l_offset) nBlockYOff * nLineOffset,
> SEEK_SET ) == -1 )

It seems that the IRIX MIPSPro compiler used in the first test on IRIX
may handle the automatic datatype conversion differently than GCC. The
nBlockYOff * nLineOffset result overflowed without the typecast.

o port/cpl_vsil_unix_stdio_64.cpp
< return( fseek64( fp, nOffset, nWhence ) );
> return( fseeko64( fp, nOffset, nWhence ) );
< return( ftell64( fp ) );
> return( ftello64( fp ) );

The long file position offset functions fseek64 and ftell64 do not
exist on our Solaris 2.7 system. That may well be due to outdated
include files. That system is not a primary development system, but
acts as our Sun test platform. The "o" versions of the calls are what
we use in LAS for large file support.

I also had to override the configure test for large file support using
"--with-unix-stdio-64" since the test looks for fseek64 in stdio.h.

So I'm not sure whether this impacts other users or if it something
that we need to deal with locally.

> > One thing I've noticed is that the RawRasterBand->nImgOffset member is of
> > type unsigned int. That will take us up to 4GB, but what after
> > that? Should this be a long long?
> The type of nImgOffset in the constructor calls, and within the class for
> RawRasterBand (as found in rawdataset.h:1.7) is vsi_l_offset which should
> be a 64bit type on platforms where a 64bit type is available. Note that
> the default configuration on Linux does not (as far as I know) support large
> file IO. It has been successfully tested on Solaris, NT and IRIX. Some
> linux versions can also support large file IO, but I doubt the configure
> logic finds it.

Ah, yes. My primary development is being conducted on a Linux 2.2 box, so
it is being set to unsigned int.

Talk to you later.

Lowell D. Johnson
Software Engineering
Raytheon Systems
EROS Data Center
47914 252nd Street, Mundt Federal Bldg
Sioux Falls, SD 57198

More information about the Gdal-dev mailing list