[gdal-dev] Large Raw Files

[ldjohn at u...]

On Wed, 12 Dec 2001, Frank Warmerdam wrote:

> ldjohn at u... wrote:
> 
> > I just tested with LDF format and it seems to work fine. Created a
> > 3-band image with:
> > 
> > band 1: 40,000 lines x 60,000 samples, GDT_Byte
> > band 2: 250 x 250 , GDT_Int16
> > band 3: 500 x 500 , GDT_Int32
> > 
> > Image file (.img) size: 2,401,125,000
> 
> 
> Lowell,
> 
> Excellent! It would be good to test over 4GB too. What platform was
> this testing on?

This testing was on IRIX.

I just finished running a test case similar to the above, except band 1 is
50,000 lines x 100,000 samples (~5GB). I ran this test on Solaris, using
the GCC compiler. I did have to make the following modifications to get
this to work:

o frmts/raw/rawdataset.cpp
184c184
< if( Seek( nImgOffset + iLine * nLineOffset, SEEK_SET ) == -1
---
> if( Seek( nImgOffset + (vsi_l_offset) iLine * nLineOffset,
SEEK_SET ) == -1 
292c292,293
< if( Seek( nImgOffset + nBlockYOff * nLineOffset, SEEK_SET ) == -1 )
---
> if( Seek( nImgOffset + (vsi_l_offset) nBlockYOff * nLineOffset,
> SEEK_SET ) == -1 )

It seems that the IRIX MIPSPro compiler used in the first test on IRIX
may handle the automatic datatype conversion differently than GCC. The
nBlockYOff * nLineOffset result overflowed without the typecast.

o port/cpl_vsil_unix_stdio_64.cpp
76c76
< return( fseek64( fp, nOffset, nWhence ) );
---
> return( fseeko64( fp, nOffset, nWhence ) );
86c86
< return( ftell64( fp ) );
---
> return( ftello64( fp ) );

The long file position offset functions fseek64 and ftell64 do not
exist on our Solaris 2.7 system. That may well be due to outdated
include files. That system is not a primary development system, but
acts as our Sun test platform. The "o" versions of the calls are what
we use in LAS for large file support.

I also had to override the configure test for large file support using
"--with-unix-stdio-64" since the test looks for fseek64 in stdio.h.

So I'm not sure whether this impacts other users or if it something
that we need to deal with locally.

> > One thing I've noticed is that the RawRasterBand->nImgOffset member is of
> > type unsigned int. That will take us up to 4GB, but what after
> > that? Should this be a long long?
> 
> 
> The type of nImgOffset in the constructor calls, and within the class for
> RawRasterBand (as found in rawdataset.h:1.7) is vsi_l_offset which should
> be a 64bit type on platforms where a 64bit type is available. Note that
> the default configuration on Linux does not (as far as I know) support large
> file IO. It has been successfully tested on Solaris, NT and IRIX. Some
> linux versions can also support large file IO, but I doubt the configure
> logic finds it.

Ah, yes. My primary development is being conducted on a Linux 2.2 box, so
it is being set to unsigned int.

Talk to you later.

-- 
--------------------------------------------------
Lowell D. Johnson
Software Engineering
Raytheon Systems
EROS Data Center
47914 252nd Street, Mundt Federal Bldg
Sioux Falls, SD 57198
--------------------------------------------------