[Gdal-dev] Format limitations in 64-bit architecture

[Frank Warmerdam]

Simon Perkins wrote:
> Ozy Sjahputera wrote:
>> From the list of file formats currently supported by GDAL, GeoTIFF is 
>> listed to have a 4 GB size limit.  Is this limitation due to the 
>> assumed 32-bit machine?  Or, will the size limit be bigger if we use a 
>> 64-bit machine and OS?
> 
> AFAIK, that's a fundamental limitation of the GeoTIFF file format. Does 
> anyone know if there are any plans to fix this in the future?

Ozy / Simon,

Traditional TIFF is limited to 4GB in size by use of 32bit file offsets.
There is a BigTIFF project underway now and the result should be deployed
in GDAL this summer.

   http://libtiff.maptools.org/BigTIFFProposal.html

>> What about other formats such as NITF and MEM?  Because the data type 
>> size_t in a 32-bit system is defined as a 4-byte type, whereas in a 
>> 64-bit system it is defined as an 8-byte.   Does this mean the MEM 
>> format is not limited to 2 GB anymore once we  work in the 64-bit system?
> 
> That would seem like a reasonable assumption though I don't know for 
> sure. I would think that NITF should be able to handle larger than 4GB 
> files even on 32-bit OSes, on any modern filesystem.

Skimming the MEM driver code, it does not appear to be careful about having
a very large backing image.  That is, offsets into it's memory image are
computed using 32bit offsets.  If you want to have a very large MEM dataset
some improvements to the MEM driver will be required.

I believe that NITF uses ascii encoded offsets for image chunks, and that
these are wide enough to support files substantially larger than 4GB though
I haven't reviewed the limits.  GDAL does use the "large file API" for access
and I have had reports of people working with large NITF files.  But I don't
have much direct experience so there may be issues.

Hmm, on review I see the NITFCollectSegmentInfo() keeps track of the
offsets in a 32bit integer.  So now I am not at all confident that large
files would necessarily work.  It would depend on the particulars of the file.
This is really something that should be fixed in the driver, but I'd be more
comfortable fixing it with a large example file to work with.

Best regards,
-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | President OSGeo, http://osgeo.org