[Gdal-dev] Format limitations in 64-bit architecture

[Mateusz Loskot]

Frank Warmerdam wrote:
> Mateusz Loskot wrote:
>> Mateusz Loskot wrote:
>>> Simon Perkins wrote:
>>>>> What about other formats such as NITF and MEM?  Because the
>>>>> data type size_t in a 32-bit system is defined as a 4-byte
>>>>> type, whereas in a 64-bit system it is defined as an 8-byte.
>>>>> Does this mean the MEM format is not limited to 2 GB anymore
>>>>> once we work in the 64-bit system?
>>>> That would seem like a reasonable assumption though I don't
>>>> know for sure. I would think that NITF should be able to handle
>>>> larger than 4GB files even on 32-bit OSes, on any modern
>>>> filesystem.
>>> Yes, I think it's correct too.
>> 
>> Unfortunately, this assumption above *may* be incorrect. Big Thanks
>> to Norman Vine, who enlighten me about NITF format, and told that
>> the size limit depends on size of pointer type used by C++ streams.
>>  The C++ standard specifies general requirements about type of file
>>  pointer used by streams, but the type size is not fixed there.
>> 
>> All constraints like type used to represent stream position 
>> (traits::pos_type) and offset (traits::off_type). and others, are
>> determined by char_traits used during instantiation of particular
>> stream type. Finally, position/offset in streams (not only file
>> stream) is convertible to integral type with sign (sic).
>> 
>> According to the C++ standard, behavior of pos_type and off_type 
>> are implementation defined.
>> 
>> Assuming the C++ standard doesn't define 64 bit integers, maximum 
>> value of pos_type or off_type can be maximum value of int type, so
>> these values are different on different platforms as well.
> 
> Folks,
> 
> For reference, GDAL "built in" formats like NITF use the VSI API for 
> file access, and in this case the VSI*L functions which use 64bit
> file offsets on platforms where they are available (essentially all).
> I am not aware of any GDAL format drivers using the c++ stream type
> for io.

Frank,

Thanks for another enlightenment.

As we've already discussed on IRC, Norman was referring to
libnitf library.
The libnitf does use C++ file streams so the limitations explained
above apply to this library, but not to GDAL certainly what Frank has
explained.

Cheers
-- 
Mateusz Loskot
http://mateusz.loskot.net