[Gdal-dev] Large File Support in GDAL?

[Andrey Kiselev]

Hi, Simon,

On Wed, Mar 26, 2003 at 10:08:37AM -0700, Simon Perkins wrote:
> I know I've seen some discussion of this topic before, but can someone
> tell me what the current situation is regarding suppport for large image
> files (> 4GB) in GDAL? Which formats, if any, support such large files?
> Is there a limit in GDAL? Or just in the drivers? Or in the OS? And is
> it necessary to be running on a 64-bit processor/OS to get large file
> support?

Not all GDAL drivers are tested for large files support, but some are
known to work. Large files supported by the HFA (Erdas Imagine) and ENVI
drivers. Other dirvers may work as well, but it should be tested. Most
drivers use wrapper I/O calls implemented in /port/cpl_vsisimple.cpp,
/port/cpl_vsil_unix_stdio_64.cpp, /port/cpl_vsil_win32.cpp. But not all
of them are properly tested and sometimes calls from simple and large
APIs mixed in the driver and it results in crashes in Windows
environment. I shall try to clarify this issue shortly and test most
drivers.

You should remember that some file formats has size limits. For example,
TIFF files cannot be larger than 4GB (in fact, you can create larger
files using special tricks, but it is simpler just think that you can't
do it).

Of course, OS support is a keystone for determining I/O capabilities.
Below I just quote my notes from gdal_building.html:

``GDAL supports reading and writing large files (> 2GiB) if it is possible
in your operating system. Notes about large file support in Linux available
here: http://www.suse.de/~aj/linux_lfs.html. In short: if you have kernel
2.4.x and glibc 2.2.x your are in luck. Maximum file size depends on
cluster size. For ext2 filesystem with 1 KiB clusters it is 16448
MiB, for ext2 with 4 KiB clusters it is 2048 GiB. Other filesystems can
handle even larger files.

Information about large file support in Windows could be obtained at
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base/supported_file_systems.asp.
In short, maximum file size at NTFS is limited by (2^64 - 1) bytes, at
FAT32 and FAT16 it is (2^32 - 1) bytes. So don't try to create files,
larger than 4 GiB on FAT32. In some cases you will not get an error
message during jumping over 4GiB barrier, but all your data will be lost.
This is not GDAL bug, this is Windows problem.''

					Best regards,
					Andrey

-- 
Andrey V. Kiselev
Home phone:  +7 812 5274898  ICQ# 26871517