[gdal-dev] Re: Disk or database full error from Rasterlite driver

[Jukka Rahkonen]

 <a.furieri <at> lqt.it> writes:

> 
> On Wed, 9 May 2012 19:50:15 +0000 (UTC), Jukka Rahkonen wrote:
> > However, the Rasterlite file size has probably been
> > around 4 gigabytes when the error occurred. Did I meet some file size
> > limit or tile count limit of the Rasterlite driver?
> >
> 
> Hi Jukka,
> 
> on any platform there is an intrinsic physical size limit;
> this is simply due to the word size measured in bits.
> on 32 bit platforms the max address you can express is 2^32,
> i.e. 4 GB; as you probably remember, on earlier versions of
> Linux and Win98 this was the max file-size allowed by their
> file-systems.
> 
> Happily enough, any modern 32 bit Linux supports a file-system
> based on long-addressing, i.e. using two 32 bit words in order
> to support 64 bit file offsets; and this effectively allows to
> store files widely exceeding the 4 GB barrier.
> 
> More or less the same is on WinXP, WinVista and Win7 if you are
> using the NTFS file-system.
> But if you are using instead the obsolete FAT file-system (the
> default choice on many USB pen drivers and USB external HDDs) the
> old 32 bit addressing is the only one supported, thus still imposing
> a file-size limit of 4 GB.
> 
> A final consideration: on both Linux 32 bit and Windows 32 bit you
> must explicitly set few compile-time flags in order to build an
> executable effectively supporting 64 bit extended file addressing;
> on both GCC and MSVC these compiler options are:
> 
> -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE=1 -D_LARGE_FILE=1
> 
> conclusion: SQLite, SpatiaLite and RasterLite have no intrinsic
> file-size limits (I've successfully built several impressively huge
> DB-files 100+ GB).
> - using 64 bit sw this assumption is surely true
> - using 32 bit sw it depends: you are surely limited to 4 GB max,
>    if your binaries were built without setting the above mentioned 
> flags.
> - if you are using the infamous FAT file-system there is absolutely no
>    way to escape from the 4 GB barrier because this is the physical 
> limit
>    imposed by this file-system.


Thanks Sandro.

I am tied to use ready made binaries and Windows and I tend to test with pretty
old computers so that raw computing power will not hide the efficiency problems.
But NTFS drives I do have.

Is there a way for me as a non-programmer to check if the components used for
Rasterlite conversion in the Windows 32-bit binaries from gisinternals.com 
have been built with or without
 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE=1 -D_LARGE_FILE=1 ?

4 GB is rather small as a geospatial image nowadays. I think that a good
practice GDAL would be to to use those large file flags as defaults when
compiling for 32-bit systems.

I have now another trial going on with the rasterlite_load tool. After a few
hours I will see if it was successful. I did my first test with GDAL for saving
some time because rasterlite_load consumes only geotiffs and I had just tiffs
and .tfw files.

-Jukka Rahkonen-