[Gdal-dev] large file support, gdal_merge with 8bit images (was: GDAL data model...)

[Frank Warmerdam]

Hannu Koivisto wrote:
> I have now assembled the files into an uncompressed PNG.

Hannu,

Is there such a thing as an uncompressed PNG file?  I believe all
image segments in PNG files are internally compressed using zlib.

> Unfortunately, if I try to convert that to a compressed GeoTIFF
> file using gdal_translate, it cannot open the input file because it
> is larger than 4GB, i.e. it does not seem use large file support.
>>From the output of strace:
> 
> 30339 open("foo.png", O_RDONLY) = -1 EFBIG (File too large)
> 
> Given that you suggested using an uncompressed format, although a
> different one, I assumed that this would be because my gdal was
> compiled to use system's libpng, which is unlikely to have large
> file support.  So I recompiled gdal with internal PNG support but I
> still get the same error.  I guess this means that somehow the PNG
> driver does not get compiled with correct definitions.  I do see
> calls to stat64 before that open call, so it would seem that some
> parts are using 64bit calls.  Unless you have some insight into the
> issue, I'll investigate this more closely later today or tomorrow.

Yow!  That's a big png file.

First, I should note that the internal version of libpng is a bit
antique. If you are going to do a bunch of work getting it going in
64 bit IO mode it might be better that we upgrade first to the latest
libpng.

There are parts of GDAL that using the large file API (abstracted
within GDAL as the VSI*L functions, such as VSIOpenL(), VSIFSeekL(),
etc.  For libraries like libtiff that allow easy replacement of the
low level IO functions I try to insert using the VSI functions (or
VSI*L) to take advantage of large file access where practical.

It might be possible to do this with libpng but I haven't looked into
it.

Let us know how work on large-file PNG support goes.  With luck it
might be something that can be incorporated back.  If you need it, I
could also dig into this and try to make libpng work with large files
within GDAL.

>>My suggestion would be to let gdaladdo generate the overviews itself
>>unless you have some specific algorithm for generating the overviews
>>that you feel strongly about.
> 
> 
> It's not really an algorithm issue, the overviews I already have
> have been generated from different data and serve much better as
> overviews.  If I generated highly downscaled versions of the high
> resolution data, I couldn't read text or make much sense of the
> data otherwise either.  But I can work around this problem easily
> by keeping the overviews in separate files and operating on them as
> separate datasets.  

Understood.  If you find you want to assemble GeoTIFF files with these
overviews let me know.  It can be done using libtiff utilities but it
is a hassle.

 >I am slightly worried whether GDAL's caching
> works well in a situation like this, i.e. is it designed to work
> with multiple simultaneously open datasets, but given that I
> eventually need multiple simultaneously open datasets anyway
> ("real" datasets, not just overviews of an other dataset), I guess
> I'll just make it work if it doesn't.

GDAL's raster caching is global.  That is one memory pool is shared
amoung all datasets.  If you have two files open, read all of one and
then read all of the second, the cached blocks of the first will
be expired as memory is needed for the second dataset.

I presume this is the behaviour you would want?

Best regards,

-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | Geospatial Programmer for Rent