[Gdal-dev] GDAL data model, caching, GDALProjDef

[Frank Warmerdam]

Hannu Koivisto wrote:
> Does this limitation of TIFF apply to the size of data in
> uncompressed form or in compressed form?  

Hannu,

The limitation is on the compressed form, so you should be
fine.


> Also, I wonder what would be the best way to make that large TIFF
> file.  All I can think of is to use netpbm to concatenate all the
> files into one really huge (it would be three times that 4GB+) ppm
> file, then convert that to 8bpp PNG (I noticed that gdal_translate
> didn't figure out that a ppm converted from 8bpp PNG had <256
> colours, so it seems I need to go through some 8bpp format) and
> then use gdal_translate.  This sounds like it could work but
> probably isn't exactly the most space and time efficient way :)

I would suggest using the gdal_merge.py script.  This will build
a mosaic from a set of georeferenced input files.  You would need
to have .wld (ESRI world files) for each of your PNG chunks for it
to place them properly in the result.

Because of the way that gdal_merge.py updates the output file, it
is likely impossible to have the output file be a compressed TIFF
file.  You will likely need to assemble the files into an uncompressed
file (perhaps of some format like Erdas Imagine - driver=HFA) and
then convert that to a compressed GeoTIFF file using gdal_translate
as a separate step.

This approach has the same "large intermediate mosaic" problem as the
method that you suggest, but might be easier.

> Oh, and those overviews.  How should I add them to that TIFF file?
> I note that gdaladdo utility can be used to generate overviews from
> the same data but I already have them.  Then again, perhaps it does
> not make sense to add overviews generated from other data to the
> same file to begin with?  The covered area is not the same even
> though GDAL (maybe also the GeoTIFF format) wants it to be, if I
> interpret the GDAL documentation correctly, but I guess this can be
> fixed.  I also note that the example overview levels in gdaladdo
> documentation are integers but, on the other hand, I couldn't see
> that GDAL API would restrict them that way, so perhaps there is no
> need to prescale them as well.

My suggestion would be to let gdaladdo generate the overviews itself
unless you have some specific algorithm for generating the overviews
that you feel strongly about.

You are correct that it is not really necesary for the overviews to
be at integral resolutions.

>>A second approach would be to use your PNG files, but to prepare
>>a ".vrt" file describing how they should all be used as one large
>>virtual dataset.  However, I think there would be some performance
>>issues with using a .vrt file with *alot* of files listed.  Also, the
>>.vrt format is rather insensitive to the ideal block size of the
>>underlying images, and maintains a secondary level of caching which
>>does not really add much value.
> 
> 
> Right...  I also couldn't find any information about this ".vrt"
> format (I grepped GDAL documentation and sources plus tried
> Google).

There is some info in the archive, but I have yet to write user
documentation on the driver and the format.

>>You are correct that it is obsoleted by the OGR library - in particular
>>use of the OGRSpatialReference and OGRCoordinateTransformation classes.
>>I have updated the documentation accordingly.
> 
> 
> Ack, I'll learn more about those classes, then.
> 
> [1] When I converted one of my PNGs to GeoTIFF format with
> 
> gdal_translate -of GTiff -co "TILED=YES" -co "COMPRESSION=DEFLATE" foo.png foo.tif
> 
> I got this message right before gdal_translate finished:
> 
> Error 1: foo.tif: Can't open a compressed TIFF file with compression for update.
> 
> I wonder why it says that?  I was able to open the resulting TIFF
> with Gimp so I guess the conversion succeeded at least to some
> extent despite that error.

The file would still have been created properly.  I have also fixed the
problem where GDAL (and libtiff) was unable to open a compressed GeoTIFF file
in update mode as it prevented obviously necessary things like adding
overviews to a compressed file.

However, this fix is quite recent (the last couple of weeks), so you will
need a GDAL "nightly snapshot" or to build it from CVS.  Also, the fix
requires changes in libtiff so if you aren't using the internal libtiff you
will need to update your libtiff from CVS.

Best regards,

-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | Geospatial Programmer for Rent