[Gdal-dev] GDAL data model, caching, GDALProjDef

[Hannu Koivisto]

Frank Warmerdam <warmerdam at pobox.com> writes:

> I would actually suggest handling the image as one large TIFF file
> with tiling, pre-built overviews and deflate/zip compression.  Then you

Ah, I didn't realize to consider TIFF at all, I probably dismissed
it unconsciously because programs that I have used have not
supported compression and thus produced humongous files or
something like that.

I tested[1] TIFF with deflate compression on one PNG and the result
was even a bit smaller than the source!  This sounds like a good
choice indeed, except that...

> could use the existing TIFF driver for the whole mosaic, but with
> similar performance characteristics.  This assumes the result is
> not larger than 4GB.

Does this limitation of TIFF apply to the size of data in
uncompressed form or in compressed form?  The PNGs I have take
under 150MB and even the uncompressed size of the data is under
3GB, I recall, _but_ the area that the images cover is not
rectangular.  So if I combine the PNGs into one TIFF, there will be
empty areas and that entire area in uncompressed form is over 4GB.
I assume that compression can handle those areas so that they are
not a problem considering disk space consumption but if the
limitation applies to uncompressed data, this obviously is of
little comfort.

Also, I wonder what would be the best way to make that large TIFF
file.  All I can think of is to use netpbm to concatenate all the
files into one really huge (it would be three times that 4GB+) ppm
file, then convert that to 8bpp PNG (I noticed that gdal_translate
didn't figure out that a ppm converted from 8bpp PNG had <256
colours, so it seems I need to go through some 8bpp format) and
then use gdal_translate.  This sounds like it could work but
probably isn't exactly the most space and time efficient way :)

Oh, and those overviews.  How should I add them to that TIFF file?
I note that gdaladdo utility can be used to generate overviews from
the same data but I already have them.  Then again, perhaps it does
not make sense to add overviews generated from other data to the
same file to begin with?  The covered area is not the same even
though GDAL (maybe also the GeoTIFF format) wants it to be, if I
interpret the GDAL documentation correctly, but I guess this can be
fixed.  I also note that the example overview levels in gdaladdo
documentation are integers but, on the other hand, I couldn't see
that GDAL API would restrict them that way, so perhaps there is no
need to prescale them as well.

> A second approach would be to use your PNG files, but to prepare
> a ".vrt" file describing how they should all be used as one large
> virtual dataset.  However, I think there would be some performance
> issues with using a .vrt file with *alot* of files listed.  Also, the
> .vrt format is rather insensitive to the ideal block size of the
> underlying images, and maintains a secondary level of caching which
> does not really add much value.

Right...  I also couldn't find any information about this ".vrt"
format (I grepped GDAL documentation and sources plus tried
Google).

> A third option is to implement a custom multi-file driver that can be
> used with any regularly tiled set of GDAL supported files such as your
> PNGs.  Doing this well would be a bit of work, perhaps a day or two.

Ok.  Well, let's hope that the TIFF approach can be used.

>> Another thing, the documentation of GDALDataset::GetProjectionRef
>> refers to GDALProjDef ("The returned string defines the projection
...
> You are correct that it is obsoleted by the OGR library - in particular
> use of the OGRSpatialReference and OGRCoordinateTransformation classes.
> I have updated the documentation accordingly.

Ack, I'll learn more about those classes, then.

[1] When I converted one of my PNGs to GeoTIFF format with

gdal_translate -of GTiff -co "TILED=YES" -co "COMPRESSION=DEFLATE" foo.png foo.tif

I got this message right before gdal_translate finished:

Error 1: foo.tif: Can't open a compressed TIFF file with compression for update.

I wonder why it says that?  I was able to open the resulting TIFF
with Gimp so I guess the conversion succeeded at least to some
extent despite that error.

Thanks for your very helpful response,
-- 
Hannu