[gdal-dev] Re: gdalwarp -co COMPRESSION issues

[Even Rouault]

Maciek,

we are well aware of the problem but I'm afraid that complaining more will not 
help to fix it quicker.

I've analyzed the problem for my own needs a few months ago and I came with 
the conclusion that it looks like a complex interaction between GDAL and 
libtiff. The strip/tile allocation strategy in libtiff is surely at stake, 
but it could also involve the GeoTIFF driver in GDAL, or the way that some 
algorithms such as gdalwarp work.

Changing libtiff strip/tile allocation strategy to enable it to recycle wasted 
space or to reserve space to anticipated partial writes is more a (complex) 
feature development than a bug fix...

Let imagine the following scenario :
- Create a LZW compress GTiff with GDALCreate
- Write sequentially each strip with zeros. You'll get a small image
- Then write sequentially each strip with first half of the line being zeros 
and second half random data. Libtiff will try to reuse the previously written 
strips, but, of course, strips with random data compressed will not fit into 
the very small strips produced. So libtiff must allocate new space at the end 
of the file to put the new data.
- Let do it again but with only random data. Again, the last allocated strip 
space will not be enough.

And I'm not mentionning that I/O in GDAL are cached, that gdalwrap doesn't 
write sequentially strips...

So, we could now imagine to recycle wasted space, but that wouldn't help a lot 
as you're going to have fragmentation and strips not organized in the file as 
they are in the image. Thus frequent read seeks forth and back, leading to 
poor read performance. So you need some sort of defragmentation utility. And 
guess what ? I know one that already does more or less that job : it's called 
gdal_translate....

Of course I can be wrong and over-exaggerating the difficulties...

Best regards,
Even

Le Friday 14 March 2008 22:17:06 Maciej Sieczka, vous avez écrit :
> Adam Nowacki pisze:
> > Add -co "TILED=YES" to produce tiled tiff (and a lot smaller that way).
>
> 1. There are software that perform better with stripped TIFFs, eg.
> ArcPad [1], QGIS (latest SVN trunk) too - the tiff at [2] if tiled with
> gdal_translate takes about 10 (ten) times longer to render that it's
> stripped version in QGIS.
>
> 2. There are software which don't support tiled TIFF at all. To stay on
> the safe side I prefer to create stripped TIFFs in case I might be
> sharing/selling them. I might be not the only one.
>
> 2. User will be surprised that gdalwarp -co "TILED=NO" -co
> "COMPRESS=LZW" (or other compression alg.) creates tiffs bigger than
> "COMPRESS=NONE".
>
> 3. Mind that gdal_translate -co "TILED=YES" still outperforms gdalwarp
> -co "TILED=YES", any compression. With the sample [2] it creates a TIFF
> over 8% smaller. Not *very* much, but with 13 such tiffs gdal_translate
> gives you a room on your hard disk for another one, compared to gdalwarp.
>
> [1]http://arcpadteam.blogspot.com/2006/08/raster-tips-1-choosing-raster-for
>mat.html
> [2]http://www.biol.uni.wroc.pl/sieczka/udostepnione/M-33-21-A-b-4.tif
>
> Maciek
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev