[gdal-dev] Question on 0th IFD offset when producing a JPEG-Compressed GeoTIFF (GDAL 1.11 vs 2.x)

[Even Rouault]

On mercredi 14 décembre 2016 15:01:51 CET Daniele Romagnoli wrote:
> Hi List,
> I'm investigating on some GeoTIFF data being produced with GDAL 1.11.3.
> Basically, when setting COMPRESS=JPEG, the generated GTIFF has the 0th IFD
> offset set to some KB.
> 
> these are the first 16 bytes of a GeoTIFF being generated like that:
> 49 49 2A 00 88 44 00 00 *10* 00 00 01 03 00 01
> 
> so:
> 49 49 is the endianness
> 00 2A is the magic number
> 00 00 44 88 is the offset of the 0th IFD = (17544 bytes)
> Then there are several TAGs (baseline, extended and so on).
> However, I think that any TIFF reader will jump to the 00 00 44 88 offset.
> Moreover, the offset 00 00 00 08 (referring to the red bolded byte in my
> previous sample) seems not referred by any other referencing IFD.
> 
> So it seems like that GeoTIFF has some useless bytes right after the first
> 8s.
> 
> I have then tried to use GDAL 2.2.1 and GDAL 2.1.0 and with that version I
> see that the GeoTIFF being produced using same exact command is smaller and
> the 0th IFD offset is 8 with these recent versions.
> (This information is confirmed by AsTIFFTagViewer and TiffInfo too).
> 
> When activating CPL_DEBUG on all versions I see this output on GDAL
> 1.11.3/1.11.4 whilst I don't see it on GDAL 2.x
> 
> *GTiff: directory moved during flush in FlushDirectory()*
> 
> I have quickly checked some reference in the issue tracker but I was unable
> to find anything, especially due to the fact that when searching for "IFD"
> it returns hundreds of results related to "IFDefined" :-)
> 
> Do you have any feedback or comment on this?
> Is it related to any bug in old code or some version update? (is a
> different libtiff version somehow involved?)

Yes this is an enhancement I did during 2.0 cycle.

I think it is linked to this NEWS item (although it is clearly not obvious), which brings other 
space enhancements:
 * for JPEG-compressed TIFF, avoid quantization tables to be emitted in each strip/tile and 
use optimized Huffman coding by default

Previously we indeed lost space since the IFD was written twice. The handling of JPEG 
quantization tables in libtiff is quite tricky, since they are emitted only when the first pixel is 
encoded, but at that time, you have already had to write a first version of the IFD without the 
tables. So the trick is to generate a in-memory very small JPEG-compressed TIFF with the 
same settings as the real image, extract the JpegTables tag from it, and apply it right away to 
the real IFD. Obvious, isn't it ;-) ?

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20161214/aa7c705f/attachment.html>