[gdal-dev] Question on 0th IFD offset when producing a JPEG-Compressed GeoTIFF (GDAL 1.11 vs 2.x)

Fri Dec 16 04:05:31 PST 2016

Hi Even,
thanks for the feedback!

Since I'm here, I'm bothering you again with another question.
I have regenerated that GeoTIFF using GDAL 2.1.2.
Right now there isn't a useless IFD at the beginning anymore, due to the
improvement you was talking about.
While still checking the data bytes (with an Hex editor and a calculator,
... doh!) I have noticed that there is an unused bytes section between the
last data tile of the main page and the first byte of the second page (it's
a GeoTIFF with overviews).

Here below, some numbers to make you lost :-) You can skip them and go to
the "--------" section.

So the latest TileOffsets of page1 is 0x6BF9, the latest TileByteCounts is
0x095D resulting into last byte for Page1 is 0x7555.
The next referred IFD is 0xBBF0 which for the TileOffsets refer back the
0x7A64 byte.
Checking IFD, TAGs and what else, there is some useless bytes in the range
0x7556-0x7A63.
Going to 0x7556 I have found 13 TAG Definitions (some of them are also
TileOffsets/TileByteCounts Tag referring to an address containing nothing).
----------
Wondering if when generating overviews we still have a problem similar to
the one you have fixed in 2.0 or some data-updating issue.

These are my hypothesis on what is going on.
A simple translate with no overviews makes the file ending at 0x7555 with
offset of next IFD saying 0x0000.
When you add overviews, it writes a new IFD at 0x7556 with TAG entries,
then it encodes something and then it writes a new IFD.
At the end, it goes back updating the offset of the 1st overview IFD
(0xBBF0) making the one at 0x7556 being not referred anymore.
Does it make sense?

Please let me know and sorry for all these numbers.
Daniele

On Wed, Dec 14, 2016 at 3:23 PM, Even Rouault <even.rouault at spatialys.com>
wrote:

> On mercredi 14 décembre 2016 15:01:51 CET Daniele Romagnoli wrote:
>
> > Hi List,
>
> > I'm investigating on some GeoTIFF data being produced with GDAL 1.11.3.
>
> > Basically, when setting COMPRESS=JPEG, the generated GTIFF has the 0th
> IFD
>
> > offset set to some KB.
>
> >
>
> > these are the first 16 bytes of a GeoTIFF being generated like that:
>
> > 49 49 2A 00 88 44 00 00 *10* 00 00 01 03 00 01
>
> >
>
> > so:
>
> > 49 49 is the endianness
>
> > 00 2A is the magic number
>
> > 00 00 44 88 is the offset of the 0th IFD = (17544 bytes)
>
> > Then there are several TAGs (baseline, extended and so on).
>
> > However, I think that any TIFF reader will jump to the 00 00 44 88
> offset.
>
> > Moreover, the offset 00 00 00 08 (referring to the red bolded byte in my
>
> > previous sample) seems not referred by any other referencing IFD.
>
> >
>
> > So it seems like that GeoTIFF has some useless bytes right after the
> first
>
> > 8s.
>
> >
>
> > I have then tried to use GDAL 2.2.1 and GDAL 2.1.0 and with that version
> I
>
> > see that the GeoTIFF being produced using same exact command is smaller
> and
>
> > the 0th IFD offset is 8 with these recent versions.
>
> > (This information is confirmed by AsTIFFTagViewer and TiffInfo too).
>
> >
>
> > When activating CPL_DEBUG on all versions I see this output on GDAL
>
> > 1.11.3/1.11.4 whilst I don't see it on GDAL 2.x
>
> >
>
> > *GTiff: directory moved during flush in FlushDirectory()*
>
> >
>
> > I have quickly checked some reference in the issue tracker but I was
> unable
>
> > to find anything, especially due to the fact that when searching for
> "IFD"
>
> > it returns hundreds of results related to "IFDefined" :-)
>
> >
>
> > Do you have any feedback or comment on this?
>
> > Is it related to any bug in old code or some version update? (is a
>
> > different libtiff version somehow involved?)
>
>
>
> Yes this is an enhancement I did during 2.0 cycle.
>
>
>
> I think it is linked to this NEWS item (although it is clearly not
> obvious), which brings other space enhancements:
>
> * for JPEG-compressed TIFF, avoid quantization tables to be emitted in
> each strip/tile and use optimized Huffman coding by default
>
>
>
> Previously we indeed lost space since the IFD was written twice. The
> handling of JPEG quantization tables in libtiff is quite tricky, since they
> are emitted only when the first pixel is encoded, but at that time, you
> have already had to write a first version of the IFD without the tables. So
> the trick is to generate a in-memory very small JPEG-compressed TIFF with
> the same settings as the real image, extract the JpegTables tag from it,
> and apply it right away to the real IFD. Obvious, isn't it ;-) ?
>
>
>
> Even
>
>
>
> --
>
> Spatialys - Geospatial professional services
>
> http://www.spatialys.com
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>

-- 
==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.
==

Ing. Daniele Romagnoli
Senior Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax:      +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility  for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20161216/5e5b2bdc/attachment-0001.html>