<div dir="ltr">Hi Even,<div>thanks for the feedback! </div><div><br></div><div>Since I'm here, I'm bothering you again with another question.</div><div>I have regenerated that GeoTIFF using GDAL 2.1.2.</div><div>Right now there isn't a useless IFD at the beginning anymore, due to the improvement you was talking about.</div><div>While still checking the data bytes (with an Hex editor and a calculator, ... doh!) I have noticed that there is an unused bytes section between the last data tile of the main page and the first byte of the second page (it's a GeoTIFF with overviews).</div><div><br></div><div>Here below, some numbers to make you lost :-) You can skip them and go to the "--------" section.</div><div><br></div><div>So the latest TileOffsets of page1 is 0x6BF9, the latest TileByteCounts is 0x095D resulting into last byte for Page1 is 0x7555.</div><div>The next referred IFD is 0xBBF0 which for the TileOffsets refer back the 0x7A64 byte.</div><div>Checking IFD, TAGs and what else, there is some useless bytes in the range 0x7556-0x7A63.</div><div>Going to 0x7556 I have found 13 TAG Definitions (some of them are also TileOffsets/TileByteCounts Tag referring to an address containing nothing).</div><div>----------<br></div><div>Wondering if when generating overviews we still have a problem similar to the one you have fixed in 2.0 or some data-updating issue.</div><div><br></div><div>These are my hypothesis on what is going on.</div><div>A simple translate with no overviews makes the file ending at 0x7555 with offset of next IFD saying 0x0000.</div><div>When you add overviews, it writes a new IFD at 0x7556 with TAG entries, then it encodes something and then it writes a new IFD.</div><div>At the end, it goes back updating the offset of the 1st overview IFD (0xBBF0) making the one at 0x7556 being not referred anymore.</div><div>Does it make sense?</div><div><br></div><div>Please let me know and sorry for all these numbers.</div><div>Daniele <br></div><div> </div><div><br></div><div><br></div><div><br></div><div> </div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Dec 14, 2016 at 3:23 PM, Even Rouault <span dir="ltr"><<a href="mailto:even.rouault@spatialys.com" target="_blank">even.rouault@spatialys.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><u></u>
<div style="font-family:'monospace';font-size:9pt;font-weight:400;font-style:normal"><span class="">
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">On mercredi 14 décembre 2016 15:01:51 CET Daniele Romagnoli wrote:</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> Hi List,</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> I'm investigating on some GeoTIFF data being produced with GDAL 1.11.3.</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> Basically, when setting COMPRESS=JPEG, the generated GTIFF has the 0th IFD</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> offset set to some KB.</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> these are the first 16 bytes of a GeoTIFF being generated like that:</p>
</span><p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> 49 49 2A 00 88 44 00 00 *10* 00 00 01 03 00 01</p><span class="">
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> so:</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> 49 49 is the endianness</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> 00 2A is the magic number</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> 00 00 44 88 is the offset of the 0th IFD = (17544 bytes)</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> Then there are several TAGs (baseline, extended and so on).</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> However, I think that any TIFF reader will jump to the 00 00 44 88 offset.</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> Moreover, the offset 00 00 00 08 (referring to the red bolded byte in my</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> previous sample) seems not referred by any other referencing IFD.</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> So it seems like that GeoTIFF has some useless bytes right after the first</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> 8s.</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> I have then tried to use GDAL 2.2.1 and GDAL 2.1.0 and with that version I</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> see that the GeoTIFF being produced using same exact command is smaller and</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> the 0th IFD offset is 8 with these recent versions.</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> (This information is confirmed by AsTIFFTagViewer and TiffInfo too).</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> When activating CPL_DEBUG on all versions I see this output on GDAL</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> 1.11.3/1.11.4 whilst I don't see it on GDAL 2.x</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> </p>
</span><p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> *GTiff: directory moved during flush in FlushDirectory()*</p><span class="">
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> I have quickly checked some reference in the issue tracker but I was unable</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> to find anything, especially due to the fact that when searching for "IFD"</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> it returns hundreds of results related to "IFDefined" :-)</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> Do you have any feedback or comment on this?</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> Is it related to any bug in old code or some version update? (is a</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> different libtiff version somehow involved?)</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
</span><p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">Yes this is an enhancement I did during 2.0 cycle.</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">I think it is linked to this NEWS item (although it is clearly not obvious), which brings other space enhancements:</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> * for JPEG-compressed TIFF, avoid quantization tables to be emitted in each strip/tile and use optimized Huffman coding by default</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">Previously we indeed lost space since the IFD was written twice. The handling of JPEG quantization tables in libtiff is quite tricky, since they are emitted only when the first pixel is encoded, but at that time, you have already had to write a first version of the IFD without the tables. So the trick is to generate a in-memory very small JPEG-compressed TIFF with the same settings as the real image, extract the JpegTables tag from it, and apply it right away to the real IFD. Obvious, isn't it ;-) ?</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">Even</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">-- </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">Spatialys - Geospatial professional services</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"><a href="http://www.spatialys.com" target="_blank">http://www.spatialys.com</a></p></div><br>______________________________<wbr>_________________<br>
gdal-dev mailing list<br>
<a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a><br>
<a href="http://lists.osgeo.org/mailman/listinfo/gdal-dev" rel="noreferrer" target="_blank">http://lists.osgeo.org/<wbr>mailman/listinfo/gdal-dev</a><br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><span style="font-size:12.8px">==</span><br></div><div dir="ltr"><span style="font-size:12.8000001907349px">GeoServer Professional Services from the experts! Visit</span><br style="font-size:12.8000001907349px"><a href="http://goo.gl/it488V" style="color:rgb(17,85,204);font-size:12.8000001907349px" target="_blank">http://goo.gl/it488V</a><span style="font-size:12.8000001907349px"> for more information.</span><br>
==<br>
<br>Ing. Daniele Romagnoli<br>Senior Software Engineer<br><br>GeoSolutions S.A.S.<br><span style="font-size:12.8px">Via di Montramito 3/A</span><br>55054 Massarosa (LU)<br>Italy<br>phone: +39 0584 962313<br>fax: +39 0584 1660272<br><br><a href="http://www.geo-solutions.it" target="_blank">http://www.geo-solutions.it</a><br><a href="http://twitter.com/geosolutions_it" target="_blank">http://twitter.com/geosolutions_it</a><br><br>-------------------------------------------------------<br><p><span lang="IT"><font size="1"><b>AVVERTENZE AI SENSI DEL D.Lgs. 196/2003</b></font></span></p><p><span lang="IT"><font size="1">Le
informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio
stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti,
copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento
contrario ai principi dettati dal D.Lgs. 196/2003.</font></span></p><p><span lang="IT"><font size="1"> </font></span></p><p><font size="1">The
information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential
or proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure,
reproduction, copying, distribution, or either dissemination, either
whole or partial, is strictly forbidden except previous formal approval
of the named addressee(s). If you are not the intended recipient, please
contact immediately the sender by telephone, fax or e-mail and delete
the information in this message that has been received in error. The
sender does not give any warranty or accept liability as the content,
accuracy or completeness of sent messages and accepts no responsibility
for changes made after they were sent or for other risks which arise as
a result of e-mail transmission, viruses, etc.</font></p><br></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
</div>