<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Jesse,</p>
<p>This would break interoperability with other TIFF readers... Even
adding a new TIFF tag to advertize that bit shuffling is applied
would probably not be a sufficient guard, as existing readers
wouldn't read it, and would just display garbage, which is worth
that not being able to open the file at all. The only way I can
think off of doing that in a safe way would be to use new values
for the Compression tag, which isn't pretty either.<br>
</p>
<p>You should probably try Zarr which has such capability with the
Blosc codec. Cf <a class="moz-txt-link-freetext" href="https://gdal.org/drivers/raster/zarr.html">https://gdal.org/drivers/raster/zarr.html</a> :
BLOSC_SHUFFLE<br>
</p>
<p>I'm curious however to know which typical compression gain you
get with that.</p>
<p>Even<br>
</p>
<p><br>
</p>
<div class="moz-cite-prefix">Le 08/12/2023 à 18:06, Meyer, Jesse R.
(GSFC-618.0)[SCIENCE SYSTEMS AND APPLICATIONS INC] via gdal-dev a
écrit :<br>
</div>
<blockquote type="cite"
cite="mid:FF2AF56D-2A2A-4CB7-8259-8A2527392BF8@ndc.nasa.gov">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="Generator"
content="Microsoft Word 15 (filtered medium)">
<style>@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-ligatures:standardcontextual;}span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}div.WordSection1
{page:WordSection1;}</style>
<div class="WordSection1">
<p class="MsoNormal">Hi,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">When using horizonal differencing to reduce
the numerical range of band data, the upper bytes in the
produced stream are typically 0 which leverages LZ’s byte
based compression model. But the least significant bytes can
still have many significant bits as 0. Unless the whole byte
is replicated, LZ compressors can’t do much to leverage the
pattern however. For data with temporal and or spatial
coherence, ‘shuffling’ is another effective strategy to
losslessly reform the data stream to be favorable to LZ style
compressors. And plays nicely off gains already provided by
the PREDICTOR functionality.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The notion is to arrange the bit stream
where the Nth “shuffled” byte contains the Nth bit from each
byte in the sequence. The sequence length is usually
determined by the data type bit length.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">For example (for brevity, assume bytes are
4 bits long)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Byte 1, Byte 2, Byte 3, Byte 4<o:p></o:p></p>
<p class="MsoNormal">0001, 0011, 0111, 0001<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">They all share the top 0 bit and the bottom
1 bit,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">“Shuffled”<o:p></o:p></p>
<p class="MsoNormal">0000, 0010, 0110, 1111<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The algorithm is pretty simple to
implement, and can be SIMD accelerated for high performance.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">While we specifically are users of the
GTIFF format, such a strategy could be employed generically
for most raster and even vector formats.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Best,<o:p></o:p></p>
<p class="MsoNormal">Jesse<o:p></o:p></p>
</div>
<br>
<fieldset class="moz-mime-attachment-header"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
gdal-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a>
<a class="moz-txt-link-freetext" href="https://lists.osgeo.org/mailman/listinfo/gdal-dev">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a>
</pre>
</blockquote>
<pre class="moz-signature" cols="72">--
<a class="moz-txt-link-freetext" href="http://www.spatialys.com">http://www.spatialys.com</a>
My software is free, but my time generally not.</pre>
</body>
</html>