<!DOCTYPE html><html><head><title></title><style type="text/css">#qt p.qt-MsoNormal{margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;font-size:11pt;font-family:Calibri, sans-serif;}
p.MsoNormal,p.MsoNoSpacing{margin:0}
p.MsoNormal,p.MsoNoSpacing{margin:0}</style></head><body><div style="font-family:Arial;">Hi,<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">In my (limited) experience, it can really make a difference for floating-point rasters. Testing with a small one I have on hand (10980x10980x1, Float32), I get:<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;"> - GeoTIFF DEFLATE 280 MB<br></div><div style="font-family:Arial;"> - Zarr BLOSC zlib NONE 281 MB<br></div><div style="font-family:Arial;"> - Zarr BLOSC zlib BIT 253 MB<br></div><div style="font-family:Arial;"> - Zarr BLOSC zlib BYTE 249 MB<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">Laurentiu<br></div><div style="font-family:Arial;"><br></div><div>On Fri, Dec 8, 2023, at 19:19, Even Rouault via gdal-dev wrote:<br></div><blockquote type="cite" id="qt" style=""><p>Jesse,<br></p><p>This would break interoperability with other TIFF readers... Even
adding a new TIFF tag to advertize that bit shuffling is applied
would probably not be a sufficient guard, as existing readers
wouldn't read it, and would just display garbage, which is worth
that not being able to open the file at all. The only way I can
think off of doing that in a safe way would be to use new values
for the Compression tag, which isn't pretty either.<br></p><p>You should probably try Zarr which has such capability with the
Blosc codec. Cf <a class="qt-moz-txt-link-freetext" href="https://gdal.org/drivers/raster/zarr.html">https://gdal.org/drivers/raster/zarr.html</a> :
BLOSC_SHUFFLE<br></p><p>I'm curious however to know which typical compression gain you
get with that.<br></p><p>Even<br></p><p><br></p><div class="qt-moz-cite-prefix">Le 08/12/2023 à 18:06, Meyer, Jesse R.
(GSFC-618.0)[SCIENCE SYSTEMS AND APPLICATIONS INC] via gdal-dev a
écrit :<br></div><blockquote type="cite" cite="mid:FF2AF56D-2A2A-4CB7-8259-8A2527392BF8@ndc.nasa.gov"><div class="qt-WordSection1"><p class="qt-MsoNormal">Hi,<br></p><p class="qt-MsoNormal"> <br></p><p class="qt-MsoNormal">When using horizonal differencing to reduce
the numerical range of band data, the upper bytes in the
produced stream are typically 0 which leverages LZ’s byte
based compression model. But the least significant bytes can
still have many significant bits as 0. Unless the whole byte
is replicated, LZ compressors can’t do much to leverage the
pattern however. For data with temporal and or spatial
coherence, ‘shuffling’ is another effective strategy to
losslessly reform the data stream to be favorable to LZ style
compressors. And plays nicely off gains already provided by
the PREDICTOR functionality.<br></p><p class="qt-MsoNormal"> <br></p><p class="qt-MsoNormal">The notion is to arrange the bit stream
where the Nth “shuffled” byte contains the Nth bit from each
byte in the sequence. The sequence length is usually
determined by the data type bit length.<br></p><p class="qt-MsoNormal"> <br></p><p class="qt-MsoNormal">For example (for brevity, assume bytes are
4 bits long)<br></p><p class="qt-MsoNormal"> <br></p><p class="qt-MsoNormal">Byte 1, Byte 2, Byte 3, Byte 4<br></p><p class="qt-MsoNormal">0001, 0011, 0111, 0001<br></p><p class="qt-MsoNormal"> <br></p><p class="qt-MsoNormal">They all share the top 0 bit and the bottom
1 bit,<br></p><p class="qt-MsoNormal"> <br></p><p class="qt-MsoNormal">“Shuffled”<br></p><p class="qt-MsoNormal">0000, 0010, 0110, 1111<br></p><p class="qt-MsoNormal"> <br></p><p class="qt-MsoNormal">The algorithm is pretty simple to
implement, and can be SIMD accelerated for high performance.<br></p><p class="qt-MsoNormal"> <br></p><p class="qt-MsoNormal">While we specifically are users of the
GTIFF format, such a strategy could be employed generically
for most raster and even vector formats.<br></p><p class="qt-MsoNormal"> <br></p><p class="qt-MsoNormal">Best,<br></p><p class="qt-MsoNormal">Jesse<br></p></div><div><br></div><pre class="qt-moz-quote-pre">_______________________________________________
gdal-dev mailing list
<a class="qt-moz-txt-link-abbreviated" href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a>
<a class="qt-moz-txt-link-freetext" href="https://lists.osgeo.org/mailman/listinfo/gdal-dev">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a>
<br></pre></blockquote><pre class="qt-moz-signature" cols="72">--
<a class="qt-moz-txt-link-freetext" href="http://www.spatialys.com">http://www.spatialys.com</a>
My software is free, but my time generally not.<br></pre><div>_______________________________________________<br></div><div>gdal-dev mailing list<br></div><div><a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a><br></div><div><a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a><br></div><div><br></div></blockquote><div style="font-family:Arial;"><br></div></body></html>