[gdal-dev] [EXTERNAL] [BULK] Re: GTiff bit shuffle compression feature request

[Meyer, Jesse R. (GSFC-618.0)[SCIENCE SYSTEMS AND APPLICATIONS INC]]

The underlying network file system is opaque to us and can change on occasion.  But recently our team were asked to cull unused files due to inode counts.

We’re excited to explore SOZip on our vector data where random seek is important to us, but we’re waiting for that branch to be merged into master.   I don’t trust standard zip libraries to be performant for this usage case but I’m willing to be shown otherwise.

Jesse

From: gdal-dev <gdal-dev-bounces at lists.osgeo.org> on behalf of Laurențiu Nicola via gdal-dev <gdal-dev at lists.osgeo.org>
Reply-To: Laurențiu Nicola <lnicola at dend.ro>
Date: Friday, December 8, 2023 at 3:01 PM
To: gdallists <gdal-dev at lists.osgeo.org>
Subject: [EXTERNAL] [BULK] Re: [gdal-dev] GTiff bit shuffle compression feature request

CAUTION: This email originated from outside of NASA.  Please take care when clicking links or opening attachments.  Use the "Report Message" button to report suspicious messages to the NASA SOC.


On Fri, Dec 8, 2023, at 21:32, Even Rouault wrote:

yes, poor wording of mine. I meant that if using PREDICTOR=3, one should compare with FILTER=DELTA. But looking more closely, they are not strictly equivalent. PREDICTOR=3 applies the delta as b[0]-a[0], b[1]-a[1], b[2]-a[2], b[3]-a[3] where a[0...3] and b[0...3] are seen as the 4 byte representation of the float32, whereas FILTER=DELTA does the difference b_float - a_float as floating point. This isn't the same...
https://www.blosc.org/posts/bytedelta-enhance-compression-toolset/ seems to be the equivalent.

> inode allocation

XFS or ZIP?

> extra step to decompress Zarr out of ZIP

Most libraries should be able to read Zarr directly from a ZIP archive.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20231208/c62d0a0a/attachment.htm>