<div dir='auto'>Thank you! <div dir="auto">This is amazing.</div><div dir="auto"><br></div><div dir="auto">Will it be enabled by default?</div></div><div class="gmail_extra"><br><div class="gmail_quote">Il 9 gen 2023 15:19, Even Rouault <even.rouault@spatialys.com> ha scritto:<br type="attribution" /><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">Hi,</p>
<p dir="ltr">It is my pleasure to announce ( <br>
https://github.com/sozip/sozip-spec/blob/master/blog/01-announcement.md <br>
) the initial release of the specification ( <br>
https://github.com/sozip/sozip-spec/blob/master/sozip_specification.md ) <br>
for the SOZip (Seek-Optimized Zip) profile to the ZIP file format, as <br>
well as its GDAL implementation.</p>
<p dir="ltr">What is SOZip ?<br>
----------------------</p>
<p dir="ltr">A Seek-Optimized ZIP file (SOZip) is a ZIP file that contains one or <br>
several Deflate-compressed files that are organized and annotated such <br>
that a SOZip-aware reader can perform very fast random access (seek) <br>
within a compressed file.</p>
<p dir="ltr">SOZip makes it possible to access large compressed files directly from a <br>
.zip file without prior decompression. It is not a new file format, but <br>
a profile of the existing ZIP format, done in a fully backward <br>
compatible way. ZIP readers that are non-SOZip aware can read a <br>
SOZip-enabled file normally and ignore the extended features that <br>
support efficient seek capability.</p>
<p dir="ltr">Use cases<br>
--------------</p>
<p dir="ltr">The SOZip specification is intended to be general purpose / not domain <br>
specific. It was first developed to serve geospatial use cases, which <br>
commonly have large compressed files inside of ZIP archives. In <br>
particular, it makes it possible for users to read large GIS files using <br>
the Shapefile, GeoPackage or FlatGeobuf formats (which have no native <br>
provision for compression) compressed in .zip files without prior <br>
decompression.</p>
<p dir="ltr">Efficient random access and selective decompression are a requirement to <br>
provide acceptable performance in many usage scenarios: spatial index <br>
filtering, access to a feature by its identifier, etc.</p>
<p dir="ltr">Performance<br>
------------------</p>
<p dir="ltr">SOZip is efficient:</p>
<p dir="ltr">* The overhead of using a file from a SOZip archive, compared to using <br>
it uncompressed, is of the order of 10% for common read operations.<br>
* Generation of a SOZip file can be much faster than regular ZIP <br>
generation when using multithreading.<br>
* SOZip files are typically only ~ 5% larger than regular ZIPs <br>
(dependent on content, and chunk size)</p>
<p dir="ltr">Have a look at benchmarking results: <br>
https://github.com/sozip/sozip-spec/blob/master/README.md#benchmarking</p>
<p dir="ltr">Other ZIP related specification<br>
------------------------------------------</p>
<p dir="ltr">The SOZip GitHub organization also hosts the KeyValuePairs extra-field <br>
specification ( <br>
https://github.com/sozip/keyvaluepairs-spec/blob/master/zip_keyvalue_extra_field_specification.md <br>
), to be able to encode arbitrary key-value pairs of metadata associated <br>
with a file within a ZIP. For example to store the Content-Type of a file.</p>
<p dir="ltr">How does this relate to GDAL ?<br>
-------------------------------------------</p>
<p dir="ltr">Pull request https://github.com/OSGeo/gdal/pull/7042 has been submitted <br>
with the following enhancements:</p>
<p dir="ltr">* The /vsizip/ virtual file system uses the SOZip index to perform fast<br>
random access within a compressed SOZip-enabled file.</p>
<p dir="ltr">* The Shapefile and GPKG drivers can directly generate SOZip-enabled <br>
.shz/.shp.zip or .gpkg.zip files.</p>
<p dir="ltr">* Addition of the CPLAddFileInZip() C function that can compress a file <br>
and add<br>
it to an new or existing ZIP file, and enable the SOZip <br>
optimization when relevant.</p>
<p dir="ltr">* The existed VSIGetFileMetadata() method can be called on a filename of<br>
the form /vsizip/path/to/the/file.zip/path/inside/the/zip/file and<br>
with domain = "ZIP" to get information if a SOZip index is <br>
available for that file.</p>
<p dir="ltr">* The sozip <br>
(https://github.com/rouault/gdal/blob/sozip/doc/source/programs/sozip.rst) <br>
new command line utility<br>
can be used to create a seek-optimized ZIP file, to append files to <br>
an existing ZIP file, list the<br>
contents of a ZIP file and display the SOZip optimization status or <br>
validate a SOZip file.</p>
<p dir="ltr">Best regards,</p>
<p dir="ltr">Even</p>
<p dir="ltr">-- </p>
<p dir="ltr">http://www.spatialys.com<br>
My software is free, but my time generally not.</p>
<p dir="ltr">_______________________________________________<br>
gdal-dev mailing list<br>
gdal-dev@lists.osgeo.org<br>
https://lists.osgeo.org/mailman/listinfo/gdal-dev<br>
</p>
</blockquote></div><br></div>