[gdal-dev] How to wrap a zarr in a zip and read it with vsizip?

[Kurt Schwehr]

Mike,

That's an interesting one for sure.

Even before the ESA talk of Zarr, we have a good amount of Zarr data
around. Just one I know of:
https://cloud.google.com/storage/docs/public-datasets/era5

On Wed, Feb 26, 2025 at 8:07 PM Michael Sumner <mdsumner at gmail.com> wrote:

> Just clueing into why you might be working with this, have you seen this
> critique?  (was a bit shocked to see that this is apparently going forward
> for Sentinel 2, let alone that it was even considered!)
>
> https://github.com/csaybar/ESA-zar-zip-decision
>
> Also glad to see a working example outlined that I can follow, Thanks!
>
> Cheers, Mike
>
>
>
> On Tue, Feb 25, 2025 at 7:10 AM Kurt Schwehr via gdal-dev <
> gdal-dev at lists.osgeo.org> wrote:
>
>> Thanks Laurentiu and Scott!
>>
>> I can't believe 1) I left off -r and 2) I didn't think to look at what's
>> in the zip. Doh!
>>
>> And thanks for reminding me about sozip. I was one of the reviewers of
>> the initial spec and still don't have it front-of-mind.
>>
>> unzip -l nczarr_v2.zarr.zip
>> Archive:  nczarr_v2.zarr.zip
>>   Length      Date    Time    Name
>> ---------  ---------- -----   ----
>>         0  2024-12-23 02:00   nczarr_v2.zarr/
>> ---------                     -------
>>         0                     1 file
>>
>> Now happily able to see the contents:
>>
>> gdalinfo /vsizip/nczarr_v2.zarr.zip/nczarr_v2.zarr
>> Driver: Zarr/Zarr
>> Files: /vsizip/nczarr_v2.zarr.zip/nczarr_v2.zarr
>> Size is 512, 512
>> Subdatasets:
>>
>> SUBDATASET_1_NAME=ZARR:"/vsizip/nczarr_v2.zarr.zip/nczarr_v2.zarr":/MyGroup/lon
>>   SUBDATASET_1_DESC=Array /MyGroup/lon
>>
>> SUBDATASET_2_NAME=ZARR:"/vsizip/nczarr_v2.zarr.zip/nczarr_v2.zarr":/MyGroup/lat
>>   SUBDATASET_2_DESC=Array /MyGroup/lat
>>
>> SUBDATASET_3_NAME=ZARR:"/vsizip/nczarr_v2.zarr.zip/nczarr_v2.zarr":/MyGroup/dset1
>>   SUBDATASET_3_DESC=Array /MyGroup/dset1
>>
>> SUBDATASET_4_NAME=ZARR:"/vsizip/nczarr_v2.zarr.zip/nczarr_v2.zarr":/MyGroup/Group_A/dset2
>>   SUBDATASET_4_DESC=Array /MyGroup/Group_A/dset2
>>
>> SUBDATASET_5_NAME=ZARR:"/vsizip/nczarr_v2.zarr.zip/nczarr_v2.zarr":/MyGroup/Group_A/dset3
>>   SUBDATASET_5_DESC=Array /MyGroup/Group_A/dset3
>> [SNIP]
>>
>> -Kurt
>>
>> On Mon, Feb 24, 2025 at 11:21 AM Laurențiu Nicola via gdal-dev <
>> gdal-dev at lists.osgeo.org> wrote:
>>
>>> I suspect it won't make a lot of difference. SOZIP is designed to allow
>>> seeking within a compressed file, but Zarr is tile- (block) based, where
>>> each of those is stored in a different file. So you end up uncompressing a
>>> whole file anyway.
>>>
>>> Laurentiu
>>>
>>> On Mon, Feb 24, 2025, at 21:17, Scott via gdal-dev wrote:
>>> > There's GDAL's sozip (Search Optimized Zip) utility as well. I have no
>>> > idea if it works with .zarr. I'm sure someone will correct me! ;)
>>> >
>>> > cd nczarr_v2.zarr
>>> > sozip ../nczarr_v2.zarr.zip .
>>> > sozip -l /vsizip/../nczarr_v2.zarr.zip
>>> >
>>> > On 2/24/25 10:09, Kurt Schwehr via gdal-dev wrote:
>>> >> Hi all,
>>> >>
>>> >> I seem to be having trouble exactly how to correctly make a zip of a
>>> >> zarr and how to correctly specify the vsizip url.
>>> >>
>>> >> e.g from autotest/gdrivers/data/zarr
>>> >>
>>> >> gdalinfo nczarr_v2.zarr # Works
>>> >> tar cf nczarr_v2.zarr.zip nczarr_v2.zarr
>>> >>
>>> >> Then what? I've tried lots of variations and not had any success.
>>> >>
>>> >> Thanks!
>>> >> -Kurt
>>> >>
>>> >> _______________________________________________
>>> >> gdal-dev mailing list
>>> >> gdal-dev at lists.osgeo.org
>>> >> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>> >
>>> > _______________________________________________
>>> > gdal-dev mailing list
>>> > gdal-dev at lists.osgeo.org
>>> > https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>> _______________________________________________
>>> gdal-dev mailing list
>>> gdal-dev at lists.osgeo.org
>>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>>
>> _______________________________________________
>> gdal-dev mailing list
>> gdal-dev at lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>
>
>
> --
> Michael Sumner
> Research Software Engineer
> Australian Antarctic Division
> Hobart, Australia
> e-mail: mdsumner at gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20250226/a2d009cf/attachment.htm>