[gdal-dev] How to wrap a zarr in a zip and read it with vsizip?

[Michael Sumner]

nice, I've been hitting that up with the multidim model and it works pretty
well. (i'm still scratchy in the C++ but getting what I want out of it).

We were checking out CMIP6 holdings too that look similarly impressive.

M

On Thu, Feb 27, 2025 at 5:27 PM Kurt Schwehr <schwehr at gmail.com> wrote:

> Mike,
>
> That's an interesting one for sure.
>
> Even before the ESA talk of Zarr, we have a good amount of Zarr data
> around. Just one I know of:
> https://cloud.google.com/storage/docs/public-datasets/era5
>
> On Wed, Feb 26, 2025 at 8:07 PM Michael Sumner <mdsumner at gmail.com> wrote:
>
>> Just clueing into why you might be working with this, have you seen this
>> critique?  (was a bit shocked to see that this is apparently going forward
>> for Sentinel 2, let alone that it was even considered!)
>>
>> https://github.com/csaybar/ESA-zar-zip-decision
>>
>> Also glad to see a working example outlined that I can follow, Thanks!
>>
>> Cheers, Mike
>>
>>
>>
>> On Tue, Feb 25, 2025 at 7:10 AM Kurt Schwehr via gdal-dev <
>> gdal-dev at lists.osgeo.org> wrote:
>>
>>> Thanks Laurentiu and Scott!
>>>
>>> I can't believe 1) I left off -r and 2) I didn't think to look at what's
>>> in the zip. Doh!
>>>
>>> And thanks for reminding me about sozip. I was one of the reviewers of
>>> the initial spec and still don't have it front-of-mind.
>>>
>>> unzip -l nczarr_v2.zarr.zip
>>> Archive:  nczarr_v2.zarr.zip
>>>   Length      Date    Time    Name
>>> ---------  ---------- -----   ----
>>>         0  2024-12-23 02:00   nczarr_v2.zarr/
>>> ---------                     -------
>>>         0                     1 file
>>>
>>> Now happily able to see the contents:
>>>
>>> gdalinfo /vsizip/nczarr_v2.zarr.zip/nczarr_v2.zarr
>>> Driver: Zarr/Zarr
>>> Files: /vsizip/nczarr_v2.zarr.zip/nczarr_v2.zarr
>>> Size is 512, 512
>>> Subdatasets:
>>>
>>> SUBDATASET_1_NAME=ZARR:"/vsizip/nczarr_v2.zarr.zip/nczarr_v2.zarr":/MyGroup/lon
>>>   SUBDATASET_1_DESC=Array /MyGroup/lon
>>>
>>> SUBDATASET_2_NAME=ZARR:"/vsizip/nczarr_v2.zarr.zip/nczarr_v2.zarr":/MyGroup/lat
>>>   SUBDATASET_2_DESC=Array /MyGroup/lat
>>>
>>> SUBDATASET_3_NAME=ZARR:"/vsizip/nczarr_v2.zarr.zip/nczarr_v2.zarr":/MyGroup/dset1
>>>   SUBDATASET_3_DESC=Array /MyGroup/dset1
>>>
>>> SUBDATASET_4_NAME=ZARR:"/vsizip/nczarr_v2.zarr.zip/nczarr_v2.zarr":/MyGroup/Group_A/dset2
>>>   SUBDATASET_4_DESC=Array /MyGroup/Group_A/dset2
>>>
>>> SUBDATASET_5_NAME=ZARR:"/vsizip/nczarr_v2.zarr.zip/nczarr_v2.zarr":/MyGroup/Group_A/dset3
>>>   SUBDATASET_5_DESC=Array /MyGroup/Group_A/dset3
>>> [SNIP]
>>>
>>> -Kurt
>>>
>>> On Mon, Feb 24, 2025 at 11:21 AM Laurențiu Nicola via gdal-dev <
>>> gdal-dev at lists.osgeo.org> wrote:
>>>
>>>> I suspect it won't make a lot of difference. SOZIP is designed to allow
>>>> seeking within a compressed file, but Zarr is tile- (block) based, where
>>>> each of those is stored in a different file. So you end up uncompressing a
>>>> whole file anyway.
>>>>
>>>> Laurentiu
>>>>
>>>> On Mon, Feb 24, 2025, at 21:17, Scott via gdal-dev wrote:
>>>> > There's GDAL's sozip (Search Optimized Zip) utility as well. I have
>>>> no
>>>> > idea if it works with .zarr. I'm sure someone will correct me! ;)
>>>> >
>>>> > cd nczarr_v2.zarr
>>>> > sozip ../nczarr_v2.zarr.zip .
>>>> > sozip -l /vsizip/../nczarr_v2.zarr.zip
>>>> >
>>>> > On 2/24/25 10:09, Kurt Schwehr via gdal-dev wrote:
>>>> >> Hi all,
>>>> >>
>>>> >> I seem to be having trouble exactly how to correctly make a zip of a
>>>> >> zarr and how to correctly specify the vsizip url.
>>>> >>
>>>> >> e.g from autotest/gdrivers/data/zarr
>>>> >>
>>>> >> gdalinfo nczarr_v2.zarr # Works
>>>> >> tar cf nczarr_v2.zarr.zip nczarr_v2.zarr
>>>> >>
>>>> >> Then what? I've tried lots of variations and not had any success.
>>>> >>
>>>> >> Thanks!
>>>> >> -Kurt
>>>> >>
>>>> >> _______________________________________________
>>>> >> gdal-dev mailing list
>>>> >> gdal-dev at lists.osgeo.org
>>>> >> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>>> >
>>>> > _______________________________________________
>>>> > gdal-dev mailing list
>>>> > gdal-dev at lists.osgeo.org
>>>> > https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>>> _______________________________________________
>>>> gdal-dev mailing list
>>>> gdal-dev at lists.osgeo.org
>>>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>>>
>>> _______________________________________________
>>> gdal-dev mailing list
>>> gdal-dev at lists.osgeo.org
>>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>>
>>
>>
>> --
>> Michael Sumner
>> Research Software Engineer
>> Australian Antarctic Division
>> Hobart, Australia
>> e-mail: mdsumner at gmail.com
>>
>

-- 
Michael Sumner
Research Software Engineer
Australian Antarctic Division
Hobart, Australia
e-mail: mdsumner at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20250227/e1f2c892/attachment-0001.htm>