[gdal-dev] kerchunk

[Joe Lee]

Hi, Michael!

It's an interesting idea since Kerchunk can't handle HDF4 yet [1].
OPeNDAP DMR++ now can handle HDF4 
so I think Kerchunk can do, too.

For GDAL, is there C++ binding for Kerchunk?
I think that will be the main blocker for GDAL driver development.

[1] https://github.com/hyoklee/kerchunk/wiki

---
Reality is hierarchical. Store scientific reality in HDF for Spatial Computing.



________________________________________
From: gdal-dev <gdal-dev-bounces at lists.osgeo.org> on behalf of Michael Sumner via gdal-dev <gdal-dev at lists.osgeo.org>
Sent: Tuesday, July 23, 2024 17:37
To: gdal-dev
Subject: [gdal-dev] kerchunk

Hi, is there any effort or thought into something like Python's kerchunk in GDAL?   (my summary of kerchunk is below)

https://github.com/fsspec/kerchunk

I'll be exploring the python outputs in detail and looking for hooks into where we might bring some of this tighter into GDAL.  This would work nicely inside the GTI driver, for example. But,  a *kerchunk-driver*? That would be in the family of raw/ drivers, my skillset won't have much to offer but I'm going to explore with some simpler examples.   It could even bring old HDF4 files into the fold, I think.

It's a bit weird from a GDAL perspective to map the chunks in a format for which we have a driver, but there's definitely performance advantages and convenience for virtualizing huge disparate collections (even the simplest time-series-of-files in netcdf is nicely abstracted here for xarray, a super-charged VRT for xarray).

Interested in any thoughts, feedback, pointers to related efforts ... thanks!

(my take on) A description of kerchunk:

kerchunk replaces the actual binary blobs on file in a Zarr with json references to a file/uri/object and the byte start and end values, in this way kerchunk brings formats like hdf/netcdf/grib into the fold of "cloud readiness" by having a complete separation of metadata from the actual storage. The information about those chunks (compression, type, orientation etc is stored in json also).

(a Zarr  is a multidimensional version of a single-zoom-level image tiling, imagine every image tile as a potentially n-dimensional child block of a larger array. The blobs are stored like one zoom of an z/y/x tile server [[[v/]w/]y/]x way (with a position for each dimension of the array,  1, 2, 3, 4, or n, and z is not special, and with more general encoding possibilities than tif/png/jpeg provide.)  This scheme is extremely general,  literally a virtualized array-like abstraction on any storage, and with kerchunk you can transcend many legacy issues with actual formats.

Cheers, Mike


--
Michael Sumner
Research Software Engineer
Australian Antarctic Division
Hobart, Australia
e-mail: mdsumner at gmail.com<mailto:mdsumner at gmail.com>