[gdal-dev] Proposal for GDAL Driver: EOPF Zarr (Earth Observation Product Format)

Wed Apr 2 05:18:27 PDT 2025

Yuvraj,

Having the spec publicly available seems to be a pre-condition before 
submitting a pull request implementing it, otherwise this will make 
review close to impossible.

Regarding a separate EOPF driver vs enhancing the Zarr one, I don't have 
enough elements to make a judgement. There are other options:

- the EOPF driver uses internally the Zarr one. It seems to be the most 
attractive option assuming that EOPF Zarr is valid Zarr (which I hope it is)

- if the above hypothesis isn't completely true, the Zarr driver code is 
restructured so that relevant parts can be reused by EOPF Zarr.

In any case, we want to avoid copy&paste of code between the 2 drivers 
to ease maintenance.

Even

Le 02/04/2025 à 13:50, Adagale Yuvraj Bhagwan a écrit :
> Hi Even,
>
> 1. Maturity of the EOPF Enhancements
> EOPF (Earth Observation Processing Framework) is still in a relatively 
> early stage of development. While the core concepts have been tested 
> in prototype systems, EOPF’s final production readiness is not yet 
> established. We are currently refining specifications and partial read 
> logic, with an aim to gradually move toward production use in the 
> coming months.
>
> 2. Location of the Format Specification
> The EOPF format specification is outlined in internal documentation 
> maintained by the EOPF development team (originally used in ESA’s EOPF 
> Sample Service activities). We plan to release a public-facing version 
> of this spec once it stabilizes, so that interested developers can 
> reference it directly.
>
> 3. Why Not Enhance the Existing Zarr Driver?
> While the EOPF data layout leverages Zarr-like chunking, it carries 
> additional, domain-specific metadata and conventions—such as 
> specialized Earth Observation fields, naming conventions, and possibly 
> integration with SAFE or STAC metadata. These specialized aspects 
> don’t neatly map onto the existing Zarr driver’s scope.
>
>  *
>     *Technical*: Enhancing the Zarr driver to interpret EOPF’s custom
>     logic and metadata might introduce significant complexity or break
>     typical Zarr assumptions.
>  *
>     *Maintainability*: A standalone EOPF driver is less fragile, as we
>     can manage EOPF’s domain rules independently of any future Zarr
>     driver changes.
>  *
>     *Flexibility*: This also allows the EOPF plugin to evolve at a
>     different pace than the generic Zarr driver.
>
>
> 4. Current Skeleton Status
> Yes, the posted code is indeed just a skeleton right now. It 
> demonstrates driver registration, basic dataset creation, and minimal 
> chunk-read placeholders. This is not yet production-ready. Our 
> near-term roadmap includes fleshing out actual reading of Zarr chunks, 
> robust metadata handling, and multi-band support. The skeleton is a 
> starting point that will expand into a more complete EOPF driver in 
> subsequent development stages.
>
> Thanks,
> Yuvraj
>
> ------------------------------------------------------------------------
> *From:* Even Rouault <even.rouault at spatialys.com>
> *Sent:* Tuesday, April 1, 2025 9:41 PM
> *To:* Adagale Yuvraj Bhagwan <Yuvraj.Adagale at eurac.edu>; 
> gdal-dev at lists.osgeo.org <gdal-dev at lists.osgeo.org>
> *Subject:* Re: [gdal-dev] Proposal for GDAL Driver: EOPF Zarr (Earth 
> Observation Product Format)
>
> 	
> You don't often get email from even.rouault at spatialys.com. Learn why 
> this is important <https://aka.ms/LearnAboutSenderIdentification>
> 	
>
> Hi Yuvraj,
>
>
> A few questions:
>
> - what is the maturity of the EOPF enhancements ?
>
> - where are they described ? (i.e. the format specification)
>
> - why not enhancing the existing Zarr driver ?
>
> - from what I can currently see in 
> https://github.com/EOPF-Sample-Service/GDAL-ZARR-EOPF/tree/main/src 
> <https://github.com/EOPF-Sample-Service/GDAL-ZARR-EOPF/tree/main/src>, 
> this seems more to be a skeleton of a driver that something production 
> ready
>
>
> Even
>
>
>
> Le 01/04/2025 à 18:45, Adagale Yuvraj Bhagwan via gdal-dev a écrit :
>> Hello GDAL Community,
>> We’re developing a GDAL driver for the Earth Observation Product 
>> Format (EOPF), a cloud-optimized Zarr-based format tailored for 
>> large-scale EO data.
>> This driver aims to enable seamless access to EOPF datasets and their 
>> metadata through GDAL, supporting features like chunked I/O, and  
>> compatibility with STAC metadata.
>> Key features:
>> - Support for Zarr V2/V3 structures with EOPF-specific enhancements.
>> - Integration with cloud storage (S3, GCS, etc.).
>> - Alignment with ESA/Copernicus data standards.
>> We’d appreciate your feedback on integration requirements and best 
>> practices. The code is available at 
>> EOPF-Sample-Service/GDAL-ZARR-EOPF 
>> <https://github.com/EOPF-Sample-Service/GDAL-ZARR-EOPF>, and we plan 
>> to submit a PR soon.
>> Best regards,
>>
>> *Yuvraj Adagale*
>>
>> *Eurac Research*
>>
>>
>> *Researcher*
>>
>> Institute for Earth Observation
>> *T* +39 344 584 4031
>>
>> yuvraj.adagale at eurac.edu <mailto:yuvraj.adagale at eurac.edu>
>>
>> Drususallee/Viale Druso 1
>>
>> I-39100 Bozen/Bolzano
>>
>> Legal Seat
>>
>> Drususallee/Viale Druso 1
>>
>> I-39100 Bozen/Bolzano
>> *_www.eurac.edu <http://www.eurac.edu/>_*
>>
>> *_Facebook <https://facebook.com/eurac.research>_ | _YouTube 
>> <https://www.youtube.com/EURACtv>_ | _X 
>> <https://twitter.com/eurac>_ | _LinkedIn 
>> <https://www.linkedin.com/company/euracresearch>_ | _Instagram 
>> <https://www.instagram.com/euracresearch/>_****| CV*
>>
>> _signature_1401579056 <https://www.eurac.edu/en>_
>>
>> According to regulation (EU) 2016/679 this transmission is intended only
>>
>> for the use of the addressee and may contain confidential information.
>>
>> If you receive this transmission in error, please notify the sender 
>> immediately
>>
>> by email and delete all copies of this message and any attachments.
>>
>> Diese Nachricht ist im Sinne der Verordnung (EU) 2016/679 
>> ausschließlich für
>>
>> den Adressaten bestimmt und kann vertrauliche Informationen enthalten.
>>
>> Sollten Sie diese Nachricht irrtümlich erhalten haben, bitten wir 
>> Sie, den
>>
>> Absender darüber unverzüglich per E-Mail in Kenntnis zu setzen sowie die
>>
>> Nachricht und etwaige Kopien und Anlagen zu vernichten.
>>
>> Ai sensi del Regolamento UE 679/2016 questo messaggio è ad uso esclusivo
>>
>> del destinatario e può contenere informazioni riservate. Qualora Le fosse
>>
>> pervenuto per errore, Le chiediamo gentilmente di comunicarcelo 

>>
>> immediatamente via e-mail ed eliminare qualsiasi copia e allegato.
>>
>>
>>
>> _______________________________________________
>> gdal-dev mailing list
>> gdal-dev at lists.osgeo.org  <mailto:gdal-dev at lists.osgeo.org>
>> https://lists.osgeo.org/mailman/listinfo/gdal-dev  <https://lists.osgeo.org/mailman/listinfo/gdal-dev>
> -- 
> http://www.spatialys.com  <http://www.spatialys.com/>
> My software is free, but my time generally not.

-- 
http://www.spatialys.com
My software is free, but my time generally not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20250402/33c0311c/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Outlook-signature_.png
Type: image/png
Size: 17457 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20250402/33c0311c/attachment-0001.png>