[gdal-dev] Errors while creating sozip zarr file

[Even Rouault]

Erik,

I don't think it is really worth sozip'ing a zipped Zarr, given that 
zarr is made of many relatively small files, and sozip shines with big 
compressed files.  Generally, even when creating a zipped (sozip or not) 
Zarr file, you need to make sure that your writing pattern matches 
chunks boundaries, to avoid chunk files to be rewritten several times 
and making the zip bigger than needed. Please file an issue about the 
error not being transmitted up to the caller

Even

Le 19/07/2025 à 17:44, Erik Schnetter via gdal-dev a écrit :
> I am using GDAL to create a multidimensional zarr file that is sozip 
> compressed. I see this error when creating the file:
>
> ERROR 1: dish_positions.00000000.zarr/zarr.json already exists in ZIP file
> ERROR 8: Open file 
> /vsizip/data/fengine_init_pathfinder/cx66_dish_positions.00000000.zarr.zip/dish_positions.00000000.zarr/zarr.json 
> to write failed
>
> Everything is working fine when I do not use sozip compression. I 
> enable sozip compression by adding a "/vsizip" prefix to the file 
> name. Although there is an error reported on screen, I do not see an 
> error code reported by the function creating or closing the 
> multidimensional dataset. The resulting file ("*.zarr.zip") is created 
> fine and looks almost correct, but all attributes seem to be missing.
>
> I wonder – is it actually possible to create a zarr file that is sozip 
> compressed, given that zarr probably writes to each of its file 
> multiple times? If not, what is the preferred way to create a 
> sozip-compressed zarr file efficiently?
>
> Some details:
>
> I create the dataset (i.e. the file) via
>
>                 const auto driver_manager = GetGDALDriverManager();
>                 const auto driver = 
> driver_manager->GetDriverByName("Zarr");
>                 const auto dataset = 
> std::unique_ptr<GDALDataset>(driver->CreateMultiDimensional(
>                     full_path.c_str(), root_group_options_c.data(), 
> options_c.data()));
>
> where "full_path" is 
> "/vsizip/data/fengine_init_pathfinder/cx66_dish_positions.00000000.zarr.zip/dish_positions.00000000.zarr".
>
> I then create multiple attributes ("CreateAttribute") and then
>
>                 const auto mdarray = 
> group->CreateMDArray(meta->get_name(), dimensions, datatype,
> array_options_c.data());
>                     const bool success = mdarray->Write(
>                         arrayStart.data(), count.data(), nullptr, 
> bufferStride.data(), datatype,
>                         frame + datatypesize * meta->offset, frame, 
> buffer->frame_size);
>
> and finish with
>
>                 const CPLErr err = dataset->Close();
>                 assert(!err);
>
> The full code is available at 
> <https://github.com/kotekan/kotekan/blob/eschnett/updates-2/lib/stages/gdalFileWrite.cpp>.
>
> -erik
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev

-- 
http://www.spatialys.com
My software is free, but my time generally not.