[gdal-dev] writing parquet to /vsiaz/ fails when chunked

[snorris at hillcrestgeo.ca]

I've posted the data - round-tripping still produces the noted error (PUT as AppendBlob results in InvalidBlobType):

ogr2ogr -f Parquet \
  /vsiaz/bcfishpass/crossings.parquet \
  -nln crossings \
  --debug ON \
  --config CPL_CURL_VERBOSE=YES \
  /vsicurl/https://cabdbcfishpassstorage.blob.core.windows.net/bcfishpass/test.parquet

But.... now I've noticed that if I change the output file/blob name it works fine?! (uses AppendBlob)

ogr2ogr -f Parquet \
  /vsiaz/bcfishpass/temp1.parquet \
  -nln crossings \
  --debug ON \
  --config CPL_CURL_VERBOSE=YES \
  /vsicurl/https://cabdbcfishpassstorage.blob.core.windows.net/bcfishpass/test.parquet

If pass a single point to the failing file/blob name, it succeeds (using BlockBlob):

echo '{"type": "FeatureCollection", "features": [{"type": "Feature", "geometry": {"type": "Point", "coordinates": [30, 10]}, "properties": {}}]}' | \
ogr2ogr -f Parquet \
  /vsiaz/bcfishpass/crossings.parquet \
  -nln crossings \
  --debug ON \
  --config CPL_CURL_VERBOSE=YES \
 /vsistdin/

Full log for the three commands above is linked below... but this must be an Azure issue.
https://gist.github.com/smnorris/427ab56914b5bd55ad8f797c107aea73


> On Apr 4, 2025, at 2:52 PM, Even Rouault <even.rouault at spatialys.com> wrote:
> 
> 
> 
> Le 04/04/2025 à 20:40, Simon Norris via gdal-dev a écrit :
>> Perhaps another hapless user question:
>> 
>> Using ogr2ogr, I can write very small parquet files to /vsiaz/<container>/<file>.parquet without issue.
>> But with larger files (that ogr2ogr tries to write in chunks), Azure does not accept the first PUT request and ogr2ogr fails with:
> Can't reproduce with a dataset of mine. Maybe this is specific to your input dataset ?
> 
> 
>> I'd file an issue but if it was a bug I'd think it would have already have been found by other users?
> ah ah, if everybody had the same thinking, nobody would report bugs ;-)
>> thanks
>> Simon
>> 
>> _______________________________________________
>> gdal-dev mailing list
>> gdal-dev at lists.osgeo.org <mailto:gdal-dev at lists.osgeo.org>
>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
> -- 
> http://www.spatialys.com <http://www.spatialys.com/>
> My software is free, but my time generally not.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20250404/9e7c99f0/attachment.htm>