[gdal-dev] Expected behavior wrt .properties files with /vsigzip/

[Simon Eves]

Dear GDAL,

We are seeing some strange behavior when reading and writing files with the
GZip VSI, and I want to ask in the dev forum first before assuming it's a
bug and filing an issue. I can't find anything in the docs or the forum
that seems relevant.

When writing (say) a .geojson.gz using the original GeoJSON driver, it
seems to create, and then delete, a sibling .properties file during the
write. Based on the comment at...

https://github.com/OSGeo/gdal/blob/master/gdal/port/cpl_vsil_gzip.cpp#L68

...this seems fair enough.

It also seems to try to create the .properties file when READING a GZip
file, which is also fair enough, if it can.

In the case of a read-only file-system, or other read-only VSI source,
however, I assume a failure to write this file is supposed to be benign,
and indeed I wouldn't expect any user-facing message or error at all.

We are able to read (say) a .geojson.gz from an S3 bucket with no issues.

However, when we try to read a file from within a gzipped TAR file on an S3
bucket, there is a problem. GDAL throws the following error:

DoSinglePartPUT of
/vsis3/omnisci-import-test/example.geojson.tar.gz.properties failed (1)

...and in our CI environment with an ASAN test build, ASAN reports a bunch
of leaks, which appear to be to do with Curl. I don't have these messages
to hand, but I can get them from my colleague if necessary.

>From the code (we're still running 2.4.2, but 3.1.2 appears to be the same)
it seem that DoSinglePartPUT is only called from VSIS3WriteHandle::Close(),
so I'm wondering if it's trying to flush out the write of the .properties
file while closing the data source, and perhaps not knowing to squash any
error.

Here is the path to the file in what should be an open public bucket (in
region us-west-1) if you want to try it yourself:

s3://omnisci-import-test/example.geojson.tar.gz with subpath
/example/example.geojson

Thanks in advance!

-- 
<http://www.omnisci.com/>
Simon Eves
Senior Graphics Engineer, Rendering Group
100 Montgomery St (5th Floor), San Francisco, CA 94104, USA


Email: simon.eves at omnisci.com | Cell:  +1 (415) 902-1996
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20200908/ac5e9c76/attachment.html>