[gdal-dev] Ignore content-length in vsicurl?

[Rahkonen Jukka]

Hi,

Have you tried with configuration option "CPL_VSIL_CURL_USE_HEAD=[YES/NO]: Defaults to YES. Controls whether to use a HEAD request when opening a remote URL."

-Jukka Rahkonen-

Lähettäjä: gdal-dev <gdal-dev-bounces at lists.osgeo.org> Puolesta Daniel Evans via gdal-dev
Lähetetty: tiistai 10. syyskuuta 2024 16.57
Vastaanottaja: 'gdal-dev at lists.osgeo.org' (gdal-dev at lists.osgeo.org) <gdal-dev at lists.osgeo.org>
Aihe: [gdal-dev] Ignore content-length in vsicurl?

Hi all,

I am attempting to read a dataset via /vsicurl/ where I believe the server is incorrectly returning `content-length: 0` in response to HEAD requests. This causes GDAL to believe it's a zero-length file, and it therefore can't be read.

If I download the file via HTTP GET, it's valid, and GDAL can read it locally. I've also confirmed I can use /vsicurl/ on some test datasets in the GDAL repo.

Is it possible to force GDAL to work around the faulty content-length header, or is it too fundamental a problem to ignore?

I've separately got in touch with the data provider to see if they are able to fix the issue at their end.

Cheers,
Daniel


URL of the troublesome dataset:
https://data.source.coop/earthgenome/sentinel2-temporal-mosaics/20NMH_2024-04-01_2024-08-01/B08.tif


Example HTTP header responses I'm seeing:

GET

HTTP/2 200
date: Tue, 10 Sep 2024 13:47:54 GMT
content-type: binary/octet-stream
content-length: 278198294
vary: Origin, Access-Control-Request-Method, Access-Control-Request-Headers
etag: "a79f3f685281d6681e4d362536c5b3eb-34"
last-modified: Thu, 25 Jul 2024 13:16:08 GMT
x-version: 0.0.16
access-control-allow-credentials: true

HEAD

HTTP/2 200
date: Tue, 10 Sep 2024 13:48:08 GMT
content-type: binary/octet-stream
content-length: 0
x-version: 0.0.16
access-control-allow-credentials: true
etag: "a79f3f685281d6681e4d362536c5b3eb-34"
last-modified: Thu, 25 Jul 2024 13:16:08 GMT
vary: Origin, Access-Control-Request-Method, Access-Control-Request-Headers
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240910/bcf7a925/attachment.htm>