<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>The server seems to have been fixed, but on reflection, I won't
merge the workaround PR as this is (thanksfully) quite an unusual
case. At least the PR is there if someone badly needed to patch
their build with such a workaround.<br>
</p>
<div class="moz-cite-prefix">Le 10/09/2024 à 17:11, thomas bonfort a
écrit :<br>
</div>
<blockquote type="cite"
cite="mid:CAOM3y2jcv=+_xsTDWoh9DfCMn6=aD4ppoPL3Epn2rdwGb6pxLw@mail.gmail.com">
<div dir="ltr">
<div dir="ltr">I'm not sure that providing a fix to work around
this very broken behavior is the best way of action to make
them fix their server... <br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, Sep 10, 2024 at
5:07 PM Even Rouault via gdal-dev <<a
href="mailto:gdal-dev@lists.osgeo.org"
moz-do-not-send="true" class="moz-txt-link-freetext">gdal-dev@lists.osgeo.org</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote">
<div>
<p><br>
</p>
<div>Le 10/09/2024 à 16:10, Rahkonen Jukka via gdal-dev a
écrit :<br>
</div>
<blockquote type="cite">
<div>
<p class="MsoNormal"><span>Hi,</span></p>
<p class="MsoNormal"><span> </span></p>
<p class="MsoNormal"><span lang="EN-US">Have you tried
with configuration option
“CPL_VSIL_CURL_USE_HEAD=[YES/NO]: Defaults to YES.
Controls whether to use a HEAD request when
opening a remote URL.”</span></p>
</div>
</blockquote>
<p>I was just going to suggest that too. It "works", but
not really. It just postpones the core issue: the server
doesn't support GET Range requests, so can't be used
with /vsicurl/</p>
<p>As it has a COG organization with overview data first
in the file, If you want to read the smallest
overview(s), you can use /vsicurl_streaming/ instead,
but that won't be efficient to read the bottom-right
most tile of the full resoultion late, which will
require reading the whole file...</p>
<p>Nothing GDAL can do about that.</p>
<p>Actually... digging further... it somehow supports
Range requests, but in what I believe a non-compliant
way. It does return the expected content, but returns
HTTP 200 and not HTTP 206 (Partial content). And it
never returns the Content-Length header.</p>
<p>Well, I've implemented a workaround in <a
href="https://github.com/OSGeo/gdal/pull/10760"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://github.com/OSGeo/gdal/pull/10760</a>
that might be useful in other similar cases too.</p>
<p>With that, the following works:</p>
<pre class="gmail-notranslate"><code
class="gmail-notranslate">gdal_translate "/vsicurl?file_size=unlimited&url=<a
href="https://data.source.coop/earthgenome/sentinel2-temporal-mosaics/20NMH_2024-04-01_2024-08-01/B08.tif"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://data.source.coop/earthgenome/sentinel2-temporal-mosaics/20NMH_2024-04-01_2024-08-01/B08.tif</a>" --config GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR out.tif -srcwin 5000 5000 50 50</code></pre>
<p>file_size=unlimited works here since the GTiff driver
doesn't really need to have the right file size, it will
just check we don't try to read beyond at some points,
so unlimited is OK. In other situations/drivers, the
exact value could be needed.<br>
</p>
<p>But they should really fix their servers<br>
</p>
<p>Even<br>
</p>
<blockquote type="cite">
<div>
<p class="MsoNormal"><span lang="EN-US"></span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">-Jukka
Rahkonen-</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<div>
<p class="MsoNormal"><b>Lähettäjä:</b> gdal-dev <a
href="mailto:gdal-dev-bounces@lists.osgeo.org"
target="_blank" moz-do-not-send="true"><gdal-dev-bounces@lists.osgeo.org></a>
<b>Puolesta </b>Daniel Evans via gdal-dev<br>
<b>Lähetetty:</b> tiistai 10. syyskuuta 2024 16.57<br>
<b>Vastaanottaja:</b> '<a
href="mailto:gdal-dev@lists.osgeo.org"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">gdal-dev@lists.osgeo.org</a>'
(<a href="mailto:gdal-dev@lists.osgeo.org"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">gdal-dev@lists.osgeo.org</a>)
<a href="mailto:gdal-dev@lists.osgeo.org"
target="_blank" moz-do-not-send="true"><gdal-dev@lists.osgeo.org></a><br>
<b>Aihe:</b> [gdal-dev] Ignore content-length in
vsicurl?</p>
</div>
<p class="MsoNormal"> </p>
<div>
<p class="MsoNormal">Hi all,</p>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">I am attempting to read a
dataset via /vsicurl/ where I believe the server
is incorrectly returning `content-length: 0` in
response to HEAD requests. This causes GDAL to
believe it's a zero-length file, and it
therefore can't be read.</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">If I download the file via
HTTP GET, it's valid, and GDAL can read it
locally. I've also confirmed I can use /vsicurl/
on some test datasets in the GDAL repo.</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">Is it possible to force GDAL
to work around the faulty content-length header,
or is it too fundamental a problem to ignore?</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">I've separately got in touch
with the data provider to see if they are able
to fix the issue at their end.</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">Cheers,</p>
</div>
<div>
<p class="MsoNormal">Daniel</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">URL of the troublesome
dataset:</p>
</div>
<div>
<p class="MsoNormal"><a
href="https://data.source.coop/earthgenome/sentinel2-temporal-mosaics/20NMH_2024-04-01_2024-08-01/B08.tif"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://data.source.coop/earthgenome/sentinel2-temporal-mosaics/20NMH_2024-04-01_2024-08-01/B08.tif</a></p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">Example HTTP header responses
I'm seeing:</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">GET</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">HTTP/2 200<br>
date: Tue, 10 Sep 2024 13:47:54 GMT<br>
content-type: binary/octet-stream<br>
content-length: 278198294<br>
vary: Origin, Access-Control-Request-Method,
Access-Control-Request-Headers<br>
etag: "a79f3f685281d6681e4d362536c5b3eb-34"<br>
last-modified: Thu, 25 Jul 2024 13:16:08 GMT<br>
x-version: 0.0.16<br>
access-control-allow-credentials: true</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">HEAD</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">HTTP/2 200<br>
date: Tue, 10 Sep 2024 13:48:08 GMT<br>
content-type: binary/octet-stream<br>
content-length: 0<br>
x-version: 0.0.16<br>
access-control-allow-credentials: true<br>
etag: "a79f3f685281d6681e4d362536c5b3eb-34"<br>
last-modified: Thu, 25 Jul 2024 13:16:08 GMT<br>
vary: Origin, Access-Control-Request-Method,
Access-Control-Request-Headers</p>
</div>
</div>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
gdal-dev mailing list
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">gdal-dev@lists.osgeo.org</a>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a>
</pre>
</blockquote>
<pre cols="72">--
<a href="http://www.spatialys.com" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">http://www.spatialys.com</a>
My software is free, but my time generally not.</pre>
</div>
_______________________________________________<br>
gdal-dev mailing list<br>
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">gdal-dev@lists.osgeo.org</a><br>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev"
rel="noreferrer" target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a><br>
</blockquote>
</div>
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
<a class="moz-txt-link-freetext" href="http://www.spatialys.com">http://www.spatialys.com</a>
My software is free, but my time generally not.</pre>
</body>
</html>