<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p><br>
</p>
<div class="moz-cite-prefix">Le 10/09/2024 à 16:10, Rahkonen Jukka
via gdal-dev a écrit :<br>
</div>
<blockquote type="cite"
cite="mid:DB9PR09MB6887A6E392A9DDB7EB558AB3FD9A2@DB9PR09MB6887.eurprd09.prod.outlook.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="Generator"
content="Microsoft Word 15 (filtered medium)">
<style>@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}span.Shkpostityyli18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}.MsoChpDefault
{mso-style-type:export-only;
mso-ligatures:none;
mso-fareast-language:EN-US;}div.WordSection1
{page:WordSection1;}</style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Hi,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"
lang="EN-US">Have you tried with configuration option
“CPL_VSIL_CURL_USE_HEAD=[YES/NO]: Defaults to YES. Controls
whether to use a HEAD request when opening a remote URL.”</span></p>
</div>
</blockquote>
<p>I was just going to suggest that too. It "works", but not really.
It just postpones the core issue: the server doesn't support GET
Range requests, so can't be used with /vsicurl/</p>
<p>As it has a COG organization with overview data first in the
file, If you want to read the smallest overview(s), you can use
/vsicurl_streaming/ instead, but that won't be efficient to read
the bottom-right most tile of the full resoultion late, which will
require reading the whole file...</p>
<p>Nothing GDAL can do about that.</p>
<p>Actually... digging further... it somehow supports Range
requests, but in what I believe a non-compliant way. It does
return the expected content, but returns HTTP 200 and not HTTP 206
(Partial content). And it never returns the Content-Length header.</p>
<p>Well, I've implemented a workaround in
<a class="moz-txt-link-freetext" href="https://github.com/OSGeo/gdal/pull/10760">https://github.com/OSGeo/gdal/pull/10760</a> that might be useful in
other similar cases too.</p>
<p>With that, the following works:</p>
<pre class="notranslate"><code class="notranslate">gdal_translate "/vsicurl?file_size=unlimited&url=<a class="moz-txt-link-freetext" href="https://data.source.coop/earthgenome/sentinel2-temporal-mosaics/20NMH_2024-04-01_2024-08-01/B08.tif">https://data.source.coop/earthgenome/sentinel2-temporal-mosaics/20NMH_2024-04-01_2024-08-01/B08.tif</a>" --config GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR out.tif -srcwin 5000 5000 50 50</code></pre>
<p></p>
<p>file_size=unlimited works here since the GTiff driver doesn't
really need to have the right file size, it will just check we
don't try to read beyond at some points, so unlimited is OK. In
other situations/drivers, the exact value could be needed.<br>
</p>
<p>But they should really fix their servers<br>
</p>
<p>Even<br>
</p>
<blockquote type="cite"
cite="mid:DB9PR09MB6887A6E392A9DDB7EB558AB3FD9A2@DB9PR09MB6887.eurprd09.prod.outlook.com">
<div class="WordSection1">
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"
lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"
lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"
lang="EN-US">-Jukka Rahkonen-<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"
lang="EN-US"><o:p> </o:p></span></p>
<div
style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b>Lähettäjä:</b> gdal-dev
<a class="moz-txt-link-rfc2396E" href="mailto:gdal-dev-bounces@lists.osgeo.org"><gdal-dev-bounces@lists.osgeo.org></a>
<b>Puolesta </b>Daniel Evans via gdal-dev<br>
<b>Lähetetty:</b> tiistai 10. syyskuuta 2024 16.57<br>
<b>Vastaanottaja:</b> '<a class="moz-txt-link-abbreviated" href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a>'
(<a class="moz-txt-link-abbreviated" href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a>) <a class="moz-txt-link-rfc2396E" href="mailto:gdal-dev@lists.osgeo.org"><gdal-dev@lists.osgeo.org></a><br>
<b>Aihe:</b> [gdal-dev] Ignore content-length in vsicurl?<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Hi all,<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I am attempting to read a dataset via
/vsicurl/ where I believe the server is incorrectly
returning `content-length: 0` in response to HEAD
requests. This causes GDAL to believe it's a zero-length
file, and it therefore can't be read.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">If I download the file via HTTP GET,
it's valid, and GDAL can read it locally. I've also
confirmed I can use /vsicurl/ on some test datasets in the
GDAL repo.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Is it possible to force GDAL to work
around the faulty content-length header, or is it too
fundamental a problem to ignore?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I've separately got in touch with the
data provider to see if they are able to fix the issue at
their end.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Cheers,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Daniel<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">URL of the troublesome dataset:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a
href="https://data.source.coop/earthgenome/sentinel2-temporal-mosaics/20NMH_2024-04-01_2024-08-01/B08.tif"
moz-do-not-send="true" class="moz-txt-link-freetext">https://data.source.coop/earthgenome/sentinel2-temporal-mosaics/20NMH_2024-04-01_2024-08-01/B08.tif</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Example HTTP header responses I'm
seeing:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">GET<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">HTTP/2 200<br>
date: Tue, 10 Sep 2024 13:47:54 GMT<br>
content-type: binary/octet-stream<br>
content-length: 278198294<br>
vary: Origin, Access-Control-Request-Method,
Access-Control-Request-Headers<br>
etag: "a79f3f685281d6681e4d362536c5b3eb-34"<br>
last-modified: Thu, 25 Jul 2024 13:16:08 GMT<br>
x-version: 0.0.16<br>
access-control-allow-credentials: true<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">HEAD<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">HTTP/2 200<br>
date: Tue, 10 Sep 2024 13:48:08 GMT<br>
content-type: binary/octet-stream<br>
content-length: 0<br>
x-version: 0.0.16<br>
access-control-allow-credentials: true<br>
etag: "a79f3f685281d6681e4d362536c5b3eb-34"<br>
last-modified: Thu, 25 Jul 2024 13:16:08 GMT<br>
vary: Origin, Access-Control-Request-Method,
Access-Control-Request-Headers<o:p></o:p></p>
</div>
</div>
</div>
<br>
<fieldset class="moz-mime-attachment-header"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
gdal-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a>
<a class="moz-txt-link-freetext" href="https://lists.osgeo.org/mailman/listinfo/gdal-dev">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a>
</pre>
</blockquote>
<pre class="moz-signature" cols="72">--
<a class="moz-txt-link-freetext" href="http://www.spatialys.com">http://www.spatialys.com</a>
My software is free, but my time generally not.</pre>
</body>
</html>