<div dir="ltr"><div>Hey Even,</div><div><br></div><div>Sorry, yeah, the cookies warning goes away if I add the full path. The rest of the log stays the same though.</div><div><br></div><div>-Aaron<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Oct 1, 2021 at 6:09 AM Even Rouault <<a href="mailto:even.rouault@spatialys.com">even.rouault@spatialys.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Aaron,</p>
<p>I see a suspicious "* WARNING: failed to save cookies in
~/cookies.txt: Failed writing received data to disk/application"
that comes from curl. I'm not sure if it is due to the ~/ not
being expanded or something else. This might be dependent of your
curl version.</p>
<p>Using --config CPL_VSIL_CURL_USE_HEAD FALSE is also needed for me
with GDAL master on Ubuntu 20.04 / curl 7.68.0 to make your
request work. The behavior of the HEAD request with CloudFront is
quite strange (returning a 206 code)</p>
<p>If you use the very latest curl 7.79.0, you need to use GDAL
master (or top of release/3.3 branch) to get
<a href="https://github.com/OSGeo/gdal/pull/4502" target="_blank">https://github.com/OSGeo/gdal/pull/4502</a> , but while testing your
use case, I realized this fix caused an issue with the
CPL_VSIL_CURL_USE_HEAD=FALSE case, which is going to be fixed per
<a href="https://github.com/OSGeo/gdal/pull/4578" target="_blank">https://github.com/OSGeo/gdal/pull/4578</a>.</p>
<p>Even<br>
</p>
<div>Le 30/09/2021 à 23:20, Aaron Friesz a
écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Hi,</div>
<div><br>
</div>
<div>I was recently updating the HLS tutorial to access version
2.0 and discovered that the tutorial is no longer able to
access the HLS data in the cloud from a local Python
environment. I've been trying to troubleshoot for a couple
days now with no positive results. The failure can be boiled
down to just accessing an HLS COG file using rasterio in
Python (note, a .netrc file with Earthdata login credentials
is needed to access data).<br>
<br>
```<br>
import rasterio as rio<br>
import os<br>
<br>
s30_url = '<a href="https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TEK.2021261T175011.v2.0/HLS.S30.T13TEK.2021261T175011.v2.0.B04.tif" target="_blank">https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TEK.2021261T175011.v2.0/HLS.S30.T13TEK.2021261T175011.v2.0.B04.tif</a>'<br>
<br>
rio_env = rio.Env(GDAL_DISABLE_READDIR_ON_OPEN='EMPTY_DIR',<br>
#CPL_VSIL_CURL_ALLOWED_EXTENSIONS='tif',<br>
#CPL_VSIL_CURL_USE_HEAD='FALSE',<br>
CPL_DEBUG='ON',<br>
CPL_CURL_VERBOSE='ON',<br>
GDAL_HTTP_UNSAFESSL='YES',<br>
GDAL_HTTP_COOKIEFILE=os.path.expanduser('~/.edl_cookies'),<br>
GDAL_HTTP_COOKIEJAR=os.path.expanduser('~/.edl_cookies'))<br>
<br>
rio_env.__enter__()<br>
<br>
ds = rio.open(s30_url) <br>
<br>
```<br>
While running a local python env (via conda) the result of the
rio.open() is a '206 status', which essentially fails the
command. If I add CPL_VSIL_CURL_USE_HEAD='FALSE' to the rio
environment, I get a 'file format not supported' error, which
seems to be a red herring. I'm curious if you've run into this
type of thing before with COGs in the cloud.</div>
<div><br>
</div>
<div>The same behavior can be observed when just using the gdal
command line utilities.</div>
<div>```</div>
<div>#without --config CPL_VSIL_CURL_USE_HEAD FALSE</div>
<div>> gdalinfo /vsicurl/<a href="https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T10TEK.2021192T184511.v2.0/HLS.L30.T10TEK.2021192T184511.v2.0.B04.tif" target="_blank">https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T10TEK.2021192T184511.v2.0/HLS.L30.T10TEK.2021192T184511.v2.0.B04.tif</a>
--config GDAL_HTTP_COOKIEFILE ~/cookies.txt --config
GDAL_HTTP_COOKIEJAR ~/cookies.txt --config
GDAL_DISABLE_READDIR_ON_OPEN EMPTY_DIR --config
CPL_CURL_VERBOSE ON</div>
<div><br>
</div>
<div>...</div>
<div>#output<br>
</div>
<div>* Mark bundle as not supporting multiuse<br>
< HTTP/1.1 206 Partial Content<br>
< Content-Type: image/tiff<br>
< Content-Length: 1<br>
< Connection: keep-alive<br>
< Date: Thu, 30 Sep 2021 20:52:14 GMT<br>
< Last-Modified: Tue, 20 Jul 2021 20:27:33 GMT<br>
< ETag: "85d0b4b089083135d26188a51e187788-1"<br>
< x-amz-server-side-encryption: AES256<br>
< Accept-Ranges: bytes<br>
< Content-Range: bytes 0-0/14178318<br>
< Server: AmazonS3<br>
< X-Edge-Origin-Shield-Skipped: 0<br>
< X-Cache: Miss from cloudfront<br>
< Via: 1.1 <a href="http://130ce7c752c5865952ded89032560b33.cloudfront.net" target="_blank">130ce7c752c5865952ded89032560b33.cloudfront.net</a>
(CloudFront)<br>
< X-Amz-Cf-Pop: MIA3-C3<br>
< X-Amz-Cf-Id:
qEq_2x3pF25dN5YMclkgRwhyTozXDexPdaQwOyJaI2VvIKhNnajozQ==<br>
<<br>
* Connection #1 to host <a href="http://d1nklfio7vscoe.cloudfront.net" target="_blank">d1nklfio7vscoe.cloudfront.net</a>
left intact<br>
ERROR 11: HTTP response code: 206<br>
gdalinfo failed - unable to open '/vsicurl/<a href="https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T10TEK.2021192T184511.v2.0/HLS.L30.T10TEK.2021192T184511.v2.0.B04.tif" target="_blank">https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T10TEK.2021192T184511.v2.0/HLS.L30.T10TEK.2021192T184511.v2.0.B04.tif</a>'.</div>
<div>```</div>
<div><br>
</div>
<div>```</div>
<div>
<div>#with --config CPL_VSIL_CURL_USE_HEAD FALSE</div>
<div>> gdalinfo /vsicurl/<a href="https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T10TEK.2021192T184511.v2.0/HLS.L30.T10TEK.2021192T184511.v2.0.B04.tif" target="_blank">https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T10TEK.2021192T184511.v2.0/HLS.L30.T10TEK.2021192T184511.v2.0.B04.tif</a>
--config GDAL_HTTP_COOKIEFILE ~/cookies.txt --config
GDAL_HTTP_COOKIEJAR ~/cookies.txt --config
GDAL_DISABLE_READDIR_ON_OPEN EMPTY_DIR --config
CPL_VSIL_CURL_USE_HEAD FALSE --config CPL_CURL_VERBOSE ON</div>
<div><br>
</div>
<div>...</div>
<div>#output<br>
</div>
<div>* Mark bundle as not supporting multiuse<br>
< HTTP/1.1 200 OK<br>
< Content-Type: image/tiff<br>
< Content-Length: 14178318<br>
< Connection: keep-alive<br>
< Date: Thu, 30 Sep 2021 21:12:52 GMT<br>
< Last-Modified: Tue, 20 Jul 2021 20:27:33 GMT<br>
< ETag: "85d0b4b089083135d26188a51e187788-1"<br>
< x-amz-server-side-encryption: AES256<br>
< Accept-Ranges: bytes<br>
< Server: AmazonS3<br>
< X-Edge-Origin-Shield-Skipped: 0<br>
< X-Cache: Miss from cloudfront<br>
< Via: 1.1 <a href="http://8a771ca27e5a3c9e06b12b7af5d25aa4.cloudfront.net" target="_blank">8a771ca27e5a3c9e06b12b7af5d25aa4.cloudfront.net</a>
(CloudFront)<br>
< X-Amz-Cf-Pop: MIA3-C3<br>
< X-Amz-Cf-Id:
Xzv3KWP6glYk68odo5d4KAPnzxsNUFjHRHe90EFXOKNGeYL3ea7jiA==<br>
* Failed writing header<br>
* Closing connection 2<br>
* schannel: shutting down SSL/TLS connection with <a href="http://d1nklfio7vscoe.cloudfront.net" target="_blank">d1nklfio7vscoe.cloudfront.net</a>
port 443<br>
* WARNING: failed to save cookies in ~/cookies.txt: Failed
writing received data to disk/application<br>
ERROR 4: `/vsicurl/<a href="https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T10TEK.2021192T184511.v2.0/HLS.L30.T10TEK.2021192T184511.v2.0.B04.tif" target="_blank">https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T10TEK.2021192T184511.v2.0/HLS.L30.T10TEK.2021192T184511.v2.0.B04.tif</a>'
not recognized as a supported file format.<br>
gdalinfo failed - unable to open '/vsicurl/<a href="https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T10TEK.2021192T184511.v2.0/HLS.L30.T10TEK.2021192T184511.v2.0.B04.tif" target="_blank">https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T10TEK.2021192T184511.v2.0/HLS.L30.T10TEK.2021192T184511.v2.0.B04.tif</a>'.</div>
<div>```<br>
</div>
</div>
<div><br>
Strangely, I'm able to use the original rio/gdal configuration
options if I'm running the code in a cloud environment, that
is:</div>
<div><br>
</div>
<div>> gdalinfo /vsicurl/<a href="https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T10TEK.2021192T184511.v2.0/HLS.L30.T10TEK.2021192T184511.v2.0.B04.tif" target="_blank">https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T10TEK.2021192T184511.v2.0/HLS.L30.T10TEK.2021192T184511.v2.0.B04.tif</a>
--config GDAL_HTTP_COOKIEFILE ~/cookies.txt --config
GDAL_HTTP_COOKIEJAR ~/cookies.txt --config
GDAL_DISABLE_READDIR_ON_OPEN EMPTY_DIR --config
CPL_CURL_VERBOSE ON</div>
<div><br>
</div>
<div>...works fine in the USGS Pangeo and Openscapes 2i2c
instance in us-west2. I get the failures on both Windows and
MacOS, but my colleagues seem to be able to run the code just
fine from linux within a Docker container. <br>
</div>
<div><br>
</div>
<div>I've tried the gdalinfo commands using the gdal versions
3.1.4 through 3.2.2. Oddly, v3.2.0 succeeds in printing out
the gdalinfo content, but I've been unable to create a conda
environment with that version of gdal.<br>
</div>
<div><br>
I'm talking to our NASA cloud team as well to see if there was
anything on the backend that changed recently just to cover
that angle. I'm hoping, though, that I'm missing something
obvious in the local configurations.<br>
<br>
Any guidance would be appreciated.</div>
<div><br>
</div>
<div>-Aaron<br>
</div>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
gdal-dev mailing list
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a>
</pre>
</blockquote>
<pre cols="72">--
<a href="http://www.spatialys.com" target="_blank">http://www.spatialys.com</a>
My software is free, but my time generally not.</pre>
</div>
_______________________________________________<br>
gdal-dev mailing list<br>
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a><br>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" rel="noreferrer" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a><br>
</blockquote></div>