<div dir="ltr"><div>Hi,</div><div><br></div><div>I was recently updating the HLS tutorial to access version 2.0 and discovered that the tutorial is no longer able to access the HLS data in the cloud from a local Python environment. I've been trying to troubleshoot for a couple days now with no positive results. The failure can be boiled down to just accessing an HLS COG file using rasterio in Python (note, a .netrc file with Earthdata login credentials is needed to access data).<br><br>```<br>import rasterio as rio<br>import os<br><br>s30_url = '<a href="https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TEK.2021261T175011.v2.0/HLS.S30.T13TEK.2021261T175011.v2.0.B04.tif">https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TEK.2021261T175011.v2.0/HLS.S30.T13TEK.2021261T175011.v2.0.B04.tif</a>'<br><br>rio_env = rio.Env(GDAL_DISABLE_READDIR_ON_OPEN='EMPTY_DIR',<br> #CPL_VSIL_CURL_ALLOWED_EXTENSIONS='tif',<br> #CPL_VSIL_CURL_USE_HEAD='FALSE',<br> CPL_DEBUG='ON',<br> CPL_CURL_VERBOSE='ON',<br> GDAL_HTTP_UNSAFESSL='YES',<br> GDAL_HTTP_COOKIEFILE=os.path.expanduser('~/.edl_cookies'),<br> GDAL_HTTP_COOKIEJAR=os.path.expanduser('~/.edl_cookies'))<br><br>rio_env.__enter__()<br><br>ds = rio.open(s30_url) <br><br>```<br>While running a local python env (via conda) the result of the rio.open() is a '206 status', which essentially fails the command. If I add CPL_VSIL_CURL_USE_HEAD='FALSE' to the rio environment, I get a 'file format not supported' error, which seems to be a red herring. I'm curious if you've run into this type of thing before with COGs in the cloud.</div><div><br></div><div>The same behavior can be observed when just using the gdal command line utilities.</div><div>```</div><div>#without --config CPL_VSIL_CURL_USE_HEAD FALSE</div><div>> gdalinfo /vsicurl/<a href="https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T10TEK.2021192T184511.v2.0/HLS.L30.T10TEK.2021192T184511.v2.0.B04.tif">https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T10TEK.2021192T184511.v2.0/HLS.L30.T10TEK.2021192T184511.v2.0.B04.tif</a> --config GDAL_HTTP_COOKIEFILE ~/cookies.txt --config GDAL_HTTP_COOKIEJAR ~/cookies.txt --config GDAL_DISABLE_READDIR_ON_OPEN EMPTY_DIR --config CPL_CURL_VERBOSE ON</div><div><br></div><div>...</div><div>#output<br></div><div>* Mark bundle as not supporting multiuse<br>< HTTP/1.1 206 Partial Content<br>< Content-Type: image/tiff<br>< Content-Length: 1<br>< Connection: keep-alive<br>< Date: Thu, 30 Sep 2021 20:52:14 GMT<br>< Last-Modified: Tue, 20 Jul 2021 20:27:33 GMT<br>< ETag: "85d0b4b089083135d26188a51e187788-1"<br>< x-amz-server-side-encryption: AES256<br>< Accept-Ranges: bytes<br>< Content-Range: bytes 0-0/14178318<br>< Server: AmazonS3<br>< X-Edge-Origin-Shield-Skipped: 0<br>< X-Cache: Miss from cloudfront<br>< Via: 1.1 <a href="http://130ce7c752c5865952ded89032560b33.cloudfront.net">130ce7c752c5865952ded89032560b33.cloudfront.net</a> (CloudFront)<br>< X-Amz-Cf-Pop: MIA3-C3<br>< X-Amz-Cf-Id: qEq_2x3pF25dN5YMclkgRwhyTozXDexPdaQwOyJaI2VvIKhNnajozQ==<br><<br>* Connection #1 to host <a href="http://d1nklfio7vscoe.cloudfront.net">d1nklfio7vscoe.cloudfront.net</a> left intact<br>ERROR 11: HTTP response code: 206<br>gdalinfo failed - unable to open '/vsicurl/<a href="https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T10TEK.2021192T184511.v2.0/HLS.L30.T10TEK.2021192T184511.v2.0.B04.tif">https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T10TEK.2021192T184511.v2.0/HLS.L30.T10TEK.2021192T184511.v2.0.B04.tif</a>'.</div><div>```</div><div><br></div><div>```</div><div>
<div>#with --config CPL_VSIL_CURL_USE_HEAD FALSE</div><div>>
gdalinfo
/vsicurl/<a href="https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T10TEK.2021192T184511.v2.0/HLS.L30.T10TEK.2021192T184511.v2.0.B04.tif">https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T10TEK.2021192T184511.v2.0/HLS.L30.T10TEK.2021192T184511.v2.0.B04.tif</a>
--config GDAL_HTTP_COOKIEFILE ~/cookies.txt --config
GDAL_HTTP_COOKIEJAR ~/cookies.txt --config GDAL_DISABLE_READDIR_ON_OPEN
EMPTY_DIR
--config CPL_VSIL_CURL_USE_HEAD FALSE
--config CPL_CURL_VERBOSE ON</div><div><br></div><div>...</div><div>#output<br></div><div>* Mark bundle as not supporting multiuse<br>< HTTP/1.1 200 OK<br>< Content-Type: image/tiff<br>< Content-Length: 14178318<br>< Connection: keep-alive<br>< Date: Thu, 30 Sep 2021 21:12:52 GMT<br>< Last-Modified: Tue, 20 Jul 2021 20:27:33 GMT<br>< ETag: "85d0b4b089083135d26188a51e187788-1"<br>< x-amz-server-side-encryption: AES256<br>< Accept-Ranges: bytes<br>< Server: AmazonS3<br>< X-Edge-Origin-Shield-Skipped: 0<br>< X-Cache: Miss from cloudfront<br>< Via: 1.1 <a href="http://8a771ca27e5a3c9e06b12b7af5d25aa4.cloudfront.net">8a771ca27e5a3c9e06b12b7af5d25aa4.cloudfront.net</a> (CloudFront)<br>< X-Amz-Cf-Pop: MIA3-C3<br>< X-Amz-Cf-Id: Xzv3KWP6glYk68odo5d4KAPnzxsNUFjHRHe90EFXOKNGeYL3ea7jiA==<br>* Failed writing header<br>* Closing connection 2<br>* schannel: shutting down SSL/TLS connection with <a href="http://d1nklfio7vscoe.cloudfront.net">d1nklfio7vscoe.cloudfront.net</a> port 443<br>* WARNING: failed to save cookies in ~/cookies.txt: Failed writing received data to disk/application<br>ERROR 4: `/vsicurl/<a href="https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T10TEK.2021192T184511.v2.0/HLS.L30.T10TEK.2021192T184511.v2.0.B04.tif">https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T10TEK.2021192T184511.v2.0/HLS.L30.T10TEK.2021192T184511.v2.0.B04.tif</a>' not recognized as a supported file format.<br>gdalinfo failed - unable to open '/vsicurl/<a href="https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T10TEK.2021192T184511.v2.0/HLS.L30.T10TEK.2021192T184511.v2.0.B04.tif">https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T10TEK.2021192T184511.v2.0/HLS.L30.T10TEK.2021192T184511.v2.0.B04.tif</a>'.</div><div>```<br></div>
</div><div><br>Strangely, I'm able to use the original rio/gdal configuration options if I'm running the code in a cloud environment, that is:</div><div><br></div><div>> gdalinfo /vsicurl/<a href="https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T10TEK.2021192T184511.v2.0/HLS.L30.T10TEK.2021192T184511.v2.0.B04.tif">https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T10TEK.2021192T184511.v2.0/HLS.L30.T10TEK.2021192T184511.v2.0.B04.tif</a> --config GDAL_HTTP_COOKIEFILE ~/cookies.txt --config GDAL_HTTP_COOKIEJAR ~/cookies.txt --config GDAL_DISABLE_READDIR_ON_OPEN EMPTY_DIR --config CPL_CURL_VERBOSE ON</div><div><br></div><div>...works fine in the USGS Pangeo and Openscapes 2i2c instance in us-west2. I get the failures on both Windows and MacOS, but my colleagues seem to be able to run the code just fine from linux within a Docker container. <br></div><div><br></div><div>I've tried the gdalinfo commands using the gdal versions 3.1.4 through 3.2.2. Oddly, v3.2.0 succeeds in printing out the gdalinfo content, but I've been unable to create a conda environment with that version of gdal.<br></div><div><br>I'm talking to our NASA cloud team as well to see if there was anything on the backend that changed recently just to cover that angle. I'm hoping, though, that I'm missing something obvious in the local configurations.<br><br>Any guidance would be appreciated.</div><div><br></div><div>-Aaron<br></div></div>