<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body>
    <p>Aaron,</p>
    <p>I see a suspicious "* WARNING: failed to save cookies in
      ~/cookies.txt: Failed writing received data to disk/application"
      that comes from curl. I'm not sure if it is due to the ~/ not
      being expanded or something else. This might be dependent of your
      curl version.</p>
    <p>Using --config CPL_VSIL_CURL_USE_HEAD FALSE is also needed for me
      with GDAL master on Ubuntu 20.04 / curl 7.68.0 to make your
      request work. The behavior of the HEAD request with CloudFront is
      quite strange (returning a 206 code)</p>
    <p>If you use the very latest curl 7.79.0, you need to use GDAL
      master (or top of release/3.3 branch) to get
      <a class="moz-txt-link-freetext" href="https://github.com/OSGeo/gdal/pull/4502">https://github.com/OSGeo/gdal/pull/4502</a> , but while testing your
      use case, I realized this fix caused an issue with the
      CPL_VSIL_CURL_USE_HEAD=FALSE case, which is going to be fixed per
      <a class="moz-txt-link-freetext" href="https://github.com/OSGeo/gdal/pull/4578">https://github.com/OSGeo/gdal/pull/4578</a>.</p>
    <p>Even<br>
    </p>
    <div class="moz-cite-prefix">Le 30/09/2021 à 23:20, Aaron Friesz a
      écrit :<br>
    </div>
    <blockquote type="cite"
cite="mid:CAHAWBU4hERm-CGoN47ZPDaM6AT6D+eZMXq7rOLKLwdiyq+6L9A@mail.gmail.com">
      <meta http-equiv="content-type" content="text/html; charset=UTF-8">
      <div dir="ltr">
        <div>Hi,</div>
        <div><br>
        </div>
        <div>I was recently updating the HLS tutorial to access version
          2.0 and discovered that the tutorial is no longer able to
          access the HLS data in the cloud from a local Python
          environment. I've been trying to troubleshoot for a couple
          days now with no positive results. The failure can be boiled
          down to just accessing an HLS COG file using rasterio in
          Python (note, a .netrc file with Earthdata login credentials
          is needed to access data).<br>
          <br>
          ```<br>
          import rasterio as rio<br>
          import os<br>
          <br>
          s30_url = '<a
href="https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TEK.2021261T175011.v2.0/HLS.S30.T13TEK.2021261T175011.v2.0.B04.tif"
            moz-do-not-send="true">https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T13TEK.2021261T175011.v2.0/HLS.S30.T13TEK.2021261T175011.v2.0.B04.tif</a>'<br>
          <br>
          rio_env = rio.Env(GDAL_DISABLE_READDIR_ON_OPEN='EMPTY_DIR',<br>
                            #CPL_VSIL_CURL_ALLOWED_EXTENSIONS='tif',<br>
                            #CPL_VSIL_CURL_USE_HEAD='FALSE',<br>
                            CPL_DEBUG='ON',<br>
                            CPL_CURL_VERBOSE='ON',<br>
                            GDAL_HTTP_UNSAFESSL='YES',<br>
                           
          GDAL_HTTP_COOKIEFILE=os.path.expanduser('~/.edl_cookies'),<br>
                           
          GDAL_HTTP_COOKIEJAR=os.path.expanduser('~/.edl_cookies'))<br>
          <br>
          rio_env.__enter__()<br>
          <br>
          ds = rio.open(s30_url) <br>
          <br>
          ```<br>
          While running a local python env (via conda) the result of the
          rio.open() is a '206 status', which essentially fails the
          command. If I add CPL_VSIL_CURL_USE_HEAD='FALSE' to the rio
          environment, I get a 'file format not supported' error, which
          seems to be a red herring. I'm curious if you've run into this
          type of thing before with COGs in the cloud.</div>
        <div><br>
        </div>
        <div>The same behavior can be observed when just using the gdal
          command line utilities.</div>
        <div>```</div>
        <div>#without --config CPL_VSIL_CURL_USE_HEAD FALSE</div>
        <div>> gdalinfo /vsicurl/<a
href="https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T10TEK.2021192T184511.v2.0/HLS.L30.T10TEK.2021192T184511.v2.0.B04.tif"
            moz-do-not-send="true">https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T10TEK.2021192T184511.v2.0/HLS.L30.T10TEK.2021192T184511.v2.0.B04.tif</a>
          --config GDAL_HTTP_COOKIEFILE ~/cookies.txt --config
          GDAL_HTTP_COOKIEJAR ~/cookies.txt --config
          GDAL_DISABLE_READDIR_ON_OPEN EMPTY_DIR --config
          CPL_CURL_VERBOSE ON</div>
        <div><br>
        </div>
        <div>...</div>
        <div>#output<br>
        </div>
        <div>* Mark bundle as not supporting multiuse<br>
          < HTTP/1.1 206 Partial Content<br>
          < Content-Type: image/tiff<br>
          < Content-Length: 1<br>
          < Connection: keep-alive<br>
          < Date: Thu, 30 Sep 2021 20:52:14 GMT<br>
          < Last-Modified: Tue, 20 Jul 2021 20:27:33 GMT<br>
          < ETag: "85d0b4b089083135d26188a51e187788-1"<br>
          < x-amz-server-side-encryption: AES256<br>
          < Accept-Ranges: bytes<br>
          < Content-Range: bytes 0-0/14178318<br>
          < Server: AmazonS3<br>
          < X-Edge-Origin-Shield-Skipped: 0<br>
          < X-Cache: Miss from cloudfront<br>
          < Via: 1.1 <a
            href="http://130ce7c752c5865952ded89032560b33.cloudfront.net"
            moz-do-not-send="true">130ce7c752c5865952ded89032560b33.cloudfront.net</a>
          (CloudFront)<br>
          < X-Amz-Cf-Pop: MIA3-C3<br>
          < X-Amz-Cf-Id:
          qEq_2x3pF25dN5YMclkgRwhyTozXDexPdaQwOyJaI2VvIKhNnajozQ==<br>
          <<br>
          * Connection #1 to host <a
            href="http://d1nklfio7vscoe.cloudfront.net"
            moz-do-not-send="true">d1nklfio7vscoe.cloudfront.net</a>
          left intact<br>
          ERROR 11: HTTP response code: 206<br>
          gdalinfo failed - unable to open '/vsicurl/<a
href="https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T10TEK.2021192T184511.v2.0/HLS.L30.T10TEK.2021192T184511.v2.0.B04.tif"
            moz-do-not-send="true">https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T10TEK.2021192T184511.v2.0/HLS.L30.T10TEK.2021192T184511.v2.0.B04.tif</a>'.</div>
        <div>```</div>
        <div><br>
        </div>
        <div>```</div>
        <div>
          <div>#with --config CPL_VSIL_CURL_USE_HEAD FALSE</div>
          <div>> gdalinfo /vsicurl/<a
href="https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T10TEK.2021192T184511.v2.0/HLS.L30.T10TEK.2021192T184511.v2.0.B04.tif"
              moz-do-not-send="true">https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T10TEK.2021192T184511.v2.0/HLS.L30.T10TEK.2021192T184511.v2.0.B04.tif</a>
            --config GDAL_HTTP_COOKIEFILE ~/cookies.txt --config
            GDAL_HTTP_COOKIEJAR ~/cookies.txt --config
            GDAL_DISABLE_READDIR_ON_OPEN EMPTY_DIR  --config
            CPL_VSIL_CURL_USE_HEAD FALSE --config CPL_CURL_VERBOSE ON</div>
          <div><br>
          </div>
          <div>...</div>
          <div>#output<br>
          </div>
          <div>* Mark bundle as not supporting multiuse<br>
            < HTTP/1.1 200 OK<br>
            < Content-Type: image/tiff<br>
            < Content-Length: 14178318<br>
            < Connection: keep-alive<br>
            < Date: Thu, 30 Sep 2021 21:12:52 GMT<br>
            < Last-Modified: Tue, 20 Jul 2021 20:27:33 GMT<br>
            < ETag: "85d0b4b089083135d26188a51e187788-1"<br>
            < x-amz-server-side-encryption: AES256<br>
            < Accept-Ranges: bytes<br>
            < Server: AmazonS3<br>
            < X-Edge-Origin-Shield-Skipped: 0<br>
            < X-Cache: Miss from cloudfront<br>
            < Via: 1.1 <a
              href="http://8a771ca27e5a3c9e06b12b7af5d25aa4.cloudfront.net"
              moz-do-not-send="true">8a771ca27e5a3c9e06b12b7af5d25aa4.cloudfront.net</a>
            (CloudFront)<br>
            < X-Amz-Cf-Pop: MIA3-C3<br>
            < X-Amz-Cf-Id:
            Xzv3KWP6glYk68odo5d4KAPnzxsNUFjHRHe90EFXOKNGeYL3ea7jiA==<br>
            * Failed writing header<br>
            * Closing connection 2<br>
            * schannel: shutting down SSL/TLS connection with <a
              href="http://d1nklfio7vscoe.cloudfront.net"
              moz-do-not-send="true">d1nklfio7vscoe.cloudfront.net</a>
            port 443<br>
            * WARNING: failed to save cookies in ~/cookies.txt: Failed
            writing received data to disk/application<br>
            ERROR 4: `/vsicurl/<a
href="https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T10TEK.2021192T184511.v2.0/HLS.L30.T10TEK.2021192T184511.v2.0.B04.tif"
              moz-do-not-send="true">https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T10TEK.2021192T184511.v2.0/HLS.L30.T10TEK.2021192T184511.v2.0.B04.tif</a>'
            not recognized as a supported file format.<br>
            gdalinfo failed - unable to open '/vsicurl/<a
href="https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T10TEK.2021192T184511.v2.0/HLS.L30.T10TEK.2021192T184511.v2.0.B04.tif"
              moz-do-not-send="true">https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T10TEK.2021192T184511.v2.0/HLS.L30.T10TEK.2021192T184511.v2.0.B04.tif</a>'.</div>
          <div>```<br>
          </div>
        </div>
        <div><br>
          Strangely, I'm able to use the original rio/gdal configuration
          options if I'm running the code in a cloud environment, that
          is:</div>
        <div><br>
        </div>
        <div>> gdalinfo /vsicurl/<a
href="https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T10TEK.2021192T184511.v2.0/HLS.L30.T10TEK.2021192T184511.v2.0.B04.tif"
            moz-do-not-send="true">https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T10TEK.2021192T184511.v2.0/HLS.L30.T10TEK.2021192T184511.v2.0.B04.tif</a>
          --config GDAL_HTTP_COOKIEFILE ~/cookies.txt --config
          GDAL_HTTP_COOKIEJAR ~/cookies.txt --config
          GDAL_DISABLE_READDIR_ON_OPEN EMPTY_DIR --config
          CPL_CURL_VERBOSE ON</div>
        <div><br>
        </div>
        <div>...works fine in the USGS Pangeo and Openscapes 2i2c
          instance in us-west2.  I get the failures on both Windows and
          MacOS, but my colleagues seem to be able to run the code just
          fine from linux within a Docker container. <br>
        </div>
        <div><br>
        </div>
        <div>I've tried the gdalinfo commands using the gdal versions
          3.1.4 through 3.2.2. Oddly, v3.2.0 succeeds in printing out
          the gdalinfo content, but I've been unable to create a conda
          environment with that version of gdal.<br>
        </div>
        <div><br>
          I'm talking to our NASA cloud team as well to see if there was
          anything on the backend that changed recently just to cover
          that angle. I'm hoping, though, that I'm missing something
          obvious in the local configurations.<br>
          <br>
          Any guidance would be appreciated.</div>
        <div><br>
        </div>
        <div>-Aaron<br>
        </div>
      </div>
      <br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <pre class="moz-quote-pre" wrap="">_______________________________________________
gdal-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a>
<a class="moz-txt-link-freetext" href="https://lists.osgeo.org/mailman/listinfo/gdal-dev">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a>
</pre>
    </blockquote>
    <pre class="moz-signature" cols="72">-- 
<a class="moz-txt-link-freetext" href="http://www.spatialys.com">http://www.spatialys.com</a>
My software is free, but my time generally not.</pre>
  </body>
</html>