<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p><br>
</p>
<div class="moz-cite-prefix">Le 19/11/2022 à 16:00,
<a class="moz-txt-link-abbreviated" href="mailto:michael.smith.erdc@gmail.com">michael.smith.erdc@gmail.com</a> a écrit :<br>
</div>
<blockquote type="cite"
cite="mid:D84600C8-F681-4BA9-B610-FDBA35DBA758@gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
Correct, not a public bucket, which is why the IAM credentials are
needed. If I set them manually, it all works fine.</blockquote>
<p>That's super weird if the result of a range request changes
depending on how credentials have been set... Perhaps enable
CPL_CURL_VERBOSE=ON env variable and diff the logs ?</p>
<p>You could also try the gdal_cp.py sample script at
<a class="moz-txt-link-freetext" href="https://github.com/OSGeo/gdal/blob/master/swig/python/gdal-utils/osgeo_utils/samples/gdal_cp.py">https://github.com/OSGeo/gdal/blob/master/swig/python/gdal-utils/osgeo_utils/samples/gdal_cp.py</a>
, which is a cp-like utility working with GDAL virtual file
systems, with the 2 authentication methods<br>
</p>
<p>python gdal_cp.py
/vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif out.tif</p>
<p>(you can interrupt it with ctrl-c after a few seconds. that will
be enough to get the first bytes)<br>
</p>
<p>you might need to run an hexadecimal editor to inspect a bit the
content.<br>
</p>
<blockquote type="cite"
cite="mid:D84600C8-F681-4BA9-B610-FDBA35DBA758@gmail.com">
<div><br>
</div>
<div>
<div><font face="Courier New"><span style="font-style: normal;
font-size: 15px;">[ u02]$ export AWS_ACCESS_KEY_ID=xxxxx</span></font></div>
<div><font face="Courier New"><span style="font-style: normal;
font-size: 15px;">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8">
<p>Yes, a 206 response code means success here as we are
requesting only bytes 0-16383. So maybe the file is not
a valid TIFF ?</p>
<p>( "grid-dev-publiclidar" must not be so public I guess,
because when trying with my credentials, I get a Access
Denied)</p>
<div class="moz-cite-prefix">Le 19/11/2022 à 15:40, <a
class="moz-txt-link-abbreviated moz-txt-link-freetext"
href="mailto:michael.smith.erdc@gmail.com"
moz-do-not-send="true">michael.smith.erdc@gmail.com</a>
a écrit :<br>
</div>
<blockquote type="cite"
cite="mid:8F735EF0-188A-4F70-A9C7-6B3CD9ACD718@gmail.com">
<meta http-equiv="content-type" content="text/html;
charset=UTF-8">
I’m seeing that it’s getting a 206 response code, so
wouldn’t that indicate auth is working?
<div><br>
</div>
<div>
<div><font size="3" face="Courier New"><span
style="font-style: normal;"> gdalinfo
/vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif</span></font></div>
<div><font size="3" face="Courier New"><span
style="font-style: normal;">HTTP: Fetch(<a
class="moz-txt-link-freetext"
href="http://169.254.169.254/latest/api/token"
moz-do-not-send="true">http://169.254.169.254/latest/api/token</a>)</span></font></div>
<div><font size="3" face="Courier New"><span
style="font-style: normal;">HTTP: libcurl/7.86.0
OpenSSL/3.0.7 zlib/1.2.13 libssh2/1.10.0
nghttp2/1.47.0</span></font></div>
<div><font size="3" face="Courier New"><span
style="font-style: normal;">HTTP: These HTTP
headers were set:
X-aws-ec2-metadata-token-ttl-seconds: 10</span></font></div>
<div><font size="3" face="Courier New"><span
style="font-style: normal;">HTTP: Fetch(<a
class="moz-txt-link-freetext"
href="http://169.254.169.254/latest/meta-data/iam/security-credentials/"
moz-do-not-send="true">http://169.254.169.254/latest/meta-data/iam/security-credentials/</a>)</span></font></div>
<div><font size="3" face="Courier New"><span
style="font-style: normal;">HTTP:
Fetch(<a class="moz-txt-link-freetext"
href="http://169.254.169.254/latest/meta-data/iam/security-credentials/iam-grid-s3"
moz-do-not-send="true">http://169.254.169.254/latest/meta-data/iam/security-credentials/iam-grid-s3</a>)</span></font></div>
<div><font size="3" face="Courier New"><span
style="font-style: normal;">AWS: Storing AIM
credentials until 2022-11-19T20:42:58Z</span></font></div>
<div><font size="3" face="Courier New"><span
style="font-style: normal;">S3: Downloading
0-16383
(<a class="moz-txt-link-freetext"
href="https://grid-dev-publiclidar.s3.amazonaws.com/estonia/dtm/estonia_dtm_5m.tif"
moz-do-not-send="true">https://grid-dev-publiclidar.s3.amazonaws.com/estonia/dtm/estonia_dtm_5m.tif</a>)...</span></font></div>
<div><font size="3" face="Courier New"><span
style="font-style: normal;">S3: Got
response_code=206</span></font></div>
<div><font size="3" face="Courier New"><span
style="font-style: normal;">gdalinfo failed -
unable to open
'/vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif'.</span></font></div>
<div><br>
</div>
<br>
<div dir="ltr">Mike</div>
<div dir="ltr"><br>
</div>
<div dir="ltr"><br>
</div>
<div dir="ltr"><br>
<blockquote type="cite">On Nov 19, 2022, at 9:26 AM,
Even Rouault <a class="moz-txt-link-rfc2396E"
href="mailto:even.rouault@spatialys.com"
moz-do-not-send="true"><even.rouault@spatialys.com></a>
wrote:<br>
<br>
</blockquote>
</div>
<blockquote type="cite">
<div dir="ltr"><span>Hi Mike,</span><br>
<span></span><br>
<span>could you send the output of</span><br>
<span></span><br>
<span>curl
<a class="moz-txt-link-freetext"
href="http://169.254.169.254/latest/meta-data/iam/security-credentials/iam-grid-s3"
moz-do-not-send="true">http://169.254.169.254/latest/meta-data/iam/security-credentials/iam-grid-s3</a></span><br>
<span></span><br>
<span>Slightly redacted of course, but with the
exact formatting. This part of thee code
currently uses a "simple JSON parser"
(<a class="moz-txt-link-freetext"
href="https://github.com/OSGeo/gdal/blob/c61d116a469821b769630a112dee7f1a61fed885/port/cpl_aws.cpp#L554"
moz-do-not-send="true">https://github.com/OSGeo/gdal/blob/c61d116a469821b769630a112dee7f1a61fed885/port/cpl_aws.cpp#L554</a>),
which is actually just a non JSON-aware string
tokenizer, and I suspect it could be defeated by
a new formatting of S3 or something specific to
your credentials.</span><br>
<span></span><br>
<span>It could also be that something unhandled by
that parser appears inside quoted strings, like
an escaped double quote or some other JSON
escaped character (like an escaped forward slash
\/ )</span><br>
<span></span><br>
<span>If that was the case we should likely switch
to proper JSON deserialization (that part of the
code must predate libjson-c being a build
requirement of GDAL).</span><br>
<span></span><br>
<span>Even</span><br>
<span></span><br>
<span></span><br>
<span>-- </span><br>
<span><a class="moz-txt-link-freetext"
href="http://www.spatialys.com"
moz-do-not-send="true">http://www.spatialys.com</a></span><br>
<span>My software is free, but my time generally
not.</span><br>
<span></span><br>
</div>
</blockquote>
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
<a class="moz-txt-link-freetext" href="http://www.spatialys.com" moz-do-not-send="true">http://www.spatialys.com</a>
My software is free, but my time generally not.</pre>
</span></font></div>
</div>
<font face="Courier New">
</font></blockquote>
<pre class="moz-signature" cols="72">--
<a class="moz-txt-link-freetext" href="http://www.spatialys.com">http://www.spatialys.com</a>
My software is free, but my time generally not.</pre>
</body>
</html>