<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Mike,</p>
<p>I've ended up firing a EC2 instance and I did replicate with my
private bucket too.</p>
<p>With a EC2 Ubuntu 22.04 instance, on a minimal GDAL master debug
build, things work fine with IAM authentication, in the VM itself</p>
<p>But if I run inside a ubuntu:22.04 Docker image the same build
(mounting my home directory), then I got exactly the same symptoms
as you!</p>
<p>I finally figured out that the "curl -X PUT
<a class="moz-txt-link-rfc2396E" href="http://169.254.169.254/latest/api/token">"http://169.254.169.254/latest/api/token"</a> -H
"X-aws-ec2-metadata-token-ttl-seconds: 10" request for IDMSv2 was
failing inside Docker but not in the outside EC2 VM. Known
(intended) issue according to <a
href="https://community.grafana.com/t/imdsv2-is-not-working-from-docker/65944"
rel="nofollow" class="moz-txt-link-freetext">https://community.grafana.com/t/imdsv2-is-not-working-from-docker/65944</a>
</p>
<p>The solution/workaround is to run docker with --network=host (I
guess it could be possible to restrict it a bit to just
169.254.169.254)</p>
<p>In <a class="moz-txt-link-freetext" href="https://github.com/OSGeo/gdal/pull/6752">https://github.com/OSGeo/gdal/pull/6752</a>, I've pushed a proper
fix, that makes sure that the fallback to IDMSv1 works by clearing
the GDAL error state (which was the reason for the weird behaviour
that GDALOpen() only worked the second time), and emits a
hopefully debug hint when it sees that IDMSv2 fails on timeout
inside a Docker container.</p>
<p>With the fix, when trying inside a Docker with default networking
rules:<br>
</p>
<p># gdalinfo /vsis3/XXXXXXX/byte.tif --debug on<br>
AWS: AWS_ROLE_ARN configuration option not defined<br>
HTTP: Fetch(<a class="moz-txt-link-freetext" href="http://169.254.169.254/latest/api/token">http://169.254.169.254/latest/api/token</a>)<br>
HTTP: libcurl/7.81.0 GnuTLS/3.7.3 zlib/1.2.11 brotli/1.0.9
zstd/1.4.8 libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2)
libssh/0.9.6/openssl/zlib nghttp2/1.43.0 librtmp/2.3
OpenLDAP/2.5.13<br>
HTTP: Fetch(<a class="moz-txt-link-freetext" href="http://169.254.169.254/latest/meta-data">http://169.254.169.254/latest/meta-data</a>)<br>
S3: /latest/api/token EC2 IDMSv2 request timed out, but
/latest/metadata succeeded. Trying with IDMSv1. Try running your
Docker container with --network=host.<br>
HTTP:
Fetch(<a class="moz-txt-link-freetext" href="http://169.254.169.254/latest/meta-data/iam/security-credentials/">http://169.254.169.254/latest/meta-data/iam/security-credentials/</a>)<br>
HTTP:
Fetch(<a class="moz-txt-link-freetext" href="http://169.254.169.254/latest/meta-data/iam/security-credentials/ec2_s3">http://169.254.169.254/latest/meta-data/iam/security-credentials/ec2_s3</a>)<br>
AWS: Storing AIM credentials until 2022-11-20T23:38:10Z<br>
S3: Switching to region eu-central-1<br>
S3: Downloading 0-1796
(<a class="moz-txt-link-freetext" href="https://XXXXXXX.s3.amazonaws.com/byte.tif">https://XXXXXXX.s3.amazonaws.com/byte.tif</a>)...<br>
S3: Got response_code=206<br>
GDAL: GDALOpen(/vsis3/XXXXXXX/byte.tif, this=0x5599cfbce470)
succeeds as GTiff.<br>
</p>
<p>It was an "interesting" bug...<br>
</p>
<p>Even<br>
</p>
<div class="moz-cite-prefix">Le 20/11/2022 à 14:47, Michael Smith a
écrit :<br>
</div>
<blockquote type="cite"
cite="mid:EA855DA6-53F8-446E-97CA-ABF415E0B5AB@gmail.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style>@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
font-size:10.0pt;
font-family:"Courier New",serif;}span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}div.WordSection1
{page:WordSection1;}</style>
<div class="WordSection1">
<p class="MsoNormal">Is there a reason why OpenEx would work but
Open wouldn’t?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">In [1]: from osgeo import gdal<o:p></o:p></p>
<p class="MsoNormal">In [2]: pszFilename =
"/vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif"<o:p></o:p></p>
<p class="MsoNormal">In [3]: hDataset = gdal.Open(pszFilename,
gdal.GA_ReadOnly)<o:p></o:p></p>
<p class="MsoNormal">In [4]: hDataset<o:p></o:p></p>
<p class="MsoNormal">In [5]: hDataset = gdal.OpenEx(pszFilename,
gdal.GA_ReadOnly)<o:p></o:p></p>
<p class="MsoNormal">In [6]: hDataset<o:p></o:p></p>
<p class="MsoNormal">Out[6]: <osgeo.gdal.Dataset; proxy of
<Swig Object of type 'GDALDatasetShadow *' at
0x7f827217c450> ><o:p></o:p></p>
<p class="MsoNormal">In [7]: hDataset.GetGeoTransform()<o:p></o:p></p>
<p class="MsoNormal">Out[7]: (365000.0, 5.0, 0.0, 6635000.0,
0.0, -5.0)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Mike<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span
style="font-size:12.0pt;color:black">From: </span></b><span
style="font-size:12.0pt;color:black">Even Rouault
<a class="moz-txt-link-rfc2396E" href="mailto:even.rouault@spatialys.com"><even.rouault@spatialys.com></a><br>
<b>Date: </b>Saturday, November 19, 2022 at 10:08 AM<br>
<b>To: </b><a class="moz-txt-link-rfc2396E" href="mailto:michael.smith.erdc@gmail.com"><michael.smith.erdc@gmail.com></a><br>
<b>Cc: </b>gdal-dev <a class="moz-txt-link-rfc2396E" href="mailto:gdal-dev@lists.osgeo.org"><gdal-dev@lists.osgeo.org></a><br>
<b>Subject: </b>Re: [gdal-dev] errors using IAM instance
profile auth in s3<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p><o:p> </o:p></p>
<div>
<p class="MsoNormal">Le 19/11/2022 à 16:00, <a
href="mailto:michael.smith.erdc@gmail.com"
moz-do-not-send="true" class="moz-txt-link-freetext">michael.smith.erdc@gmail.com</a>
a écrit :<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">Correct, not a public bucket, which is
why the IAM credentials are needed. If I set them manually,
it all works fine.<o:p></o:p></p>
</blockquote>
<p>That's super weird if the result of a range request changes
depending on how credentials have been set... Perhaps enable
CPL_CURL_VERBOSE=ON env variable and diff the logs ?<o:p></o:p></p>
<p>You could also try the gdal_cp.py sample script at <a
href="https://github.com/OSGeo/gdal/blob/master/swig/python/gdal-utils/osgeo_utils/samples/gdal_cp.py"
moz-do-not-send="true" class="moz-txt-link-freetext">https://github.com/OSGeo/gdal/blob/master/swig/python/gdal-utils/osgeo_utils/samples/gdal_cp.py</a>
, which is a cp-like utility working with GDAL virtual file
systems, with the 2 authentication methods<o:p></o:p></p>
<p>python gdal_cp.py
/vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif
out.tif<o:p></o:p></p>
<p>(you can interrupt it with ctrl-c after a few seconds. that
will be enough to get the first bytes)<o:p></o:p></p>
<p>you might need to run an hexadecimal editor to inspect a bit
the content.<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><span
style="font-size:11.5pt;font-family:"Courier
New",serif">[ u02]$ export
AWS_ACCESS_KEY_ID=xxxxx</span><o:p></o:p></p>
</div>
<div>
<p><span style="font-size:11.5pt;font-family:"Courier
New",serif">Yes, a 206 response code means
success here as we are requesting only bytes 0-16383.
So maybe the file is not a valid TIFF ?<o:p></o:p></span></p>
<p><span style="font-size:11.5pt;font-family:"Courier
New",serif">( "grid-dev-publiclidar" must not be
so public I guess, because when trying with my
credentials, I get a Access Denied)<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span
style="font-size:11.5pt;font-family:"Courier
New",serif">Le 19/11/2022 à 15:40, <a
href="mailto:michael.smith.erdc@gmail.com"
moz-do-not-send="true"
class="moz-txt-link-freetext">michael.smith.erdc@gmail.com</a>
a écrit :<o:p></o:p></span></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span
style="font-size:11.5pt;font-family:"Courier
New",serif">I’m seeing that it’s getting a 206
response code, so wouldn’t that indicate auth is
working? <o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span
style="font-size:11.5pt;font-family:"Courier
New",serif"><o:p> </o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Courier
New",serif"> gdalinfo
/vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif</span><span
style="font-size:11.5pt;font-family:"Courier New",serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Courier
New",serif">HTTP: Fetch(<a
href="http://169.254.169.254/latest/api/token"
moz-do-not-send="true"
class="moz-txt-link-freetext">http://169.254.169.254/latest/api/token</a>)</span><span
style="font-size:11.5pt;font-family:"Courier New",serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Courier
New",serif">HTTP: libcurl/7.86.0
OpenSSL/3.0.7 zlib/1.2.13 libssh2/1.10.0
nghttp2/1.47.0</span><span
style="font-size:11.5pt;font-family:"Courier
New",serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Courier
New",serif">HTTP: These HTTP headers were
set: X-aws-ec2-metadata-token-ttl-seconds: 10</span><span
style="font-size:11.5pt;font-family:"Courier New",serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Courier
New",serif">HTTP: Fetch(<a
href="http://169.254.169.254/latest/meta-data/iam/security-credentials/"
moz-do-not-send="true"
class="moz-txt-link-freetext">http://169.254.169.254/latest/meta-data/iam/security-credentials/</a>)</span><span
style="font-size:11.5pt;font-family:"Courier New",serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Courier
New",serif">HTTP: Fetch(<a
href="http://169.254.169.254/latest/meta-data/iam/security-credentials/iam-grid-s3"
moz-do-not-send="true"
class="moz-txt-link-freetext">http://169.254.169.254/latest/meta-data/iam/security-credentials/iam-grid-s3</a>)</span><span
style="font-size:11.5pt;font-family:"Courier New",serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Courier
New",serif">AWS: Storing AIM credentials
until 2022-11-19T20:42:58Z</span><span
style="font-size:11.5pt;font-family:"Courier
New",serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Courier
New",serif">S3: Downloading 0-16383 (<a
href="https://grid-dev-publiclidar.s3.amazonaws.com/estonia/dtm/estonia_dtm_5m.tif"
moz-do-not-send="true"
class="moz-txt-link-freetext">https://grid-dev-publiclidar.s3.amazonaws.com/estonia/dtm/estonia_dtm_5m.tif</a>)...</span><span
style="font-size:11.5pt;font-family:"Courier New",serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Courier
New",serif">S3: Got response_code=206</span><span
style="font-size:11.5pt;font-family:"Courier New",serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Courier
New",serif">gdalinfo failed - unable to
open
'/vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif'.</span><span
style="font-size:11.5pt;font-family:"Courier New",serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:11.5pt;font-family:"Courier
New",serif"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span
style="font-size:11.5pt;font-family:"Courier
New",serif"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span
style="font-size:11.5pt;font-family:"Courier
New",serif">Mike<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:11.5pt;font-family:"Courier
New",serif"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:11.5pt;font-family:"Courier
New",serif"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:11.5pt;font-family:"Courier
New",serif"><br>
<br>
<o:p></o:p></span></p>
<blockquote
style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-bottom:12.0pt"><span
style="font-size:11.5pt;font-family:"Courier New",serif">On
Nov 19, 2022, at 9:26 AM, Even Rouault <a
href="mailto:even.rouault@spatialys.com"
moz-do-not-send="true"><even.rouault@spatialys.com></a>
wrote:<o:p></o:p></span></p>
</blockquote>
</div>
<blockquote
style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span
style="font-size:11.5pt;font-family:"Courier New",serif">Hi
Mike,<br>
<br>
could you send the output of<br>
<br>
curl <a
href="http://169.254.169.254/latest/meta-data/iam/security-credentials/iam-grid-s3"
moz-do-not-send="true"
class="moz-txt-link-freetext">http://169.254.169.254/latest/meta-data/iam/security-credentials/iam-grid-s3</a><br>
<br>
Slightly redacted of course, but with the
exact formatting. This part of thee code
currently uses a "simple JSON parser" (<a
href="https://github.com/OSGeo/gdal/blob/c61d116a469821b769630a112dee7f1a61fed885/port/cpl_aws.cpp#L554"
moz-do-not-send="true"
class="moz-txt-link-freetext">https://github.com/OSGeo/gdal/blob/c61d116a469821b769630a112dee7f1a61fed885/port/cpl_aws.cpp#L554</a>),
which is actually just a non JSON-aware string
tokenizer, and I suspect it could be defeated
by a new formatting of S3 or something
specific to your credentials.<br>
<br>
It could also be that something unhandled by
that parser appears inside quoted strings,
like an escaped double quote or some other
JSON escaped character (like an escaped
forward slash \/ )<br>
<br>
If that was the case we should likely switch
to proper JSON deserialization (that part of
the code must predate libjson-c being a build
requirement of GDAL).<br>
<br>
Even<br>
<br>
<br>
-- <br>
<a href="http://www.spatialys.com"
moz-do-not-send="true"
class="moz-txt-link-freetext">http://www.spatialys.com</a><br>
My software is free, but my time generally
not.<o:p></o:p></span></p>
</div>
</blockquote>
</div>
</blockquote>
<pre>-- <o:p></o:p></pre>
<pre><a href="http://www.spatialys.com" moz-do-not-send="true" class="moz-txt-link-freetext">http://www.spatialys.com</a><o:p></o:p></pre>
<pre>My software is free, but my time generally not.<o:p></o:p></pre>
</div>
</div>
</blockquote>
<pre>-- <o:p></o:p></pre>
<pre><a href="http://www.spatialys.com" moz-do-not-send="true" class="moz-txt-link-freetext">http://www.spatialys.com</a><o:p></o:p></pre>
<pre>My software is free, but my time generally not.<o:p></o:p></pre>
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
<a class="moz-txt-link-freetext" href="http://www.spatialys.com">http://www.spatialys.com</a>
My software is free, but my time generally not.</pre>
</body>
</html>