<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:"Courier New \,serif";
panose-1:2 7 3 9 2 2 5 2 4 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
font-size:10.0pt;
font-family:"Courier New",serif;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style></head><body lang=EN-US link=blue vlink=purple style='word-wrap:break-word'><div class=WordSection1><p class=MsoNormal>Cool, I was looking at the PR and trying to decide if it was the same error. The error state being cleared and then working did sound like the same issue. It all makes sense now. Thanks for the fix!<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Mike<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><div><div><p class=MsoNormal>-- <o:p></o:p></p></div><div><p class=MsoNormal>Michael Smith<o:p></o:p></p><p class=MsoNormal>US Army Corps / Remote Sensing GIS Center<o:p></o:p></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:12.0pt;color:black'>From: </span></b><span style='font-size:12.0pt;color:black'>Even Rouault <even.rouault@spatialys.com><br><b>Date: </b>Sunday, November 20, 2022 at 12:52 PM<br><b>To: </b>Michael Smith <michael.smith.erdc@gmail.com><br><b>Cc: </b>gdal-dev <gdal-dev@lists.osgeo.org><br><b>Subject: </b>Re: [gdal-dev] errors using IAM instance profile auth in s3<o:p></o:p></span></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><p>Mike,<o:p></o:p></p><p>I've ended up firing a EC2 instance and I did replicate with my private bucket too.<o:p></o:p></p><p>With a EC2 Ubuntu 22.04 instance, on a minimal GDAL master debug build, things work fine with IAM authentication, in the VM itself<o:p></o:p></p><p>But if I run inside a ubuntu:22.04 Docker image the same build (mounting my home directory), then I got exactly the same symptoms as you!<o:p></o:p></p><p>I finally figured out that the "curl -X PUT <a href="http://169.254.169.254/latest/api/token">"http://169.254.169.254/latest/api/token"</a> -H "X-aws-ec2-metadata-token-ttl-seconds: 10" request for IDMSv2 was failing inside Docker but not in the outside EC2 VM. Known (intended) issue according to <a href="https://community.grafana.com/t/imdsv2-is-not-working-from-docker/65944">https://community.grafana.com/t/imdsv2-is-not-working-from-docker/65944</a> <o:p></o:p></p><p>The solution/workaround is to run docker with --network=host (I guess it could be possible to restrict it a bit to just 169.254.169.254)<o:p></o:p></p><p>In <a href="https://github.com/OSGeo/gdal/pull/6752">https://github.com/OSGeo/gdal/pull/6752</a>, I've pushed a proper fix, that makes sure that the fallback to IDMSv1 works by clearing the GDAL error state (which was the reason for the weird behaviour that GDALOpen() only worked the second time), and emits a hopefully debug hint when it sees that IDMSv2 fails on timeout inside a Docker container.<o:p></o:p></p><p>With the fix, when trying inside a Docker with default networking rules:<o:p></o:p></p><p># gdalinfo /vsis3/XXXXXXX/byte.tif --debug on<br>AWS: AWS_ROLE_ARN configuration option not defined<br>HTTP: Fetch(<a href="http://169.254.169.254/latest/api/token">http://169.254.169.254/latest/api/token</a>)<br>HTTP: libcurl/7.81.0 GnuTLS/3.7.3 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.13<br>HTTP: Fetch(<a href="http://169.254.169.254/latest/meta-data">http://169.254.169.254/latest/meta-data</a>)<br>S3: /latest/api/token EC2 IDMSv2 request timed out, but /latest/metadata succeeded. Trying with IDMSv1. Try running your Docker container with --network=host.<br>HTTP: Fetch(<a href="http://169.254.169.254/latest/meta-data/iam/security-credentials/">http://169.254.169.254/latest/meta-data/iam/security-credentials/</a>)<br>HTTP: Fetch(<a href="http://169.254.169.254/latest/meta-data/iam/security-credentials/ec2_s3">http://169.254.169.254/latest/meta-data/iam/security-credentials/ec2_s3</a>)<br>AWS: Storing AIM credentials until 2022-11-20T23:38:10Z<br>S3: Switching to region eu-central-1<br>S3: Downloading 0-1796 (<a href="https://XXXXXXX.s3.amazonaws.com/byte.tif">https://XXXXXXX.s3.amazonaws.com/byte.tif</a>)...<br>S3: Got response_code=206<br>GDAL: GDALOpen(/vsis3/XXXXXXX/byte.tif, this=0x5599cfbce470) succeeds as GTiff.<o:p></o:p></p><p>It was an "interesting" bug...<o:p></o:p></p><p>Even<o:p></o:p></p><div><p class=MsoNormal>Le 20/11/2022 à 14:47, Michael Smith a écrit :<o:p></o:p></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal>Is there a reason why OpenEx would work but Open wouldn’t?<o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal>In [1]: from osgeo import gdal<o:p></o:p></p><p class=MsoNormal>In [2]: pszFilename = "/vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif"<o:p></o:p></p><p class=MsoNormal>In [3]: hDataset = gdal.Open(pszFilename, gdal.GA_ReadOnly)<o:p></o:p></p><p class=MsoNormal>In [4]: hDataset<o:p></o:p></p><p class=MsoNormal>In [5]: hDataset = gdal.OpenEx(pszFilename, gdal.GA_ReadOnly)<o:p></o:p></p><p class=MsoNormal>In [6]: hDataset<o:p></o:p></p><p class=MsoNormal>Out[6]: <osgeo.gdal.Dataset; proxy of <Swig Object of type 'GDALDatasetShadow *' at 0x7f827217c450> ><o:p></o:p></p><p class=MsoNormal>In [7]: hDataset.GetGeoTransform()<o:p></o:p></p><p class=MsoNormal>Out[7]: (365000.0, 5.0, 0.0, 6635000.0, 0.0, -5.0)<o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal>Mike<o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:12.0pt;color:black'>From: </span></b><span style='font-size:12.0pt;color:black'>Even Rouault <a href="mailto:even.rouault@spatialys.com"><even.rouault@spatialys.com></a><br><b>Date: </b>Saturday, November 19, 2022 at 10:08 AM<br><b>To: </b><a href="mailto:michael.smith.erdc@gmail.com"><michael.smith.erdc@gmail.com></a><br><b>Cc: </b>gdal-dev <a href="mailto:gdal-dev@lists.osgeo.org"><gdal-dev@lists.osgeo.org></a><br><b>Subject: </b>Re: [gdal-dev] errors using IAM instance profile auth in s3</span><o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><p> <o:p></o:p></p><div><p class=MsoNormal>Le 19/11/2022 à 16:00, <a href="mailto:michael.smith.erdc@gmail.com">michael.smith.erdc@gmail.com</a> a écrit :<o:p></o:p></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal>Correct, not a public bucket, which is why the IAM credentials are needed. If I set them manually, it all works fine.<o:p></o:p></p></blockquote><p>That's super weird if the result of a range request changes depending on how credentials have been set... Perhaps enable CPL_CURL_VERBOSE=ON env variable and diff the logs ?<o:p></o:p></p><p>You could also try the gdal_cp.py sample script at <a href="https://github.com/OSGeo/gdal/blob/master/swig/python/gdal-utils/osgeo_utils/samples/gdal_cp.py">https://github.com/OSGeo/gdal/blob/master/swig/python/gdal-utils/osgeo_utils/samples/gdal_cp.py</a> , which is a cp-like utility working with GDAL virtual file systems, with the 2 authentication methods<o:p></o:p></p><p>python gdal_cp.py /vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif out.tif<o:p></o:p></p><p>(you can interrupt it with ctrl-c after a few seconds. that will be enough to get the first bytes)<o:p></o:p></p><p>you might need to run an hexadecimal editor to inspect a bit the content.<o:p></o:p></p><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><div><p class=MsoNormal> <o:p></o:p></p></div><div><div><p class=MsoNormal><span style='font-size:11.5pt;font-family:"Courier New \,serif"'>[ u02]$ export AWS_ACCESS_KEY_ID=xxxxx</span><o:p></o:p></p></div><div><p><span style='font-size:11.5pt;font-family:"Courier New \,serif"'>Yes, a 206 response code means success here as we are requesting only bytes 0-16383. So maybe the file is not a valid TIFF ?</span><o:p></o:p></p><p><span style='font-size:11.5pt;font-family:"Courier New \,serif"'>( "grid-dev-publiclidar" must not be so public I guess, because when trying with my credentials, I get a Access Denied)</span><o:p></o:p></p><div><p class=MsoNormal><span style='font-size:11.5pt;font-family:"Courier New \,serif"'>Le 19/11/2022 à 15:40, <a href="mailto:michael.smith.erdc@gmail.com">michael.smith.erdc@gmail.com</a> a écrit :</span><o:p></o:p></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal><span style='font-size:11.5pt;font-family:"Courier New \,serif"'>I’m seeing that it’s getting a 206 response code, so wouldn’t that indicate auth is working? </span><o:p></o:p></p><div><p class=MsoNormal><span style='font-size:11.5pt;font-family:"Courier New \,serif"'> </span><o:p></o:p></p></div><div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Courier New \,serif"'> gdalinfo /vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif</span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Courier New \,serif"'>HTTP: Fetch(<a href="http://169.254.169.254/latest/api/token">http://169.254.169.254/latest/api/token</a>)</span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Courier New \,serif"'>HTTP: libcurl/7.86.0 OpenSSL/3.0.7 zlib/1.2.13 libssh2/1.10.0 nghttp2/1.47.0</span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Courier New \,serif"'>HTTP: These HTTP headers were set: X-aws-ec2-metadata-token-ttl-seconds: 10</span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Courier New \,serif"'>HTTP: Fetch(<a href="http://169.254.169.254/latest/meta-data/iam/security-credentials/">http://169.254.169.254/latest/meta-data/iam/security-credentials/</a>)</span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Courier New \,serif"'>HTTP: Fetch(<a href="http://169.254.169.254/latest/meta-data/iam/security-credentials/iam-grid-s3">http://169.254.169.254/latest/meta-data/iam/security-credentials/iam-grid-s3</a>)</span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Courier New \,serif"'>AWS: Storing AIM credentials until 2022-11-19T20:42:58Z</span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Courier New \,serif"'>S3: Downloading 0-16383 (<a href="https://grid-dev-publiclidar.s3.amazonaws.com/estonia/dtm/estonia_dtm_5m.tif">https://grid-dev-publiclidar.s3.amazonaws.com/estonia/dtm/estonia_dtm_5m.tif</a>)...</span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Courier New \,serif"'>S3: Got response_code=206</span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Courier New \,serif"'>gdalinfo failed - unable to open '/vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif'.</span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:11.5pt;font-family:"Courier New \,serif"'> </span><o:p></o:p></p></div><p class=MsoNormal><span style='font-size:11.5pt;font-family:"Courier New \,serif"'> </span><o:p></o:p></p><div><p class=MsoNormal><span style='font-size:11.5pt;font-family:"Courier New \,serif"'>Mike</span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:11.5pt;font-family:"Courier New \,serif"'> </span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:11.5pt;font-family:"Courier New \,serif"'> </span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:11.5pt;font-family:"Courier New \,serif"'><br><br><br></span><o:p></o:p></p><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal style='margin-bottom:12.0pt'><span style='font-size:11.5pt;font-family:"Courier New",serif'>On Nov 19, 2022, at 9:26 AM, Even Rouault <a href="mailto:even.rouault@spatialys.com"><even.rouault@spatialys.com></a> wrote:</span><o:p></o:p></p></blockquote></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><div><p class=MsoNormal style='margin-bottom:12.0pt'><span style='font-size:11.5pt;font-family:"Courier New",serif'>Hi Mike,<br><br>could you send the output of<br><br>curl <a href="http://169.254.169.254/latest/meta-data/iam/security-credentials/iam-grid-s3">http://169.254.169.254/latest/meta-data/iam/security-credentials/iam-grid-s3</a><br><br>Slightly redacted of course, but with the exact formatting. This part of thee code currently uses a "simple JSON parser" (<a href="https://github.com/OSGeo/gdal/blob/c61d116a469821b769630a112dee7f1a61fed885/port/cpl_aws.cpp#L554">https://github.com/OSGeo/gdal/blob/c61d116a469821b769630a112dee7f1a61fed885/port/cpl_aws.cpp#L554</a>), which is actually just a non JSON-aware string tokenizer, and I suspect it could be defeated by a new formatting of S3 or something specific to your credentials.<br><br>It could also be that something unhandled by that parser appears inside quoted strings, like an escaped double quote or some other JSON escaped character (like an escaped forward slash \/ )<br><br>If that was the case we should likely switch to proper JSON deserialization (that part of the code must predate libjson-c being a build requirement of GDAL).<br><br>Even<br><br><br>-- <br><a href="http://www.spatialys.com">http://www.spatialys.com</a><br>My software is free, but my time generally not.</span><o:p></o:p></p></div></blockquote></div></blockquote><pre>-- <o:p></o:p></pre><pre><a href="http://www.spatialys.com">http://www.spatialys.com</a><o:p></o:p></pre><pre>My software is free, but my time generally not.<o:p></o:p></pre></div></div></blockquote><pre>-- <o:p></o:p></pre><pre><a href="http://www.spatialys.com">http://www.spatialys.com</a><o:p></o:p></pre><pre>My software is free, but my time generally not.<o:p></o:p></pre></blockquote><pre>-- <o:p></o:p></pre><pre><a href="http://www.spatialys.com">http://www.spatialys.com</a><o:p></o:p></pre><pre>My software is free, but my time generally not.<o:p></o:p></pre></div></body></html>