<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
font-size:10.0pt;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Consolas",serif;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style></head><body lang=EN-US link=blue vlink=purple style='word-wrap:break-word'><div class=WordSection1><p class=MsoNormal>Using gdal_cp.py, it copies using IAM credentials (and using key credentials from the curl call). Now I’m even more confused.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I was able to replicate with the docker image osgeo/gdal:latest, so its not just my own docker image although it could be my own environment.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Will continue to debug.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Mike<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:12.0pt;color:black'>From: </span></b><span style='font-size:12.0pt;color:black'>Even Rouault <even.rouault@spatialys.com><br><b>Date: </b>Saturday, November 19, 2022 at 10:08 AM<br><b>To: </b><michael.smith.erdc@gmail.com><br><b>Cc: </b>gdal-dev <gdal-dev@lists.osgeo.org><br><b>Subject: </b>Re: [gdal-dev] errors using IAM instance profile auth in s3<o:p></o:p></span></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><p><o:p> </o:p></p><div><p class=MsoNormal>Le 19/11/2022 à 16:00, <a href="mailto:michael.smith.erdc@gmail.com">michael.smith.erdc@gmail.com</a> a écrit :<o:p></o:p></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal>Correct, not a public bucket, which is why the IAM credentials are needed. If I set them manually, it all works fine.<o:p></o:p></p></blockquote><p>That's super weird if the result of a range request changes depending on how credentials have been set... Perhaps enable CPL_CURL_VERBOSE=ON env variable and diff the logs ?<o:p></o:p></p><p>You could also try the gdal_cp.py sample script at <a href="https://github.com/OSGeo/gdal/blob/master/swig/python/gdal-utils/osgeo_utils/samples/gdal_cp.py">https://github.com/OSGeo/gdal/blob/master/swig/python/gdal-utils/osgeo_utils/samples/gdal_cp.py</a> , which is a cp-like utility working with GDAL virtual file systems, with the 2 authentication methods<o:p></o:p></p><p>python gdal_cp.py /vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif out.tif<o:p></o:p></p><p>(you can interrupt it with ctrl-c after a few seconds. that will be enough to get the first bytes)<o:p></o:p></p><p>you might need to run an hexadecimal editor to inspect a bit the content.<o:p></o:p></p><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><div><p class=MsoNormal><o:p> </o:p></p></div><div><div><p class=MsoNormal><span style='font-size:11.5pt;font-family:"Courier New"'>[ u02]$ export AWS_ACCESS_KEY_ID=xxxxx</span><o:p></o:p></p></div><div><p><span style='font-size:11.5pt;font-family:"Courier New"'>Yes, a 206 response code means success here as we are requesting only bytes 0-16383. So maybe the file is not a valid TIFF ?<o:p></o:p></span></p><p><span style='font-size:11.5pt;font-family:"Courier New"'>( "grid-dev-publiclidar" must not be so public I guess, because when trying with my credentials, I get a Access Denied)<o:p></o:p></span></p><div><p class=MsoNormal><span style='font-size:11.5pt;font-family:"Courier New"'>Le 19/11/2022 à 15:40, <a href="mailto:michael.smith.erdc@gmail.com">michael.smith.erdc@gmail.com</a> a écrit :<o:p></o:p></span></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal><span style='font-size:11.5pt;font-family:"Courier New"'>I’m seeing that it’s getting a 206 response code, so wouldn’t that indicate auth is working? <o:p></o:p></span></p><div><p class=MsoNormal><span style='font-size:11.5pt;font-family:"Courier New"'><o:p> </o:p></span></p></div><div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Courier New"'> gdalinfo /vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif</span><span style='font-size:11.5pt;font-family:"Courier New"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Courier New"'>HTTP: Fetch(<a href="http://169.254.169.254/latest/api/token">http://169.254.169.254/latest/api/token</a>)</span><span style='font-size:11.5pt;font-family:"Courier New"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Courier New"'>HTTP: libcurl/7.86.0 OpenSSL/3.0.7 zlib/1.2.13 libssh2/1.10.0 nghttp2/1.47.0</span><span style='font-size:11.5pt;font-family:"Courier New"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Courier New"'>HTTP: These HTTP headers were set: X-aws-ec2-metadata-token-ttl-seconds: 10</span><span style='font-size:11.5pt;font-family:"Courier New"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Courier New"'>HTTP: Fetch(<a href="http://169.254.169.254/latest/meta-data/iam/security-credentials/">http://169.254.169.254/latest/meta-data/iam/security-credentials/</a>)</span><span style='font-size:11.5pt;font-family:"Courier New"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Courier New"'>HTTP: Fetch(<a href="http://169.254.169.254/latest/meta-data/iam/security-credentials/iam-grid-s3">http://169.254.169.254/latest/meta-data/iam/security-credentials/iam-grid-s3</a>)</span><span style='font-size:11.5pt;font-family:"Courier New"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Courier New"'>AWS: Storing AIM credentials until 2022-11-19T20:42:58Z</span><span style='font-size:11.5pt;font-family:"Courier New"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Courier New"'>S3: Downloading 0-16383 (<a href="https://grid-dev-publiclidar.s3.amazonaws.com/estonia/dtm/estonia_dtm_5m.tif">https://grid-dev-publiclidar.s3.amazonaws.com/estonia/dtm/estonia_dtm_5m.tif</a>)...</span><span style='font-size:11.5pt;font-family:"Courier New"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Courier New"'>S3: Got response_code=206</span><span style='font-size:11.5pt;font-family:"Courier New"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Courier New"'>gdalinfo failed - unable to open '/vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif'.</span><span style='font-size:11.5pt;font-family:"Courier New"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:11.5pt;font-family:"Courier New"'><o:p> </o:p></span></p></div><p class=MsoNormal><span style='font-size:11.5pt;font-family:"Courier New"'><o:p> </o:p></span></p><div><p class=MsoNormal><span style='font-size:11.5pt;font-family:"Courier New"'>Mike<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:11.5pt;font-family:"Courier New"'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-size:11.5pt;font-family:"Courier New"'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-size:11.5pt;font-family:"Courier New"'><br><br><o:p></o:p></span></p><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal style='margin-bottom:12.0pt'><span style='font-size:11.5pt;font-family:"Courier New"'>On Nov 19, 2022, at 9:26 AM, Even Rouault <a href="mailto:even.rouault@spatialys.com"><even.rouault@spatialys.com></a> wrote:<o:p></o:p></span></p></blockquote></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><div><p class=MsoNormal style='margin-bottom:12.0pt'><span style='font-size:11.5pt;font-family:"Courier New"'>Hi Mike,<br><br>could you send the output of<br><br>curl <a href="http://169.254.169.254/latest/meta-data/iam/security-credentials/iam-grid-s3">http://169.254.169.254/latest/meta-data/iam/security-credentials/iam-grid-s3</a><br><br>Slightly redacted of course, but with the exact formatting. This part of thee code currently uses a "simple JSON parser" (<a href="https://github.com/OSGeo/gdal/blob/c61d116a469821b769630a112dee7f1a61fed885/port/cpl_aws.cpp#L554">https://github.com/OSGeo/gdal/blob/c61d116a469821b769630a112dee7f1a61fed885/port/cpl_aws.cpp#L554</a>), which is actually just a non JSON-aware string tokenizer, and I suspect it could be defeated by a new formatting of S3 or something specific to your credentials.<br><br>It could also be that something unhandled by that parser appears inside quoted strings, like an escaped double quote or some other JSON escaped character (like an escaped forward slash \/ )<br><br>If that was the case we should likely switch to proper JSON deserialization (that part of the code must predate libjson-c being a build requirement of GDAL).<br><br>Even<br><br><br>-- <br><a href="http://www.spatialys.com">http://www.spatialys.com</a><br>My software is free, but my time generally not.<o:p></o:p></span></p></div></blockquote></div></blockquote><pre>-- <o:p></o:p></pre><pre><a href="http://www.spatialys.com">http://www.spatialys.com</a><o:p></o:p></pre><pre>My software is free, but my time generally not.<o:p></o:p></pre></div></div></blockquote><pre>-- <o:p></o:p></pre><pre><a href="http://www.spatialys.com">http://www.spatialys.com</a><o:p></o:p></pre><pre>My software is free, but my time generally not.<o:p></o:p></pre></div></body></html>