<div dir="ltr">Oh, its just the second open that works<div><br></div><div>In [1]: from osgeo import gdal<br>In [2]: pszFilename = "/vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif"<br>In [3]: hDataset = gdal.OpenEx(pszFilename, gdal.GA_ReadOnly)<br>In [4]: hDataset<br>In [5]: hDataset = gdal.Open(pszFilename, gdal.GA_ReadOnly)<br>In [6]: hDataset<br>Out[6]: <osgeo.gdal.Dataset; proxy of <Swig Object of type 'GDALDatasetShadow *' at 0x7fef5b3fe1c0> ><br>In [7]: hDataset.GetProjectionRef()<br>Out[7]: 'PROJCS["Estonian Coordinate System of 1997",GEOGCS["EST97",DATUM["Estonia_1997",SPHEROID["GRS 1980",6378137,298.257222101,AUTHORITY["EPSG","7019"]],AUTHORITY["EPSG","6180"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4180"]],PROJECTION["Lambert_Conformal_Conic_2SP"],PARAMETER["latitude_of_origin",57.5175539305556],PARAMETER["central_meridian",24],PARAMETER["standard_parallel_1",59.3333333333333],PARAMETER["standard_parallel_2",58],PARAMETER["false_easting",500000],PARAMETER["false_northing",6375000],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["Northing",NORTH],AXIS["Easting",EAST],AUTHORITY["EPSG","3301"]]'<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Nov 20, 2022 at 8:47 AM Michael Smith <<a href="mailto:michael.smith.erdc@gmail.com">michael.smith.erdc@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg364253227305395635"><div lang="EN-US" style="overflow-wrap: break-word;"><div class="m_364253227305395635WordSection1"><p class="MsoNormal">Is there a reason why OpenEx would work but Open wouldn’t?<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">In [1]: from osgeo import gdal<u></u><u></u></p><p class="MsoNormal">In [2]: pszFilename = "/vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif"<u></u><u></u></p><p class="MsoNormal">In [3]: hDataset = gdal.Open(pszFilename, gdal.GA_ReadOnly)<u></u><u></u></p><p class="MsoNormal">In [4]: hDataset<u></u><u></u></p><p class="MsoNormal">In [5]: hDataset = gdal.OpenEx(pszFilename, gdal.GA_ReadOnly)<u></u><u></u></p><p class="MsoNormal">In [6]: hDataset<u></u><u></u></p><p class="MsoNormal">Out[6]: <osgeo.gdal.Dataset; proxy of <Swig Object of type 'GDALDatasetShadow *' at 0x7f827217c450> ><u></u><u></u></p><p class="MsoNormal">In [7]: hDataset.GetGeoTransform()<u></u><u></u></p><p class="MsoNormal">Out[7]: (365000.0, 5.0, 0.0, 6635000.0, 0.0, -5.0)<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">Mike<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal"><u></u> <u></u></p><div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(181,196,223);padding:3pt 0in 0in"><p class="MsoNormal"><b><span style="font-size:12pt;color:black">From: </span></b><span style="font-size:12pt;color:black">Even Rouault <<a href="mailto:even.rouault@spatialys.com" target="_blank">even.rouault@spatialys.com</a>><br><b>Date: </b>Saturday, November 19, 2022 at 10:08 AM<br><b>To: </b><<a href="mailto:michael.smith.erdc@gmail.com" target="_blank">michael.smith.erdc@gmail.com</a>><br><b>Cc: </b>gdal-dev <<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a>><br><b>Subject: </b>Re: [gdal-dev] errors using IAM instance profile auth in s3<u></u><u></u></span></p></div><div><p class="MsoNormal"><u></u> <u></u></p></div><p><u></u> <u></u></p><div><p class="MsoNormal">Le 19/11/2022 à 16:00, <a href="mailto:michael.smith.erdc@gmail.com" target="_blank">michael.smith.erdc@gmail.com</a> a écrit :<u></u><u></u></p></div><blockquote style="margin-top:5pt;margin-bottom:5pt"><p class="MsoNormal">Correct, not a public bucket, which is why the IAM credentials are needed. If I set them manually, it all works fine.<u></u><u></u></p></blockquote><p>That's super weird if the result of a range request changes depending on how credentials have been set... Perhaps enable CPL_CURL_VERBOSE=ON env variable and diff the logs ?<u></u><u></u></p><p>You could also try the gdal_cp.py sample script at <a href="https://github.com/OSGeo/gdal/blob/master/swig/python/gdal-utils/osgeo_utils/samples/gdal_cp.py" target="_blank">https://github.com/OSGeo/gdal/blob/master/swig/python/gdal-utils/osgeo_utils/samples/gdal_cp.py</a> , which is a cp-like utility working with GDAL virtual file systems, with the 2 authentication methods<u></u><u></u></p><p>python gdal_cp.py /vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif out.tif<u></u><u></u></p><p>(you can interrupt it with ctrl-c after a few seconds. that will be enough to get the first bytes)<u></u><u></u></p><p>you might need to run an hexadecimal editor to inspect a bit the content.<u></u><u></u></p><blockquote style="margin-top:5pt;margin-bottom:5pt"><div><p class="MsoNormal"><u></u> <u></u></p></div><div><div><p class="MsoNormal"><span style="font-size:11.5pt;font-family:"Courier New",serif">[ u02]$ export AWS_ACCESS_KEY_ID=xxxxx</span><u></u><u></u></p></div><div><p><span style="font-size:11.5pt;font-family:"Courier New",serif">Yes, a 206 response code means success here as we are requesting only bytes 0-16383. So maybe the file is not a valid TIFF ?<u></u><u></u></span></p><p><span style="font-size:11.5pt;font-family:"Courier New",serif">( "grid-dev-publiclidar" must not be so public I guess, because when trying with my credentials, I get a Access Denied)<u></u><u></u></span></p><div><p class="MsoNormal"><span style="font-size:11.5pt;font-family:"Courier New",serif">Le 19/11/2022 à 15:40, <a href="mailto:michael.smith.erdc@gmail.com" target="_blank">michael.smith.erdc@gmail.com</a> a écrit :<u></u><u></u></span></p></div><blockquote style="margin-top:5pt;margin-bottom:5pt"><p class="MsoNormal"><span style="font-size:11.5pt;font-family:"Courier New",serif">I’m seeing that it’s getting a 206 response code, so wouldn’t that indicate auth is working? <u></u><u></u></span></p><div><p class="MsoNormal"><span style="font-size:11.5pt;font-family:"Courier New",serif"><u></u> <u></u></span></p></div><div><div><p class="MsoNormal"><span style="font-size:12pt;font-family:"Courier New",serif"> gdalinfo /vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif</span><span style="font-size:11.5pt;font-family:"Courier New",serif"><u></u><u></u></span></p></div><div><p class="MsoNormal"><span style="font-size:12pt;font-family:"Courier New",serif">HTTP: Fetch(<a href="http://169.254.169.254/latest/api/token" target="_blank">http://169.254.169.254/latest/api/token</a>)</span><span style="font-size:11.5pt;font-family:"Courier New",serif"><u></u><u></u></span></p></div><div><p class="MsoNormal"><span style="font-size:12pt;font-family:"Courier New",serif">HTTP: libcurl/7.86.0 OpenSSL/3.0.7 zlib/1.2.13 libssh2/1.10.0 nghttp2/1.47.0</span><span style="font-size:11.5pt;font-family:"Courier New",serif"><u></u><u></u></span></p></div><div><p class="MsoNormal"><span style="font-size:12pt;font-family:"Courier New",serif">HTTP: These HTTP headers were set: X-aws-ec2-metadata-token-ttl-seconds: 10</span><span style="font-size:11.5pt;font-family:"Courier New",serif"><u></u><u></u></span></p></div><div><p class="MsoNormal"><span style="font-size:12pt;font-family:"Courier New",serif">HTTP: Fetch(<a href="http://169.254.169.254/latest/meta-data/iam/security-credentials/" target="_blank">http://169.254.169.254/latest/meta-data/iam/security-credentials/</a>)</span><span style="font-size:11.5pt;font-family:"Courier New",serif"><u></u><u></u></span></p></div><div><p class="MsoNormal"><span style="font-size:12pt;font-family:"Courier New",serif">HTTP: Fetch(<a href="http://169.254.169.254/latest/meta-data/iam/security-credentials/iam-grid-s3" target="_blank">http://169.254.169.254/latest/meta-data/iam/security-credentials/iam-grid-s3</a>)</span><span style="font-size:11.5pt;font-family:"Courier New",serif"><u></u><u></u></span></p></div><div><p class="MsoNormal"><span style="font-size:12pt;font-family:"Courier New",serif">AWS: Storing AIM credentials until 2022-11-19T20:42:58Z</span><span style="font-size:11.5pt;font-family:"Courier New",serif"><u></u><u></u></span></p></div><div><p class="MsoNormal"><span style="font-size:12pt;font-family:"Courier New",serif">S3: Downloading 0-16383 (<a href="https://grid-dev-publiclidar.s3.amazonaws.com/estonia/dtm/estonia_dtm_5m.tif" target="_blank">https://grid-dev-publiclidar.s3.amazonaws.com/estonia/dtm/estonia_dtm_5m.tif</a>)...</span><span style="font-size:11.5pt;font-family:"Courier New",serif"><u></u><u></u></span></p></div><div><p class="MsoNormal"><span style="font-size:12pt;font-family:"Courier New",serif">S3: Got response_code=206</span><span style="font-size:11.5pt;font-family:"Courier New",serif"><u></u><u></u></span></p></div><div><p class="MsoNormal"><span style="font-size:12pt;font-family:"Courier New",serif">gdalinfo failed - unable to open '/vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif'.</span><span style="font-size:11.5pt;font-family:"Courier New",serif"><u></u><u></u></span></p></div><div><p class="MsoNormal"><span style="font-size:11.5pt;font-family:"Courier New",serif"><u></u> <u></u></span></p></div><p class="MsoNormal"><span style="font-size:11.5pt;font-family:"Courier New",serif"><u></u> <u></u></span></p><div><p class="MsoNormal"><span style="font-size:11.5pt;font-family:"Courier New",serif">Mike<u></u><u></u></span></p></div><div><p class="MsoNormal"><span style="font-size:11.5pt;font-family:"Courier New",serif"><u></u> <u></u></span></p></div><div><p class="MsoNormal"><span style="font-size:11.5pt;font-family:"Courier New",serif"><u></u> <u></u></span></p></div><div><p class="MsoNormal"><span style="font-size:11.5pt;font-family:"Courier New",serif"><br><br><u></u><u></u></span></p><blockquote style="margin-top:5pt;margin-bottom:5pt"><p class="MsoNormal" style="margin-bottom:12pt"><span style="font-size:11.5pt;font-family:"Courier New",serif">On Nov 19, 2022, at 9:26 AM, Even Rouault <a href="mailto:even.rouault@spatialys.com" target="_blank"><even.rouault@spatialys.com></a> wrote:<u></u><u></u></span></p></blockquote></div><blockquote style="margin-top:5pt;margin-bottom:5pt"><div><p class="MsoNormal" style="margin-bottom:12pt"><span style="font-size:11.5pt;font-family:"Courier New",serif">Hi Mike,<br><br>could you send the output of<br><br>curl <a href="http://169.254.169.254/latest/meta-data/iam/security-credentials/iam-grid-s3" target="_blank">http://169.254.169.254/latest/meta-data/iam/security-credentials/iam-grid-s3</a><br><br>Slightly redacted of course, but with the exact formatting. This part of thee code currently uses a "simple JSON parser" (<a href="https://github.com/OSGeo/gdal/blob/c61d116a469821b769630a112dee7f1a61fed885/port/cpl_aws.cpp#L554" target="_blank">https://github.com/OSGeo/gdal/blob/c61d116a469821b769630a112dee7f1a61fed885/port/cpl_aws.cpp#L554</a>), which is actually just a non JSON-aware string tokenizer, and I suspect it could be defeated by a new formatting of S3 or something specific to your credentials.<br><br>It could also be that something unhandled by that parser appears inside quoted strings, like an escaped double quote or some other JSON escaped character (like an escaped forward slash \/ )<br><br>If that was the case we should likely switch to proper JSON deserialization (that part of the code must predate libjson-c being a build requirement of GDAL).<br><br>Even<br><br><br>-- <br><a href="http://www.spatialys.com" target="_blank">http://www.spatialys.com</a><br>My software is free, but my time generally not.<u></u><u></u></span></p></div></blockquote></div></blockquote><pre>-- <u></u><u></u></pre><pre><a href="http://www.spatialys.com" target="_blank">http://www.spatialys.com</a><u></u><u></u></pre><pre>My software is free, but my time generally not.<u></u><u></u></pre></div></div></blockquote><pre>-- <u></u><u></u></pre><pre><a href="http://www.spatialys.com" target="_blank">http://www.spatialys.com</a><u></u><u></u></pre><pre>My software is free, but my time generally not.<u></u><u></u></pre></div></div>
</div></blockquote></div>