[gdal-dev] errors using IAM instance profile auth in s3

[Michael Smith]

Oh, its just the second open that works

In [1]: from osgeo import gdal
In [2]: pszFilename =
"/vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif"
In [3]: hDataset = gdal.OpenEx(pszFilename, gdal.GA_ReadOnly)
In [4]: hDataset
In [5]: hDataset = gdal.Open(pszFilename, gdal.GA_ReadOnly)
In [6]: hDataset
Out[6]: <osgeo.gdal.Dataset; proxy of <Swig Object of type
'GDALDatasetShadow *' at 0x7fef5b3fe1c0> >
In [7]: hDataset.GetProjectionRef()
Out[7]: 'PROJCS["Estonian Coordinate System of
1997",GEOGCS["EST97",DATUM["Estonia_1997",SPHEROID["GRS
1980",6378137,298.257222101,AUTHORITY["EPSG","7019"]],AUTHORITY["EPSG","6180"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4180"]],PROJECTION["Lambert_Conformal_Conic_2SP"],PARAMETER["latitude_of_origin",57.5175539305556],PARAMETER["central_meridian",24],PARAMETER["standard_parallel_1",59.3333333333333],PARAMETER["standard_parallel_2",58],PARAMETER["false_easting",500000],PARAMETER["false_northing",6375000],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["Northing",NORTH],AXIS["Easting",EAST],AUTHORITY["EPSG","3301"]]'

On Sun, Nov 20, 2022 at 8:47 AM Michael Smith <michael.smith.erdc at gmail.com>
wrote:

> Is there a reason why OpenEx would work but Open wouldn’t?
>
>
>
> In [1]: from osgeo import gdal
>
> In [2]: pszFilename =
> "/vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif"
>
> In [3]: hDataset = gdal.Open(pszFilename, gdal.GA_ReadOnly)
>
> In [4]: hDataset
>
> In [5]: hDataset = gdal.OpenEx(pszFilename, gdal.GA_ReadOnly)
>
> In [6]: hDataset
>
> Out[6]: <osgeo.gdal.Dataset; proxy of <Swig Object of type
> 'GDALDatasetShadow *' at 0x7f827217c450> >
>
> In [7]: hDataset.GetGeoTransform()
>
> Out[7]: (365000.0, 5.0, 0.0, 6635000.0, 0.0, -5.0)
>
>
>
> Mike
>
>
>
>
>
> *From: *Even Rouault <even.rouault at spatialys.com>
> *Date: *Saturday, November 19, 2022 at 10:08 AM
> *To: *<michael.smith.erdc at gmail.com>
> *Cc: *gdal-dev <gdal-dev at lists.osgeo.org>
> *Subject: *Re: [gdal-dev] errors using IAM instance profile auth in s3
>
>
>
>
>
> Le 19/11/2022 à 16:00, michael.smith.erdc at gmail.com a écrit :
>
> Correct, not a public bucket, which is why the IAM credentials are needed.
> If I set them manually, it all works fine.
>
> That's super weird if the result of a range request changes depending on
> how credentials have been set... Perhaps enable CPL_CURL_VERBOSE=ON env
> variable and diff the logs ?
>
> You could also try the gdal_cp.py sample script at
> https://github.com/OSGeo/gdal/blob/master/swig/python/gdal-utils/osgeo_utils/samples/gdal_cp.py
> , which is a cp-like utility working with GDAL virtual file systems, with
> the 2 authentication methods
>
> python gdal_cp.py
> /vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif out.tif
>
> (you can interrupt it with ctrl-c after a few seconds. that will be enough
> to get the first bytes)
>
> you might need to run an hexadecimal editor to inspect a bit the content.
>
>
>
> [ u02]$ export AWS_ACCESS_KEY_ID=xxxxx
>
> Yes, a 206 response code means success here as we are requesting only
> bytes 0-16383. So maybe the file is not a valid TIFF ?
>
> ( "grid-dev-publiclidar" must not be so public I guess, because when
> trying with my credentials, I get a Access Denied)
>
> Le 19/11/2022 à 15:40, michael.smith.erdc at gmail.com a écrit :
>
> I’m seeing that it’s getting a 206 response code, so wouldn’t that
> indicate auth is working?
>
>
>
>  gdalinfo /vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif
>
> HTTP: Fetch(http://169.254.169.254/latest/api/token)
>
> HTTP: libcurl/7.86.0 OpenSSL/3.0.7 zlib/1.2.13 libssh2/1.10.0
> nghttp2/1.47.0
>
> HTTP: These HTTP headers were set: X-aws-ec2-metadata-token-ttl-seconds: 10
>
> HTTP: Fetch(
> http://169.254.169.254/latest/meta-data/iam/security-credentials/)
>
> HTTP: Fetch(
> http://169.254.169.254/latest/meta-data/iam/security-credentials/iam-grid-s3
> )
>
> AWS: Storing AIM credentials until 2022-11-19T20:42:58Z
>
> S3: Downloading 0-16383 (
> https://grid-dev-publiclidar.s3.amazonaws.com/estonia/dtm/estonia_dtm_5m.tif
> )...
>
> S3: Got response_code=206
>
> gdalinfo failed - unable to open
> '/vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif'.
>
>
>
>
>
> Mike
>
>
>
>
>
>
>
> On Nov 19, 2022, at 9:26 AM, Even Rouault <even.rouault at spatialys.com>
> <even.rouault at spatialys.com> wrote:
>
> Hi Mike,
>
> could you send the output of
>
> curl
> http://169.254.169.254/latest/meta-data/iam/security-credentials/iam-grid-s3
>
> Slightly redacted of course, but with the exact formatting. This part of
> thee code currently uses a "simple JSON parser" (
> https://github.com/OSGeo/gdal/blob/c61d116a469821b769630a112dee7f1a61fed885/port/cpl_aws.cpp#L554),
> which is actually just a non JSON-aware string tokenizer, and I suspect it
> could be defeated by a new formatting of S3 or something specific to your
> credentials.
>
> It could also be that something unhandled by that parser appears inside
> quoted strings, like an escaped double quote or some other JSON escaped
> character (like an escaped forward slash \/ )
>
> If that was the case we should likely switch to proper JSON
> deserialization (that part of the code must predate libjson-c being a build
> requirement of GDAL).
>
> Even
>
>
> --
> http://www.spatialys.com
> My software is free, but my time generally not.
>
> --
>
> http://www.spatialys.com
>
> My software is free, but my time generally not.
>
> --
>
> http://www.spatialys.com
>
> My software is free, but my time generally not.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20221120/0637504f/attachment.htm>