[gdal-dev] errors using IAM instance profile auth in s3

[Michael Smith]

Using gdal_cp.py, it copies using IAM credentials (and using key credentials from the curl call). Now I’m even more confused.

 

I was able to replicate with the docker image osgeo/gdal:latest, so its not just my own docker image although it could be my own environment.

 

Will continue to debug.

 

Mike

 

From: Even Rouault <even.rouault at spatialys.com>
Date: Saturday, November 19, 2022 at 10:08 AM
To: <michael.smith.erdc at gmail.com>
Cc: gdal-dev <gdal-dev at lists.osgeo.org>
Subject: Re: [gdal-dev] errors using IAM instance profile auth in s3

 

 

Le 19/11/2022 à 16:00, michael.smith.erdc at gmail.com a écrit :

Correct, not a public bucket, which is why the IAM credentials are needed. If I set them manually, it all works fine.

That's super weird if the result of a range request changes depending on how credentials have been set... Perhaps enable CPL_CURL_VERBOSE=ON env variable and diff the logs ?

You could also try the gdal_cp.py sample script at https://github.com/OSGeo/gdal/blob/master/swig/python/gdal-utils/osgeo_utils/samples/gdal_cp.py , which is a cp-like utility working with GDAL virtual file systems, with the 2 authentication methods

python gdal_cp.py /vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif out.tif

(you can interrupt it with ctrl-c after a few seconds. that will be enough to get the first bytes)

you might need to run an hexadecimal editor to inspect a bit the content.

 

[ u02]$ export AWS_ACCESS_KEY_ID=xxxxx

Yes, a 206 response code means success here as we are requesting only bytes 0-16383. So maybe the file is not a valid TIFF ?

( "grid-dev-publiclidar" must not be so public I guess, because when trying with my credentials, I get a Access Denied)

Le 19/11/2022 à 15:40, michael.smith.erdc at gmail.com a écrit :

I’m seeing that it’s getting a 206 response code, so wouldn’t that indicate auth is working? 

 

 gdalinfo /vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif

HTTP: Fetch(http://169.254.169.254/latest/api/token)

HTTP: libcurl/7.86.0 OpenSSL/3.0.7 zlib/1.2.13 libssh2/1.10.0 nghttp2/1.47.0

HTTP: These HTTP headers were set: X-aws-ec2-metadata-token-ttl-seconds: 10

HTTP: Fetch(http://169.254.169.254/latest/meta-data/iam/security-credentials/)

HTTP: Fetch(http://169.254.169.254/latest/meta-data/iam/security-credentials/iam-grid-s3)

AWS: Storing AIM credentials until 2022-11-19T20:42:58Z

S3: Downloading 0-16383 (https://grid-dev-publiclidar.s3.amazonaws.com/estonia/dtm/estonia_dtm_5m.tif)...

S3: Got response_code=206

gdalinfo failed - unable to open '/vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif'.

 

 

Mike

 

 




On Nov 19, 2022, at 9:26 AM, Even Rouault <even.rouault at spatialys.com> wrote:

Hi Mike,

could you send the output of

curl http://169.254.169.254/latest/meta-data/iam/security-credentials/iam-grid-s3

Slightly redacted of course, but with the exact formatting. This part of thee code currently uses a "simple JSON parser" (https://github.com/OSGeo/gdal/blob/c61d116a469821b769630a112dee7f1a61fed885/port/cpl_aws.cpp#L554), which is actually just a non JSON-aware string tokenizer, and I suspect it could be defeated by a new formatting of S3 or something specific to your credentials.

It could also be that something unhandled by that parser appears inside quoted strings, like an escaped double quote or some other JSON escaped character (like an escaped forward slash \/ )

If that was the case we should likely switch to proper JSON deserialization (that part of the code must predate libjson-c being a build requirement of GDAL).

Even


-- 
http://www.spatialys.com
My software is free, but my time generally not.
-- 
http://www.spatialys.com
My software is free, but my time generally not.
-- 
http://www.spatialys.com
My software is free, but my time generally not.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20221119/56cb8f2f/attachment-0001.htm>