[gdal-dev] errors using IAM instance profile auth in s3

[Even Rouault]

Le 19/11/2022 à 16:00, michael.smith.erdc at gmail.com a écrit :
> Correct, not a public bucket, which is why the IAM credentials are 
> needed. If I set them manually, it all works fine.

That's super weird if the result of a range request changes depending on 
how credentials have been set... Perhaps enable CPL_CURL_VERBOSE=ON env 
variable and diff the logs ?

You could also try the gdal_cp.py sample script at 
https://github.com/OSGeo/gdal/blob/master/swig/python/gdal-utils/osgeo_utils/samples/gdal_cp.py 
, which is a cp-like utility working with GDAL virtual file systems, 
with the 2 authentication methods

python gdal_cp.py 
/vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif out.tif

(you can interrupt it with ctrl-c after a few seconds. that will be 
enough to get the first bytes)

you might need to run an hexadecimal editor to inspect a bit the content.

>
> [ u02]$ export AWS_ACCESS_KEY_ID=xxxxx
>
> Yes, a 206 response code means success here as we are requesting only 
> bytes 0-16383. So maybe the file is not a valid TIFF ?
>
> ( "grid-dev-publiclidar" must not be so public I guess, because when 
> trying with my credentials, I get a Access Denied)
>
> Le 19/11/2022 à 15:40, michael.smith.erdc at gmail.com a écrit :
>> I’m seeing that it’s getting a 206 response code, so wouldn’t that 
>> indicate auth is working?
>>
>>  gdalinfo /vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif
>> HTTP: Fetch(http://169.254.169.254/latest/api/token)
>> HTTP: libcurl/7.86.0 OpenSSL/3.0.7 zlib/1.2.13 libssh2/1.10.0 
>> nghttp2/1.47.0
>> HTTP: These HTTP headers were set: 
>> X-aws-ec2-metadata-token-ttl-seconds: 10
>> HTTP: 
>> Fetch(http://169.254.169.254/latest/meta-data/iam/security-credentials/)
>> HTTP: 
>> Fetch(http://169.254.169.254/latest/meta-data/iam/security-credentials/iam-grid-s3)
>> AWS: Storing AIM credentials until 2022-11-19T20:42:58Z
>> S3: Downloading 0-16383 
>> (https://grid-dev-publiclidar.s3.amazonaws.com/estonia/dtm/estonia_dtm_5m.tif)...
>> S3: Got response_code=206
>> gdalinfo failed - unable to open 
>> '/vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif'.
>>
>>
>> Mike
>>
>>
>>
>>> On Nov 19, 2022, at 9:26 AM, Even Rouault 
>>> <even.rouault at spatialys.com> wrote:
>>>
>>> Hi Mike,
>>>
>>> could you send the output of
>>>
>>> curl 
>>> http://169.254.169.254/latest/meta-data/iam/security-credentials/iam-grid-s3
>>>
>>> Slightly redacted of course, but with the exact formatting. This 
>>> part of thee code currently uses a "simple JSON parser" 
>>> (https://github.com/OSGeo/gdal/blob/c61d116a469821b769630a112dee7f1a61fed885/port/cpl_aws.cpp#L554), 
>>> which is actually just a non JSON-aware string tokenizer, and I 
>>> suspect it could be defeated by a new formatting of S3 or something 
>>> specific to your credentials.
>>>
>>> It could also be that something unhandled by that parser appears 
>>> inside quoted strings, like an escaped double quote or some other 
>>> JSON escaped character (like an escaped forward slash \/ )
>>>
>>> If that was the case we should likely switch to proper JSON 
>>> deserialization (that part of the code must predate libjson-c being 
>>> a build requirement of GDAL).
>>>
>>> Even
>>>
>>>
>>> -- 
>>> http://www.spatialys.com
>>> My software is free, but my time generally not.
>>>
> -- 
> http://www.spatialys.com
> My software is free, but my time generally not.

-- 
http://www.spatialys.com
My software is free, but my time generally not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20221119/92703a05/attachment.htm>