[gdal-dev] Gdal.Open on S3 using IAM roles

[Patrick Young]

That should be the behavior, and I use it all the time so I am confident it
works...

Is it possible you have set any of the AWS_* variables in your
EC2/container environment? I think that could spoil it for you.

You can set CPL_CURL_VERBOSE to YES and get an idea of the network requests
GDAL is doing.   If things are just slow, you might need to set
GDAL_DISABLE_READDIR_ON_OPEN=YES
and CPL_VSIL_CURL_ALLOWED_EXTENSIONS=tif  as described here:

https://trac.osgeo.org/gdal/wiki/CloudOptimizedGeoTIFF

On Wed, Apr 14, 2021 at 3:01 PM Jeannie May <jeannie_may at trimble.com> wrote:

> Further to this discussion, I guess we had hoped that GDal would pick up
> the EC2 instance profile credentials as the s3Client upload does as per:
>
> https://gdal.org/user/virtual_file_systems.html#vsis3
>     5. If none of the above method succeeds, instance profile credentials
> will be retrieved when GDAL is used on EC2 instances.
>
> On Thu, Apr 15, 2021 at 8:58 AM Jeannie May <jeannie_may at trimble.com>
> wrote:
>
>> Thank you Patrick for your prompt reply.
>>
>> Our app runs on an EC2 instance and has no user context. It uses the
>> s3Client SDK upload, utilizing the existing IAM role/policy already setup.
>>
>> Are you saying here that I need to as a 1-time process generate a secret
>> and access key for our existing policy/role, store it in environment
>> variables (AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID only?) and the
>> gdal SDK will use that?
>>
>> Jeannie May
>>
>>
>>
>> Jeannie M
>>
>>
>>
>> ay <jeannie_may at trimble.com>
>> Tue, Apr 13, 2:44 PM (2 days ago)
>> Reply
>> to gdal-dev
>> I'm new to using Gdal. How do I configure gdal to use an IAM role, rather
>> than defining an aws-Profile?
>>
>> I get a timeout doing a gdal.Open() on a tiff file on S3 using vsis3,
>> while trying to use an IAM role.
>>
>> I'm using MaxRev.Gdal.Core 3.2.0.250. Netcore 3.1 c#, running in a Linux
>> container.
>>
>> Note that defining an AWS_Profile etc works ok, but I need to use IAM
>> roles. Prior to trying to open the file with gdal, I successfully upload
>> using s3Client (which uses the IAM role), so it's something specific to the
>> Gdal.Open().
>>
>> using GetConfigOption() I can see that none of the following are set
>> AWS_PROFILE; AWS_ACCESS_KEY_ID; AWS_SECRET_ACCESS_KEY; AWS_SESSION_TOKEN;
>> AWS_NO_SIGN_REQUEST; AWS_DEFAULT_PROFILE
>>
>>
>> --
>> *Jeannie May*
>> Senior Software Engineer
>> 11 Birmingham Drive, Christchurch | 963 5305 Office
>> www.trimble.com
>>
>> Connect with us!
>> Patrick Young <patrick.mckendree.young at gmail.com>
>> Wed, Apr 14, 3:47 AM (1 day ago)
>> Reply
>> to me, gdal
>> See
>> https://aws.amazon.com/premiumsupport/knowledge-center/iam-assume-role-cli/ ,
>> they describe how to assume a role and set the usual AWS_*
>> environment variables that GDAL should pick up.
>>
>> There's discussion on vsis3 related stuff (e.g. authentication) here:
>> https://gdal.org/user/virtual_file_systems.html#vsis3
>>
>> P
>>
>>
>>
>>
>>
>
> --
> *Jeannie May*
> Senior Software Engineer
> 11 Birmingham Drive, Christchurch | 963 5305 Office
> www.trimble.com
>
> Connect with us!
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20210414/fbe3af7c/attachment-0001.html>