[gdal-dev] Gdal.Open on S3 using IAM roles

Wed Apr 14 14:28:21 PDT 2021

AWS_REGION shouldn't cause problems.

If it was me, I'd just hop onto the box and test by running gdalinfo
/vsis3/... with verbose curl config options set and so on and make sure I
could get that working as a sanity check.

On Wed, Apr 14, 2021 at 3:21 PM Jeannie May <jeannie_may at trimble.com> wrote:

> This is the gdal setup. None of the AWS keys are set excep region (could
> that be an issue). Where would I expect to see the logging - none is
> showing up in our logs. Do we need to configure the output to our log?
>
>     public GdalClient(IConfigurationStore configStore, ILoggerFactory
> loggerFactory)
>     {
>       _log = loggerFactory.CreateLogger<GdalClient>();
>       GdalBase.ConfigureAll();
>       Gdal.AllRegister();
>
>       //For debugging s3 access problems
>       Gdal.SetConfigOption("CPL_CURL_VERBOSE", "YES");
>       Gdal.SetConfigOption("CPL_DEBUG", "YES");
>       Gdal.SetConfigOption("AWS_REGION", "us-west-2");
>
>       _log.LogDebug($"#{nameof(GdalClient)} AWS_PROFILE: {
> Gdal.GetConfigOption("AWS_PROFILE", "NotFound")} " +
>                     $"AWS_ACCESS_KEY_ID: {
> Gdal.GetConfigOption("AWS_ACCESS_KEY_ID", "NotFound")} " +
>                     $"AWS_SECRET_ACCESS_KEY: {
> Gdal.GetConfigOption("AWS_SECRET_ACCESS_KEY", "NotFound")} " +
>                     $"AWS_SESSION_TOKEN: {
> Gdal.GetConfigOption("AWS_SESSION_TOKEN", "NotFound")} " +
>                     $"AWS_NO_SIGN_REQUEST: {
> Gdal.GetConfigOption("AWS_NO_SIGN_REQUEST", "NotFound")} " +
>                     $"AWS_DEFAULT_PROFILE: {
> Gdal.GetConfigOption("AWS_DEFAULT_PROFILE", "NotFound")} ");
>       _awsBucketName =
> configStore.GetValueString("AWS_DESIGNIMPORT_BUCKET_NAME");
>     }
>
> and this is the open which times out:
>  public string GetMetadata(string fileName, bool s3=true)
>     {
>       var path = s3 ? $"/vsis3/{_awsBucketName}/{fileName}" : fileName;
>       var dataset = Gdal.Open(path, Access.GA_ReadOnly);
> .........
>
> On Thu, Apr 15, 2021 at 9:11 AM Patrick Young <
> patrick.mckendree.young at gmail.com> wrote:
>
>> That should be the behavior, and I use it all the time so I am confident
>> it works...
>>
>> Is it possible you have set any of the AWS_* variables in your
>> EC2/container environment? I think that could spoil it for you.
>>
>> You can set CPL_CURL_VERBOSE to YES and get an idea of the network
>> requests GDAL is doing.   If things are just slow, you might need to set GDAL_DISABLE_READDIR_ON_OPEN=YES
>> and CPL_VSIL_CURL_ALLOWED_EXTENSIONS=tif  as described here:
>>
>> https://trac.osgeo.org/gdal/wiki/CloudOptimizedGeoTIFF
>>
>> On Wed, Apr 14, 2021 at 3:01 PM Jeannie May <jeannie_may at trimble.com>
>> wrote:
>>
>>> Further to this discussion, I guess we had hoped that GDal would pick up
>>> the EC2 instance profile credentials as the s3Client upload does as per:
>>>
>>> https://gdal.org/user/virtual_file_systems.html#vsis3
>>>     5. If none of the above method succeeds, instance profile
>>> credentials will be retrieved when GDAL is used on EC2 instances.
>>>
>>> On Thu, Apr 15, 2021 at 8:58 AM Jeannie May <jeannie_may at trimble.com>
>>> wrote:
>>>
>>>> Thank you Patrick for your prompt reply.
>>>>
>>>> Our app runs on an EC2 instance and has no user context. It uses the
>>>> s3Client SDK upload, utilizing the existing IAM role/policy already setup.
>>>>
>>>> Are you saying here that I need to as a 1-time process generate a
>>>> secret and access key for our existing policy/role, store it in environment
>>>> variables (AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID only?) and the
>>>> gdal SDK will use that?
>>>>
>>>> Jeannie May
>>>>
>>>>
>>>>
>>>> Jeannie M
>>>>
>>>>
>>>>
>>>> ay <jeannie_may at trimble.com>
>>>> Tue, Apr 13, 2:44 PM (2 days ago)
>>>> Reply
>>>> to gdal-dev
>>>> I'm new to using Gdal. How do I configure gdal to use an IAM role,
>>>> rather than defining an aws-Profile?
>>>>
>>>> I get a timeout doing a gdal.Open() on a tiff file on S3 using vsis3,
>>>> while trying to use an IAM role.
>>>>
>>>> I'm using MaxRev.Gdal.Core 3.2.0.250. Netcore 3.1 c#, running in a
>>>> Linux container.
>>>>
>>>> Note that defining an AWS_Profile etc works ok, but I need to use IAM
>>>> roles. Prior to trying to open the file with gdal, I successfully upload
>>>> using s3Client (which uses the IAM role), so it's something specific to the
>>>> Gdal.Open().
>>>>
>>>> using GetConfigOption() I can see that none of the following are set
>>>> AWS_PROFILE; AWS_ACCESS_KEY_ID; AWS_SECRET_ACCESS_KEY; AWS_SESSION_TOKEN;
>>>> AWS_NO_SIGN_REQUEST; AWS_DEFAULT_PROFILE
>>>>
>>>>
>>>> --
>>>> *Jeannie May*
>>>> Senior Software Engineer
>>>> 11 Birmingham Drive, Christchurch | 963 5305 Office
>>>> www.trimble.com
>>>>
>>>> Connect with us!
>>>> Patrick Young <patrick.mckendree.young at gmail.com>
>>>> Wed, Apr 14, 3:47 AM (1 day ago)
>>>> Reply
>>>> to me, gdal
>>>> See
>>>> https://aws.amazon.com/premiumsupport/knowledge-center/iam-assume-role-cli/ ,
>>>> they describe how to assume a role and set the usual AWS_*
>>>> environment variables that GDAL should pick up.
>>>>
>>>> There's discussion on vsis3 related stuff (e.g. authentication) here:
>>>> https://gdal.org/user/virtual_file_systems.html#vsis3
>>>>
>>>> P
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>> --
>>> *Jeannie May*
>>> Senior Software Engineer
>>> 11 Birmingham Drive, Christchurch | 963 5305 Office
>>> www.trimble.com
>>>
>>> Connect with us!
>>>
>>
>
> --
> *Jeannie May*
> Senior Software Engineer
> 11 Birmingham Drive, Christchurch | 963 5305 Office
> www.trimble.com
>
> Connect with us!
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20210414/32fa98f2/attachment-0001.html>