[gdal-dev] errors using IAM instance profile auth in s3

Michael Smith michael.smith.erdc at gmail.com
Sun Nov 20 05:56:52 PST 2022 

If I duplicate the open line in gdalinfo.py,

    hDataset = gdal.Open(pszFilename, gdal.GA_ReadOnly)
    hDataset = gdal.Open(pszFilename, gdal.GA_ReadOnly)

python
/opt/conda/envs/grid/lib/python3.11/site-packages/osgeo_utils/samples/gdalinfo.py
/vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif
Driver: GTiff/GeoTIFF
Files: /vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif
Size is 75000, 52000
Coordinate System is:
PROJCS["Estonian Coordinate System of 1997",
    GEOGCS["EST97",
        DATUM["Estonia_1997",
            SPHEROID["GRS 1980",6378137,298.257222101,
                AUTHORITY["EPSG","7019"]],
            AUTHORITY["EPSG","6180"]],
        PRIMEM["Greenwich",0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.0174532925199433,
            AUTHORITY["EPSG","9122"]],
        AUTHORITY["EPSG","4180"]],
    PROJECTION["Lambert_Conformal_Conic_2SP"],
    PARAMETER["latitude_of_origin",57.5175539305556],
    PARAMETER["central_meridian",24],
    PARAMETER["standard_parallel_1",59.3333333333333],
    PARAMETER["standard_parallel_2",58],
    PARAMETER["false_easting",500000],
    PARAMETER["false_northing",6375000],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    AXIS["Northing",NORTH],
    AXIS["Easting",EAST],
    AUTHORITY["EPSG","3301"]]
Origin = (365000.000000000000000,6635000.000000000000000)
Pixel Size = (5.000000000000000,-5.000000000000000)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  COMPRESSION=LZW
  INTERLEAVE=BAND
  LAYOUT=COG
Corner Coordinates:
Upper Left  (  365000.000, 6635000.000) (  1d39'11.60"W, 60d49'17.85"N)
Lower Left  (  365000.000, 6375000.000) (  0d52'55.22"W, 59d32'29.94"N)
Upper Right (  740000.000, 6635000.000) (  0d11'58.07"E, 61d58' 8.49"N)
Lower Right (  740000.000, 6375000.000) (  1d 0'53.53"E, 60d39'59.57"N)
Center      (  552500.000, 6505000.000) (  0d19'56.63"W, 60d44'44.22"N)
Band 1 Block=256x256 Type=Float32, ColorInterp=Gray
  NoData Value=-9999
  Overviews: 37500x26000, 18750x13000, 9375x6500, 4688x3250, 2344x1625,
1172x813, 586x407, 293x204

On Sun, Nov 20, 2022 at 8:53 AM Michael Smith <michael.smith.erdc at gmail.com>
wrote:

> Oh, its just the second open that works
>
> In [1]: from osgeo import gdal
> In [2]: pszFilename =
> "/vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif"
> In [3]: hDataset = gdal.OpenEx(pszFilename, gdal.GA_ReadOnly)
> In [4]: hDataset
> In [5]: hDataset = gdal.Open(pszFilename, gdal.GA_ReadOnly)
> In [6]: hDataset
> Out[6]: <osgeo.gdal.Dataset; proxy of <Swig Object of type
> 'GDALDatasetShadow *' at 0x7fef5b3fe1c0> >
> In [7]: hDataset.GetProjectionRef()
> Out[7]: 'PROJCS["Estonian Coordinate System of
> 1997",GEOGCS["EST97",DATUM["Estonia_1997",SPHEROID["GRS
> 1980",6378137,298.257222101,AUTHORITY["EPSG","7019"]],AUTHORITY["EPSG","6180"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4180"]],PROJECTION["Lambert_Conformal_Conic_2SP"],PARAMETER["latitude_of_origin",57.5175539305556],PARAMETER["central_meridian",24],PARAMETER["standard_parallel_1",59.3333333333333],PARAMETER["standard_parallel_2",58],PARAMETER["false_easting",500000],PARAMETER["false_northing",6375000],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["Northing",NORTH],AXIS["Easting",EAST],AUTHORITY["EPSG","3301"]]'
>
> On Sun, Nov 20, 2022 at 8:47 AM Michael Smith <
> michael.smith.erdc at gmail.com> wrote:
>
>> Is there a reason why OpenEx would work but Open wouldn’t?
>>
>>
>>
>> In [1]: from osgeo import gdal
>>
>> In [2]: pszFilename =
>> "/vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif"
>>
>> In [3]: hDataset = gdal.Open(pszFilename, gdal.GA_ReadOnly)
>>
>> In [4]: hDataset
>>
>> In [5]: hDataset = gdal.OpenEx(pszFilename, gdal.GA_ReadOnly)
>>
>> In [6]: hDataset
>>
>> Out[6]: <osgeo.gdal.Dataset; proxy of <Swig Object of type
>> 'GDALDatasetShadow *' at 0x7f827217c450> >
>>
>> In [7]: hDataset.GetGeoTransform()
>>
>> Out[7]: (365000.0, 5.0, 0.0, 6635000.0, 0.0, -5.0)
>>
>>
>>
>> Mike
>>
>>
>>
>>
>>
>> *From: *Even Rouault <even.rouault at spatialys.com>
>> *Date: *Saturday, November 19, 2022 at 10:08 AM
>> *To: *<michael.smith.erdc at gmail.com>
>> *Cc: *gdal-dev <gdal-dev at lists.osgeo.org>
>> *Subject: *Re: [gdal-dev] errors using IAM instance profile auth in s3
>>
>>
>>
>>
>>
>> Le 19/11/2022 à 16:00, michael.smith.erdc at gmail.com a écrit :
>>
>> Correct, not a public bucket, which is why the IAM credentials are
>> needed. If I set them manually, it all works fine.
>>
>> That's super weird if the result of a range request changes depending on
>> how credentials have been set... Perhaps enable CPL_CURL_VERBOSE=ON env
>> variable and diff the logs ?
>>
>> You could also try the gdal_cp.py sample script at
>> https://github.com/OSGeo/gdal/blob/master/swig/python/gdal-utils/osgeo_utils/samples/gdal_cp.py
>> , which is a cp-like utility working with GDAL virtual file systems, with
>> the 2 authentication methods
>>
>> python gdal_cp.py
>> /vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif out.tif
>>
>> (you can interrupt it with ctrl-c after a few seconds. that will be
>> enough to get the first bytes)
>>
>> you might need to run an hexadecimal editor to inspect a bit the content.
>>
>>
>>
>> [ u02]$ export AWS_ACCESS_KEY_ID=xxxxx
>>
>> Yes, a 206 response code means success here as we are requesting only
>> bytes 0-16383. So maybe the file is not a valid TIFF ?
>>
>> ( "grid-dev-publiclidar" must not be so public I guess, because when
>> trying with my credentials, I get a Access Denied)
>>
>> Le 19/11/2022 à 15:40, michael.smith.erdc at gmail.com a écrit :
>>
>> I’m seeing that it’s getting a 206 response code, so wouldn’t that
>> indicate auth is working?
>>
>>
>>
>>  gdalinfo /vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif
>>
>> HTTP: Fetch(http://169.254.169.254/latest/api/token)
>>
>> HTTP: libcurl/7.86.0 OpenSSL/3.0.7 zlib/1.2.13 libssh2/1.10.0
>> nghttp2/1.47.0
>>
>> HTTP: These HTTP headers were set: X-aws-ec2-metadata-token-ttl-seconds:
>> 10
>>
>> HTTP: Fetch(
>> http://169.254.169.254/latest/meta-data/iam/security-credentials/)
>>
>> HTTP: Fetch(
>> http://169.254.169.254/latest/meta-data/iam/security-credentials/iam-grid-s3
>> )
>>
>> AWS: Storing AIM credentials until 2022-11-19T20:42:58Z
>>
>> S3: Downloading 0-16383 (
>> https://grid-dev-publiclidar.s3.amazonaws.com/estonia/dtm/estonia_dtm_5m.tif
>> )...
>>
>> S3: Got response_code=206
>>
>> gdalinfo failed - unable to open
>> '/vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif'.
>>
>>
>>
>>
>>
>> Mike
>>
>>
>>
>>
>>
>>
>>
>> On Nov 19, 2022, at 9:26 AM, Even Rouault <even.rouault at spatialys.com>
>> <even.rouault at spatialys.com> wrote:
>>
>> Hi Mike,
>>
>> could you send the output of
>>
>> curl
>> http://169.254.169.254/latest/meta-data/iam/security-credentials/iam-grid-s3
>>
>> Slightly redacted of course, but with the exact formatting. This part of
>> thee code currently uses a "simple JSON parser" (
>> https://github.com/OSGeo/gdal/blob/c61d116a469821b769630a112dee7f1a61fed885/port/cpl_aws.cpp#L554),
>> which is actually just a non JSON-aware string tokenizer, and I suspect it
>> could be defeated by a new formatting of S3 or something specific to your
>> credentials.
>>
>> It could also be that something unhandled by that parser appears inside
>> quoted strings, like an escaped double quote or some other JSON escaped
>> character (like an escaped forward slash \/ )
>>
>> If that was the case we should likely switch to proper JSON
>> deserialization (that part of the code must predate libjson-c being a build
>> requirement of GDAL).
>>
>> Even
>>
>>
>> --
>> http://www.spatialys.com
>> My software is free, but my time generally not.
>>
>> --
>>
>> http://www.spatialys.com
>>
>> My software is free, but my time generally not.
>>
>> --
>>
>> http://www.spatialys.com
>>
>> My software is free, but my time generally not.
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20221120/5f0f2271/attachment-0001.htm>

More information about the gdal-dev mailing list