<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-ligatures:standardcontextual;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style></head><body lang=en-NL link="#0563C1" vlink="#954F72" style='word-wrap:break-word'><div class=WordSection1><p class=MsoNormal><span lang=EN-US style='color:#212121'>Hello,</span><span style='color:#212121'><o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(33, 33, 33);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> </span><span style='color:#212121'><o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(33, 33, 33);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='color:#212121'>I'm trying to access data from NASA's Earthdata S3 buckets, but I get a `"<filename> does not exist in the file system, and is not recognized as a supported dataset name."` error after waiting a long time (± 50 minutes, the process is downloading some data the whole time) doing the following:</span><span style='color:#212121'><o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(33, 33, 33);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> </span><span style='color:#212121'><o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt;caret-color: rgb(33, 33, 33);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='color:#212121'>from osgeo import gdal</span><span style='color:#212121'><o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt;caret-color: rgb(33, 33, 33);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> </span><span style='color:#212121'><o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt;caret-color: rgb(33, 33, 33);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='color:#212121'>gdal_config_options = {</span><span style='color:#212121'><o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt;caret-color: rgb(33, 33, 33);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> "AWS_ACCESS_KEY_ID": creds["accessKeyId"],</span><span style='color:#212121'><o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt;caret-color: rgb(33, 33, 33);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> "AWS_SESSION_TOKEN": creds["sessionToken"],</span><span style='color:#212121'><o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt;caret-color: rgb(33, 33, 33);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> "AWS_SECRET_ACCESS_KEY": creds["secretAccessKey"],</span><span style='color:#212121'><o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt;caret-color: rgb(33, 33, 33);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> "AWS_REGION": "us-west-2",</span><span style='color:#212121'><o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt;caret-color: rgb(33, 33, 33);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='color:#212121'>}</span><span style='color:#212121'><o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt;caret-color: rgb(33, 33, 33);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> </span><span style='color:#212121'><o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt;caret-color: rgb(33, 33, 33);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='color:#212121'>url = "/vsis3/prod-lads/VNP02IMG/VNP02IMG.A2023193.1942.002.2023194025636.nc"</span><span style='color:#212121'><o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt;caret-color: rgb(33, 33, 33);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> <span class=apple-converted-space> </span></span><span style='color:#212121'><o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt;caret-color: rgb(33, 33, 33);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='color:#212121'>for k, v in gdal_config_options.items():</span><span style='color:#212121'><o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt;caret-color: rgb(33, 33, 33);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> gdal.SetConfigOption(k, v)</span><span style='color:#212121'><o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt;caret-color: rgb(33, 33, 33);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> </span><span style='color:#212121'><o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt;caret-color: rgb(33, 33, 33);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='color:#212121'>out =<span class=apple-converted-space> </span>gdal.Info(url)</span><span style='color:#212121'><o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(33, 33, 33);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> </span><span style='color:#212121'><o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(33, 33, 33);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='color:#212121'>The `creds` variable is a dictionary with temporary credential information that I get from [here](</span><span lang=NL style='color:#212121'><a href="https://data.laadsdaac.earthdatacloud.nasa.gov/s3credentials" title="https://data.laadsdaac.earthdatacloud.nasa.gov/s3credentials"><span lang=EN-US style='color:#0078D7'>https://data.laadsdaac.earthdatacloud.nasa.gov/s3credentials</span></a></span><span lang=EN-US style='color:#212121'>), you need a free account to get them.</span><span style='color:#212121'><o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(33, 33, 33);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> </span><span style='color:#212121'><o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(33, 33, 33);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='color:#212121'>When I introduce an error in one of the keys/tokens (e.g. `"AWS_ACCESS_KEY_ID": creds["accessKeyId"] + "x"`, I do get a message immediately saying my credentials are unknown. So I do think they are being ingested correctly. I’m using GDAL version 3.7.1.</span><span style='color:#212121'><o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(33, 33, 33);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> </span><span style='color:#212121'><o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(33, 33, 33);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='color:#212121'>I also managed to download the entire file using `boto3`, by doing the following:</span><span style='color:#212121'><o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(33, 33, 33);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> </span><span style='color:#212121'><o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(33, 33, 33);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> import boto3</span><span style='color:#212121'><o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(33, 33, 33);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> </span><span style='color:#212121'><o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(33, 33, 33);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> client = boto3.client(</span><span style='color:#212121'><o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(33, 33, 33);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> 's3',</span><span style='color:#212121'><o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(33, 33, 33);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> aws_access_key_id=creds["accessKeyId"],</span><span style='color:#212121'><o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(33, 33, 33);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> aws_secret_access_key=creds["secretAccessKey"],</span><span style='color:#212121'><o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(33, 33, 33);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> aws_session_token=creds["sessionToken"]</span><span style='color:#212121'><o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(33, 33, 33);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> )</span><span style='color:#212121'><o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(33, 33, 33);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> <span class=apple-converted-space> </span></span><span style='color:#212121'><o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(33, 33, 33);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> client.download_file('prod-lads', 'VNP02IMG/VNP02IMG.A2023193.1942.002.2023194025636.nc', 'test.nc')</span><span style='color:#212121'><o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(33, 33, 33);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> <span class=apple-converted-space> </span></span><span style='color:#212121'><o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(33, 33, 33);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='color:#212121'>Any ideas what I'm doing wrong or how to make this work? In the end I'm interested in accessing the files metadata without downloading the entire file.</span><span style='color:#212121'><o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(33, 33, 33);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> </span><span style='color:#212121'><o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(33, 33, 33);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='color:#212121'>Regards,</span><span style='color:#212121'><o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(33, 33, 33);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span lang=EN-US style='color:#212121'>Bert</span><span style='color:#212121'><o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p></div></body></html>