<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-ligatures:standardcontextual;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
font-size:10.0pt;
font-family:"Courier New";
mso-ligatures:none;
mso-fareast-language:EN-GB;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Consolas",serif;
mso-ligatures:standardcontextual;
mso-fareast-language:EN-US;}
span.EmailStyle23
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style></head><body lang=en-NL link="#0563C1" vlink=purple style='word-wrap:break-word'><div class=WordSection1><p class=MsoNormal><span lang=EN-US>That does it, thank you so much!<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><div id=mail-editor-reference-message-container><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal style='margin-bottom:12.0pt'><b><span style='font-size:12.0pt;color:black'>From: </span></b><span style='font-size:12.0pt;color:black'>Even Rouault <even.rouault@spatialys.com><br><b>Date: </b>Thursday, 20 July 2023 at 11:44<br><b>To: </b>bcoerver@mailbox.org <b.coerver@mailbox.org>, gdal-dev@lists.osgeo.org <gdal-dev@lists.osgeo.org><br><b>Subject: </b>Re: [gdal-dev] /vsis3/ on NetCDF from Earthdata</span><span style='font-size:12.0pt;color:black;mso-ligatures:none;mso-fareast-language:EN-GB'><o:p></o:p></span></p></div><p>Bert,<o:p></o:p></p><p>Also set the GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR config option, otherwise the generic open mechanism of GDAL tries to list the content of the VNP02IMG/ directory and it seems there are tons of files there<o:p></o:p></p><p>When doing that, I get a result within a few seconds<o:p></o:p></p><p>Even<o:p></o:p></p><div><p class=MsoNormal>Le 20/07/2023 à 09:59, b.coerver--- via gdal-dev a écrit :<o:p></o:p></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal><span lang=EN-US style='color:#212121'>Hello,</span><o:p></o:p></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> </span><o:p></o:p></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='color:#212121'>I'm trying to access data from NASA's Earthdata S3 buckets, but I get a `"<filename> does not exist in the file system, and is not recognized as a supported dataset name."` error after waiting a long time (± 50 minutes, the process is downloading some data the whole time) doing the following:</span><o:p></o:p></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> </span><o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt;caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='color:#212121'>from osgeo import gdal</span><o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt;caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> </span><o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt;caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='color:#212121'>gdal_config_options = {</span><o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt;caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> "AWS_ACCESS_KEY_ID": creds["accessKeyId"],</span><o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt;caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> "AWS_SESSION_TOKEN": creds["sessionToken"],</span><o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt;caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> "AWS_SECRET_ACCESS_KEY": creds["secretAccessKey"],</span><o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt;caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> "AWS_REGION": "us-west-2",</span><o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt;caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='color:#212121'>}</span><o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt;caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> </span><o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt;caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='color:#212121'>url = "/vsis3/prod-lads/VNP02IMG/VNP02IMG.A2023193.1942.002.2023194025636.nc"</span><o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt;caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> <span class=apple-converted-space> </span></span><o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt;caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='color:#212121'>for k, v in gdal_config_options.items():</span><o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt;caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> gdal.SetConfigOption(k, v)</span><o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt;caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> </span><o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt;caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='color:#212121'>out =<span class=apple-converted-space> </span>gdal.Info(url)</span><o:p></o:p></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> </span><o:p></o:p></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='color:#212121'>The `creds` variable is a dictionary with temporary credential information that I get from [here](</span><span lang=NL style='color:#212121'><a href="https://data.laadsdaac.earthdatacloud.nasa.gov/s3credentials" title="https://data.laadsdaac.earthdatacloud.nasa.gov/s3credentials"><span lang=EN-US style='color:#0078D7'>https://data.laadsdaac.earthdatacloud.nasa.gov/s3credentials</span></a></span><span lang=EN-US style='color:#212121'>), you need a free account to get them.</span><o:p></o:p></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> </span><o:p></o:p></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='color:#212121'>When I introduce an error in one of the keys/tokens (e.g. `"AWS_ACCESS_KEY_ID": creds["accessKeyId"] + "x"`, I do get a message immediately saying my credentials are unknown. So I do think they are being ingested correctly. I’m using GDAL version 3.7.1.</span><o:p></o:p></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> </span><o:p></o:p></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='color:#212121'>I also managed to download the entire file using `boto3`, by doing the following:</span><o:p></o:p></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> </span><o:p></o:p></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> import boto3</span><o:p></o:p></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> </span><o:p></o:p></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> client = boto3.client(</span><o:p></o:p></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> 's3',</span><o:p></o:p></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> aws_access_key_id=creds["accessKeyId"],</span><o:p></o:p></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> aws_secret_access_key=creds["secretAccessKey"],</span><o:p></o:p></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> aws_session_token=creds["sessionToken"]</span><o:p></o:p></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> )</span><o:p></o:p></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> <span class=apple-converted-space> </span></span><o:p></o:p></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> client.download_file('prod-lads', 'VNP02IMG/VNP02IMG.A2023193.1942.002.2023194025636.nc', 'test.nc')</span><o:p></o:p></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> <span class=apple-converted-space> </span></span><o:p></o:p></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='color:#212121'>Any ideas what I'm doing wrong or how to make this work? In the end I'm interested in accessing the files metadata without downloading the entire file.</span><o:p></o:p></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='color:#212121'> </span><o:p></o:p></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='color:#212121'>Regards,</span><o:p></o:p></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='color:#212121'>Bert</span><o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal><span style='mso-ligatures:none;mso-fareast-language:EN-GB'><br><br><o:p></o:p></span></p><pre>_______________________________________________<o:p></o:p></pre><pre>gdal-dev mailing list<o:p></o:p></pre><pre><a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a><o:p></o:p></pre><pre><a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a><o:p></o:p></pre></blockquote><pre>-- <o:p></o:p></pre><pre><a href="http://www.spatialys.com">http://www.spatialys.com</a><o:p></o:p></pre><pre>My software is free, but my time generally not.<o:p></o:p></pre></div></div></div></body></html>