<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Bert,</p>
<p>Also set the GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR config
option, otherwise the generic open mechanism of GDAL tries to list
the content of the VNP02IMG/ directory and it seems there are tons
of files there</p>
<p>When doing that, I get a result within a few seconds</p>
<p>Even<br>
</p>
<div class="moz-cite-prefix">Le 20/07/2023 à 09:59, b.coerver--- via
gdal-dev a écrit :<br>
</div>
<blockquote type="cite"
cite="mid:69C83C86-2257-5245-94C6-FF0C220C778E@hxcore.ol">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style>@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-ligatures:standardcontextual;
mso-fareast-language:EN-US;}a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}span.apple-converted-space
{mso-style-name:apple-converted-space;}MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}div.WordSection1
{page:WordSection1;}</style>
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#212121" lang="EN-US">Hello,</span><span
style="color:#212121"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(33, 33,
33);font-variant-caps: normal;orphans:
auto;text-align:start;widows: auto;-webkit-text-stroke-width:
0px;word-spacing:0px"><span style="color:#212121" lang="EN-US"> </span><span
style="color:#212121"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(33, 33,
33);font-variant-caps: normal;orphans:
auto;text-align:start;widows: auto;-webkit-text-stroke-width:
0px;word-spacing:0px"><span style="color:#212121" lang="EN-US">I'm
trying to access data from NASA's Earthdata S3 buckets, but
I get a `"<filename> does not exist in the file
system, and is not recognized as a supported dataset name."`
error after waiting a long time (± 50 minutes, the process
is downloading some data the whole time) doing the
following:</span><span style="color:#212121"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(33, 33,
33);font-variant-caps: normal;orphans:
auto;text-align:start;widows: auto;-webkit-text-stroke-width:
0px;word-spacing:0px"><span style="color:#212121" lang="EN-US"> </span><span
style="color:#212121"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt;caret-color:
rgb(33, 33, 33);font-variant-caps: normal;orphans:
auto;text-align:start;widows: auto;-webkit-text-stroke-width:
0px;word-spacing:0px"><span style="color:#212121" lang="EN-US">from
osgeo import gdal</span><span style="color:#212121"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt;caret-color:
rgb(33, 33, 33);font-variant-caps: normal;orphans:
auto;text-align:start;widows: auto;-webkit-text-stroke-width:
0px;word-spacing:0px"><span style="color:#212121" lang="EN-US"> </span><span
style="color:#212121"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt;caret-color:
rgb(33, 33, 33);font-variant-caps: normal;orphans:
auto;text-align:start;widows: auto;-webkit-text-stroke-width:
0px;word-spacing:0px"><span style="color:#212121" lang="EN-US">gdal_config_options
= {</span><span style="color:#212121"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt;caret-color:
rgb(33, 33, 33);font-variant-caps: normal;orphans:
auto;text-align:start;widows: auto;-webkit-text-stroke-width:
0px;word-spacing:0px"><span style="color:#212121" lang="EN-US">
"AWS_ACCESS_KEY_ID": creds["accessKeyId"],</span><span
style="color:#212121"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt;caret-color:
rgb(33, 33, 33);font-variant-caps: normal;orphans:
auto;text-align:start;widows: auto;-webkit-text-stroke-width:
0px;word-spacing:0px"><span style="color:#212121" lang="EN-US">
"AWS_SESSION_TOKEN": creds["sessionToken"],</span><span
style="color:#212121"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt;caret-color:
rgb(33, 33, 33);font-variant-caps: normal;orphans:
auto;text-align:start;widows: auto;-webkit-text-stroke-width:
0px;word-spacing:0px"><span style="color:#212121" lang="EN-US">
"AWS_SECRET_ACCESS_KEY": creds["secretAccessKey"],</span><span
style="color:#212121"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt;caret-color:
rgb(33, 33, 33);font-variant-caps: normal;orphans:
auto;text-align:start;widows: auto;-webkit-text-stroke-width:
0px;word-spacing:0px"><span style="color:#212121" lang="EN-US">
"AWS_REGION": "us-west-2",</span><span style="color:#212121"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt;caret-color:
rgb(33, 33, 33);font-variant-caps: normal;orphans:
auto;text-align:start;widows: auto;-webkit-text-stroke-width:
0px;word-spacing:0px"><span style="color:#212121" lang="EN-US">}</span><span
style="color:#212121"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt;caret-color:
rgb(33, 33, 33);font-variant-caps: normal;orphans:
auto;text-align:start;widows: auto;-webkit-text-stroke-width:
0px;word-spacing:0px"><span style="color:#212121" lang="EN-US"> </span><span
style="color:#212121"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt;caret-color:
rgb(33, 33, 33);font-variant-caps: normal;orphans:
auto;text-align:start;widows: auto;-webkit-text-stroke-width:
0px;word-spacing:0px"><span style="color:#212121" lang="EN-US">url
=
"/vsis3/prod-lads/VNP02IMG/VNP02IMG.A2023193.1942.002.2023194025636.nc"</span><span
style="color:#212121"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt;caret-color:
rgb(33, 33, 33);font-variant-caps: normal;orphans:
auto;text-align:start;widows: auto;-webkit-text-stroke-width:
0px;word-spacing:0px"><span style="color:#212121" lang="EN-US"> <span
class="apple-converted-space"> </span></span><span
style="color:#212121"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt;caret-color:
rgb(33, 33, 33);font-variant-caps: normal;orphans:
auto;text-align:start;widows: auto;-webkit-text-stroke-width:
0px;word-spacing:0px"><span style="color:#212121" lang="EN-US">for
k, v in gdal_config_options.items():</span><span
style="color:#212121"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt;caret-color:
rgb(33, 33, 33);font-variant-caps: normal;orphans:
auto;text-align:start;widows: auto;-webkit-text-stroke-width:
0px;word-spacing:0px"><span style="color:#212121" lang="EN-US">
gdal.SetConfigOption(k, v)</span><span style="color:#212121"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt;caret-color:
rgb(33, 33, 33);font-variant-caps: normal;orphans:
auto;text-align:start;widows: auto;-webkit-text-stroke-width:
0px;word-spacing:0px"><span style="color:#212121" lang="EN-US"> </span><span
style="color:#212121"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt;caret-color:
rgb(33, 33, 33);font-variant-caps: normal;orphans:
auto;text-align:start;widows: auto;-webkit-text-stroke-width:
0px;word-spacing:0px"><span style="color:#212121" lang="EN-US">out
=<span class="apple-converted-space"> </span>gdal.Info(url)</span><span
style="color:#212121"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(33, 33,
33);font-variant-caps: normal;orphans:
auto;text-align:start;widows: auto;-webkit-text-stroke-width:
0px;word-spacing:0px"><span style="color:#212121" lang="EN-US"> </span><span
style="color:#212121"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(33, 33,
33);font-variant-caps: normal;orphans:
auto;text-align:start;widows: auto;-webkit-text-stroke-width:
0px;word-spacing:0px"><span style="color:#212121" lang="EN-US">The
`creds` variable is a dictionary with temporary credential
information that I get from [here](</span><span
style="color:#212121" lang="NL"><a
href="https://data.laadsdaac.earthdatacloud.nasa.gov/s3credentials"
title="https://data.laadsdaac.earthdatacloud.nasa.gov/s3credentials"
moz-do-not-send="true"><span style="color:#0078D7"
lang="EN-US">https://data.laadsdaac.earthdatacloud.nasa.gov/s3credentials</span></a></span><span
style="color:#212121" lang="EN-US">), you need a free
account to get them.</span><span style="color:#212121"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(33, 33,
33);font-variant-caps: normal;orphans:
auto;text-align:start;widows: auto;-webkit-text-stroke-width:
0px;word-spacing:0px"><span style="color:#212121" lang="EN-US"> </span><span
style="color:#212121"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(33, 33,
33);font-variant-caps: normal;orphans:
auto;text-align:start;widows: auto;-webkit-text-stroke-width:
0px;word-spacing:0px"><span style="color:#212121" lang="EN-US">When
I introduce an error in one of the keys/tokens (e.g.
`"AWS_ACCESS_KEY_ID": creds["accessKeyId"] + "x"`, I do get
a message immediately saying my credentials are unknown. So
I do think they are being ingested correctly. I’m using GDAL
version 3.7.1.</span><span style="color:#212121"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(33, 33,
33);font-variant-caps: normal;orphans:
auto;text-align:start;widows: auto;-webkit-text-stroke-width:
0px;word-spacing:0px"><span style="color:#212121" lang="EN-US"> </span><span
style="color:#212121"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(33, 33,
33);font-variant-caps: normal;orphans:
auto;text-align:start;widows: auto;-webkit-text-stroke-width:
0px;word-spacing:0px"><span style="color:#212121" lang="EN-US">I
also managed to download the entire file using `boto3`, by
doing the following:</span><span style="color:#212121"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(33, 33,
33);font-variant-caps: normal;orphans:
auto;text-align:start;widows: auto;-webkit-text-stroke-width:
0px;word-spacing:0px"><span style="color:#212121" lang="EN-US"> </span><span
style="color:#212121"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(33, 33,
33);font-variant-caps: normal;orphans:
auto;text-align:start;widows: auto;-webkit-text-stroke-width:
0px;word-spacing:0px"><span style="color:#212121" lang="EN-US">
import boto3</span><span style="color:#212121"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(33, 33,
33);font-variant-caps: normal;orphans:
auto;text-align:start;widows: auto;-webkit-text-stroke-width:
0px;word-spacing:0px"><span style="color:#212121" lang="EN-US"> </span><span
style="color:#212121"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(33, 33,
33);font-variant-caps: normal;orphans:
auto;text-align:start;widows: auto;-webkit-text-stroke-width:
0px;word-spacing:0px"><span style="color:#212121" lang="EN-US">
client = boto3.client(</span><span style="color:#212121"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(33, 33,
33);font-variant-caps: normal;orphans:
auto;text-align:start;widows: auto;-webkit-text-stroke-width:
0px;word-spacing:0px"><span style="color:#212121" lang="EN-US">
's3',</span><span style="color:#212121"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(33, 33,
33);font-variant-caps: normal;orphans:
auto;text-align:start;widows: auto;-webkit-text-stroke-width:
0px;word-spacing:0px"><span style="color:#212121" lang="EN-US">
aws_access_key_id=creds["accessKeyId"],</span><span
style="color:#212121"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(33, 33,
33);font-variant-caps: normal;orphans:
auto;text-align:start;widows: auto;-webkit-text-stroke-width:
0px;word-spacing:0px"><span style="color:#212121" lang="EN-US">
aws_secret_access_key=creds["secretAccessKey"],</span><span
style="color:#212121"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(33, 33,
33);font-variant-caps: normal;orphans:
auto;text-align:start;widows: auto;-webkit-text-stroke-width:
0px;word-spacing:0px"><span style="color:#212121" lang="EN-US">
aws_session_token=creds["sessionToken"]</span><span
style="color:#212121"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(33, 33,
33);font-variant-caps: normal;orphans:
auto;text-align:start;widows: auto;-webkit-text-stroke-width:
0px;word-spacing:0px"><span style="color:#212121" lang="EN-US">
)</span><span style="color:#212121"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(33, 33,
33);font-variant-caps: normal;orphans:
auto;text-align:start;widows: auto;-webkit-text-stroke-width:
0px;word-spacing:0px"><span style="color:#212121" lang="EN-US"> <span
class="apple-converted-space"> </span></span><span
style="color:#212121"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(33, 33,
33);font-variant-caps: normal;orphans:
auto;text-align:start;widows: auto;-webkit-text-stroke-width:
0px;word-spacing:0px"><span style="color:#212121" lang="EN-US"> client.download_file('prod-lads',
'VNP02IMG/VNP02IMG.A2023193.1942.002.2023194025636.nc',
'test.nc')</span><span style="color:#212121"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(33, 33,
33);font-variant-caps: normal;orphans:
auto;text-align:start;widows: auto;-webkit-text-stroke-width:
0px;word-spacing:0px"><span style="color:#212121" lang="EN-US"> <span
class="apple-converted-space"> </span></span><span
style="color:#212121"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(33, 33,
33);font-variant-caps: normal;orphans:
auto;text-align:start;widows: auto;-webkit-text-stroke-width:
0px;word-spacing:0px"><span style="color:#212121" lang="EN-US">Any
ideas what I'm doing wrong or how to make this work? In the
end I'm interested in accessing the files metadata without
downloading the entire file.</span><span
style="color:#212121"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(33, 33,
33);font-variant-caps: normal;orphans:
auto;text-align:start;widows: auto;-webkit-text-stroke-width:
0px;word-spacing:0px"><span style="color:#212121" lang="EN-US"> </span><span
style="color:#212121"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(33, 33,
33);font-variant-caps: normal;orphans:
auto;text-align:start;widows: auto;-webkit-text-stroke-width:
0px;word-spacing:0px"><span style="color:#212121" lang="EN-US">Regards,</span><span
style="color:#212121"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(33, 33,
33);font-variant-caps: normal;orphans:
auto;text-align:start;widows: auto;-webkit-text-stroke-width:
0px;word-spacing:0px"><span style="color:#212121" lang="EN-US">Bert</span><span
style="color:#212121"><o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<br>
<fieldset class="moz-mime-attachment-header"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
gdal-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a>
<a class="moz-txt-link-freetext" href="https://lists.osgeo.org/mailman/listinfo/gdal-dev">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a>
</pre>
</blockquote>
<pre class="moz-signature" cols="72">--
<a class="moz-txt-link-freetext" href="http://www.spatialys.com">http://www.spatialys.com</a>
My software is free, but my time generally not.</pre>
</body>
</html>