<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
font-size:10.0pt;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
mso-ligatures:standardcontextual;
mso-fareast-language:EN-US;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle23
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
p.p1, li.p1, div.p1
{mso-style-name:p1;
margin:0cm;
font-size:8.5pt;
font-family:Menlo;
color:black;}
MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style></head><body lang=en-NL link="#0563C1" vlink=purple style='word-wrap:break-word'><div class=WordSection1><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;mso-fareast-language:EN-US'>I found a solution in the meantime, so no need to reply anymore:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=p1><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext;mso-fareast-language:EN-US'>Using gdalmdimtranslate</span><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'> <span lang=EN-US>with the -array flag to select the group and array as e.g.</span></span><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext;mso-fareast-language:EN-US'> </span><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext;mso-fareast-language:EN-US'>“/geolocation_data/longitude” downloads the array and nothing else (bytes transferred around 80MB)</span><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext;mso-fareast-language:EN-US'>.<o:p></o:p></span></p><p class=p1><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=p1><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext;mso-fareast-language:EN-US'>Regards,<o:p></o:p></span></p><p class=p1><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext;mso-fareast-language:EN-US'>Bert<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;mso-fareast-language:EN-US'><o:p> </o:p></span></p><div id=mail-editor-reference-message-container><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal style='margin-bottom:12.0pt'><b><span style='font-size:12.0pt;color:black'>From: </span></b><span style='font-size:12.0pt;color:black'>gdal-dev <gdal-dev-bounces@lists.osgeo.org> on behalf of bcoerver--- via gdal-dev <gdal-dev@lists.osgeo.org><br><b>Date: </b>Wednesday, 30 August 2023 at 10:47<br><b>To: </b>Even Rouault <even.rouault@spatialys.com>, gdal-dev@lists.osgeo.org <gdal-dev@lists.osgeo.org><br><b>Subject: </b>Re: [gdal-dev] /vsis3/ on NetCDF from Earthdata<o:p></o:p></span></p></div><p class=MsoNormal><span lang=NL style='font-size:11.0pt;mso-fareast-language:EN-US'>Hi Even,</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal><span lang=NL style='font-size:11.0pt;mso-fareast-language:EN-US'> </span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;mso-fareast-language:EN-US'>Thanks for you response. Is there any way to disable this automatic sampling of the latitude and longitude subdatasets?</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;mso-fareast-language:EN-US'> </span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;mso-fareast-language:EN-US'>The file I’m accessing is 182.2MB in size, when I download one of its subdatasets using gdal_translate I can see on my network that the process transfers roughly the same amount (176Mbytes received according to Mac’s activity monitor) I’m interested in accessing the latitude and longitude subdatasets, so opening these two subdatasets would already result in about 350MB transferred…</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;mso-fareast-language:EN-US'> </span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;mso-fareast-language:EN-US'>Also, when I run gdal_translate to download the file using the -sds flag, the data transferred greatly exceeds the total file size, it seems like the lat/lon subdatasets get re-downloaded for each of the subdatasets in the file.</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;mso-fareast-language:EN-US'> </span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;mso-fareast-language:EN-US'>Regards,</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;mso-fareast-language:EN-US'>Bert</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;mso-fareast-language:EN-US'> </span><span style='font-size:11.0pt'><o:p></o:p></span></p><div id=mail-editor-reference-message-container><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal style='margin-bottom:12.0pt'><b><span style='font-size:12.0pt;color:black'>From: </span></b><span style='font-size:12.0pt;color:black'>Even Rouault <even.rouault@spatialys.com><br><b>Date: </b>Friday, 21 July 2023 at 15:43<br><b>To: </b>b.coerver@mailbox.org <b.coerver@mailbox.org>, gdal-dev@lists.osgeo.org <gdal-dev@lists.osgeo.org><br><b>Subject: </b>Re: [gdal-dev] /vsis3/ on NetCDF from Earthdata</span><span style='font-size:11.0pt'><o:p></o:p></span></p></div><p> <o:p></o:p></p><div><p class=MsoNormal><span style='font-size:11.0pt'>Le 21/07/2023 à 11:46, <a href="mailto:b.coerver@mailbox.org">b.coerver@mailbox.org</a> a écrit :<o:p></o:p></span></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;mso-fareast-language:EN-US'>One more follow-up question:</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;mso-fareast-language:EN-US'> </span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;mso-fareast-language:EN-US'>The datasets that I’m interested in contains subdatasets. I can get the info of a subdataset like this:</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;mso-fareast-language:EN-US'> </span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;mso-fareast-language:EN-US'> sub_ds_path </span><span style='font-size:11.0pt;mso-fareast-language:EN-US'>= 'HDF5:"/vsis3/prod</span><span lang=EN-US style='font-size:11.0pt;mso-fareast-language:EN-US'>-</span><span style='font-size:11.0pt;mso-fareast-language:EN-US'>lads/VNP02IMG/VNP02IMG.A2021064.2342.002.2021128145323.nc"://observation_data/I04'</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;mso-fareast-language:EN-US'> info = gdal.Info(sub_ds_path)</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;mso-fareast-language:EN-US'> </span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;mso-fareast-language:EN-US'>This works fine and finishes in a few seconds. However, when I do the same thing for a different dataset (which contains the geolocation of the dataset above):</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;mso-fareast-language:EN-US'> </span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;mso-fareast-language:EN-US'> </span><span style='font-size:11.0pt;mso-fareast-language:EN-US'>sub_ds_path = 'HDF5:"/vsis3/prod</span><span lang=EN-US style='font-size:11.0pt;mso-fareast-language:EN-US'>-</span><span style='font-size:11.0pt;mso-fareast-language:EN-US'>lads/VNP03IMG/VNP03IMG.A2021065.2324.002.2021127011303.nc"://geolocation_data/latitude'</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;mso-fareast-language:EN-US'> </span><span style='font-size:11.0pt;mso-fareast-language:EN-US'>info = gdal.Info(sub_ds_path)</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;mso-fareast-language:EN-US'> </span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;mso-fareast-language:EN-US'>This takes about 2</span><span lang=EN-US style='font-size:11.0pt;mso-fareast-language:EN-US'>.</span><span style='font-size:11.0pt;mso-fareast-language:EN-US'>5 minutes and I can see on my network that Python is downloading data at about 1MB/s the whole time. The info from this subdataset contains a lot of ground-control-points, so I tried setting “showGCPs=</span><span lang=EN-US style='font-size:11.0pt;mso-fareast-language:EN-US'>False</span><span style='font-size:11.0pt;mso-fareast-language:EN-US'>”</span><span lang=EN-US style='font-size:11.0pt;mso-fareast-language:EN-US'>, but that doesn’t solve it. I’m not sure if it’s really the GCPs that’s causing this (when I save the info as a json, it is about 750kb in size).</span><span style='font-size:11.0pt'><o:p></o:p></span></p></blockquote><p>The second product has georeferencing information, and upon opening of one of its subsdataset, GDAL samples the latitude and longitude arrays to expose ground control points, hence it reads those arrays.<o:p></o:p></p><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;mso-fareast-language:EN-US'> </span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;mso-fareast-language:EN-US'>Any ideas what else can cause this difference in execution time?</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;mso-fareast-language:EN-US'> </span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;mso-fareast-language:EN-US'>Regards,</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;mso-fareast-language:EN-US'>Bert</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;mso-fareast-language:EN-US'> </span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;mso-fareast-language:EN-US'> </span><span style='font-size:11.0pt'><o:p></o:p></span></p><div id=mail-editor-reference-message-container><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal style='margin-bottom:12.0pt'><b><span style='font-size:12.0pt;color:black'>From: </span></b><span style='font-size:12.0pt;color:black'>gdal-dev <a href="mailto:gdal-dev-bounces@lists.osgeo.org"><gdal-dev-bounces@lists.osgeo.org></a> on behalf of b.coerver--- via gdal-dev <a href="mailto:gdal-dev@listsosgeo.org"><gdal-dev@lists.osgeo.org></a><br><b>Date: </b>Thursday, 20 July 2023 at 11:51<br><b>To: </b>Even Rouault <a href="mailto:even.rouault@spatialys.com"><even.rouault@spatialys.com></a>, <a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a> <a href="mailto:gdal-dev@lists.osgeo.org"><gdal-dev@lists.osgeo.org></a><br><b>Subject: </b>Re: [gdal-dev] /vsis3/ on NetCDF from Earthdata</span><span style='font-size:11.0pt'><o:p></o:p></span></p></div><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'>That does it, thank you so much!</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'> </span><span style='font-size:11.0pt'><o:p></o:p></span></p><div id=mail-editor-reference-message-container><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal style='margin-bottom:12.0pt'><b><span style='font-size:12.0pt;color:black;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'>From: </span></b><span style='font-size:12.0pt;color:black;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'>Even Rouault <a href="mailto:even.rouault@spatialys.com"><even.rouault@spatialys.com></a><br><b>Date: </b>Thursday, 20 July 2023 at 11:44<br><b>To: </b><a href="mailto:bcoerver@mailbox.org">bcoerver@mailbox.org</a> <a href="mailto:b.coerver@mailbox.org"><b.coerver@mailbox.org></a>, <a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a> <a href="mailto:gdal-dev@lists.osgeo.org"><gdal-dev@lists.osgeo.org></a><br><b>Subject: </b>Re: [gdal-dev] /vsis3/ on NetCDF from Earthdata</span><span style='font-size:11.0pt'><o:p></o:p></span></p></div><p>Bert,<o:p></o:p></p><p>Also set the GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR config option, otherwise the generic open mechanism of GDAL tries to list the content of the VNP02IMG/ directory and it seems there are tons of files there<o:p></o:p></p><p>When doing that, I get a result within a few seconds<o:p></o:p></p><p>Even<o:p></o:p></p><div><p class=MsoNormal><span style='font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'>Le 20/07/2023 à 09:59, b.coerver--- via gdal-dev a écrit :</span><span style='font-size:11.0pt'><o:p></o:p></span></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'>Hello,</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'> </span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'>I'm trying to access data from NASA's Earthdata S3 buckets, but I get a `"<filename> does not exist in the file system, and is not recognized as a supported dataset name."` error after waiting a long time (± 50 minutes, the process is downloading some data the whole time) doing the following:</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'> </span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt;caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'>from osgeo import gdal</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt;caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'> </span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt;caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'>gdal_config_options = {</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt;caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'> "AWS_ACCESS_KEY_ID": creds["accessKeyId"],</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt;caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'> "AWS_SESSION_TOKEN": creds["sessionToken"],</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt;caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'> "AWS_SECRET_ACCESS_KEY": creds["secretAccessKey"],</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt;caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'> "AWS_REGION": "us-west-2",</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt;caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'>}</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt;caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'> </span><span style='font-size:110pt'><o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt;caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'>url = "/vsis3/prod-lads/VNP02IMG/VNP02IMG.A2023193.1942.0022023194025636.nc"</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt;caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'> <span class=apple-converted-space> </span></span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt;caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'>for k, v in gdal_config_options.items():</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt;caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'> gdal.SetConfigOption(k, v)</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt;caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'> </span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt;caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'>out =<span class=apple-converted-space> </span>gdal.Info(url)</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'> </span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'>The `creds` variable is a dictionary with temporary credential information that I get from [here](</span><span lang=NL style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'><a href="https://data.laadsdaac.earthdatacloud.nasagov/s3credentials" title="https://data.laadsdaac.earthdatacloud.nasa.gov/s3credentials"><span lang=EN-US style='color:#0078D7'>https://data.laadsdaac.earthdatacloud.nasa.gov/s3credentials</span></a></span><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'>), you need a free account to get them.</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'> </span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'>When I introduce an error in one of the keys/tokens (e.g. `"AWS_ACCESS_KEY_ID": creds["accessKeyId"] + "x"`, I do get a message immediately saying my credentials are unknown. So I do think they are being ingested correctly. I’m using GDAL version 3.7.1.</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'> </span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'>I also managed to download the entire file using `boto3`, by doing the following:</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'> </span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'> import boto3</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'> </span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'> client = boto3.client(</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'> 's3',</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'> aws_access_key_id=creds["accessKeyId"],</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'> aws_secret_access_key=creds["secretAccessKey"],</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'> aws_session_token=creds["sessionToken"]</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'> )</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'> <span class=apple-converted-space> </span></span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'> client.download_file('prod-lads', 'VNP02IMG/VNP02IMG.A2023193.1942.002.2023194025636.nc', 'test.nc')</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'> <span class=apple-converted-space> </span></span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'>Any ideas what I'm doing wrong or how to make this work? In the end I'm interested in accessing the files metadata without downloading the entire file</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'> </span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'>Regards,</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal style='caret-color:rgb(33,33,33);font-variant-caps:normal;orphans:auto;text-align:start;widows:auto;-webkit-text-stroke-width:0px;word-spacing:0px'><span lang=EN-US style='font-size:11.0pt;color:#212121;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'>Bert</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'> </span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal style='margin-bottom:12.0pt'><span style='font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:EN-US'> </span><span style='font-size:11.0pt'><o:p></o:p></span></p><pre>_______________________________________________<o:p></o:p></pre><pre>gdal-dev mailing list<o:p></o:p></pre><pre><a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a><o:p></o:p></pre><pre><a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a><o:p></o:p></pre></blockquote><pre>-- <o:p></o:p></pre><pre><a href="http://wwwspatialys.com">http://www.spatialys.com</a><o:p></o:p></pre><pre>My software is free, but my time generally not.<o:p></o:p></pre></div></div></div></div></blockquote><pre>-- <o:p></o:p></pre><pre><a href="http://www.spatialys.com">http://www.spatialys.com</a><o:p></o:p></pre><pre>My software is free, but my time generally not.<o:p></o:p></pre></div></div></div></div></div></body></html>