[gdal-dev] /vsis3/ on NetCDF from Earthdata

[Even Rouault]

Bert,

Also set the GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR config option, 
otherwise the generic open mechanism of GDAL tries to list the content 
of the VNP02IMG/ directory and it seems there are tons of files there

When doing that, I get a result within a few seconds

Even

Le 20/07/2023 à 09:59, b.coerver--- via gdal-dev a écrit :
>
> Hello,
>
> I'm trying to access data from NASA's Earthdata S3 buckets, but I get 
> a `"<filename> does not exist in the file system, and is not 
> recognized as a supported dataset name."` error after waiting a long 
> time (± 50 minutes, the process is downloading some data the whole 
> time) doing the following:
>
> from osgeo import gdal
>
> gdal_config_options = {
>
> "AWS_ACCESS_KEY_ID": creds["accessKeyId"],
>
> "AWS_SESSION_TOKEN":  creds["sessionToken"],
>
> "AWS_SECRET_ACCESS_KEY": creds["secretAccessKey"],
>
> "AWS_REGION": "us-west-2",
>
> }
>
> url = 
> "/vsis3/prod-lads/VNP02IMG/VNP02IMG.A2023193.1942.002.2023194025636.nc"
>
> for k, v in gdal_config_options.items():
>
> gdal.SetConfigOption(k, v)
>
> out =gdal.Info(url)
>
> The `creds` variable is a dictionary with temporary credential 
> information that I get from 
> [here](https://data.laadsdaac.earthdatacloud.nasa.gov/s3credentials 
> <https://data.laadsdaac.earthdatacloud.nasa.gov/s3credentials>), you 
> need a free account to get them.
>
> When I introduce an error in one of the keys/tokens (e.g. 
> `"AWS_ACCESS_KEY_ID": creds["accessKeyId"] + "x"`, I do get a message 
> immediately saying my credentials are unknown. So I do think they are 
> being ingested correctly. I’m using GDAL version 3.7.1.
>
> I also managed to download the entire file using `boto3`, by doing the 
> following:
>
> import boto3
>
> client = boto3.client(
>
> 's3',
>
> aws_access_key_id=creds["accessKeyId"],
>
> aws_secret_access_key=creds["secretAccessKey"],
>
> aws_session_token=creds["sessionToken"]
>
> )
>
>     client.download_file('prod-lads', 
> 'VNP02IMG/VNP02IMG.A2023193.1942.002.2023194025636.nc', 'test.nc')
>
> Any ideas what I'm doing wrong or how to make this work? In the end 
> I'm interested in accessing the files metadata without downloading the 
> entire file.
>
> Regards,
>
> Bert
>
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev

-- 
http://www.spatialys.com
My software is free, but my time generally not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20230720/48f31f28/attachment.htm>