[gdal-dev] How to access remote data that requires authentication?

Tue Oct 5 12:58:02 PDT 2021

Hmm, very tricky situations. Apparently from Julia things remain in memory and it remembers previous errors.

From a clean Juia REPL start this works (even with authentication)

julia> set_config_option("GDAL_HTTP_COOKIEFILE", joinpath(tempdir(), "cookies.txt"))
julia> set_config_option("GDAL_HTTP_COOKIEJAR", joinpath(tempdir(), "cookies.txt"))
julia> set_config_option("GDAL_DISABLE_READDIR_ON_OPEN","YES")
julia> set_config_option("CPL_VSIL_CURL_ALLOWED_EXTENSIONS","TIF")
julia> set_config_option("CPL_VSIL_CURL_USE_HEAD","FALSE")

gdalinfo("/vsicurl/https://lpdaac.earthdata.nasa.gov/lp-prod-protected/HLSS30.015/HLS.S30.T10TEK.2020273T190109.v1.5.B8A.tif")
"Driver: GTiff/GeoTIFF\nFiles: /vsicurl/https://lpdaac.earthdata.nasa.gov/lp-prod-protected/HLSS30.015/HLS.S30.T10TEK.2020273T190109.v1.5.B8A.tif\nSize is 3660, 3660\nCoordinate System is:\nPROJCRS[\"UTM Zone 10, Northern Hemisphere\",\n    BASEGEOGCRS[\"Unknown datum based upon the WGS 84 ellipsoid\",\n        DATUM[\"Not_specified_based_on_WGS_84_spheroid\",\n …..

However, if run the gdalinfo(…) first and set the options after, then any posterior call to gdalinfo will return that “not supported file format” error.

In fact the errors evolve like this (after another clean start)

julia> gdalinfo("/vsicurl/https://lpdaac.earthdata.nasa.gov/lp-prod-protected/HLSS30.015/HLS.S30.T10TEK.2020273T190109.v1.5.B8A.tif")
ERROR 11: HTTP response code: 206

julia> gdalinfo("/vsicurl/https://lpdaac.earthdata.nasa.gov/lp-prod-protected/HLSS30.015/HLS.S30.T10TEK.2020273T190109.v1.5.B8A.tif")
ERROR 4: `/vsicurl/https://lpdaac.earthdata.nasa.gov/lp-prod-protected/HLSS30.015/HLS.S30.T10TEK.2020273T190109.v1.5.B8A.tif' does not exist in the file system, and is not recognized as a supported dataset name.

From: Even Rouault <even.rouault at spatialys.com>
Sent: Tuesday, October 5, 2021 8:11 PM
To: Joaquim Manuel Freire Luís <jluis at ualg.pt>; gdal-dev at lists.osgeo.org
Subject: Re: [gdal-dev] How to access remote data that requires authentication?

There are a few historically exported VSIInstaller symbols, but VSIInstallCurlFileHandler is never exported. None of those symbols need to be explicitly called as GDAL will automatically register the builtin-handlers. Perhaps your gdal.dll lacks curl support ? I assumed you pass the needed config options / env variables for that service. Perhaps you could try with something simpler that doesn't require authentication.
Le 05/10/2021 à 21:05, Joaquim Manuel Freire Luís a écrit :
Even, sorry to continue this.

Now is from Julia

julia> gdalinfo("/vsicurl/https://lpdaac.earthdata.nasa.gov/lp-prod-protected/HLSS30.015/HLS.S30.T10TEK.2020273T190109.v1.5.B8A.tif")
ERROR 4: `/vsicurl/https://lpdaac.earthdata.nasa.gov/lp-prod-protected/HLSS30.015/HLS.S30.T10TEK.2020273T190109.v1.5.B8A.tif' not recognized as a supported file format.

I saw a reference to VSIInstallCurlFileHandler() so I thought “right I need this guy too” but when I try to install it

julia> GMT.Gdal.VSIInstallCurlFileHandler()
ERROR: could not load symbol "VSIInstallCurlFileHandler":
The specified procedure could not be found.

And indeed my gdal.dll has a couple of VSIInstaller symbols but not this one (nor zip, gzip and others).

Is the “not supported file format” related to this? Shouldn’t that symbol be exported to the dll?

From: gdal-dev <gdal-dev-bounces at lists.osgeo.org><mailto:gdal-dev-bounces at lists.osgeo.org> On Behalf Of Joaquim Manuel Freire Luís
Sent: Tuesday, October 5, 2021 5:59 PM
To: Even Rouault <even.rouault at spatialys.com><mailto:even.rouault at spatialys.com>; gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org>
Subject: Re: [gdal-dev] How to access remote data that requires authentication?

>The mention to _netrc is weird...

On Windows is _netrc, not .netrc

And I finally made it (twisted). The problem was that I detest where MS decides is my home dir (and all the hidden dirs stuff that it puts there) and always have had a home at “HOME=c:\j” and it’s there that libcurl (I presume) seeks for the netrc file.

Thanks for all the tips. But I still wonder how GMT manages to read the file without anything of this.

For the record if any other Windows user needs it

gdalinfo /vsicurl/https://lpdaac.earthdata.nasa.gov/lp-prod-protected/HLSS30.015/HLS.S30.T10TEK.2020273T190109.v1.5.B8A.tif --config GDAL_DISABLE_READDIR_ON_OPEN EMPTY_DIR --config CPL_VSIL_CURL_USE_HEAD FALSE --config GDAL_HTTP_COOKIEFILE c:/TEMP/cookies.txt --config GDAL_HTTP_COOKIEJAR c:/TEMP/cookies.txt

Driver: GTiff/GeoTIFF

Files: /vsicurl/https://lpdaac.earthdata.nasa.gov/lp-prod-protected/HLSS30.015/HLS.S30.T10TEK.2020273T190109.v1.5.B8A.tif

Size is 3660, 3660

Coordinate System is:

PROJCRS["UTM Zone 10, Northern Hemisphere",

Regarding authentication issues, this service is quite annoying and requires enabling cookies. See https://lists.osgeo.org/pipermail/gdal-dev/2021-October/054728.html

You need to add things like --config GDAL_HTTP_COOKIEFILE /tmp/cookies.txt --config GDAL_HTTP_COOKIEJAR /tmp/cookies.txt
Le 05/10/2021 à 18:20, Joaquim Manuel Freire Luís a écrit :

OK, tried more things from that thread.

gdalinfo /vsicurl/https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T10TEK.2021192T184511.v2.0/HLS.L30.T10TEK.2021192T184511.v2.0.B04.tif --config GDAL_DISABLE_READDIR_ON_OPEN EMPTY_DIR --config CPL_CURL_VERBOSE ON --config CPL_VSIL_CURL_USE_HEAD FALSE

* Couldn't find host data.lpdaac.earthdatacloud.nasa.gov in the _netrc file; using defaults ...

OK, right it's not there.

But latter down it says

* Couldn't find host urs.earthdata.nasa.gov in the _netrc file; using defaults

Now this is not right.

< HTTP/1.1 401 Unauthorized

...

< WWW-Authenticate: Basic realm="Please enter your Earthdata Login credentials. If you do not have a Earthdata Login, create one at https://urs.earthdata.nasa.gov//users/new<https://urs.earthdata.nasa.gov/users/new>"

...

ERROR 11: HTTP response code: 401

So I'm back to the Authentication problem. I do have an _netrc file (and .netrc btw) in my home dir as well as current dir but it does seem to find it. Is there something else that I must to in order to that file be found/used?

-----Original Message-----

From: thomas bonfort <thomas.bonfort at gmail.com><mailto:thomas.bonfort at gmail.com>

Sent: Tuesday, October 5, 2021 4:31 PM

To: Joaquim Manuel Freire Luís <jluis at ualg.pt><mailto:jluis at ualg.pt>

Subject: Re: [gdal-dev] How to access remote data that requires authentication?

for the 206 there seems to be a similar issue posted here a few days ago, search for "Problem accessing NASA Cloud Optimized GeoTIFF data"

in the archives

On Tue, Oct 5, 2021 at 5:26 PM Joaquim Manuel Freire Luís <jluis at ualg.pt><mailto:jluis at ualg.pt> wrote:

you should also change your password, now you have posted it on a

public mailing list :/

Shit, thanks for spotting it.

But it doesn't work with /vsicur/ neither (had tried it  before). Now

the error is  206

gdalinfo

/vsicurl/https://user:pass@lpdaac.earthdata.nasa.gov/lp-prod-protected

/HLSS30.015/HLS.S30.T10TEK.2020273T190109.v1.5.B8A.tif

ERROR 11: HTTP response code: 206

On Tue, Oct 5, 2021 at 5:12 PM Joaquim Manuel Freire Luís <jluis at ualg.pt><mailto:jluis at ualg.pt> wrote:

Hi,

I’ve read a lot of the docs, tried many -co options but can’t get through this mystery.

I can access the data through GMT, which uses GDAL to do this job,

but can’t do it with GDAL directly

gdalinfo

https://user:pass@lpdaac.earthdata.nasa.gov/lp-prod-protected/HLSS30

.0 15/HLS.S30.T10TEK.2020273T190109.v1.5.B8A.tif

ERROR 1: HTTP error code : 401

gdalinfo failed - unable to open 'https://jluis:abaixo0Earthdata@lpdaac.earthdata.nasa.gov/lp-prod-protected/HLSS30.015/HLS.S30.T10TEK.2020273T190109.v1.5.B8A.tif'.

So passing longin:password via url does not work either. But it does

if indirectly used

grdinfo

https://user:pass@lpdaac.earthdata.nasa.gov/lp-prod-protected/HLSS30

.0 15/HLS.S30.T10TEK.2020273T190109.v1.5.B8A.tif

HLS.S30.T10TEK.2020273T190109.v1.5.B8A.tif: Title: Grid imported via

GDAL

HLS.S30.T10TEK.2020273T190109.v1.5.B8A.tif: Command:

HLS.S30.T10TEK.2020273T190109.v1.5.B8A.tif: Remark:

HLS.S30.T10TEK.2020273T190109.v1.5.B8A.tif: Pixel node registration

used [Cartesian grid]

HLS.S30.T10TEK.2020273T190109.v1.5.B8A.tif: Grid file format: gd =

Import/export through GDAL

HLS.S30.T10TEK.2020273T190109.v1.5.B8A.tif: x_min: 499980 x_max:

609780 x_inc: 30 name: x n_columns: 3660

HLS.S30.T10TEK.2020273T190109.v1.5.B8A.tif: y_min: 4390200 y_max:

4500000 y_inc: 30 name: y n_rows: 3660

HLS.S30.T10TEK.2020273T190109.v1.5.B8A.tif: v_min: -0.0149 v_max:

0.8833 name: z

HLS.S30.T10TEK.2020273T190109.v1.5.B8A.tif: scale_factor: 0.0001

add_offset: 0 packed z-range: [-149,8833]

+proj=utm +zone=10 +ellps=WGS84 +units=m +no_defs

Joaquim

_______________________________________________

gdal-dev mailing list

gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org>

https://lists.osgeo.org/mailman/listinfo/gdal-dev

_______________________________________________

gdal-dev mailing list

gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org>

https://lists.osgeo.org/mailman/listinfo/gdal-dev

--

http://www.spatialys.com

My software is free, but my time generally not.

--

http://www.spatialys.com

My software is free, but my time generally not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20211005/e5a44da4/attachment-0001.html>