[gdal-dev] 503 Errors when accessing Public Datasets on Google Cloud Storage with automatic auth

Fri Oct 18 01:22:25 PDT 2019

Hi,

I'm in contact with google about the problem. It might be a problem on
their side.

But during testing I discovered something else which I found unusual.
Maybe I understand the meaning of the GDAL_DISABLE_READDIR_ON_OPEN variable
incorrectly but I still wanted to double check.

Using:

#!/bin/bash
export GS_ACCESS_KEY_ID=<OUR_KEY>
export GS_SECRET_ACCESS_KEY=<OUR_KEY>
export GDAL_DISABLE_READDIR_ON_OPEN=TRUE
export CPL_DEBUG=ON

gdalinfo
/vsigs/gcp-public-data-sentinel-2/tiles/29/T/NE/S2B_MSIL1C_20170719T112119_N0205_R037_T29TNE_20170719T112511.SAFE/GRANULE/L1C_T29TNE_A001922_20170719T112511/IMG_DATA/T29TNE_20170719T112119_B01.jp2

Tries to open tons of files like e.g.
https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/29/T/NE/S2B_MSIL1C_20170719T112119_N0205_R037_T29TNE_20170719T112511.SAFE/GRANULE/L1C_T29TNE_A001922_20170719T112511/IMG_DATA/HDR_20170719T112119_B01.txt

The other to settings FALSE and EMPTY_DIR still try to open some auxiliary
files. But not as many.

In this case it seems that setting GDAL_DISABLE_READDIR_ON_OPEN=TRUE is by
far the slowest option.
Is this expected?

Thanks,
Christoph

On Wed, 16 Oct 2019 at 17:38, Christoph Paulik <cpaulik at vandersat.com>
wrote:

> Hi,
>
> The thing is that one request works fine in any case.
> But google seems to rate limit on the public bucket in the following cases:
>
> - no authentication (/vsicurl/ driver)
> - authentication with automatic detection by gdal that it is running on a
> GCE VM.
>
> but not when using the explicit authentication with
> GS_ACCESS_KEY_ID/GS_SECRET_ACCESS_KEY
>
> Christoph
>
>
> On Wed, 16 Oct 2019 at 17:31, Even Rouault <even.rouault at spatialys.com>
> wrote:
>
>> Hi,
>>
>> did you try with Google gsutil tool ?
>>
>> Anyway as it is a public resource that doesn't require any signing, you
>> could
>> just use
>>
>> /vsicurl/
>> https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/29/T/
>>
>> NE/S2B_MSIL1C_20170719T112119_N0205_R037_T29TNE_20170719T112511.SAFE/GRANULE/
>> L1C_T29TNE_A001922_20170719T112511/IMG_DATA/T29TNE_20170719T112119_B01.jp2
>> <https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/29/T/NE/S2B_MSIL1C_20170719T112119_N0205_R037_T29TNE_20170719T112511.SAFE/GRANULE/L1C_T29TNE_A001922_20170719T112511/IMG_DATA/T29TNE_20170719T112119_B01.jp2>
>>
>> Even
>>
>> > Hi,
>> >
>> > When doing a few requests to the Sentinel 2 bucket of Google (
>> > https://cloud.google.com/storage/docs/public-datasets/sentinel-2) we
>> have
>> > been getting 503 Responses when using the automatic authentication. See
>> the
>> > end of this email for the full output.
>> >
>> > This behavior goes away when using e.g. the
>> > GS_ACCESS_KEY_ID/GS_SECRET_ACCESS_KEY authentication method.
>> >
>> > The Service account this machine is running with has "Full Access to All
>> > Cloud APIs"
>> >
>> > Not sure if this is a gdal problem or actually a google internal issue
>> > since accessing our own buckets works fine.
>> >
>> > Best,
>> > Christoph
>> >
>> > Output:
>> > export CPL_DEBUG=ON
>> > gdalinfo
>> >
>> /vsigs/gcp-public-data-sentinel-2/tiles/29/T/NE/S2B_MSIL1C_20170719T112119_N
>> >
>> 0205_R037_T29TNE_20170719T112511.SAFE/GRANULE/L1C_T29TNE_A001922_20170719T11
>> > 2511/IMG_DATA/T29TNE_20170719T112119_B01.jp2
>> >
>> > HTTP: Fetch(
>> >
>> http://metadata.google.internal/computeMetadata/v1/instance/service-accounts
>> > /default/token )
>> > HTTP: libcurl/7.65.3 OpenSSL/1.1.1c zlib/1.2.5 libssh2/1.8.2
>> > HTTP: GDAL was built against curl 7.64.1, but is running against 7.65.3.
>> > HTTP: These HTTP headers were set: Metadata-Flavor: Google
>> > GOA2: Refresh Token Response:
>> > {"access_token":"REMOVED IN
>> EMAIL","expires_in":2877,"token_type":"Bearer"}
>> > GOA2: Access Token : 'REMOVED IN EMAIL'
>> > GS: Using GCE inherited permissions
>> > GS: <html><head><meta http-equiv="content-type" content="text/html;
>> > charset=utf-8"/><title>Sorry...</title><style> body { font-family:
>> verdana,
>> > arial, sans-serif; background-color: #fff; color: #000;
>> > }</style></head><body><div><table><tr><td><b><font face=sans-serif
>> > size=10><font color=#4285f4>G</font><font color=#ea4335>o</font><font
>> > color=#fbbc05>o</font><font color=#4285f4>g</font><font
>> > color=#34a853>l</font><font color=#ea4335>e</font></font></b></td><td
>> > style="text-align: left; vertical-align: bottom; padding-bottom: 15px;
>> > width: 50%"><div style="border-bottom: 1px solid
>> > #dfdfdf;">Sorry...</div></td></tr></table></div><div style="margin-left:
>> > 4em;"><h1>We're sorry...</h1><p>... but your computer or network may be
>> > sending automated queries. To protect our users, we can't process your
>> > request right now.</p></div><div style="margin-left: 4em;">See <a href="
>> > https://support.google.com/websearch/answer/86640">Google Help</a> for
>> more
>> > information.<br/><br/></div><div style="text-align: center; border-top:
>> 1px
>> > solid #dfdfdf;"><a href="https://www.google.com">Google
>> > Home</a></div></body></html>
>> > VSICURL: GetFileSize(
>> >
>> https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/29/T/NE/S2B_
>> >
>> MSIL1C_20170719T112119_N0205_R037_T29TNE_20170719T112511.SAFE/GRANULE/L1C_T2
>> > 9TNE_A001922_20170719T112511/IMG_DATA/T29TNE_20170719T112119_B01.jp2)=0
>> > response_code=503
>> > GS: <html><head><meta http-equiv="content-type" content="text/html;
>> > charset=utf-8"/><title>Sorry...</title><style> body { font-family:
>> verdana,
>> > arial, sans-serif; background-color: #fff; color: #000;
>> > }</style></head><body><div><table><tr><td><b><font face=sans-serif
>> > size=10><font color=#4285f4>G</font><font color=#ea4335>o</font><font
>> > color=#fbbc05>o</font><font color=#4285f4>g</font><font
>> > color=#34a853>l</font><font color=#ea4335>e</font></font></b></td><td
>> > style="text-align: left; vertical-align: bottom; padding-bottom: 15px;
>> > width: 50%"><div style="border-bottom: 1px solid
>> > #dfdfdf;">Sorry...</div></td></tr></table></div><div style="margin-left:
>> > 4em;"><h1>We're sorry...</h1><p>... but your computer or network may be
>> > sending automated queries. To protect our users, we can't process your
>> > request right now.</p></div><div style="margin-left: 4em;">See <a href="
>> > https://support.google.com/websearch/answer/86640">Google Help</a> for
>> more
>> > information.<br/><br/></div><div style="text-align: center; border-top:
>> 1px
>> > solid #dfdfdf;"><a href="https://www.google.com">Google
>> > Home</a></div></body></html>
>> > ERROR 11: HTTP response code: 503
>> > gdalinfo failed - unable to open
>> >
>> '/vsigs/gcp-public-data-sentinel-2/tiles/29/T/NE/S2B_MSIL1C_20170719T112119_
>> >
>> N0205_R037_T29TNE_20170719T112511.SAFE/GRANULE/L1C_T29TNE_A001922_20170719T1
>> > 12511/IMG_DATA/T29TNE_20170719T112119_B01.jp2'.
>>
>>
>> --
>> Spatialys - Geospatial professional services
>> http://www.spatialys.com
>>
>
>
> --
> *Christoph Paulik* // Head of Infrastructure
> VanderSat // Satellite observed water data. Globally. Daily.
> Wilhelminastraat 43a, 2011 VK, Haarlem (NL), The Netherlands
> <https://maps.google.com/?q=Wilhelminastraat+43a,+2011+VK,+Haarlem+(NL),+The+Netherlands&entry=gmail&source=g>
> *M*  +31 6 18271928 *W*  www.vandersat.com
>
>
> --------------------------------------------------------------------------------------
>

-- 
*Christoph Paulik* // Head of Infrastructure
VanderSat // Satellite observed water data. Globally. Daily.
Wilhelminastraat 43a, 2011 VK, Haarlem (NL), The Netherlands
<https://maps.google.com/?q=Wilhelminastraat+43a,+2011+VK,+Haarlem+(NL),+The+Netherlands&entry=gmail&source=g>
*M*  +31 6 18271928 *W*  www.vandersat.com

--------------------------------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20191018/e8eff342/attachment-0001.html>