[gdal-dev] 503 Errors when accessing Public Datasets on Google Cloud Storage with automatic auth

Wed Oct 16 08:31:53 PDT 2019

Hi,

did you try with Google gsutil tool ?

Anyway as it is a public resource that doesn't require any signing, you could 
just use

/vsicurl/https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/29/T/
NE/S2B_MSIL1C_20170719T112119_N0205_R037_T29TNE_20170719T112511.SAFE/GRANULE/
L1C_T29TNE_A001922_20170719T112511/IMG_DATA/T29TNE_20170719T112119_B01.jp2

Even

> Hi,
> 
> When doing a few requests to the Sentinel 2 bucket of Google (
> https://cloud.google.com/storage/docs/public-datasets/sentinel-2) we have
> been getting 503 Responses when using the automatic authentication. See the
> end of this email for the full output.
> 
> This behavior goes away when using e.g. the
> GS_ACCESS_KEY_ID/GS_SECRET_ACCESS_KEY authentication method.
> 
> The Service account this machine is running with has "Full Access to All
> Cloud APIs"
> 
> Not sure if this is a gdal problem or actually a google internal issue
> since accessing our own buckets works fine.
> 
> Best,
> Christoph
> 
> Output:
> export CPL_DEBUG=ON
> gdalinfo
> /vsigs/gcp-public-data-sentinel-2/tiles/29/T/NE/S2B_MSIL1C_20170719T112119_N
> 0205_R037_T29TNE_20170719T112511.SAFE/GRANULE/L1C_T29TNE_A001922_20170719T11
> 2511/IMG_DATA/T29TNE_20170719T112119_B01.jp2
> 
> HTTP: Fetch(
> http://metadata.google.internal/computeMetadata/v1/instance/service-accounts
> /default/token )
> HTTP: libcurl/7.65.3 OpenSSL/1.1.1c zlib/1.2.5 libssh2/1.8.2
> HTTP: GDAL was built against curl 7.64.1, but is running against 7.65.3.
> HTTP: These HTTP headers were set: Metadata-Flavor: Google
> GOA2: Refresh Token Response:
> {"access_token":"REMOVED IN EMAIL","expires_in":2877,"token_type":"Bearer"}
> GOA2: Access Token : 'REMOVED IN EMAIL'
> GS: Using GCE inherited permissions
> GS: <html><head><meta http-equiv="content-type" content="text/html;
> charset=utf-8"/><title>Sorry...</title><style> body { font-family: verdana,
> arial, sans-serif; background-color: #fff; color: #000;
> }</style></head><body><div><table><tr><td><b><font face=sans-serif
> size=10><font color=#4285f4>G</font><font color=#ea4335>o</font><font
> color=#fbbc05>o</font><font color=#4285f4>g</font><font
> color=#34a853>l</font><font color=#ea4335>e</font></font></b></td><td
> style="text-align: left; vertical-align: bottom; padding-bottom: 15px;
> width: 50%"><div style="border-bottom: 1px solid
> #dfdfdf;">Sorry...</div></td></tr></table></div><div style="margin-left:
> 4em;"><h1>We're sorry...</h1><p>... but your computer or network may be
> sending automated queries. To protect our users, we can't process your
> request right now.</p></div><div style="margin-left: 4em;">See <a href="
> https://support.google.com/websearch/answer/86640">Google Help</a> for more
> information.<br/><br/></div><div style="text-align: center; border-top: 1px
> solid #dfdfdf;"><a href="https://www.google.com">Google
> Home</a></div></body></html>
> VSICURL: GetFileSize(
> https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/29/T/NE/S2B_
> MSIL1C_20170719T112119_N0205_R037_T29TNE_20170719T112511.SAFE/GRANULE/L1C_T2
> 9TNE_A001922_20170719T112511/IMG_DATA/T29TNE_20170719T112119_B01.jp2)=0
> response_code=503
> GS: <html><head><meta http-equiv="content-type" content="text/html;
> charset=utf-8"/><title>Sorry...</title><style> body { font-family: verdana,
> arial, sans-serif; background-color: #fff; color: #000;
> }</style></head><body><div><table><tr><td><b><font face=sans-serif
> size=10><font color=#4285f4>G</font><font color=#ea4335>o</font><font
> color=#fbbc05>o</font><font color=#4285f4>g</font><font
> color=#34a853>l</font><font color=#ea4335>e</font></font></b></td><td
> style="text-align: left; vertical-align: bottom; padding-bottom: 15px;
> width: 50%"><div style="border-bottom: 1px solid
> #dfdfdf;">Sorry...</div></td></tr></table></div><div style="margin-left:
> 4em;"><h1>We're sorry...</h1><p>... but your computer or network may be
> sending automated queries. To protect our users, we can't process your
> request right now.</p></div><div style="margin-left: 4em;">See <a href="
> https://support.google.com/websearch/answer/86640">Google Help</a> for more
> information.<br/><br/></div><div style="text-align: center; border-top: 1px
> solid #dfdfdf;"><a href="https://www.google.com">Google
> Home</a></div></body></html>
> ERROR 11: HTTP response code: 503
> gdalinfo failed - unable to open
> '/vsigs/gcp-public-data-sentinel-2/tiles/29/T/NE/S2B_MSIL1C_20170719T112119_
> N0205_R037_T29TNE_20170719T112511.SAFE/GRANULE/L1C_T29TNE_A001922_20170719T1
> 12511/IMG_DATA/T29TNE_20170719T112119_B01.jp2'.

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com