<div dir="ltr"><div>Hi,</div><div><br></div><div>I'm in contact with google about the problem. It might be a problem on their side.</div><div><br></div><div>But during testing I discovered something else which I found unusual. <br></div><div>Maybe I understand the meaning of the GDAL_DISABLE_READDIR_ON_OPEN variable incorrectly but I still wanted to double check.<br></div><div><br></div><div>Using:</div><div><br></div><div>#!/bin/bash<br>export GS_ACCESS_KEY_ID=<OUR_KEY><br>export GS_SECRET_ACCESS_KEY=<OUR_KEY><br>export GDAL_DISABLE_READDIR_ON_OPEN=TRUE<br>export CPL_DEBUG=ON<br><br>gdalinfo /vsigs/gcp-public-data-sentinel-2/tiles/29/T/NE/S2B_MSIL1C_20170719T112119_N0205_R037_T29TNE_20170719T112511.SAFE/GRANULE/L1C_T29TNE_A001922_20170719T112511/IMG_DATA/T29TNE_20170719T112119_B01.jp2</div><div><br></div><div>Tries to open tons of files like e.g. <a href="https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/29/T/NE/S2B_MSIL1C_20170719T112119_N0205_R037_T29TNE_20170719T112511.SAFE/GRANULE/L1C_T29TNE_A001922_20170719T112511/IMG_DATA/HDR_20170719T112119_B01.txt">https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/29/T/NE/S2B_MSIL1C_20170719T112119_N0205_R037_T29TNE_20170719T112511.SAFE/GRANULE/L1C_T29TNE_A001922_20170719T112511/IMG_DATA/HDR_20170719T112119_B01.txt</a></div><div><br></div><div>The other to settings FALSE and EMPTY_DIR still try to open some auxiliary files. But not as many.<br></div><div><br></div><div>In this case it seems that setting GDAL_DISABLE_READDIR_ON_OPEN=TRUE is by far the slowest option. <br></div><div>Is this expected?</div><div><br></div><div>Thanks,</div><div>Christoph<br></div><div><br></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 16 Oct 2019 at 17:38, Christoph Paulik <<a href="mailto:cpaulik@vandersat.com">cpaulik@vandersat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi,</div><div><br></div><div>The thing is that one request works fine in any case. <br></div><div>But google seems to rate limit on the public bucket in the following cases:</div><div><br></div><div>- no authentication (/vsicurl/ driver)</div><div>- authentication with automatic detection by gdal that it is running on a GCE VM.</div><div><br></div><div>but not when using the explicit authentication with GS_ACCESS_KEY_ID/GS_SECRET_ACCESS_KEY</div><div><br></div><div>Christoph<br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 16 Oct 2019 at 17:31, Even Rouault <<a href="mailto:even.rouault@spatialys.com" target="_blank">even.rouault@spatialys.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
did you try with Google gsutil tool ?<br>
<br>
Anyway as it is a public resource that doesn't require any signing, you could <br>
just use<br>
<br>
/vsicurl/<a href="https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/29/T/NE/S2B_MSIL1C_20170719T112119_N0205_R037_T29TNE_20170719T112511.SAFE/GRANULE/L1C_T29TNE_A001922_20170719T112511/IMG_DATA/T29TNE_20170719T112119_B01.jp2" rel="noreferrer" target="_blank">https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/29/T/<br>
NE/S2B_MSIL1C_20170719T112119_N0205_R037_T29TNE_20170719T112511.SAFE/GRANULE/<br>
L1C_T29TNE_A001922_20170719T112511/IMG_DATA/T29TNE_20170719T112119_B01.jp2</a><br>
<br>
Even<br>
<br>
> Hi,<br>
> <br>
> When doing a few requests to the Sentinel 2 bucket of Google (<br>
> <a href="https://cloud.google.com/storage/docs/public-datasets/sentinel-2" rel="noreferrer" target="_blank">https://cloud.google.com/storage/docs/public-datasets/sentinel-2</a>) we have<br>
> been getting 503 Responses when using the automatic authentication. See the<br>
> end of this email for the full output.<br>
> <br>
> This behavior goes away when using e.g. the<br>
> GS_ACCESS_KEY_ID/GS_SECRET_ACCESS_KEY authentication method.<br>
> <br>
> The Service account this machine is running with has "Full Access to All<br>
> Cloud APIs"<br>
> <br>
> Not sure if this is a gdal problem or actually a google internal issue<br>
> since accessing our own buckets works fine.<br>
> <br>
> Best,<br>
> Christoph<br>
> <br>
> Output:<br>
> export CPL_DEBUG=ON<br>
> gdalinfo<br>
> /vsigs/gcp-public-data-sentinel-2/tiles/29/T/NE/S2B_MSIL1C_20170719T112119_N<br>
> 0205_R037_T29TNE_20170719T112511.SAFE/GRANULE/L1C_T29TNE_A001922_20170719T11<br>
> 2511/IMG_DATA/T29TNE_20170719T112119_B01.jp2<br>
> <br>
> HTTP: Fetch(<br>
> <a href="http://metadata.google.internal/computeMetadata/v1/instance/service-accounts" rel="noreferrer" target="_blank">http://metadata.google.internal/computeMetadata/v1/instance/service-accounts</a><br>
> /default/token )<br>
> HTTP: libcurl/7.65.3 OpenSSL/1.1.1c zlib/1.2.5 libssh2/1.8.2<br>
> HTTP: GDAL was built against curl 7.64.1, but is running against 7.65.3.<br>
> HTTP: These HTTP headers were set: Metadata-Flavor: Google<br>
> GOA2: Refresh Token Response:<br>
> {"access_token":"REMOVED IN EMAIL","expires_in":2877,"token_type":"Bearer"}<br>
> GOA2: Access Token : 'REMOVED IN EMAIL'<br>
> GS: Using GCE inherited permissions<br>
> GS: <html><head><meta http-equiv="content-type" content="text/html;<br>
> charset=utf-8"/><title>Sorry...</title><style> body { font-family: verdana,<br>
> arial, sans-serif; background-color: #fff; color: #000;<br>
> }</style></head><body><div><table><tr><td><b><font face=sans-serif<br>
> size=10><font color=#4285f4>G</font><font color=#ea4335>o</font><font<br>
> color=#fbbc05>o</font><font color=#4285f4>g</font><font<br>
> color=#34a853>l</font><font color=#ea4335>e</font></font></b></td><td<br>
> style="text-align: left; vertical-align: bottom; padding-bottom: 15px;<br>
> width: 50%"><div style="border-bottom: 1px solid<br>
> #dfdfdf;">Sorry...</div></td></tr></table></div><div style="margin-left:<br>
> 4em;"><h1>We're sorry...</h1><p>... but your computer or network may be<br>
> sending automated queries. To protect our users, we can't process your<br>
> request right now.</p></div><div style="margin-left: 4em;">See <a href="<br>
> <a href="https://support.google.com/websearch/answer/86640" rel="noreferrer" target="_blank">https://support.google.com/websearch/answer/86640</a>">Google Help</a> for more<br>
> information.<br/><br/></div><div style="text-align: center; border-top: 1px<br>
> solid #dfdfdf;"><a href="<a href="https://www.google.com" rel="noreferrer" target="_blank">https://www.google.com</a>">Google<br>
> Home</a></div></body></html><br>
> VSICURL: GetFileSize(<br>
> <a href="https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/29/T/NE/S2B_" rel="noreferrer" target="_blank">https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/29/T/NE/S2B_</a><br>
> MSIL1C_20170719T112119_N0205_R037_T29TNE_20170719T112511.SAFE/GRANULE/L1C_T2<br>
> 9TNE_A001922_20170719T112511/IMG_DATA/T29TNE_20170719T112119_B01.jp2)=0<br>
> response_code=503<br>
> GS: <html><head><meta http-equiv="content-type" content="text/html;<br>
> charset=utf-8"/><title>Sorry...</title><style> body { font-family: verdana,<br>
> arial, sans-serif; background-color: #fff; color: #000;<br>
> }</style></head><body><div><table><tr><td><b><font face=sans-serif<br>
> size=10><font color=#4285f4>G</font><font color=#ea4335>o</font><font<br>
> color=#fbbc05>o</font><font color=#4285f4>g</font><font<br>
> color=#34a853>l</font><font color=#ea4335>e</font></font></b></td><td<br>
> style="text-align: left; vertical-align: bottom; padding-bottom: 15px;<br>
> width: 50%"><div style="border-bottom: 1px solid<br>
> #dfdfdf;">Sorry...</div></td></tr></table></div><div style="margin-left:<br>
> 4em;"><h1>We're sorry...</h1><p>... but your computer or network may be<br>
> sending automated queries. To protect our users, we can't process your<br>
> request right now.</p></div><div style="margin-left: 4em;">See <a href="<br>
> <a href="https://support.google.com/websearch/answer/86640" rel="noreferrer" target="_blank">https://support.google.com/websearch/answer/86640</a>">Google Help</a> for more<br>
> information.<br/><br/></div><div style="text-align: center; border-top: 1px<br>
> solid #dfdfdf;"><a href="<a href="https://www.google.com" rel="noreferrer" target="_blank">https://www.google.com</a>">Google<br>
> Home</a></div></body></html><br>
> ERROR 11: HTTP response code: 503<br>
> gdalinfo failed - unable to open<br>
> '/vsigs/gcp-public-data-sentinel-2/tiles/29/T/NE/S2B_MSIL1C_20170719T112119_<br>
> N0205_R037_T29TNE_20170719T112511.SAFE/GRANULE/L1C_T29TNE_A001922_20170719T1<br>
> 12511/IMG_DATA/T29TNE_20170719T112119_B01.jp2'.<br>
<br>
<br>
-- <br>
Spatialys - Geospatial professional services<br>
<a href="http://www.spatialys.com" rel="noreferrer" target="_blank">http://www.spatialys.com</a><br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div style="font-size:12px;color:rgb(136,136,136);font-family:Helvetica;line-height:normal;margin:0in 0in 0.0001pt"><span style="line-height:24px;font-family:Arial,sans-serif"><font color="#45818e"><b>Christoph Paulik</b></font></span><span style="line-height:24px;color:rgb(69,129,142);font-family:Arial,sans-serif"> </span><span style="line-height:24px;font-family:Arial,sans-serif"><font color="#ff9900">//</font></span><span style="line-height:24px;color:rgb(69,129,142);font-family:Arial,sans-serif"> Head of Infrastructure<br></span></div><span style="font-size:12px"><div style="line-height:normal;margin:0in 0in 0.0001pt"><span style="color:rgb(34,34,34);font-family:Arial,sans-serif;line-height:18pt"><font color="#3d85c6">VanderSat</font></span><span style="color:rgb(136,136,136);font-family:Arial,sans-serif;line-height:18pt"> </span><span style="color:rgb(34,34,34);font-family:Arial,sans-serif;line-height:18pt"><font color="#ff9900">//</font></span><span style="font-family:Arial,sans-serif;line-height:18pt"><font color="#222222"> </font><font color="#3d85c6">Satellite observed water data. Globally. Daily.</font></span></div><div style="color:rgb(34,34,34);line-height:normal;margin:0in 0in 0.0001pt"><span style="font-family:Arial,sans-serif;line-height:18pt"><div style="font-family:Helvetica;line-height:normal;margin:0in 0in 0.0001pt"><span style="font-family:Arial,sans-serif;line-height:18pt"><font color="#3d85c6"><a href="https://maps.google.com/?q=Wilhelminastraat+43a,+2011+VK,+Haarlem+(NL),+The+Netherlands&entry=gmail&source=g" style="color:rgb(17,85,204)" target="_blank">Wilhelminastraat 43a, 2011 VK, Haarlem (NL), The Netherlands</a></font></span></div></span></div></span><div style="color:rgb(0,0,0);font-size:12px;line-height:normal;margin:0in 0in 0.0001pt"><span style="color:rgb(136,136,136);font-family:Arial,sans-serif;line-height:18pt"><font color="#ff9900"><b>M</b></font></span><span style="color:rgb(92,102,112);font-family:Arial,sans-serif;line-height:18pt"> </span><font style="color:rgb(34,34,34)"><span style="color:rgb(136,136,136);font-family:Arial,sans-serif;line-height:18pt"> </span><font color="#3d85c6"><span style="font-family:Arial,sans-serif;line-height:18pt">+31 6 </span><font style="font-family:Arial,sans-serif;line-height:24px">18271928 </font></font></font><span style="font-family:Arial,sans-serif;color:rgb(136,136,136);line-height:18pt"><font color="#ff9900"><b>W</b></font></span><span style="font-family:Arial,sans-serif;color:rgb(92,102,112);line-height:18pt"> </span><font color="#45818e"><span style="font-family:Arial,sans-serif;line-height:18pt"> </span></font><a href="http://www.vandersat.com/" style="color:rgb(17,85,204);font-family:Arial,sans-serif" target="_blank"><font color="#3d85c6">www.vandersat.com</font></a></div><p style="color:rgb(0,0,0);font-size:12px;font-family:Calibri,sans-serif;line-height:normal;margin:0in 0in 0.0001pt"><span style="color:rgb(34,34,34);font-family:arial,sans-serif"></span></p><div style="color:rgb(0,0,0);font-size:12px;font-family:Helvetica;line-height:normal;margin:0in 0in 0.0001pt">--------------------------------------------------------------------------------------</div></div></div></div></div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div style="font-size:12px;color:rgb(136,136,136);font-family:Helvetica;line-height:normal;margin:0in 0in 0.0001pt"><span style="line-height:24px;font-family:Arial,sans-serif"><font color="#45818e"><b>Christoph Paulik</b></font></span><span style="line-height:24px;color:rgb(69,129,142);font-family:Arial,sans-serif"> </span><span style="line-height:24px;font-family:Arial,sans-serif"><font color="#ff9900">//</font></span><span style="line-height:24px;color:rgb(69,129,142);font-family:Arial,sans-serif"> Head of Infrastructure<br></span></div><span style="font-size:12px"><div style="line-height:normal;margin:0in 0in 0.0001pt"><span style="color:rgb(34,34,34);font-family:Arial,sans-serif;line-height:18pt"><font color="#3d85c6">VanderSat</font></span><span style="color:rgb(136,136,136);font-family:Arial,sans-serif;line-height:18pt"> </span><span style="color:rgb(34,34,34);font-family:Arial,sans-serif;line-height:18pt"><font color="#ff9900">//</font></span><span style="font-family:Arial,sans-serif;line-height:18pt"><font color="#222222"> </font><font color="#3d85c6">Satellite observed water data. Globally. Daily.</font></span></div><div style="color:rgb(34,34,34);line-height:normal;margin:0in 0in 0.0001pt"><span style="font-family:Arial,sans-serif;line-height:18pt"><div style="font-family:Helvetica;line-height:normal;margin:0in 0in 0.0001pt"><span style="font-family:Arial,sans-serif;line-height:18pt"><font color="#3d85c6"><a href="https://maps.google.com/?q=Wilhelminastraat+43a,+2011+VK,+Haarlem+(NL),+The+Netherlands&entry=gmail&source=g" style="color:rgb(17,85,204)" target="_blank">Wilhelminastraat 43a, 2011 VK, Haarlem (NL), The Netherlands</a></font></span></div></span></div></span><div style="color:rgb(0,0,0);font-size:12px;line-height:normal;margin:0in 0in 0.0001pt"><span style="color:rgb(136,136,136);font-family:Arial,sans-serif;line-height:18pt"><font color="#ff9900"><b>M</b></font></span><span style="color:rgb(92,102,112);font-family:Arial,sans-serif;line-height:18pt"> </span><font style="color:rgb(34,34,34)"><span style="color:rgb(136,136,136);font-family:Arial,sans-serif;line-height:18pt"> </span><font color="#3d85c6"><span style="font-family:Arial,sans-serif;line-height:18pt">+31 6 </span><font style="font-family:Arial,sans-serif;line-height:24px">18271928 </font></font></font><span style="font-family:Arial,sans-serif;color:rgb(136,136,136);line-height:18pt"><font color="#ff9900"><b>W</b></font></span><span style="font-family:Arial,sans-serif;color:rgb(92,102,112);line-height:18pt"> </span><font color="#45818e"><span style="font-family:Arial,sans-serif;line-height:18pt"> </span></font><a href="http://www.vandersat.com/" style="color:rgb(17,85,204);font-family:Arial,sans-serif" target="_blank"><font color="#3d85c6">www.vandersat.com</font></a></div><p style="color:rgb(0,0,0);font-size:12px;font-family:Calibri,sans-serif;line-height:normal;margin:0in 0in 0.0001pt"><span style="color:rgb(34,34,34);font-family:arial,sans-serif"></span></p><div style="color:rgb(0,0,0);font-size:12px;font-family:Helvetica;line-height:normal;margin:0in 0in 0.0001pt">--------------------------------------------------------------------------------------</div></div></div></div></div>