<div dir="ltr">In <a href="https://github.com/mapbox/rasterio/issues/2119#issuecomment-790024225">https://github.com/mapbox/rasterio/issues/2119#issuecomment-790024225</a> I've noted how s3-COG reads can use retries to handle s3 rate throttling (503) by enabling the <span style="background-color:rgb(255,255,255)"><font color="#000000">`<span style="font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,monospace;font-size:11.9px">GDAL_HTTP_MAX_RETRY`</span>.  Our dataset on </font></span>s3 only contains *.tif files and we need to prevent GDAL from issuing any GET/HEAD requests to try to read any other files.  Is there an env-var for this?<div><br></div><div>The `GDAL_DISABLE_READDIR_ON_OPEN` is `False` so that GDAL can read the s3 prefix to list objects in one GET request.  From that read, it should have already determined that there are only *.tif files and no *.aux* files available.  When it gets a 503 rate throttle response, however, it seems to auto-try to read some supplementary files, e.g. one log contains messages that indicate GDAL is trying to read an .aux file that does not exist and we want to prevent GDAL from issuing any GET or HEAD requests for anything other than *.tif files (objects).</div><div><br></div><div><pre style="box-sizing:border-box;font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,monospace;font-size:11.9px;margin-top:0px;margin-bottom:16px;padding:16px;overflow:auto;line-height:1.45;border-radius:6px;color:rgb(157,165,180)"><code style="box-sizing:border-box;background:initial;font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,monospace;font-size:11.9px;padding:0px;margin:0px;border-radius:6px;word-break:normal;border:0px;display:inline;overflow:visible;line-height:inherit;color:rgb(175,97,97)">[WARNING] 2021-03-03T19:37:40.312Z CPLE_AppDefined in HTTP error code: 503 - <a href="https://a-bucket.s3.amazonaws.com/geotiff.aux">https://a-bucket.s3.amazonaws.com/geotiff.aux</a>. Retrying again in 0.5 secs
[WARNING] 2021-03-03T19:37:40.795Z CPLE_AppDefined in HTTP error code: 503 - <a href="https://a-bucket.s3.amazonaws.com/geotiff.aux">https://a-bucket.s3.amazonaws.com/geotiff.aux</a>. Retrying again in 1.0 secs
[WARNING] 2021-03-03T19:37:41.796Z CPLE_AppDefined in HTTP error code: 503 - <a href="https://a-bucket.s3.amazonaws.com/geotiff.aux">https://a-bucket.s3.amazonaws.com/geotiff.aux</a>. Retrying again in 2.2 secs
[WARNING] 2021-03-03T19:37:44.290Z CPLE_AppDefined in HTTP response code on <a href="https://a-bucket.s3.amazonaws.com/geotiff.aux">https://a-bucket.s3.amazonaws.com/geotiff.aux</a>: 503
2021-03-03 19:37:44,155 | INFO | extract_raster_points:1036 | s3://a-bucket/geotiff.tif extract cells (1)...
2021-03-03 19:37:44,347 | INFO | extract_raster_points:1050 | s3://a-bucket/geotiff.tif extract cells (1) done</code></pre></div><div>That log suggests that there is a huge performance hit to retry reading an .aux file that does not exist and GDAL should not even try to issue any kind of request for it (i.e. we want to prevent it).</div><div><br></div><div>When running some unit tests with debug curl details available (but no ability to create 503 responses), it appears that GDAL finds just the file required and reads only partial reads from that file, e.g. this is a sample of GET requests to read from one .tif s3-COG:</div><div><br></div><div>$ grep 'GET' tmp.log<br>> GET /?delimiter=%2F&prefix=unit_tests%2Fgis%2F HTTP/1.1<br>> GET /unit_tests/gis/cea_blocks.tif HTTP/1.1<br>> GET /unit_tests/gis/cea_blocks.tif HTTP/1.1<br>> GET /unit_tests/gis/cea_blocks.tif HTTP/1.1<br>> GET /unit_tests/gis/cea_blocks.tif HTTP/1.1<br>> GET /unit_tests/gis/cea_blocks.tif HTTP/1.1<br>> GET /unit_tests/gis/cea_blocks.tif HTTP/1.1<br><div><br></div><div>TIA, Darren</div><div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div style="font-family:Arial,Helvetica,sans-serif;font-size:small;letter-spacing:normal"><span style="color:rgb(68,68,68)">Darren Weber, PhD</span><br></div><div style="font-family:Arial,Helvetica,sans-serif;font-size:small;letter-spacing:normal"><font color="#444444">Senior Software Engineer</font></div><div style="font-family:Arial,Helvetica,sans-serif;font-size:small;letter-spacing:normal"><span style="color:rgb(68,68,68)">Jupiter Intelligence</span></div><div style="font-family:Arial,Helvetica,sans-serif;font-size:small;letter-spacing:normal"><span style="font-size:12.8px"><br></span></div><div style="font-family:Arial,Helvetica,sans-serif;font-size:small;letter-spacing:normal"><span style="font-size:12.8px"><img src="https://docs.google.com/uc?export=download&id=0B3kl0-6HWUbGOUNZSkhhM0lVc3M&revid=0B3kl0-6HWUbGSTBqdTRnRGdjelFVazJQbnR6b1k4Y3loWm9rPQ" width="96" height="28"><br></span></div><div style="font-family:Arial,Helvetica,sans-serif;font-size:small;letter-spacing:normal"><font size="1"><i>"Predicting risk in a changing climate"</i></font></div></div></div></div></div></div>