<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi Darren<div class=""><br class=""></div><div class="">You can do this by setting `GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR` or by narrowing the allowed extension with "CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif"</div><div class=""><br class=""></div><div class="">Kyle is working on some docs that explains those env in TiTiler:</div><div class="">- <a href="https://github.com/developmentseed/titiler/blob/0a5288de30845865256a0124ed233f918812aa78/docs/concepts/performance_tuning.md#gdal_disable_readdir_on_open" class="">https://github.com/developmentseed/titiler/blob/0a5288de30845865256a0124ed233f918812aa78/docs/concepts/performance_tuning.md#gdal_disable_readdir_on_open</a></div><div class="">- <a href="https://github.com/developmentseed/titiler/blob/0a5288de30845865256a0124ed233f918812aa78/docs/concepts/performance_tuning.md#cpl_vsil_curl_allowed_extensions" class="">https://github.com/developmentseed/titiler/blob/0a5288de30845865256a0124ed233f918812aa78/docs/concepts/performance_tuning.md#cpl_vsil_curl_allowed_extensions</a></div><div class=""><br class=""></div><div class="">Vincent<br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Mar 3, 2021, at 6:56 PM, Darren Weber <<a href="mailto:darren.weber@jupiterintel.com" class="">darren.weber@jupiterintel.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">In <a href="https://github.com/mapbox/rasterio/issues/2119#issuecomment-790024225" class="">https://github.com/mapbox/rasterio/issues/2119#issuecomment-790024225</a> I've noted how s3-COG reads can use retries to handle s3 rate throttling (503) by enabling the <span style="background-color:rgb(255,255,255)" class=""><font class="">`<span style="font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,monospace;font-size:11.9px" class="">GDAL_HTTP_MAX_RETRY`</span>. Our dataset on </font></span>s3 only contains *.tif files and we need to prevent GDAL from issuing any GET/HEAD requests to try to read any other files. Is there an env-var for this?<div class=""><br class=""></div><div class="">The `GDAL_DISABLE_READDIR_ON_OPEN` is `False` so that GDAL can read the s3 prefix to list objects in one GET request. From that read, it should have already determined that there are only *.tif files and no *.aux* files available. When it gets a 503 rate throttle response, however, it seems to auto-try to read some supplementary files, e.g. one log contains messages that indicate GDAL is trying to read an .aux file that does not exist and we want to prevent GDAL from issuing any GET or HEAD requests for anything other than *.tif files (objects).</div><div class=""><br class=""></div><div class=""><pre style="box-sizing:border-box;font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,monospace;font-size:11.9px;margin-top:0px;margin-bottom:16px;padding:16px;overflow:auto;line-height:1.45;border-radius:6px;color:rgb(157,165,180)" class=""><code style="box-sizing:border-box;background:initial;font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,monospace;font-size:11.9px;padding:0px;margin:0px;border-radius:6px;word-break:normal;border:0px;display:inline;overflow:visible;line-height:inherit;color:rgb(175,97,97)" class="">[WARNING] 2021-03-03T19:37:40.312Z CPLE_AppDefined in HTTP error code: 503 - <a href="https://a-bucket.s3.amazonaws.com/geotiff.aux" class="">https://a-bucket.s3.amazonaws.com/geotiff.aux</a>. Retrying again in 0.5 secs
[WARNING] 2021-03-03T19:37:40.795Z CPLE_AppDefined in HTTP error code: 503 - <a href="https://a-bucket.s3.amazonaws.com/geotiff.aux" class="">https://a-bucket.s3.amazonaws.com/geotiff.aux</a>. Retrying again in 1.0 secs
[WARNING] 2021-03-03T19:37:41.796Z CPLE_AppDefined in HTTP error code: 503 - <a href="https://a-bucket.s3.amazonaws.com/geotiff.aux" class="">https://a-bucket.s3.amazonaws.com/geotiff.aux</a>. Retrying again in 2.2 secs
[WARNING] 2021-03-03T19:37:44.290Z CPLE_AppDefined in HTTP response code on <a href="https://a-bucket.s3.amazonaws.com/geotiff.aux" class="">https://a-bucket.s3.amazonaws.com/geotiff.aux</a>: 503
2021-03-03 19:37:44,155 | INFO | extract_raster_points:1036 | <a href="s3://a-bucket/geotiff.tif" class="">s3://a-bucket/geotiff.tif</a> extract cells (1)...
2021-03-03 19:37:44,347 | INFO | extract_raster_points:1050 | <a href="s3://a-bucket/geotiff.tif" class="">s3://a-bucket/geotiff.tif</a> extract cells (1) done</code></pre></div><div class="">That log suggests that there is a huge performance hit to retry reading an .aux file that does not exist and GDAL should not even try to issue any kind of request for it (i.e. we want to prevent it).</div><div class=""><br class=""></div><div class="">When running some unit tests with debug curl details available (but no ability to create 503 responses), it appears that GDAL finds just the file required and reads only partial reads from that file, e.g. this is a sample of GET requests to read from one .tif s3-COG:</div><div class=""><br class=""></div><div class="">$ grep 'GET' tmp.log<br class="">> GET /?delimiter=%2F&prefix=unit_tests%2Fgis%2F HTTP/1.1<br class="">> GET /unit_tests/gis/cea_blocks.tif HTTP/1.1<br class="">> GET /unit_tests/gis/cea_blocks.tif HTTP/1.1<br class="">> GET /unit_tests/gis/cea_blocks.tif HTTP/1.1<br class="">> GET /unit_tests/gis/cea_blocks.tif HTTP/1.1<br class="">> GET /unit_tests/gis/cea_blocks.tif HTTP/1.1<br class="">> GET /unit_tests/gis/cea_blocks.tif HTTP/1.1<br class=""><div class=""><br class=""></div><div class="">TIA, Darren</div><div class=""><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr" class=""><div style="font-family:Arial,Helvetica,sans-serif;font-size:small;letter-spacing:normal" class=""><span style="color:rgb(68,68,68)" class="">Darren Weber, PhD</span><br class=""></div><div style="font-family:Arial,Helvetica,sans-serif;font-size:small;letter-spacing:normal" class=""><font color="#444444" class="">Senior Software Engineer</font></div><div style="font-family:Arial,Helvetica,sans-serif;font-size:small;letter-spacing:normal" class=""><span style="color:rgb(68,68,68)" class="">Jupiter Intelligence</span></div><div style="font-family:Arial,Helvetica,sans-serif;font-size:small;letter-spacing:normal" class=""><span style="font-size:12.8px" class=""><br class=""></span></div><div style="font-family:Arial,Helvetica,sans-serif;font-size:small;letter-spacing:normal" class=""><span style="font-size:12.8px" class=""><img src="https://docs.google.com/uc?export=download&id=0B3kl0-6HWUbGOUNZSkhhM0lVc3M&revid=0B3kl0-6HWUbGSTBqdTRnRGdjelFVazJQbnR6b1k4Y3loWm9rPQ" width="96" height="28" class=""><br class=""></span></div><div style="font-family:Arial,Helvetica,sans-serif;font-size:small;letter-spacing:normal" class=""><font size="1" class=""><i class="">"Predicting risk in a changing climate"</i></font></div></div></div></div></div></div>
_______________________________________________<br class="">gdal-dev mailing list<br class=""><a href="mailto:gdal-dev@lists.osgeo.org" class="">gdal-dev@lists.osgeo.org</a><br class="">https://lists.osgeo.org/mailman/listinfo/gdal-dev<br class=""></div></blockquote></div><br class=""></div></body></html>