<div dir="ltr"><div>Thanks Michael and Even</div><div><br></div><div>GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR works perfectly for me, in gdalinfo and QGIS. Great.<br></div><div><br></div><div>Even. No, I was not setting GDAL_DISABLE_READDIR_ON_OPEN at all. I think I do not have directory listing permission in that
folder/bucket (that is why it returns a 403, and not a 404, right?)</div><div><br></div><div>Cheers,</div><div>Javier.<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, 27 Apr 2024 at 13:48, Even Rouault <<a href="mailto:even.rouault@spatialys.com">even.rouault@spatialys.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><u></u>
<div>
<p>yes, GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR is the trick to both
disable directory listing and preventing individual file probing.</p>
<p>Javier, do you have GDAL_DISABLE_READDIR_ON_OPEN=YES also set?
Otherwise if GDAL_DISABLE_READDIR_ON_OPEN is not set, GDAL will
normally issue a S3 directory listing call, which should prevent
later individual file probing (if we have the list of files in the
directory, we just need to check the list, and don't need to do
individual open()/GET attempts). But that director listing is
sometimes still undesirable if doing it in a folder with a large
number of files hence GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR. Or
maybe you don't have directory listing permission in that
folder/bucket?</p>
<p>Related to that, Howard filed
<a href="https://github.com/OSGeo/gdal/issues/9443" target="_blank">https://github.com/OSGeo/gdal/issues/9443</a> recently. but it is not
that simple to find a different default behavior that doesn't
result in functional regressions.</p>
<p>I was wondering if a solution, restricted to COG, couldn't be to
put a hint in the so-called "header ghost area"
(<a href="https://gdal.org/drivers/raster/cog.html#header-ghost-area" target="_blank">https://gdal.org/drivers/raster/cog.html#header-ghost-area</a>), like
SIDECAR_FILES=NO, that could be used by the GTiff driver. Not sure
if it is the right way to address the issue however (and would
work only for new files of course, and probably only on explicit
user choice at file creation, because there could be scenarios
where a user would want to provide side car files)<br>
</p>
<div>Le 27/04/2024 à 13:12, Michael Smith
via gdal-dev a écrit :<br>
</div>
<blockquote type="cite">
<div>
<p class="MsoNormal">Javier,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">You can control with the
GDAL_DISABLE_READDIR_ON_OPEN configuration parameter (<a href="https://gdal.org/user/configoptions.html" target="_blank">https://gdal.org/user/configoptions.html</a>).
Typically, I set GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR which
will disable this.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Mike<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">-- <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Michael Smith<u></u><u></u></p>
<p class="MsoNormal">Remote Sensing/GIS Center<u></u><u></u></p>
<p class="MsoNormal">US Army Corps of Engineers<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div style="border-width:1pt medium medium;border-style:solid none none;border-color:rgb(181,196,223) currentcolor currentcolor;padding:3pt 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12pt;color:black">From: </span></b><span style="font-size:12pt;color:black">gdal-dev
<a href="mailto:gdal-dev-bounces@lists.osgeo.org" target="_blank"><gdal-dev-bounces@lists.osgeo.org></a> on behalf of
Javier Jimenez Shaw via gdal-dev
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank"><gdal-dev@lists.osgeo.org></a><br>
<b>Reply-To: </b>Javier Jimenez Shaw
<a href="mailto:j1@jimenezshaw.com" target="_blank"><j1@jimenezshaw.com></a><br>
<b>Date: </b>Saturday, April 27, 2024 at 7:01 AM<br>
<b>To: </b>gdal dev <a href="mailto:gdal-dev@lists.osgeo.org" target="_blank"><gdal-dev@lists.osgeo.org></a><br>
<b>Subject: </b>[gdal-dev] don't search side car files on
s3 with gdalinfo<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<div>
<p class="MsoNormal">Hi<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I am accessing a big file in S3. All
works fine (when I learned how to set the authentication
environment variables), but gdalinfo is checking for a lot
of files that do not exist. I know there is only a GeoTIFF
- COG (no side car files)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">However gdalinfo is trying to find
several side car files, that are not there. S3 returns a
403. It takes its time to search for all those files.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Is there a way to tell to gdalinfo that
it should not check for those files?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Bonus: QGIS is also taking time to open
it (just using 'qgis' instead of 'gdalinfo' in the command
line). I think it is for the same reason. It there a hack
there as well?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">$ AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=... AWS_SESSION_TOKEN=... gdalinfo
/vsis3/blah/folder/ortho.tiff<br>
Driver: GTiff/GeoTIFF<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/ortho.tiff.aux.xml" target="_blank">https://blah.s3.amazonaws.com/folder/ortho.tiff.aux.xml</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/ortho.aux" target="_blank">https://blah.s3.amazonaws.com/folder/ortho.aux</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/ortho.AUX" target="_blank">https://blah.s3.amazonaws.com/folder/ortho.AUX</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/ortho.tiff.aux" target="_blank">https://blah.s3.amazonaws.com/folder/ortho.tiff.aux</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/ortho.tiff.AUX" target="_blank">https://blah.s3.amazonaws.com/folder/ortho.tiff.AUX</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/ortho.tiff.ovr" target="_blank">https://blah.s3.amazonaws.com/folder/ortho.tiff.ovr</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/ortho.tiff.OVR" target="_blank">https://blah.s3.amazonaws.com/folder/ortho.tiff.OVR</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/ortho.tiff.msk" target="_blank">https://blah.s3.amazonaws.com/folder/ortho.tiff.msk</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/ortho.tiff.MSK" target="_blank">https://blah.s3.amazonaws.com/folder/ortho.tiff.MSK</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/ortho.XML" target="_blank">https://blah.s3.amazonaws.com/folder/ortho.XML</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/ortho.xml" target="_blank">https://blah.s3.amazonaws.com/folder/ortho.xml</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/ortho.IMD" target="_blank">https://blah.s3.amazonaws.com/folder/ortho.IMD</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/ortho.imd" target="_blank">https://blah.s3.amazonaws.com/folder/ortho.imd</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/ortho.RPB" target="_blank">https://blah.s3.amazonaws.com/folder/ortho.RPB</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/ortho.rpb" target="_blank">https://blah.s3.amazonaws.com/folder/ortho.rpb</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/ortho.PVL" target="_blank">https://blah.s3.amazonaws.com/folder/ortho.PVL</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/ortho.pvl" target="_blank">https://blah.s3.amazonaws.com/folder/ortho.pvl</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/ortho_rpc.txt" target="_blank">https://blah.s3.amazonaws.com/folder/ortho_rpc.txt</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/ortho_RPC.TXT" target="_blank">https://blah.s3.amazonaws.com/folder/ortho_RPC.TXT</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/ortho_metadata.txt" target="_blank">https://blah.s3.amazonaws.com/folder/ortho_metadata.txt</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/ortho_METADATA.TXT" target="_blank">https://blah.s3.amazonaws.com/folder/ortho_METADATA.TXT</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/ortho_MTL.txt" target="_blank">https://blah.s3.amazonaws.com/folder/ortho_MTL.txt</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/ortho_MTL.TXT" target="_blank">https://blah.s3.amazonaws.com/folder/ortho_MTL.TXT</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/METADATA.DIM" target="_blank">https://blah.s3.amazonaws.com/folder/METADATA.DIM</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/metadata.dim" target="_blank">https://blah.s3.amazonaws.com/folder/metadata.dim</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/ortho_metadata.xml" target="_blank">https://blah.s3.amazonaws.com/folder/ortho_metadata.xml</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/ortho_METADATA.XML" target="_blank">https://blah.s3.amazonaws.com/folder/ortho_METADATA.XML</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/ortho.TXT" target="_blank">https://blah.s3.amazonaws.com/folder/ortho.TXT</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/ortho.txt" target="_blank">https://blah.s3.amazonaws.com/folder/ortho.txt</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/ortho.RPC" target="_blank">https://blah.s3.amazonaws.com/folder/ortho.RPC</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/ortho.rpc" target="_blank">https://blah.s3.amazonaws.com/folder/ortho.rpc</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/ortho.pass" target="_blank">https://blah.s3.amazonaws.com/folder/ortho.pass</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/ortho.PASS" target="_blank">https://blah.s3.amazonaws.com/folder/ortho.PASS</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/summary.txt" target="_blank">https://blah.s3.amazonaws.com/folder/summary.txt</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/SUMMARY.TXT" target="_blank">https://blah.s3.amazonaws.com/folder/SUMMARY.TXT</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/HDRdisplay.txt" target="_blank">https://blah.s3.amazonaws.com/folder/HDRdisplay.txt</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/HDRdisplay.TXT" target="_blank">https://blah.s3.amazonaws.com/folder/HDRdisplay.TXT</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/HDRho_display.txt" target="_blank">https://blah.s3.amazonaws.com/folder/HDRho_display.txt</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/HDRho_display.TXT" target="_blank">https://blah.s3.amazonaws.com/folder/HDRho_display.TXT</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/RPCdisplay.txt" target="_blank">https://blah.s3.amazonaws.com/folder/RPCdisplay.txt</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/RPCdisplay.TXT" target="_blank">https://blah.s3.amazonaws.com/folder/RPCdisplay.TXT</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/RPCho_display.txt" target="_blank">https://blah.s3.amazonaws.com/folder/RPCho_display.txt</a>:
403<br>
Warning 1: HTTP response code on <a href="https://blah.s3.amazonaws.com/folder/RPCho_display.TXT" target="_blank">https://blah.s3.amazonaws.com/folder/RPCho_display.TXT</a>:
403<br>
Files: /vsis3/blah/folder/ortho.tiff<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">...<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">.___ ._ ..._ .. . ._. .___ ..
__ . _. . __.. ... .... ._ .__<u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal">_______________________________________________
gdal-dev mailing list <a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a> <u></u><u></u></p>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
gdal-dev mailing list
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a>
</pre>
</blockquote>
<pre cols="72">--
<a href="http://www.spatialys.com" target="_blank">http://www.spatialys.com</a>
My software is free, but my time generally not.</pre>
</div>
</blockquote></div>