[gdal-dev] don't search side car files on s3 with gdalinfo

[Even Rouault]

yes, GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR is the trick to both disable 
directory listing and preventing individual file probing.

Javier, do you have GDAL_DISABLE_READDIR_ON_OPEN=YES also set? Otherwise 
if GDAL_DISABLE_READDIR_ON_OPEN is not set, GDAL will normally issue a 
S3 directory listing call, which should prevent later individual file 
probing (if we have the list of files in the directory, we just need to 
check the list, and don't need to do individual open()/GET attempts). 
But that director listing is sometimes still undesirable if doing it in 
a folder with a large number of files hence 
GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR. Or maybe you don't have 
directory listing permission in that folder/bucket?

Related to that, Howard filed https://github.com/OSGeo/gdal/issues/9443 
recently. but it is not that simple to find a different default behavior 
that doesn't result in functional regressions.

I was wondering if a solution, restricted to COG, couldn't be to put a 
hint in the so-called "header ghost area" 
(https://gdal.org/drivers/raster/cog.html#header-ghost-area), like 
SIDECAR_FILES=NO, that could be used by the GTiff driver. Not sure if it 
is the right way to address the issue however (and would work only for 
new files of course, and probably only on explicit user choice at file 
creation, because there could be scenarios where a user would want to 
provide side car files)

Le 27/04/2024 à 13:12, Michael Smith via gdal-dev a écrit :
>
> Javier,
>
> You can control with the GDAL_DISABLE_READDIR_ON_OPEN configuration 
> parameter (https://gdal.org/user/configoptions.html). Typically, I set 
> GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR which will disable this.
>
> Mike
>
> -- 
>
> Michael Smith
>
> Remote Sensing/GIS Center
>
> US Army Corps of Engineers
>
> *From: *gdal-dev <gdal-dev-bounces at lists.osgeo.org> on behalf of 
> Javier Jimenez Shaw via gdal-dev <gdal-dev at lists.osgeo.org>
> *Reply-To: *Javier Jimenez Shaw <j1 at jimenezshaw.com>
> *Date: *Saturday, April 27, 2024 at 7:01 AM
> *To: *gdal dev <gdal-dev at lists.osgeo.org>
> *Subject: *[gdal-dev] don't search side car files on s3 with gdalinfo
>
> Hi
>
> I am accessing a big file in S3. All works fine (when I learned how to 
> set the authentication environment variables), but gdalinfo is 
> checking for a lot of files that do not exist. I know there is only a 
> GeoTIFF - COG (no side car files)
>
> However gdalinfo is trying to find several side car files, that are 
> not there. S3 returns a 403. It takes its time to search for all those 
> files.
>
> Is there a way to tell to gdalinfo that it should not check for those 
> files?
>
> Bonus: QGIS is also taking time to open it (just using 'qgis' instead 
> of 'gdalinfo' in the command line). I think it is for the same reason. 
> It there a hack there as well?
>
> Thanks.
>
> $ AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... 
> AWS_SESSION_TOKEN=... gdalinfo /vsis3/blah/folder/ortho.tiff
> Driver: GTiff/GeoTIFF
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/ortho.tiff.aux.xml: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/ortho.aux: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/ortho.AUX: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/ortho.tiff.aux: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/ortho.tiff.AUX: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/ortho.tiff.ovr: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/ortho.tiff.OVR: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/ortho.tiff.msk: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/ortho.tiff.MSK: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/ortho.XML: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/ortho.xml: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/ortho.IMD: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/ortho.imd: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/ortho.RPB: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/ortho.rpb: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/ortho.PVL: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/ortho.pvl: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/ortho_rpc.txt: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/ortho_RPC.TXT: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/ortho_metadata.txt: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/ortho_METADATA.TXT: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/ortho_MTL.txt: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/ortho_MTL.TXT: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/METADATA.DIM: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/metadata.dim: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/ortho_metadata.xml: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/ortho_METADATA.XML: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/ortho.TXT: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/ortho.txt: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/ortho.RPC: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/ortho.rpc: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/ortho.pass: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/ortho.PASS: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/summary.txt: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/SUMMARY.TXT: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/HDRdisplay.txt: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/HDRdisplay.TXT: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/HDRho_display.txt: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/HDRho_display.TXT: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/RPCdisplay.txt: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/RPCdisplay.TXT: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/RPCho_display.txt: 403
> Warning 1: HTTP response code on 
> https://blah.s3.amazonaws.com/folder/RPCho_display.TXT: 403
> Files: /vsis3/blah/folder/ortho.tiff
>
> ...
>
> .___ ._ ..._ .. . ._.  .___ .. __ . _. . __..  ... .... ._ .__
>
> _______________________________________________ gdal-dev mailing list 
> gdal-dev at lists.osgeo.org 
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev

-- 
http://www.spatialys.com
My software is free, but my time generally not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240427/89a4944f/attachment.htm>