[gdal-dev] don't search side car files on s3 with gdalinfo

Sat Apr 27 05:08:33 PDT 2024

Thanks Michael and Even

GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR works perfectly for me, in gdalinfo
and QGIS. Great.

Even. No, I was not setting GDAL_DISABLE_READDIR_ON_OPEN at all. I think I
do not have directory listing permission in that folder/bucket (that is why
it returns a 403, and not a 404, right?)

Cheers,
Javier.

On Sat, 27 Apr 2024 at 13:48, Even Rouault <even.rouault at spatialys.com>
wrote:

> yes, GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR is the trick to both disable
> directory listing and preventing individual file probing.
>
> Javier, do you have GDAL_DISABLE_READDIR_ON_OPEN=YES also set? Otherwise
> if GDAL_DISABLE_READDIR_ON_OPEN is not set, GDAL will normally issue a S3
> directory listing call, which should prevent later individual file probing
> (if we have the list of files in the directory, we just need to check the
> list, and don't need to do individual open()/GET attempts). But that
> director listing is sometimes still undesirable if doing it in a folder
> with a large number of files hence GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR.
> Or maybe you don't have directory listing permission in that folder/bucket?
>
> Related to that, Howard filed https://github.com/OSGeo/gdal/issues/9443
> recently. but it is not that simple to find a different default behavior
> that doesn't result in functional regressions.
>
> I was wondering if a solution, restricted to COG, couldn't be to put a
> hint in the so-called "header ghost area" (
> https://gdal.org/drivers/raster/cog.html#header-ghost-area), like
> SIDECAR_FILES=NO, that could be used by the GTiff driver. Not sure if it is
> the right way to address the issue however (and would work only for new
> files of course, and probably only on explicit user choice at file
> creation, because there could be scenarios where a user would want to
> provide side car files)
> Le 27/04/2024 à 13:12, Michael Smith via gdal-dev a écrit :
>
> Javier,
>
>
>
> You can control with the GDAL_DISABLE_READDIR_ON_OPEN configuration
> parameter (https://gdal.org/user/configoptions.html). Typically, I set
> GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR which will disable this.
>
>
>
> Mike
>
>
>
>
>
> --
>
> Michael Smith
>
> Remote Sensing/GIS Center
>
> US Army Corps of Engineers
>
>
>
>
>
> *From: *gdal-dev <gdal-dev-bounces at lists.osgeo.org>
> <gdal-dev-bounces at lists.osgeo.org> on behalf of Javier Jimenez Shaw via
> gdal-dev <gdal-dev at lists.osgeo.org> <gdal-dev at lists.osgeo.org>
> *Reply-To: *Javier Jimenez Shaw <j1 at jimenezshaw.com> <j1 at jimenezshaw.com>
> *Date: *Saturday, April 27, 2024 at 7:01 AM
> *To: *gdal dev <gdal-dev at lists.osgeo.org> <gdal-dev at lists.osgeo.org>
> *Subject: *[gdal-dev] don't search side car files on s3 with gdalinfo
>
>
>
> Hi
>
>
>
> I am accessing a big file in S3. All works fine (when I learned how to set
> the authentication environment variables), but gdalinfo is checking for a
> lot of files that do not exist. I know there is only a GeoTIFF - COG (no
> side car files)
>
>
>
> However gdalinfo is trying to find several side car files, that are not
> there. S3 returns a 403. It takes its time to search for all those files.
>
>
>
> Is there a way to tell to gdalinfo that it should not check for those
> files?
>
>
>
> Bonus: QGIS is also taking time to open it (just using 'qgis' instead of
> 'gdalinfo' in the command line). I think it is for the same reason. It
> there a hack there as well?
>
>
>
> Thanks.
>
>
>
> $ AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... AWS_SESSION_TOKEN=...
> gdalinfo /vsis3/blah/folder/ortho.tiff
> Driver: GTiff/GeoTIFF
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/ortho.tiff.aux.xml: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/ortho.aux: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/ortho.AUX: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/ortho.tiff.aux: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/ortho.tiff.AUX: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/ortho.tiff.ovr: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/ortho.tiff.OVR: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/ortho.tiff.msk: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/ortho.tiff.MSK: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/ortho.XML: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/ortho.xml: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/ortho.IMD: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/ortho.imd: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/ortho.RPB: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/ortho.rpb: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/ortho.PVL: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/ortho.pvl: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/ortho_rpc.txt: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/ortho_RPC.TXT: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/ortho_metadata.txt: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/ortho_METADATA.TXT: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/ortho_MTL.txt: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/ortho_MTL.TXT: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/METADATA.DIM: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/metadata.dim: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/ortho_metadata.xml: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/ortho_METADATA.XML: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/ortho.TXT: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/ortho.txt: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/ortho.RPC: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/ortho.rpc: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/ortho.pass: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/ortho.PASS: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/summary.txt: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/SUMMARY.TXT: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/HDRdisplay.txt: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/HDRdisplay.TXT: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/HDRho_display.txt: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/HDRho_display.TXT: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/RPCdisplay.txt: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/RPCdisplay.TXT: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/RPCho_display.txt: 403
> Warning 1: HTTP response code on
> https://blah.s3.amazonaws.com/folder/RPCho_display.TXT: 403
> Files: /vsis3/blah/folder/ortho.tiff
>
> ...
>
>
>
> .___ ._ ..._ .. . ._.  .___ .. __ . _. . __..  ... .... ._ .__
>
> _______________________________________________ gdal-dev mailing list
> gdal-dev at lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
> _______________________________________________
> gdal-dev mailing listgdal-dev at lists.osgeo.orghttps://lists.osgeo.org/mailman/listinfo/gdal-dev
>
> -- http://www.spatialys.com
> My software is free, but my time generally not.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240427/e03bd576/attachment-0001.htm>