[gdal-dev] don't search side car files on s3 with gdalinfo

[Michael Smith]

Javier,

 

You can control with the GDAL_DISABLE_READDIR_ON_OPEN configuration parameter (https://gdal.org/user/configoptions.html). Typically, I set GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR which will disable this.

 

Mike

 

 

-- 

Michael Smith

Remote Sensing/GIS Center

US Army Corps of Engineers

 

 

From: gdal-dev <gdal-dev-bounces at lists.osgeo.org> on behalf of Javier Jimenez Shaw via gdal-dev <gdal-dev at lists.osgeo.org>
Reply-To: Javier Jimenez Shaw <j1 at jimenezshaw.com>
Date: Saturday, April 27, 2024 at 7:01 AM
To: gdal dev <gdal-dev at lists.osgeo.org>
Subject: [gdal-dev] don't search side car files on s3 with gdalinfo

 

Hi

 

I am accessing a big file in S3. All works fine (when I learned how to set the authentication environment variables), but gdalinfo is checking for a lot of files that do not exist. I know there is only a GeoTIFF - COG (no side car files)

 

However gdalinfo is trying to find several side car files, that are not there. S3 returns a 403. It takes its time to search for all those files.

 

Is there a way to tell to gdalinfo that it should not check for those files?

 

Bonus: QGIS is also taking time to open it (just using 'qgis' instead of 'gdalinfo' in the command line). I think it is for the same reason. It there a hack there as well?

 

Thanks.

 

$ AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... AWS_SESSION_TOKEN=... gdalinfo /vsis3/blah/folder/ortho.tiff
Driver: GTiff/GeoTIFF
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/ortho.tiff.aux.xml: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/ortho.aux: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/ortho.AUX: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/ortho.tiff.aux: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/ortho.tiff.AUX: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/ortho.tiff.ovr: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/ortho.tiff.OVR: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/ortho.tiff.msk: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/ortho.tiff.MSK: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/ortho.XML: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/ortho.xml: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/ortho.IMD: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/ortho.imd: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/ortho.RPB: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/ortho.rpb: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/ortho.PVL: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/ortho.pvl: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/ortho_rpc.txt: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/ortho_RPC.TXT: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/ortho_metadata.txt: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/ortho_METADATA.TXT: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/ortho_MTL.txt: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/ortho_MTL.TXT: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/METADATA.DIM: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/metadata.dim: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/ortho_metadata.xml: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/ortho_METADATA.XML: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/ortho.TXT: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/ortho.txt: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/ortho.RPC: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/ortho.rpc: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/ortho.pass: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/ortho.PASS: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/summary.txt: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/SUMMARY.TXT: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/HDRdisplay.txt: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/HDRdisplay.TXT: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/HDRho_display.txt: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/HDRho_display.TXT: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/RPCdisplay.txt: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/RPCdisplay.TXT: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/RPCho_display.txt: 403
Warning 1: HTTP response code on https://blah.s3.amazonaws.com/folder/RPCho_display.TXT: 403
Files: /vsis3/blah/folder/ortho.tiff

...

 

.___ ._ ..._ .. . ._.  .___ .. __ . _. . __..  ... .... ._ .__

_______________________________________________ gdal-dev mailing list gdal-dev at lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240427/13ced694/attachment-0001.htm>