[gdal-dev] [EXTERNAL] s3 request example

[Javier Jimenez Shaw]

This is getting even weirder. I tried removing gdal_TileDB.so before my
first email, and it didn't work.
However, now it works... but only if I opened the file before. If I wait a
couple of minutes, then it still waits 5 or 6 seconds. Is there any cache
somewhere? (I am closing QGIS, so it should not be in the memory of QGIS)

[Fri Sep 13 18:44:11 2024].3102, 0.0000: GDAL: Proxy driver TileDB *not*
registered due to gdal_TileDB.so not being found
[Fri Sep 13 18:44:12 2024].3429, 1.0326: GDAL: Proxy driver TileDB *not*
registered due to gdal_TileDB.so not being found
[Fri Sep 13 18:44:12 2024].8555, 1.5452: GDAL: Proxy driver TileDB *not*
registered due to gdal_TileDB.so not being found
[Fri Sep 13 18:44:13 2024].9017, 2.5915: HTTP: libcurl/8.9.0 OpenSSL/3.3.1
zlib/1.3.1 zstd/1.5.6 libssh2/1.11.0 nghttp2/1.58.0
[Fri Sep 13 18:44:14 2024].2537, 2.9434: S3: Switching to endpoint
s3.eu-central-1.amazonaws.com
[Fri Sep 13 18:44:19 2024].3471, 8.0369: S3: Switching to region
eu-central-1
[Fri Sep 13 18:44:19 2024].4582, 8.1480: S3: GetFileSize(
https://prod-pix4d-cloud-eu-central-1.s3.eu-central-1.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff)=66132291
 response_code=206
[Fri Sep 13 18:44:19 2024].4587, 8.1485: GDAL:
GDALOpen(/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff,
this=0x56258e57eef0) succeeds as GTiff.
[Fri Sep 13 18:44:19 2024].4593, 8.1490: GTiff: ScanDirectories()
[Fri Sep 13 18:44:19 2024].4596, 8.1494: GTiff: Opened band mask.

Another question that pops up is... why is it loading TileDB there? it is
not doing it if I just open QGIS. Is it needed for the COG?

If I open the file David suggested
AWS_NO_SIGN_REQUEST=YES CPL_DEBUG=ON CPL_TIMESTAMP=ON qgis
/vsis3/asc-pds-services/mosaic/Mars/Mars_MRO_CTX_Equi_Mosaics_Robbins/6mpp/MC16_6mpp.16bit.tif
It does not mention TileDB in the logs. Neither if I load the same COG file
from the hard drive.

If TileDB is loaded dynamically and it is the reason of the delay (that I
am not sure), is there a way to disable that load? Kind of a config
environment variable. I do not want to move the library every time (and it
checks for the library 3 times, that is a delay of 3 seconds every time, as
you can see in the logs above)

PS My connection is 250 Mbps. I don't think it is a "slow" connection


On Fri, 13 Sept 2024 at 18:30, thomas bonfort <thomas.bonfort at gmail.com>
wrote:

> I also see that you have a tiledb driver registration happening only on
> your conda version, exactly at the time of your lag. you should also try to
> remove/rename your gdal_tiledb.so plugin file to check the issue isn't
> coming from there.
>
> Le ven. 13 sept. 2024, 18:22, thomas bonfort <thomas.bonfort at gmail.com> a
> écrit :
>
>> Javier,
>> 3.9 is probably doing parallel reads, which will cause a noticeable lag
>> in requests on a slow connection (the overall processing time should not be
>> slower anyhow). There's a gtiff open option documented that can disable
>> this behavior, that you can try to use to see if it changes your timings.
>> If that is the effective cause, that would mean there is an issue with this
>> parallel implementation.
>> TB
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240913/97f35e8c/attachment.htm>