[gdal-dev] [EXTERNAL] s3 request example
thomas bonfort
thomas.bonfort at gmail.com
Fri Sep 13 09:30:31 PDT 2024
I also see that you have a tiledb driver registration happening only on
your conda version, exactly at the time of your lag. you should also try to
remove/rename your gdal_tiledb.so plugin file to check the issue isn't
coming from there.
Le ven. 13 sept. 2024, 18:22, thomas bonfort <thomas.bonfort at gmail.com> a
écrit :
> Javier,
> 3.9 is probably doing parallel reads, which will cause a noticeable lag in
> requests on a slow connection (the overall processing time should not be
> slower anyhow). There's a gtiff open option documented that can disable
> this behavior, that you can try to use to see if it changes your timings.
> If that is the effective cause, that would mean there is an issue with this
> parallel implementation.
> TB
>
> Le ven. 13 sept. 2024, 18:15, Javier Jimenez Shaw via gdal-dev <
> gdal-dev at lists.osgeo.org> a écrit :
>
>> Thanks David for the suggestion. Very interesting data.
>>
>> Unfortunately the issue I have is not reproduced with that file. So it
>> may be due to the authentication (your suggestion does not need
>> authentication).
>>
>> Some context: I am opening a COG file from S3 in QGIS, setting the url in
>> the command line. Something like
>> qgis /vsis3/blahblabla
>>
>> (more on that later)
>>
>> If I run it in my Ubuntu 22.04 (that has GDAL 3.4.1), with QGIS 3.38, no
>> problem
>> If I run with conda with QGIS 3.38 (that has GDAL 3.9.1), it is
>> noticeable slower.
>>
>> So I think it is due to the version of GDAL. If you think that I am wrong
>> and I should ask directly to QGIS, it is ok.
>>
>> This is a bit problematic to reproduce because the authorization expires
>> in 12 hours. So I post here the command I produced with a python script. If
>> it is needed, I can share the python script that is getting the
>> authentication. It is about 100 lines of code.
>>
>> Enabling the logs I see there a gap of 6 seconds during the opening with
>> conda, that is not there with the QGIS from Ubuntu that uses GDAL 3.4.1
>>
>> This is the log from conda
>> The gap is between
>> Fri Sep 13 17:56:26 2024].9370
>> and
>> [Fri Sep 13 17:56:33 2024].4445
>>
>> CPL_DEBUG=ON CPL_TIMESTAMP=ON python3 pix4d_qgis.py --project-id 1969386
>> --share-token 09c3eada-1b3b-46eb-b5de-8bd51635a5b9
>> Session expires at 2024-09-14T03:56:22Z
>> AWS_ACCESS_KEY_ID=ASIATOCJLBKS2RHGCAHB
>> AWS_SECRET_ACCESS_KEY=nNT8fTH/cGiyn4NVubp5/1LroyZ69johhXYxkCK1
>> AWS_SESSION_TOKEN=FwoGZXIvYXdzEKn//////////wEaDI1G/49ov88aGibrOCLXAilDbBuFQ/ydJF4AZ50/lYmWYmNDjM9H7XbbioyXiLQzwTqVRN1TLG6T29Anklkw47MKxdKHDF/O85r9gWJmCAqYTKalA38VM1qbixsjSXAzR7W5p1v1pgRUKtN4J03NrZOH2S6gbo4OKv+UUkPe/Xq4s8w0vEVNJLhvVx6dvpJMv7Jp9UxvLA5a+nrBKUqJujujqJnRXAKKfnpJbTDvhCuIeTxaFFt50e5UpLjSwqXS/a/AaEMJXRSsSLbKFTW/ALSCHfPv3HCStnue5CFpzVk4OhQbQ+dZyXUQp5wxQTtpr0hIh0IyB20gsoZNa0lGvFuXnzqDCH2X2mXwzDcyZk2IOX34nq+Wtnk8qYdjJxEAG0tnlYsMjmmXECnbOxl6gtNJ5hoehgak6cSfs7/rMi4jOI838tzsWWyDJeACj9qfQyfGtpk23x8Zt6KuqUjNZ1/z+/T8LwoopsGRtwYyKQbJS8pn8IGKPLtHDVxRx09yBRuqgYVFzSqpnr3T1vwWYHXiEF0sVJBx
>> GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR
>> /home/jshaw/anaconda3/envs/qgis_stable/bin/qgis
>> /vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff
>> [Fri Sep 13 17:56:26 2024].3948, 0.0000: HTTP: libcurl/8.9.0
>> OpenSSL/3.3.1 zlib/1.3.1 zstd/1.5.6 libssh2/1.11.0 nghttp2/1.58.0
>> [Fri Sep 13 17:56:26 2024].7731, 0.3783: S3: Switching to endpoint
>> s3.eu-central-1.amazonaws.com
>> [Fri Sep 13 17:56:26 2024].8415, 0.4467: S3: Switching to region
>> eu-central-1
>> [Fri Sep 13 17:56:26 2024].9230, 0.5282: S3: GetFileSize(
>> https://prod-pix4d-cloud-eu-central-1.s3.eu-central-1.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff)=66132291
>> response_code=206
>> [Fri Sep 13 17:56:26 2024].9235, 0.5288: GDAL:
>> GDALOpen(/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff,
>> this=0x556e6bb915a0) succeeds as GTiff.
>> [Fri Sep 13 17:56:26 2024].9240, 0.5292: GTiff: ScanDirectories()
>> [Fri Sep 13 17:56:26 2024].9243, 0.5296: GTiff: Opened band mask.
>> [Fri Sep 13 17:56:26 2024].9246, 0.5298: GTiff: Opened 8929x7207 overview.
>> [Fri Sep 13 17:56:26 2024].9249, 0.5301: GTiff: Opened 4464x3603 overview.
>> [Fri Sep 13 17:56:26 2024].9251, 0.5304: GTiff: Opened 2232x1801 overview.
>> [Fri Sep 13 17:56:26 2024].9254, 0.5306: GTiff: Opened 1116x900 overview.
>> [Fri Sep 13 17:56:26 2024].9257, 0.5309: GTiff: Opened 558x450 overview.
>> [Fri Sep 13 17:56:26 2024].9259, 0.5311: GTiff: Opened 279x225 overview.
>> [Fri Sep 13 17:56:26 2024].9262, 0.5314: GTiff: Opened band mask for
>> 8929x7207 overview.
>> [Fri Sep 13 17:56:26 2024].9264, 0.5316: GTiff: Opened band mask for
>> 4464x3603 overview.
>> [Fri Sep 13 17:56:26 2024].9267, 0.5319: GTiff: Opened band mask for
>> 2232x1801 overview.
>> [Fri Sep 13 17:56:26 2024].9269, 0.5321: GTiff: Opened band mask for
>> 1116x900 overview.
>> [Fri Sep 13 17:56:26 2024].9272, 0.5324: GTiff: Opened band mask for
>> 558x450 overview.
>> [Fri Sep 13 17:56:26 2024].9274, 0.5326: GTiff: Opened band mask for
>> 279x225 overview.
>> [Fri Sep 13 17:56:26 2024].9277, 0.5329: GDAL:
>> GDALClose(/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff,
>> this=0x556e6bb915a0)
>> ERROR: Status 2: File
>> /vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff
>> could not be found
>> [Fri Sep 13 17:56:26 2024].9370, 0.5422: GDAL: On-demand registering
>> /home/jshaw/anaconda3/envs/qgis_stable/lib/gdalplugins/gdal_TileDB.so using
>> GDALRegister_TileDB.
>> [Fri Sep 13 17:56:33 2024].4445, 7.0498: S3: GetFileSize(
>> https://prod-pix4d-cloud-eu-central-1.s3.eu-central-1.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff)=66132291
>> response_code=206
>> [Fri Sep 13 17:56:33 2024].4449, 7.0501: GDAL:
>> GDALOpen(/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff,
>> this=0x556e6b807d00) succeeds as GTiff.
>> GDAL ERROR 0: [Fri Sep 13 17:56:33 2024].4666, 7.0718: OGRCT: Using
>> WebMercator to WGS84 optimization
>> GDAL ERROR 0: [Fri Sep 13 17:56:33 2024].4667, 7.0719: GDAL: Computing
>> area of interest: 6.62976, 46.6391, 6.63534, 46.6422
>> [Fri Sep 13 17:56:33 2024].4667, 7.0719: GTiff: ScanDirectories()
>> [Fri Sep 13 17:56:33 2024].4669, 7.0721: GTiff: Opened band mask.
>> [Fri Sep 13 17:56:33 2024].4670, 7.0722: GTiff: Opened 8929x7207 overview.
>> [Fri Sep 13 17:56:33 2024].4671, 7.0723: GTiff: Opened 4464x3603 overview.
>> [Fri Sep 13 17:56:33 2024].4672, 7.0724: GTiff: Opened 2232x1801 overview.
>> [Fri Sep 13 17:56:33 2024].4673, 7.0725: GTiff: Opened 1116x900 overview.
>> [Fri Sep 13 17:56:33 2024].4674, 7.0726: GTiff: Opened 558x450 overview.
>> [Fri Sep 13 17:56:33 2024].4675, 7.0727: GTiff: Opened 279x225 overview.
>> [Fri Sep 13 17:56:33 2024].4676, 7.0728: GTiff: Opened band mask for
>> 8929x7207 overview.
>> [Fri Sep 13 17:56:33 2024].4677, 7.0729: GTiff: Opened band mask for
>> 4464x3603 overview.
>> [Fri Sep 13 17:56:33 2024].4677, 7.0730: GTiff: Opened band mask for
>> 2232x1801 overview.
>> [Fri Sep 13 17:56:33 2024].4678, 7.0731: GTiff: Opened band mask for
>> 1116x900 overview.
>> [Fri Sep 13 17:56:33 2024].4679, 7.0732: GTiff: Opened band mask for
>> 558x450 overview.
>> [Fri Sep 13 17:56:33 2024].4680, 7.0733: GTiff: Opened band mask for
>> 279x225 overview.
>> [Fri Sep 13 17:56:33 2024].5178, 7.1230: GDAL:
>> GDALOpen(/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff,
>> this=0x7f830c08d490) succeeds as GTiff.
>> [Fri Sep 13 17:56:33 2024].5180, 7.1232: S3: Downloading
>> 59179008-59244543 (
>> https://prod-pix4d-cloud-eu-central-1.s3.eu-central-1.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff).
>> ..
>> GDAL ERROR 0: [Fri Sep 13 17:56:33 2024].5244, 7.1296: GDAL: Computing
>> area of interest: 6.62976, 46.6391, 6.63534, 46.6422
>> [Fri Sep 13 17:56:33 2024].5245, 7.1297: GTiff: ScanDirectories()
>> [Fri Sep 13 17:56:33 2024].5246, 7.1298: GTiff: Opened band mask.
>> [Fri Sep 13 17:56:33 2024].5247, 7.1299: GTiff: Opened 8929x7207 overview.
>> [Fri Sep 13 17:56:33 2024].5247, 7.1300: GTiff: Opened 4464x3603 overview.
>> [Fri Sep 13 17:56:33 2024].5248, 7.1300: GTiff: Opened 2232x1801 overview.
>> [Fri Sep 13 17:56:33 2024].5249, 7.1301: GTiff: Opened 1116x900 overview.
>> [Fri Sep 13 17:56:33 2024].5249, 7.1302: GTiff: Opened 558x450 overview.
>> [Fri Sep 13 17:56:33 2024].5250, 7.1302: GTiff: Opened 279x225 overview.
>> [Fri Sep 13 17:56:33 2024].5251, 7.1303: GTiff: Opened band mask for
>> 8929x7207 overview.
>> [Fri Sep 13 17:56:33 2024].5251, 7.1304: GTiff: Opened band mask for
>> 4464x3603 overview.
>> [Fri Sep 13 17:56:33 2024].5252, 7.1304: GTiff: Opened band mask for
>> 2232x1801 overview.
>> [Fri Sep 13 17:56:33 2024].5253, 7.1305: GTiff: Opened band mask for
>> 1116x900 overview.
>> [Fri Sep 13 17:56:33 2024].5253, 7.1306: GTiff: Opened band mask for
>> 558x450 overview.
>> [Fri Sep 13 17:56:33 2024].5254, 7.1306: GTiff: Opened band mask for
>> 279x225 overview.
>> [Fri Sep 13 17:56:33 2024].5255, 7.1307: GDAL: Selecting overview 1116 x
>> 900
>> [Fri Sep 13 17:56:33 2024].5255, 7.1307: S3: Downloading 16384-32767 (
>> https://prod-pix4d-cloud-eu-central-1.s3.eu-central-1.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff).
>> ..
>> [Fri Sep 13 17:56:33 2024].6116, 7.2168: S3: Got response_code=206
>> [Fri Sep 13 17:56:33 2024].6118, 7.2171: S3: Downloading 180224-589823 (
>> https://prod-pix4d-cloud-eu-central-1.s3.eu-central-1.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff).
>> ..
>> [Fri Sep 13 17:56:33 2024].6955, 7.3008: S3: Got response_code=206
>> [Fri Sep 13 17:56:33 2024].6960, 7.3012: GDAL: GDAL_CACHEMAX = 1591 MB
>> [Fri Sep 13 17:56:33 2024].6977, 7.3029: S3: Got response_code=206
>>
>>
>> This is the equivalent code with QGIS 3.38 in Ubuntu and GDAL 3.4.1
>>
>> CPL_DEBUG=ON CPL_TIMESTAMP=ON python3 pix4d_qgis.py --project-id 1969386
>> --share-token 09c3eada-1b3b-46eb-b5de-8bd51635a5b9
>> Session expires at 2024-09-14T04:00:10Z
>> AWS_ACCESS_KEY_ID=ASIATOCJLBKSRCEUBKE6
>> AWS_SECRET_ACCESS_KEY=h6NhBf9VwBZoVj5Zd/8H2BnzXQu0l6bFUUv2D8E7
>> AWS_SESSION_TOKEN=FwoGZXIvYXdzEKn//////////wEaDF1xO50XL6GBe/358CLXAn3JR/uteJtarwJ/ryn6i4+lcMFEQZ/KV0FZQdKsD1PwAcqoYFY+0GKYSV+FTk81ZszsVzy8N7d2GwByvP9S/L1XOo1FrqJfDpGw3a5eQR7AoeU68EPlD/WTYnM7XBzXimHywDCyue3SOi8oZaSdbMgjBQ1N6j+qQ+EDN7RjD/Zu2CGMTzN9g/Z0FhZsuyt2HRhnOYpsvpUMZ+WQr7pbjdVdwt3PdFHsJ95fhueapJspgQ7OSgmj5VtSR8NTecohLsd2eeGkSpiqVWuBwuAu2cW6LcqmAZGnF0DZIjPWiMIDf/OLczMRCw5MJGeYpsSkkiuM+t9Miv8aN4iCo8vjl6wumOCgPqTgBBWlQdAs2mLxwpF1Bb1UjJ6eiccp17+yJF3ffn8tvxx/+1V3Q++E5Xk+jiB1keJnFcizr7qD6EY+EdDES5sQrWAhM2uR78JjNc0f+hVZEmAoisORtwYyKb0po07ytiiBv+X3j+Y6BsCHR0Es3C3xVvsgkfn1zb1Z2flFWUzwqJUD
>> GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR /usr/bin/qgis
>> /vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff
>> [Fri Sep 13 18:00:14 2024].7665, 0.0000: HTTP: libcurl/7.81.0
>> OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 libidn2/2.3.2
>> libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib nghttp2/1.43.0
>> librtmp/2.3 OpenLDAP/2.5.18
>> [Fri Sep 13 18:00:14 2024].7666, 0.0001: S3: Downloading 0-16383 (
>> https://prod-pix4d-cloud-eu-central-1.s3.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff).
>> ..
>> [Fri Sep 13 18:00:14 2024].8897, 0.1231: S3: Got response_code=400
>> [Fri Sep 13 18:00:14 2024].8898, 0.1232: S3: Switching to region
>> eu-central-1
>> [Fri Sep 13 18:00:14 2024].8899, 0.1234: S3: Downloading 0-16383 (
>> https://prod-pix4d-cloud-eu-central-1.s3.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff).
>> ..
>> [Fri Sep 13 18:00:15 2024].0287, 0.2621: S3: Got response_code=206
>> [Fri Sep 13 18:00:15 2024].0288, 0.2622: S3: GetFileSize(
>> https://prod-pix4d-cloud-eu-central-1.s3.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff)=66132291
>> response_code=206
>> [Fri Sep 13 18:00:15 2024].0292, 0.2627: GDAL:
>> GDALOpen(/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff,
>> this=0x560e4cd84920) succeeds as GTiff.
>> [Fri Sep 13 18:00:15 2024].0298, 0.2632: GTiff: ScanDirectories()
>> [Fri Sep 13 18:00:15 2024].0301, 0.2635: GTiff: Opened band mask.
>> [Fri Sep 13 18:00:15 2024].0304, 0.2638: GTiff: Opened 8929x7207 overview.
>> [Fri Sep 13 18:00:15 2024].0306, 0.2641: GTiff: Opened 4464x3603 overview.
>> [Fri Sep 13 18:00:15 2024].0309, 0.2643: GTiff: Opened 2232x1801 overview.
>> [Fri Sep 13 18:00:15 2024].0311, 0.2646: GTiff: Opened 1116x900 overview.
>> [Fri Sep 13 18:00:15 2024].0314, 0.2648: GTiff: Opened 558x450 overview.
>> [Fri Sep 13 18:00:15 2024].0316, 0.2651: GTiff: Opened 279x225 overview.
>> [Fri Sep 13 18:00:15 2024].0319, 0.2654: GTiff: Opened band mask for
>> 8929x7207 overview.
>> [Fri Sep 13 18:00:15 2024].0322, 0.2656: GTiff: Opened band mask for
>> 4464x3603 overview.
>> [Fri Sep 13 18:00:15 2024].0324, 0.2658: GTiff: Opened band mask for
>> 2232x1801 overview.
>> [Fri Sep 13 18:00:15 2024].0326, 0.2661: GTiff: Opened band mask for
>> 1116x900 overview.
>> [Fri Sep 13 18:00:15 2024].0329, 0.2663: GTiff: Opened band mask for
>> 558x450 overview.
>> [Fri Sep 13 18:00:15 2024].0331, 0.2665: GTiff: Opened band mask for
>> 279x225 overview.
>> [Fri Sep 13 18:00:15 2024].0334, 0.2669: GDAL:
>> GDALClose(/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff,
>> this=0x560e4cd84920)
>> ERROR: Status 2: File
>> /vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff
>> could not be found
>> [Fri Sep 13 18:00:15 2024].0357, 0.2691: S3: Downloading 0-16383 (
>> https://prod-pix4d-cloud-eu-central-1.s3.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff).
>> ..
>> [Fri Sep 13 18:00:15 2024].0762, 0.3096: S3: Got response_code=206
>> [Fri Sep 13 18:00:15 2024].0762, 0.3097: S3: GetFileSize(
>> https://prod-pix4d-cloud-eu-central-1.s3.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff)=66132291
>> response_code=206
>> [Fri Sep 13 18:00:15 2024].0765, 0.3099: GDAL:
>> GDALOpen(/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff,
>> this=0x560e4d0afb70) succeeds as GTiff.
>> GDAL ERROR 0: [Fri Sep 13 18:00:15 2024].0956, 0.3291: GDAL: Computing
>> area of interest: 6.62976, 46.6391, 6.63534, 46.6422
>> [Fri Sep 13 18:00:15 2024].0958, 0.3293: GTiff: ScanDirectories()
>> [Fri Sep 13 18:00:15 2024].0959, 0.3294: GTiff: Opened band mask.
>> [Fri Sep 13 18:00:15 2024].0961, 0.3295: GTiff: Opened 8929x7207 overview.
>> [Fri Sep 13 18:00:15 2024].0962, 0.3296: GTiff: Opened 4464x3603 overview.
>> [Fri Sep 13 18:00:15 2024].0962, 0.3297: GTiff: Opened 2232x1801 overview.
>> [Fri Sep 13 18:00:15 2024].0963, 0.3298: GTiff: Opened 1116x900 overview.
>> [Fri Sep 13 18:00:15 2024].0964, 0.3299: GTiff: Opened 558x450 overview.
>> [Fri Sep 13 18:00:15 2024].0965, 0.3300: GTiff: Opened 279x225 overview.
>> [Fri Sep 13 18:00:15 2024].0966, 0.3301: GTiff: Opened band mask for
>> 8929x7207 overview.
>> [Fri Sep 13 18:00:15 2024].0967, 0.3302: GTiff: Opened band mask for
>> 4464x3603 overview.
>> [Fri Sep 13 18:00:15 2024].0968, 0.3303: GTiff: Opened band mask for
>> 2232x1801 overview.
>> [Fri Sep 13 18:00:15 2024].0969, 0.3303: GTiff: Opened band mask for
>> 1116x900 overview.
>> [Fri Sep 13 18:00:15 2024].0970, 0.3304: GTiff: Opened band mask for
>> 558x450 overview.
>> [Fri Sep 13 18:00:15 2024].0971, 0.3305: GTiff: Opened band mask for
>> 279x225 overview.
>> [Fri Sep 13 18:00:15 2024].2011, 0.4346: GDAL:
>> GDALOpen(/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff,
>> this=0x7f0dfc08dfd0) succeeds as GTiff.
>> GDAL ERROR 0: [Fri Sep 13 18:00:15 2024].2120, 0.4454: GDAL: Computing
>> area of interest: 6.62976, 46.6391, 6.63534, 46.6422
>> [Fri Sep 13 18:00:15 2024].2121, 0.4456: GTiff: ScanDirectories()
>> [Fri Sep 13 18:00:15 2024].2122, 0.4457: GTiff: Opened band mask.
>> [Fri Sep 13 18:00:15 2024].2123, 0.4457: GTiff: Opened 8929x7207 overview.
>> [Fri Sep 13 18:00:15 2024].2123, 0.4458: GTiff: Opened 4464x3603 overview.
>> [Fri Sep 13 18:00:15 2024].2124, 0.4459: GTiff: Opened 2232x1801 overview.
>> [Fri Sep 13 18:00:15 2024].2125, 0.4459: GTiff: Opened 1116x900 overview.
>> [Fri Sep 13 18:00:15 2024].2125, 0.4460: GTiff: Opened 558x450 overview.
>> [Fri Sep 13 18:00:15 2024].2126, 0.4460: GTiff: Opened 279x225 overview.
>> [Fri Sep 13 18:00:15 2024].2126, 0.4461: GTiff: Opened band mask for
>> 8929x7207 overview.
>> [Fri Sep 13 18:00:15 2024].2127, 0.4462: GTiff: Opened band mask for
>> 4464x3603 overview.
>> [Fri Sep 13 18:00:15 2024].2128, 0.4462: GTiff: Opened band mask for
>> 2232x1801 overview.
>> [Fri Sep 13 18:00:15 2024].2128, 0.4463: GTiff: Opened band mask for
>> 1116x900 overview.
>> [Fri Sep 13 18:00:15 2024].2129, 0.4463: GTiff: Opened band mask for
>> 558x450 overview.
>> [Fri Sep 13 18:00:15 2024].2129, 0.4464: GTiff: Opened band mask for
>> 279x225 overview.
>> [Fri Sep 13 18:00:15 2024].2130, 0.4465: S3: Downloading 16384-32767 (
>> https://prod-pix4d-cloud-eu-central-1.s3.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff).
>> ..
>> [Fri Sep 13 18:00:15 2024].3454, 0.5789: S3: Got response_code=206
>> [Fri Sep 13 18:00:15 2024].3456, 0.5791: S3: Downloading 180224-589823 (
>> https://prod-pix4d-cloud-eu-central-1.s3.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff).
>> ..
>> [Fri Sep 13 18:00:15 2024].4272, 0.6606: S3: Got response_code=206
>> [Fri Sep 13 18:00:15 2024].4276, 0.6610: GDAL: GDAL_CACHEMAX = 1591 MB
>>
>> What can it be?
>>
>> BTW, in both cases it says
>> ERROR: Status 2: File
>> /vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff
>> could not be found
>> And later it opens it. I don't know why.
>>
>> Thank you very much
>> Javier.
>>
>> On Mon, 9 Sept 2024 at 23:49, Mayer, David P <dpmayer at usgs.gov> wrote:
>>
>>> Would ~200 GB be large enough?
>>>
>>> Here is a mosaic of a section of Mars at 6 meters/pixel in a public S3
>>> bucket:
>>>
>>> s3://asc-pds-services/mosaic/Mars/Mars_MRO_CTX_Equi_Mosaics_Robbins/6mpp/MC16_6mpp.16bit.tif
>>>
>>> Cheers,
>>> David
>>>
>>> -----------
>>> David Mayer
>>> Geographer
>>> Astrogeology Science Center
>>> US Geological Survey
>>> dpmayer at usgs.gov
>>> ------------------------------
>>> *From:* gdal-dev <gdal-dev-bounces at lists.osgeo.org> on behalf of Javier
>>> Jimenez Shaw via gdal-dev <gdal-dev at lists.osgeo.org>
>>> *Sent:* Monday, September 9, 2024 13:55
>>> *To:* gdal dev <gdal-dev at lists.osgeo.org>
>>> *Subject:* [EXTERNAL] [gdal-dev] s3 request example
>>>
>>>
>>>
>>>
>>> * This email has been received from outside of DOI - Use caution before
>>> clicking on links, opening attachments, or responding. *
>>>
>>>
>>>
>>> Hi
>>>
>>> I am experiencing a much slower load of a remote COG using vsis3 with
>>> the QGIS from conda than the QGIS in my Ubuntu (that uses GDAL 3.4.1).
>>>
>>> Because both are QGIS 3.38, I think the problem is in the version of
>>> GDAL.
>>> I would like to research it more.
>>>
>>> Is there any example of a (big) COG file in Amazon S3 I can use to
>>> compare them? Unfortunately the credentials for the files I am using expire
>>> after a few hours, so they are useless to report anything.
>>>
>>> Best regards,
>>> Javier
>>>
>> _______________________________________________
>> gdal-dev mailing list
>> gdal-dev at lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240913/1692395a/attachment-0001.htm>
More information about the gdal-dev
mailing list