[gdal-dev] [EXTERNAL] s3 request example

Fri Sep 13 09:22:10 PDT 2024

Javier,
3.9 is probably doing parallel reads, which will cause a noticeable lag in
requests on a slow connection (the overall processing time should not be
slower anyhow). There's a gtiff open option documented that can disable
this behavior, that you can try to use to see if it changes your timings.
If that is the effective cause, that would mean there is an issue with this
parallel implementation.
TB

Le ven. 13 sept. 2024, 18:15, Javier Jimenez Shaw via gdal-dev <
gdal-dev at lists.osgeo.org> a écrit :

> Thanks David for the suggestion. Very interesting data.
>
> Unfortunately the issue I have is not reproduced with that file. So it may
> be due to the authentication (your suggestion does not need authentication).
>
> Some context: I am opening a COG file from S3 in QGIS, setting the url in
> the command line. Something like
> qgis /vsis3/blahblabla
>
> (more on that later)
>
> If I run it in my Ubuntu 22.04 (that has GDAL 3.4.1), with QGIS 3.38, no
> problem
> If I run with conda with QGIS 3.38 (that has GDAL 3.9.1), it is noticeable
> slower.
>
> So I think it is due to the version of GDAL. If you think that I am wrong
> and I should ask directly to QGIS, it is ok.
>
> This is a bit problematic to reproduce because the authorization expires
> in 12 hours. So I post here the command I produced with a python script. If
> it is needed, I can share the python script that is  getting the
> authentication. It is about 100 lines of code.
>
> Enabling the logs I see there a gap of 6 seconds during the opening with
> conda, that is not there with the QGIS from Ubuntu that uses GDAL 3.4.1
>
> This is the log from conda
> The gap is between
> Fri Sep 13 17:56:26 2024].9370
> and
> [Fri Sep 13 17:56:33 2024].4445
>
> CPL_DEBUG=ON CPL_TIMESTAMP=ON python3 pix4d_qgis.py --project-id 1969386
> --share-token 09c3eada-1b3b-46eb-b5de-8bd51635a5b9
> Session expires at 2024-09-14T03:56:22Z
> AWS_ACCESS_KEY_ID=ASIATOCJLBKS2RHGCAHB
> AWS_SECRET_ACCESS_KEY=nNT8fTH/cGiyn4NVubp5/1LroyZ69johhXYxkCK1
> AWS_SESSION_TOKEN=FwoGZXIvYXdzEKn//////////wEaDI1G/49ov88aGibrOCLXAilDbBuFQ/ydJF4AZ50/lYmWYmNDjM9H7XbbioyXiLQzwTqVRN1TLG6T29Anklkw47MKxdKHDF/O85r9gWJmCAqYTKalA38VM1qbixsjSXAzR7W5p1v1pgRUKtN4J03NrZOH2S6gbo4OKv+UUkPe/Xq4s8w0vEVNJLhvVx6dvpJMv7Jp9UxvLA5a+nrBKUqJujujqJnRXAKKfnpJbTDvhCuIeTxaFFt50e5UpLjSwqXS/a/AaEMJXRSsSLbKFTW/ALSCHfPv3HCStnue5CFpzVk4OhQbQ+dZyXUQp5wxQTtpr0hIh0IyB20gsoZNa0lGvFuXnzqDCH2X2mXwzDcyZk2IOX34nq+Wtnk8qYdjJxEAG0tnlYsMjmmXECnbOxl6gtNJ5hoehgak6cSfs7/rMi4jOI838tzsWWyDJeACj9qfQyfGtpk23x8Zt6KuqUjNZ1/z+/T8LwoopsGRtwYyKQbJS8pn8IGKPLtHDVxRx09yBRuqgYVFzSqpnr3T1vwWYHXiEF0sVJBx
> GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR
> /home/jshaw/anaconda3/envs/qgis_stable/bin/qgis
> /vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff
> [Fri Sep 13 17:56:26 2024].3948, 0.0000: HTTP: libcurl/8.9.0 OpenSSL/3.3.1
> zlib/1.3.1 zstd/1.5.6 libssh2/1.11.0 nghttp2/1.58.0
> [Fri Sep 13 17:56:26 2024].7731, 0.3783: S3: Switching to endpoint
> s3.eu-central-1.amazonaws.com
> [Fri Sep 13 17:56:26 2024].8415, 0.4467: S3: Switching to region
> eu-central-1
> [Fri Sep 13 17:56:26 2024].9230, 0.5282: S3: GetFileSize(
> https://prod-pix4d-cloud-eu-central-1.s3.eu-central-1.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff)=66132291
>  response_code=206
> [Fri Sep 13 17:56:26 2024].9235, 0.5288: GDAL:
> GDALOpen(/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff,
> this=0x556e6bb915a0) succeeds as GTiff.
> [Fri Sep 13 17:56:26 2024].9240, 0.5292: GTiff: ScanDirectories()
> [Fri Sep 13 17:56:26 2024].9243, 0.5296: GTiff: Opened band mask.
> [Fri Sep 13 17:56:26 2024].9246, 0.5298: GTiff: Opened 8929x7207 overview.
> [Fri Sep 13 17:56:26 2024].9249, 0.5301: GTiff: Opened 4464x3603 overview.
> [Fri Sep 13 17:56:26 2024].9251, 0.5304: GTiff: Opened 2232x1801 overview.
> [Fri Sep 13 17:56:26 2024].9254, 0.5306: GTiff: Opened 1116x900 overview.
> [Fri Sep 13 17:56:26 2024].9257, 0.5309: GTiff: Opened 558x450 overview.
> [Fri Sep 13 17:56:26 2024].9259, 0.5311: GTiff: Opened 279x225 overview.
> [Fri Sep 13 17:56:26 2024].9262, 0.5314: GTiff: Opened band mask for
> 8929x7207 overview.
> [Fri Sep 13 17:56:26 2024].9264, 0.5316: GTiff: Opened band mask for
> 4464x3603 overview.
> [Fri Sep 13 17:56:26 2024].9267, 0.5319: GTiff: Opened band mask for
> 2232x1801 overview.
> [Fri Sep 13 17:56:26 2024].9269, 0.5321: GTiff: Opened band mask for
> 1116x900 overview.
> [Fri Sep 13 17:56:26 2024].9272, 0.5324: GTiff: Opened band mask for
> 558x450 overview.
> [Fri Sep 13 17:56:26 2024].9274, 0.5326: GTiff: Opened band mask for
> 279x225 overview.
> [Fri Sep 13 17:56:26 2024].9277, 0.5329: GDAL:
> GDALClose(/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff,
> this=0x556e6bb915a0)
> ERROR: Status 2: File
> /vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff
> could not be found
> [Fri Sep 13 17:56:26 2024].9370, 0.5422: GDAL: On-demand registering
> /home/jshaw/anaconda3/envs/qgis_stable/lib/gdalplugins/gdal_TileDB.so using
> GDALRegister_TileDB.
> [Fri Sep 13 17:56:33 2024].4445, 7.0498: S3: GetFileSize(
> https://prod-pix4d-cloud-eu-central-1.s3.eu-central-1.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff)=66132291
>  response_code=206
> [Fri Sep 13 17:56:33 2024].4449, 7.0501: GDAL:
> GDALOpen(/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff,
> this=0x556e6b807d00) succeeds as GTiff.
> GDAL ERROR 0: [Fri Sep 13 17:56:33 2024].4666, 7.0718: OGRCT: Using
> WebMercator to WGS84 optimization
> GDAL ERROR 0: [Fri Sep 13 17:56:33 2024].4667, 7.0719: GDAL: Computing
> area of interest: 6.62976, 46.6391, 6.63534, 46.6422
> [Fri Sep 13 17:56:33 2024].4667, 7.0719: GTiff: ScanDirectories()
> [Fri Sep 13 17:56:33 2024].4669, 7.0721: GTiff: Opened band mask.
> [Fri Sep 13 17:56:33 2024].4670, 7.0722: GTiff: Opened 8929x7207 overview.
> [Fri Sep 13 17:56:33 2024].4671, 7.0723: GTiff: Opened 4464x3603 overview.
> [Fri Sep 13 17:56:33 2024].4672, 7.0724: GTiff: Opened 2232x1801 overview.
> [Fri Sep 13 17:56:33 2024].4673, 7.0725: GTiff: Opened 1116x900 overview.
> [Fri Sep 13 17:56:33 2024].4674, 7.0726: GTiff: Opened 558x450 overview.
> [Fri Sep 13 17:56:33 2024].4675, 7.0727: GTiff: Opened 279x225 overview.
> [Fri Sep 13 17:56:33 2024].4676, 7.0728: GTiff: Opened band mask for
> 8929x7207 overview.
> [Fri Sep 13 17:56:33 2024].4677, 7.0729: GTiff: Opened band mask for
> 4464x3603 overview.
> [Fri Sep 13 17:56:33 2024].4677, 7.0730: GTiff: Opened band mask for
> 2232x1801 overview.
> [Fri Sep 13 17:56:33 2024].4678, 7.0731: GTiff: Opened band mask for
> 1116x900 overview.
> [Fri Sep 13 17:56:33 2024].4679, 7.0732: GTiff: Opened band mask for
> 558x450 overview.
> [Fri Sep 13 17:56:33 2024].4680, 7.0733: GTiff: Opened band mask for
> 279x225 overview.
> [Fri Sep 13 17:56:33 2024].5178, 7.1230: GDAL:
> GDALOpen(/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff,
> this=0x7f830c08d490) succeeds as GTiff.
> [Fri Sep 13 17:56:33 2024].5180, 7.1232: S3: Downloading 59179008-59244543
> (
> https://prod-pix4d-cloud-eu-central-1.s3.eu-central-1.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff).
> ..
> GDAL ERROR 0: [Fri Sep 13 17:56:33 2024].5244, 7.1296: GDAL: Computing
> area of interest: 6.62976, 46.6391, 6.63534, 46.6422
> [Fri Sep 13 17:56:33 2024].5245, 7.1297: GTiff: ScanDirectories()
> [Fri Sep 13 17:56:33 2024].5246, 7.1298: GTiff: Opened band mask.
> [Fri Sep 13 17:56:33 2024].5247, 7.1299: GTiff: Opened 8929x7207 overview.
> [Fri Sep 13 17:56:33 2024].5247, 7.1300: GTiff: Opened 4464x3603 overview.
> [Fri Sep 13 17:56:33 2024].5248, 7.1300: GTiff: Opened 2232x1801 overview.
> [Fri Sep 13 17:56:33 2024].5249, 7.1301: GTiff: Opened 1116x900 overview.
> [Fri Sep 13 17:56:33 2024].5249, 7.1302: GTiff: Opened 558x450 overview.
> [Fri Sep 13 17:56:33 2024].5250, 7.1302: GTiff: Opened 279x225 overview.
> [Fri Sep 13 17:56:33 2024].5251, 7.1303: GTiff: Opened band mask for
> 8929x7207 overview.
> [Fri Sep 13 17:56:33 2024].5251, 7.1304: GTiff: Opened band mask for
> 4464x3603 overview.
> [Fri Sep 13 17:56:33 2024].5252, 7.1304: GTiff: Opened band mask for
> 2232x1801 overview.
> [Fri Sep 13 17:56:33 2024].5253, 7.1305: GTiff: Opened band mask for
> 1116x900 overview.
> [Fri Sep 13 17:56:33 2024].5253, 7.1306: GTiff: Opened band mask for
> 558x450 overview.
> [Fri Sep 13 17:56:33 2024].5254, 7.1306: GTiff: Opened band mask for
> 279x225 overview.
> [Fri Sep 13 17:56:33 2024].5255, 7.1307: GDAL: Selecting overview 1116 x
> 900
> [Fri Sep 13 17:56:33 2024].5255, 7.1307: S3: Downloading 16384-32767 (
> https://prod-pix4d-cloud-eu-central-1.s3.eu-central-1.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff).
> ..
> [Fri Sep 13 17:56:33 2024].6116, 7.2168: S3: Got response_code=206
> [Fri Sep 13 17:56:33 2024].6118, 7.2171: S3: Downloading 180224-589823 (
> https://prod-pix4d-cloud-eu-central-1.s3.eu-central-1.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff).
> ..
> [Fri Sep 13 17:56:33 2024].6955, 7.3008: S3: Got response_code=206
> [Fri Sep 13 17:56:33 2024].6960, 7.3012: GDAL: GDAL_CACHEMAX = 1591 MB
> [Fri Sep 13 17:56:33 2024].6977, 7.3029: S3: Got response_code=206
>
>
> This is the equivalent code with QGIS 3.38 in Ubuntu and GDAL 3.4.1
>
> CPL_DEBUG=ON CPL_TIMESTAMP=ON python3 pix4d_qgis.py --project-id 1969386
> --share-token 09c3eada-1b3b-46eb-b5de-8bd51635a5b9
> Session expires at 2024-09-14T04:00:10Z
> AWS_ACCESS_KEY_ID=ASIATOCJLBKSRCEUBKE6
> AWS_SECRET_ACCESS_KEY=h6NhBf9VwBZoVj5Zd/8H2BnzXQu0l6bFUUv2D8E7
> AWS_SESSION_TOKEN=FwoGZXIvYXdzEKn//////////wEaDF1xO50XL6GBe/358CLXAn3JR/uteJtarwJ/ryn6i4+lcMFEQZ/KV0FZQdKsD1PwAcqoYFY+0GKYSV+FTk81ZszsVzy8N7d2GwByvP9S/L1XOo1FrqJfDpGw3a5eQR7AoeU68EPlD/WTYnM7XBzXimHywDCyue3SOi8oZaSdbMgjBQ1N6j+qQ+EDN7RjD/Zu2CGMTzN9g/Z0FhZsuyt2HRhnOYpsvpUMZ+WQr7pbjdVdwt3PdFHsJ95fhueapJspgQ7OSgmj5VtSR8NTecohLsd2eeGkSpiqVWuBwuAu2cW6LcqmAZGnF0DZIjPWiMIDf/OLczMRCw5MJGeYpsSkkiuM+t9Miv8aN4iCo8vjl6wumOCgPqTgBBWlQdAs2mLxwpF1Bb1UjJ6eiccp17+yJF3ffn8tvxx/+1V3Q++E5Xk+jiB1keJnFcizr7qD6EY+EdDES5sQrWAhM2uR78JjNc0f+hVZEmAoisORtwYyKb0po07ytiiBv+X3j+Y6BsCHR0Es3C3xVvsgkfn1zb1Z2flFWUzwqJUD
> GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR /usr/bin/qgis
> /vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff
> [Fri Sep 13 18:00:14 2024].7665, 0.0000: HTTP: libcurl/7.81.0
> OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 libidn2/2.3.2
> libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib nghttp2/1.43.0
> librtmp/2.3 OpenLDAP/2.5.18
> [Fri Sep 13 18:00:14 2024].7666, 0.0001: S3: Downloading 0-16383 (
> https://prod-pix4d-cloud-eu-central-1.s3.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff).
> ..
> [Fri Sep 13 18:00:14 2024].8897, 0.1231: S3: Got response_code=400
> [Fri Sep 13 18:00:14 2024].8898, 0.1232: S3: Switching to region
> eu-central-1
> [Fri Sep 13 18:00:14 2024].8899, 0.1234: S3: Downloading 0-16383 (
> https://prod-pix4d-cloud-eu-central-1.s3.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff).
> ..
> [Fri Sep 13 18:00:15 2024].0287, 0.2621: S3: Got response_code=206
> [Fri Sep 13 18:00:15 2024].0288, 0.2622: S3: GetFileSize(
> https://prod-pix4d-cloud-eu-central-1.s3.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff)=66132291
>  response_code=206
> [Fri Sep 13 18:00:15 2024].0292, 0.2627: GDAL:
> GDALOpen(/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff,
> this=0x560e4cd84920) succeeds as GTiff.
> [Fri Sep 13 18:00:15 2024].0298, 0.2632: GTiff: ScanDirectories()
> [Fri Sep 13 18:00:15 2024].0301, 0.2635: GTiff: Opened band mask.
> [Fri Sep 13 18:00:15 2024].0304, 0.2638: GTiff: Opened 8929x7207 overview.
> [Fri Sep 13 18:00:15 2024].0306, 0.2641: GTiff: Opened 4464x3603 overview.
> [Fri Sep 13 18:00:15 2024].0309, 0.2643: GTiff: Opened 2232x1801 overview.
> [Fri Sep 13 18:00:15 2024].0311, 0.2646: GTiff: Opened 1116x900 overview.
> [Fri Sep 13 18:00:15 2024].0314, 0.2648: GTiff: Opened 558x450 overview.
> [Fri Sep 13 18:00:15 2024].0316, 0.2651: GTiff: Opened 279x225 overview.
> [Fri Sep 13 18:00:15 2024].0319, 0.2654: GTiff: Opened band mask for
> 8929x7207 overview.
> [Fri Sep 13 18:00:15 2024].0322, 0.2656: GTiff: Opened band mask for
> 4464x3603 overview.
> [Fri Sep 13 18:00:15 2024].0324, 0.2658: GTiff: Opened band mask for
> 2232x1801 overview.
> [Fri Sep 13 18:00:15 2024].0326, 0.2661: GTiff: Opened band mask for
> 1116x900 overview.
> [Fri Sep 13 18:00:15 2024].0329, 0.2663: GTiff: Opened band mask for
> 558x450 overview.
> [Fri Sep 13 18:00:15 2024].0331, 0.2665: GTiff: Opened band mask for
> 279x225 overview.
> [Fri Sep 13 18:00:15 2024].0334, 0.2669: GDAL:
> GDALClose(/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff,
> this=0x560e4cd84920)
> ERROR: Status 2: File
> /vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff
> could not be found
> [Fri Sep 13 18:00:15 2024].0357, 0.2691: S3: Downloading 0-16383 (
> https://prod-pix4d-cloud-eu-central-1.s3.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff).
> ..
> [Fri Sep 13 18:00:15 2024].0762, 0.3096: S3: Got response_code=206
> [Fri Sep 13 18:00:15 2024].0762, 0.3097: S3: GetFileSize(
> https://prod-pix4d-cloud-eu-central-1.s3.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff)=66132291
>  response_code=206
> [Fri Sep 13 18:00:15 2024].0765, 0.3099: GDAL:
> GDALOpen(/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff,
> this=0x560e4d0afb70) succeeds as GTiff.
> GDAL ERROR 0: [Fri Sep 13 18:00:15 2024].0956, 0.3291: GDAL: Computing
> area of interest: 6.62976, 46.6391, 6.63534, 46.6422
> [Fri Sep 13 18:00:15 2024].0958, 0.3293: GTiff: ScanDirectories()
> [Fri Sep 13 18:00:15 2024].0959, 0.3294: GTiff: Opened band mask.
> [Fri Sep 13 18:00:15 2024].0961, 0.3295: GTiff: Opened 8929x7207 overview.
> [Fri Sep 13 18:00:15 2024].0962, 0.3296: GTiff: Opened 4464x3603 overview.
> [Fri Sep 13 18:00:15 2024].0962, 0.3297: GTiff: Opened 2232x1801 overview.
> [Fri Sep 13 18:00:15 2024].0963, 0.3298: GTiff: Opened 1116x900 overview.
> [Fri Sep 13 18:00:15 2024].0964, 0.3299: GTiff: Opened 558x450 overview.
> [Fri Sep 13 18:00:15 2024].0965, 0.3300: GTiff: Opened 279x225 overview.
> [Fri Sep 13 18:00:15 2024].0966, 0.3301: GTiff: Opened band mask for
> 8929x7207 overview.
> [Fri Sep 13 18:00:15 2024].0967, 0.3302: GTiff: Opened band mask for
> 4464x3603 overview.
> [Fri Sep 13 18:00:15 2024].0968, 0.3303: GTiff: Opened band mask for
> 2232x1801 overview.
> [Fri Sep 13 18:00:15 2024].0969, 0.3303: GTiff: Opened band mask for
> 1116x900 overview.
> [Fri Sep 13 18:00:15 2024].0970, 0.3304: GTiff: Opened band mask for
> 558x450 overview.
> [Fri Sep 13 18:00:15 2024].0971, 0.3305: GTiff: Opened band mask for
> 279x225 overview.
> [Fri Sep 13 18:00:15 2024].2011, 0.4346: GDAL:
> GDALOpen(/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff,
> this=0x7f0dfc08dfd0) succeeds as GTiff.
> GDAL ERROR 0: [Fri Sep 13 18:00:15 2024].2120, 0.4454: GDAL: Computing
> area of interest: 6.62976, 46.6391, 6.63534, 46.6422
> [Fri Sep 13 18:00:15 2024].2121, 0.4456: GTiff: ScanDirectories()
> [Fri Sep 13 18:00:15 2024].2122, 0.4457: GTiff: Opened band mask.
> [Fri Sep 13 18:00:15 2024].2123, 0.4457: GTiff: Opened 8929x7207 overview.
> [Fri Sep 13 18:00:15 2024].2123, 0.4458: GTiff: Opened 4464x3603 overview.
> [Fri Sep 13 18:00:15 2024].2124, 0.4459: GTiff: Opened 2232x1801 overview.
> [Fri Sep 13 18:00:15 2024].2125, 0.4459: GTiff: Opened 1116x900 overview.
> [Fri Sep 13 18:00:15 2024].2125, 0.4460: GTiff: Opened 558x450 overview.
> [Fri Sep 13 18:00:15 2024].2126, 0.4460: GTiff: Opened 279x225 overview.
> [Fri Sep 13 18:00:15 2024].2126, 0.4461: GTiff: Opened band mask for
> 8929x7207 overview.
> [Fri Sep 13 18:00:15 2024].2127, 0.4462: GTiff: Opened band mask for
> 4464x3603 overview.
> [Fri Sep 13 18:00:15 2024].2128, 0.4462: GTiff: Opened band mask for
> 2232x1801 overview.
> [Fri Sep 13 18:00:15 2024].2128, 0.4463: GTiff: Opened band mask for
> 1116x900 overview.
> [Fri Sep 13 18:00:15 2024].2129, 0.4463: GTiff: Opened band mask for
> 558x450 overview.
> [Fri Sep 13 18:00:15 2024].2129, 0.4464: GTiff: Opened band mask for
> 279x225 overview.
> [Fri Sep 13 18:00:15 2024].2130, 0.4465: S3: Downloading 16384-32767 (
> https://prod-pix4d-cloud-eu-central-1.s3.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff).
> ..
> [Fri Sep 13 18:00:15 2024].3454, 0.5789: S3: Got response_code=206
> [Fri Sep 13 18:00:15 2024].3456, 0.5791: S3: Downloading 180224-589823 (
> https://prod-pix4d-cloud-eu-central-1.s3.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff).
> ..
> [Fri Sep 13 18:00:15 2024].4272, 0.6606: S3: Got response_code=206
> [Fri Sep 13 18:00:15 2024].4276, 0.6610: GDAL: GDAL_CACHEMAX = 1591 MB
>
> What can it be?
>
> BTW, in both cases it says
> ERROR: Status 2: File
> /vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff
> could not be found
> And later it opens it. I don't know why.
>
> Thank you very much
> Javier.
>
> On Mon, 9 Sept 2024 at 23:49, Mayer, David P <dpmayer at usgs.gov> wrote:
>
>> Would ~200 GB be large enough?
>>
>> Here is a mosaic of a section of Mars at 6 meters/pixel in a public S3
>> bucket:
>>
>> s3://asc-pds-services/mosaic/Mars/Mars_MRO_CTX_Equi_Mosaics_Robbins/6mpp/MC16_6mpp.16bit.tif
>>
>> Cheers,
>> David
>>
>> -----------
>> David Mayer
>> Geographer
>> Astrogeology Science Center
>> US Geological Survey
>> dpmayer at usgs.gov
>> ------------------------------
>> *From:* gdal-dev <gdal-dev-bounces at lists.osgeo.org> on behalf of Javier
>> Jimenez Shaw via gdal-dev <gdal-dev at lists.osgeo.org>
>> *Sent:* Monday, September 9, 2024 13:55
>> *To:* gdal dev <gdal-dev at lists.osgeo.org>
>> *Subject:* [EXTERNAL] [gdal-dev] s3 request example
>>
>>
>>
>>
>> * This email has been received from outside of DOI - Use caution before
>> clicking on links, opening attachments, or responding.  *
>>
>>
>>
>> Hi
>>
>> I am experiencing a much slower load of a remote COG using vsis3 with the
>> QGIS from conda than the QGIS in my Ubuntu (that uses GDAL 3.4.1).
>>
>> Because both are QGIS 3.38, I think the problem is in the version of GDAL.
>> I would like to research it more.
>>
>> Is there any example of a (big) COG file in Amazon S3 I can use to
>> compare them? Unfortunately the credentials for the files I am using expire
>> after a few hours, so they are useless to report anything.
>>
>> Best regards,
>> Javier
>>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240913/8c956119/attachment-0001.htm>