[gdal-dev] (TileDB related question for you Norman/Sean) Re: [EXTERNAL] s3 request example
Even Rouault
even.rouault at spatialys.com
Fri Sep 13 10:22:54 PDT 2024
I've given this a try with my QGIS dev env and I can also reproduce, and
specifically with the TileDB driver available
The stacktrace is as following:
Thread 1 "qgis" received signal SIGINT, Interrupt.
0x00007f81cc566bcf in __GI___poll (fds=0x7fffc613b040, nfds=2,
timeout=800) at ../sysdeps/unix/sysv/linux/poll.c:29
29 ../sysdeps/unix/sysv/linux/poll.c: No such file or directory.
(gdb) bt
#0 0x00007f81cc566bcf in __GI___poll (fds=0x7fffc613b040, nfds=2,
timeout=800) at ../sysdeps/unix/sysv/linux/poll.c:29
#1 0x00007f809b5fa76a in Curl_poll () at
/home/even/tiledb/tiledb-2.15.2/lib/libtiledb.so.2.15
#2 0x00007f809b5f3684 in Curl_multi_wait.part.0 () at
/home/even/tiledb/tiledb-2.15.2/lib/libtiledb.so.2.15
#3 0x00007f809b5f3c06 in curl_multi_poll () at
/home/even/tiledb/tiledb-2.15.2/lib/libtiledb.so.2.15
#4 0x00007f809b5dfbb0 in curl_easy_perform () at
/home/even/tiledb/tiledb-2.15.2/lib/libtiledb.so.2.15
#5 0x00007f809b3d822f in
Aws::Http::CurlHttpClient::MakeRequest(std::shared_ptr<Aws::Http::HttpRequest>
const&, Aws::Utils::RateLimits::RateLimiterInterface*,
Aws::Utils::RateLimits::RateLimiterInterface*) const () at
/home/even/tiledb/tiledb-2.15.2/lib/libtiledb.so.2.15
#6 0x00007f809b3b0788 in
Aws::Internal::AWSHttpResourceClient::GetResourceWithAWSWebServiceResult(std::shared_ptr<Aws::Http::HttpRequest>
const&) const ()
at /home/even/tiledb/tiledb-2.15.2/lib/libtiledb.so.2.15
#7 0x00007f809b39c627 in
Aws::Internal::EC2MetadataClient::GetCurrentRegion() const () at
/home/even/tiledb/tiledb-2.15.2/lib/libtiledb.so.2.15
#8 0x00007f809b3a2b8b in
Aws::Client::ClientConfiguration::ClientConfiguration() () at
/home/even/tiledb/tiledb-2.15.2/lib/libtiledb.so.2.15
#9 0x00007f809aae98dd in Aws::Client::ClientConfiguration*
tiledb::common::tiledb_new<Aws::Client::ClientConfiguration>(std::string
const&) () at /home/even/tiledb/tiledb-2.15.2/lib/libtiledb.so.2.15
#10 0x00007f809aace812 in tiledb::sm::S3::init_client() const () at
/home/even/tiledb/tiledb-2.15.2/lib/libtiledb.so.2.15
#11 0x00007f809aad3d98 in tiledb::sm::S3::is_dir(tiledb::sm::URI const&,
bool*) const () at /home/even/tiledb/tiledb-2.15.2/lib/libtiledb.so.2.15
#12 0x00007f809aaf745d in tiledb::sm::VFS::is_dir(tiledb::sm::URI
const&, bool*) const () at
/home/even/tiledb/tiledb-2.15.2/lib/libtiledb.so.2.15
#13 0x00007f809b14cf86 in
tiledb::sm::StorageManagerCanonical::is_array(tiledb::sm::URI const&)
const () at /home/even/tiledb/tiledb-2.15.2/lib/libtiledb.so.2.15
#14 0x00007f809b14d1c7 in
tiledb::sm::StorageManagerCanonical::object_type(tiledb::sm::URI const&,
tiledb::sm::ObjectType*) const () at
/home/even/tiledb/tiledb-2.15.2/lib/libtiledb.so.2.15
#15 0x00007f809a9eb479 in tiledb_object_type () at
/home/even/tiledb/tiledb-2.15.2/lib/libtiledb.so.2.15
#16 0x00007f80a1a16882 in tiledb::Object::object(tiledb::Context const&,
std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> > const&)
(ctx=...,
uri="s3://prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff")
at /home/even/tiledb/tiledb-2.15.2/include/tiledb/object.h:175
#17 0x00007f80a1a1368b in TileDBDataset::Identify(GDALOpenInfo*)
(poOpenInfo=0x7fffc613ca20) at
/home/even/gdal/gdal/frmts/tiledb/tiledbcommon.cpp:157
#18 0x00007f80a1a13977 in TileDBDataset::Open(GDALOpenInfo*)
(poOpenInfo=0x7fffc613ca20) at
/home/even/gdal/gdal/frmts/tiledb/tiledbcommon.cpp:208
#19 0x00007f81d0430525 in GDALDriver::Open(GDALOpenInfo*, bool)
(this=0x562cf5138f90, poOpenInfo=0x7fffc613ca20, bSetOpenOptions=false)
at /home/even/gdal/gdal/gcore/gdaldriver.cpp:117
#20 0x00007f81d044a450 in GDALOpenEx(char const*, unsigned int, char
const* const*, char const* const*, char const* const*)
(pszFilename=0x562cf8e7e378
"/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff",
nOpenFlags=4, papszAllowedDrivers=0x0, papszOpenOptions=0x562cfb256920,
papszSiblingFiles=0x0) at /home/even/gdal/gdal/gcore/gdaldataset.cpp:3774
#21 0x00007f81d6b48489 in QgsOgrProviderUtils::GDALOpenWrapper(char
const*, bool, char**, void**)
(pszPath=0x562cf8e7e378
"/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff",
bUpdate=false, papszOpenOptionsIn=0x562cfb2533f0, phDriver=0x0) at
/home/even/qgis/qgis/src/core/providers/ogr/qgsogrproviderutils.cpp:1067
#22 0x00007f81d6b4ab7c in OpenHelper(QString const&, bool, QStringList
const&) (dsName=..., updateMode=false, options=...) at
/home/even/qgis/qgis/src/core/providers/ogr/qgsogrproviderutils.cpp:1460
#23 0x00007f81d6b4b861 in QgsOgrProviderUtils::getLayer(QString const&,
bool, QStringList const&, int, QString&, bool)
(dsName=..., updateMode=false, options=..., layerIndex=0,
errCause=..., checkModificationDateAgainstCache=true) at
/home/even/qgis/qgis/src/core/providers/ogr/qgsogrproviderutils.cpp:1603
#24 0x00007f81d6b31f80 in QgsOgrProviderMetadata::querySublayers(QString
const&, QFlags<Qgis::SublayerQueryFlag>, QgsFeedback*) const
(this=0x562cf50d8150, u=..., flags=..., feedback=0x0)
at
/home/even/qgis/qgis/src/core/providers/ogr/qgsogrprovidermetadata.cpp:970
#25 0x00007f81d6a471d7 in QgsProviderRegistry::querySublayers(QString
const&, QFlags<Qgis::SublayerQueryFlag>, QgsFeedback*) const
(this=0x562cf50d31e0, uri=..., flags=..., feedback=0x0)
at /home/even/qgis/qgis/src/core/providers/qgsproviderregistry.cpp:1114
#26 0x00007f81de8ebeed in QgsAppLayerHandling::openLayer(QString const&,
bool&, bool, bool, bool)
(fileName=..., ok=@0x7fffc613d760: false, allowInteractive=false,
suppressBulkLayerPostProcessing=false, addToLegend=true) at
/home/even/qgis/qgis/src/app/layers/qgsapplayerhandling.cpp:960
#27 0x0000562cf401167e in main(int, char**) (argc=2,
argv=0x7fffc613e1c8) at /home/even/qgis/qgis/src/app/main.cpp:1596
So, to make it short, this is when QGIS tries to open the file as a
vector one (it must tries to open both as raster and vector). And the
TileDB driver isn't currently smart enough to immediately reject that a
".tiff" file is not a vector TileDB dataset. This is not a problem when
opening as raster dataset because the GeoTIFF driver would already have
identified the file.
@Norman: only libtiledb can effectively guess if a *directory* contains
a TileDB dataset or not. But is a *file* a valid URI for a TileDB
dataset? Because at the time the Identify() method of the TileDB driver
is hit, GDAL knows already if /vsis3/something is a file or a directory,
and thus we could immediately bail out if it is a file and a file can't
be a TileDB dataset.
Even
Le 13/09/2024 à 18:30, thomas bonfort via gdal-dev a écrit :
> I also see that you have a tiledb driver registration happening only
> on your conda version, exactly at the time of your lag. you should
> also try to remove/rename your gdal_tiledb.so plugin file to check the
> issue isn't coming from there.
>
> Le ven. 13 sept. 2024, 18:22, thomas bonfort
> <thomas.bonfort at gmail.com> a écrit :
>
> Javier,
> 3.9 is probably doing parallel reads, which will cause a
> noticeable lag in requests on a slow connection (the overall
> processing time should not be slower anyhow). There's a gtiff open
> option documented that can disable this behavior, that you can try
> to use to see if it changes your timings. If that is the effective
> cause, that would mean there is an issue with this parallel
> implementation.
> TB
>
> Le ven. 13 sept. 2024, 18:15, Javier Jimenez Shaw via gdal-dev
> <gdal-dev at lists.osgeo.org> a écrit :
>
> Thanks David for the suggestion. Very interesting data.
>
> Unfortunately the issue I have is not reproduced with that
> file. So it may be due to the authentication (your suggestion
> does not need authentication).
>
> Some context: I am opening a COG file from S3 in QGIS, setting
> the url in the command line. Something like
> qgis /vsis3/blahblabla
>
> (more on that later)
>
> If I run it in my Ubuntu 22.04 (that has GDAL 3.4.1), with
> QGIS 3.38, no problem
> If I run with conda with QGIS 3.38 (that has GDAL 3.9.1), it
> is noticeable slower.
>
> So I think it is due to the version of GDAL. If you think that
> I am wrong and I should ask directly to QGIS, it is ok.
>
> This is a bit problematic to reproduce because the
> authorization expires in 12 hours. So I post here the command
> I produced with a python script. If it is needed, I can share
> the python script that is getting the authentication. It is
> about 100 lines of code.
>
> Enabling the logs I see there a gap of 6 seconds during the
> opening with conda, that is not there with the QGIS from
> Ubuntu that uses GDAL 3.4.1
>
> This is the log from conda
> The gap is between
> Fri Sep 13 17:56:26 2024].9370
> and
> [Fri Sep 13 17:56:33 2024].4445
>
> CPL_DEBUG=ON CPL_TIMESTAMP=ON python3 pix4d_qgis.py
> --project-id 1969386 --share-token
> 09c3eada-1b3b-46eb-b5de-8bd51635a5b9
> Session expires at 2024-09-14T03:56:22Z
> AWS_ACCESS_KEY_ID=ASIATOCJLBKS2RHGCAHB
> AWS_SECRET_ACCESS_KEY=nNT8fTH/cGiyn4NVubp5/1LroyZ69johhXYxkCK1
> AWS_SESSION_TOKEN=FwoGZXIvYXdzEKn//////////wEaDI1G/49ov88aGibrOCLXAilDbBuFQ/ydJF4AZ50/lYmWYmNDjM9H7XbbioyXiLQzwTqVRN1TLG6T29Anklkw47MKxdKHDF/O85r9gWJmCAqYTKalA38VM1qbixsjSXAzR7W5p1v1pgRUKtN4J03NrZOH2S6gbo4OKv+UUkPe/Xq4s8w0vEVNJLhvVx6dvpJMv7Jp9UxvLA5a+nrBKUqJujujqJnRXAKKfnpJbTDvhCuIeTxaFFt50e5UpLjSwqXS/a/AaEMJXRSsSLbKFTW/ALSCHfPv3HCStnue5CFpzVk4OhQbQ+dZyXUQp5wxQTtpr0hIh0IyB20gsoZNa0lGvFuXnzqDCH2X2mXwzDcyZk2IOX34nq+Wtnk8qYdjJxEAG0tnlYsMjmmXECnbOxl6gtNJ5hoehgak6cSfs7/rMi4jOI838tzsWWyDJeACj9qfQyfGtpk23x8Zt6KuqUjNZ1/z+/T8LwoopsGRtwYyKQbJS8pn8IGKPLtHDVxRx09yBRuqgYVFzSqpnr3T1vwWYHXiEF0sVJBx
> GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR
> /home/jshaw/anaconda3/envs/qgis_stable/bin/qgis
> /vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff
> [Fri Sep 13 17:56:26 2024].3948, 0.0000: HTTP: libcurl/8.9.0
> OpenSSL/3.3.1 zlib/1.3.1 zstd/1.5.6 libssh2/1.11.0 nghttp2/1.58.0
> [Fri Sep 13 17:56:26 2024].7731, 0.3783: S3: Switching to
> endpoint s3.eu-central-1.amazonaws.com
> <http://s3.eu-central-1.amazonaws.com>
> [Fri Sep 13 17:56:26 2024].8415, 0.4467: S3: Switching to
> region eu-central-1
> [Fri Sep 13 17:56:26 2024].9230, 0.5282: S3:
> GetFileSize(https://prod-pix4d-cloud-eu-central-1.s3.eu-central-1.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff)=66132291
> response_code=206
> [Fri Sep 13 17:56:26 2024].9235, 0.5288: GDAL:
> GDALOpen(/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff,
> this=0x556e6bb915a0) succeeds as GTiff.
> [Fri Sep 13 17:56:26 2024].9240, 0.5292: GTiff: ScanDirectories()
> [Fri Sep 13 17:56:26 2024].9243, 0.5296: GTiff: Opened band mask.
> [Fri Sep 13 17:56:26 2024].9246, 0.5298: GTiff: Opened
> 8929x7207 overview.
> [Fri Sep 13 17:56:26 2024].9249, 0.5301: GTiff: Opened
> 4464x3603 overview.
> [Fri Sep 13 17:56:26 2024].9251, 0.5304: GTiff: Opened
> 2232x1801 overview.
> [Fri Sep 13 17:56:26 2024].9254, 0.5306: GTiff: Opened
> 1116x900 overview.
> [Fri Sep 13 17:56:26 2024].9257, 0.5309: GTiff: Opened 558x450
> overview.
> [Fri Sep 13 17:56:26 2024].9259, 0.5311: GTiff: Opened 279x225
> overview.
> [Fri Sep 13 17:56:26 2024].9262, 0.5314: GTiff: Opened band
> mask for 8929x7207 overview.
> [Fri Sep 13 17:56:26 2024].9264, 0.5316: GTiff: Opened band
> mask for 4464x3603 overview.
> [Fri Sep 13 17:56:26 2024].9267, 0.5319: GTiff: Opened band
> mask for 2232x1801 overview.
> [Fri Sep 13 17:56:26 2024].9269, 0.5321: GTiff: Opened band
> mask for 1116x900 overview.
> [Fri Sep 13 17:56:26 2024].9272, 0.5324: GTiff: Opened band
> mask for 558x450 overview.
> [Fri Sep 13 17:56:26 2024].9274, 0.5326: GTiff: Opened band
> mask for 279x225 overview.
> [Fri Sep 13 17:56:26 2024].9277, 0.5329: GDAL:
> GDALClose(/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff,
> this=0x556e6bb915a0)
> ERROR: Status 2: File
> /vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff
> could not be found
> [Fri Sep 13 17:56:26 2024].9370, 0.5422: GDAL: On-demand
> registering
> /home/jshaw/anaconda3/envs/qgis_stable/lib/gdalplugins/gdal_TileDB.so
> using GDALRegister_TileDB.
> [Fri Sep 13 17:56:33 2024].4445, 7.0498: S3:
> GetFileSize(https://prod-pix4d-cloud-eu-central-1.s3.eu-central-1.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff)=66132291
> response_code=206
> [Fri Sep 13 17:56:33 2024].4449, 7.0501: GDAL:
> GDALOpen(/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff,
> this=0x556e6b807d00) succeeds as GTiff.
> GDAL ERROR 0: [Fri Sep 13 17:56:33 2024].4666, 7.0718: OGRCT:
> Using WebMercator to WGS84 optimization
> GDAL ERROR 0: [Fri Sep 13 17:56:33 2024].4667, 7.0719: GDAL:
> Computing area of interest: 6.62976, 46.6391, 6.63534, 46.6422
> [Fri Sep 13 17:56:33 2024].4667, 7.0719: GTiff: ScanDirectories()
> [Fri Sep 13 17:56:33 2024].4669, 7.0721: GTiff: Opened band mask.
> [Fri Sep 13 17:56:33 2024].4670, 7.0722: GTiff: Opened
> 8929x7207 overview.
> [Fri Sep 13 17:56:33 2024].4671, 7.0723: GTiff: Opened
> 4464x3603 overview.
> [Fri Sep 13 17:56:33 2024].4672, 7.0724: GTiff: Opened
> 2232x1801 overview.
> [Fri Sep 13 17:56:33 2024].4673, 7.0725: GTiff: Opened
> 1116x900 overview.
> [Fri Sep 13 17:56:33 2024].4674, 7.0726: GTiff: Opened 558x450
> overview.
> [Fri Sep 13 17:56:33 2024].4675, 7.0727: GTiff: Opened 279x225
> overview.
> [Fri Sep 13 17:56:33 2024].4676, 7.0728: GTiff: Opened band
> mask for 8929x7207 overview.
> [Fri Sep 13 17:56:33 2024].4677, 7.0729: GTiff: Opened band
> mask for 4464x3603 overview.
> [Fri Sep 13 17:56:33 2024].4677, 7.0730: GTiff: Opened band
> mask for 2232x1801 overview.
> [Fri Sep 13 17:56:33 2024].4678, 7.0731: GTiff: Opened band
> mask for 1116x900 overview.
> [Fri Sep 13 17:56:33 2024].4679, 7.0732: GTiff: Opened band
> mask for 558x450 overview.
> [Fri Sep 13 17:56:33 2024].4680, 7.0733: GTiff: Opened band
> mask for 279x225 overview.
> [Fri Sep 13 17:56:33 2024].5178, 7.1230: GDAL:
> GDALOpen(/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff,
> this=0x7f830c08d490) succeeds as GTiff.
> [Fri Sep 13 17:56:33 2024].5180, 7.1232: S3: Downloading
> 59179008-59244543
> (https://prod-pix4d-cloud-eu-central-1.s3.eu-central-1.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff)...
> GDAL ERROR 0: [Fri Sep 13 17:56:33 2024].5244, 7.1296: GDAL:
> Computing area of interest: 6.62976, 46.6391, 6.63534, 46.6422
> [Fri Sep 13 17:56:33 2024].5245, 7.1297: GTiff: ScanDirectories()
> [Fri Sep 13 17:56:33 2024].5246, 7.1298: GTiff: Opened band mask.
> [Fri Sep 13 17:56:33 2024].5247, 7.1299: GTiff: Opened
> 8929x7207 overview.
> [Fri Sep 13 17:56:33 2024].5247, 7.1300: GTiff: Opened
> 4464x3603 overview.
> [Fri Sep 13 17:56:33 2024].5248, 7.1300: GTiff: Opened
> 2232x1801 overview.
> [Fri Sep 13 17:56:33 2024].5249, 7.1301: GTiff: Opened
> 1116x900 overview.
> [Fri Sep 13 17:56:33 2024].5249, 7.1302: GTiff: Opened 558x450
> overview.
> [Fri Sep 13 17:56:33 2024].5250, 7.1302: GTiff: Opened 279x225
> overview.
> [Fri Sep 13 17:56:33 2024].5251, 7.1303: GTiff: Opened band
> mask for 8929x7207 overview.
> [Fri Sep 13 17:56:33 2024].5251, 7.1304: GTiff: Opened band
> mask for 4464x3603 overview.
> [Fri Sep 13 17:56:33 2024].5252, 7.1304: GTiff: Opened band
> mask for 2232x1801 overview.
> [Fri Sep 13 17:56:33 2024].5253, 7.1305: GTiff: Opened band
> mask for 1116x900 overview.
> [Fri Sep 13 17:56:33 2024].5253, 7.1306: GTiff: Opened band
> mask for 558x450 overview.
> [Fri Sep 13 17:56:33 2024].5254, 7.1306: GTiff: Opened band
> mask for 279x225 overview.
> [Fri Sep 13 17:56:33 2024].5255, 7.1307: GDAL: Selecting
> overview 1116 x 900
> [Fri Sep 13 17:56:33 2024].5255, 7.1307: S3: Downloading
> 16384-32767
> (https://prod-pix4d-cloud-eu-central-1.s3.eu-central-1.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff)...
> [Fri Sep 13 17:56:33 2024].6116, 7.2168: S3: Got response_code=206
> [Fri Sep 13 17:56:33 2024].6118, 7.2171: S3: Downloading
> 180224-589823
> (https://prod-pix4d-cloud-eu-central-1.s3.eu-central-1.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff)...
> [Fri Sep 13 17:56:33 2024].6955, 7.3008: S3: Got response_code=206
> [Fri Sep 13 17:56:33 2024].6960, 7.3012: GDAL: GDAL_CACHEMAX =
> 1591 MB
> [Fri Sep 13 17:56:33 2024].6977, 7.3029: S3: Got response_code=206
>
>
> This is the equivalent code with QGIS 3.38 in Ubuntu and GDAL
> 3.4.1
>
> CPL_DEBUG=ON CPL_TIMESTAMP=ON python3 pix4d_qgis.py
> --project-id 1969386 --share-token
> 09c3eada-1b3b-46eb-b5de-8bd51635a5b9
> Session expires at 2024-09-14T04:00:10Z
> AWS_ACCESS_KEY_ID=ASIATOCJLBKSRCEUBKE6
> AWS_SECRET_ACCESS_KEY=h6NhBf9VwBZoVj5Zd/8H2BnzXQu0l6bFUUv2D8E7
> AWS_SESSION_TOKEN=FwoGZXIvYXdzEKn//////////wEaDF1xO50XL6GBe/358CLXAn3JR/uteJtarwJ/ryn6i4+lcMFEQZ/KV0FZQdKsD1PwAcqoYFY+0GKYSV+FTk81ZszsVzy8N7d2GwByvP9S/L1XOo1FrqJfDpGw3a5eQR7AoeU68EPlD/WTYnM7XBzXimHywDCyue3SOi8oZaSdbMgjBQ1N6j+qQ+EDN7RjD/Zu2CGMTzN9g/Z0FhZsuyt2HRhnOYpsvpUMZ+WQr7pbjdVdwt3PdFHsJ95fhueapJspgQ7OSgmj5VtSR8NTecohLsd2eeGkSpiqVWuBwuAu2cW6LcqmAZGnF0DZIjPWiMIDf/OLczMRCw5MJGeYpsSkkiuM+t9Miv8aN4iCo8vjl6wumOCgPqTgBBWlQdAs2mLxwpF1Bb1UjJ6eiccp17+yJF3ffn8tvxx/+1V3Q++E5Xk+jiB1keJnFcizr7qD6EY+EdDES5sQrWAhM2uR78JjNc0f+hVZEmAoisORtwYyKb0po07ytiiBv+X3j+Y6BsCHR0Es3C3xVvsgkfn1zb1Z2flFWUzwqJUD
> GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR /usr/bin/qgis
> /vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff
> [Fri Sep 13 18:00:14 2024].7665, 0.0000: HTTP: libcurl/7.81.0
> OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8
> libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2)
> libssh/0.9.6/openssl/zlib nghttp2/1.43.0 librtmp/2.3
> OpenLDAP/2.5.18
> [Fri Sep 13 18:00:14 2024].7666, 0.0001: S3: Downloading
> 0-16383
> (https://prod-pix4d-cloud-eu-central-1.s3.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff)...
> [Fri Sep 13 18:00:14 2024].8897, 0.1231: S3: Got response_code=400
> [Fri Sep 13 18:00:14 2024].8898, 0.1232: S3: Switching to
> region eu-central-1
> [Fri Sep 13 18:00:14 2024].8899, 0.1234: S3: Downloading
> 0-16383
> (https://prod-pix4d-cloud-eu-central-1.s3.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff)...
> [Fri Sep 13 18:00:15 2024].0287, 0.2621: S3: Got response_code=206
> [Fri Sep 13 18:00:15 2024].0288, 0.2622: S3:
> GetFileSize(https://prod-pix4d-cloud-eu-central-1.s3.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff)=66132291
> response_code=206
> [Fri Sep 13 18:00:15 2024].0292, 0.2627: GDAL:
> GDALOpen(/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff,
> this=0x560e4cd84920) succeeds as GTiff.
> [Fri Sep 13 18:00:15 2024].0298, 0.2632: GTiff: ScanDirectories()
> [Fri Sep 13 18:00:15 2024].0301, 0.2635: GTiff: Opened band mask.
> [Fri Sep 13 18:00:15 2024].0304, 0.2638: GTiff: Opened
> 8929x7207 overview.
> [Fri Sep 13 18:00:15 2024].0306, 0.2641: GTiff: Opened
> 4464x3603 overview.
> [Fri Sep 13 18:00:15 2024].0309, 0.2643: GTiff: Opened
> 2232x1801 overview.
> [Fri Sep 13 18:00:15 2024].0311, 0.2646: GTiff: Opened
> 1116x900 overview.
> [Fri Sep 13 18:00:15 2024].0314, 0.2648: GTiff: Opened 558x450
> overview.
> [Fri Sep 13 18:00:15 2024].0316, 0.2651: GTiff: Opened 279x225
> overview.
> [Fri Sep 13 18:00:15 2024].0319, 0.2654: GTiff: Opened band
> mask for 8929x7207 overview.
> [Fri Sep 13 18:00:15 2024].0322, 0.2656: GTiff: Opened band
> mask for 4464x3603 overview.
> [Fri Sep 13 18:00:15 2024].0324, 0.2658: GTiff: Opened band
> mask for 2232x1801 overview.
> [Fri Sep 13 18:00:15 2024].0326, 0.2661: GTiff: Opened band
> mask for 1116x900 overview.
> [Fri Sep 13 18:00:15 2024].0329, 0.2663: GTiff: Opened band
> mask for 558x450 overview.
> [Fri Sep 13 18:00:15 2024].0331, 0.2665: GTiff: Opened band
> mask for 279x225 overview.
> [Fri Sep 13 18:00:15 2024].0334, 0.2669: GDAL:
> GDALClose(/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff,
> this=0x560e4cd84920)
> ERROR: Status 2: File
> /vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff
> could not be found
> [Fri Sep 13 18:00:15 2024].0357, 0.2691: S3: Downloading
> 0-16383
> (https://prod-pix4d-cloud-eu-central-1.s3.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff)...
> [Fri Sep 13 18:00:15 2024].0762, 0.3096: S3: Got response_code=206
> [Fri Sep 13 18:00:15 2024].0762, 0.3097: S3:
> GetFileSize(https://prod-pix4d-cloud-eu-central-1.s3.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff)=66132291
> response_code=206
> [Fri Sep 13 18:00:15 2024].0765, 0.3099: GDAL:
> GDALOpen(/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff,
> this=0x560e4d0afb70) succeeds as GTiff.
> GDAL ERROR 0: [Fri Sep 13 18:00:15 2024].0956, 0.3291: GDAL:
> Computing area of interest: 6.62976, 46.6391, 6.63534, 46.6422
> [Fri Sep 13 18:00:15 2024].0958, 0.3293: GTiff: ScanDirectories()
> [Fri Sep 13 18:00:15 2024].0959, 0.3294: GTiff: Opened band mask.
> [Fri Sep 13 18:00:15 2024].0961, 0.3295: GTiff: Opened
> 8929x7207 overview.
> [Fri Sep 13 18:00:15 2024].0962, 0.3296: GTiff: Opened
> 4464x3603 overview.
> [Fri Sep 13 18:00:15 2024].0962, 0.3297: GTiff: Opened
> 2232x1801 overview.
> [Fri Sep 13 18:00:15 2024].0963, 0.3298: GTiff: Opened
> 1116x900 overview.
> [Fri Sep 13 18:00:15 2024].0964, 0.3299: GTiff: Opened 558x450
> overview.
> [Fri Sep 13 18:00:15 2024].0965, 0.3300: GTiff: Opened 279x225
> overview.
> [Fri Sep 13 18:00:15 2024].0966, 0.3301: GTiff: Opened band
> mask for 8929x7207 overview.
> [Fri Sep 13 18:00:15 2024].0967, 0.3302: GTiff: Opened band
> mask for 4464x3603 overview.
> [Fri Sep 13 18:00:15 2024].0968, 0.3303: GTiff: Opened band
> mask for 2232x1801 overview.
> [Fri Sep 13 18:00:15 2024].0969, 0.3303: GTiff: Opened band
> mask for 1116x900 overview.
> [Fri Sep 13 18:00:15 2024].0970, 0.3304: GTiff: Opened band
> mask for 558x450 overview.
> [Fri Sep 13 18:00:15 2024].0971, 0.3305: GTiff: Opened band
> mask for 279x225 overview.
> [Fri Sep 13 18:00:15 2024].2011, 0.4346: GDAL:
> GDALOpen(/vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff,
> this=0x7f0dfc08dfd0) succeeds as GTiff.
> GDAL ERROR 0: [Fri Sep 13 18:00:15 2024].2120, 0.4454: GDAL:
> Computing area of interest: 6.62976, 46.6391, 6.63534, 46.6422
> [Fri Sep 13 18:00:15 2024].2121, 0.4456: GTiff: ScanDirectories()
> [Fri Sep 13 18:00:15 2024].2122, 0.4457: GTiff: Opened band mask.
> [Fri Sep 13 18:00:15 2024].2123, 0.4457: GTiff: Opened
> 8929x7207 overview.
> [Fri Sep 13 18:00:15 2024].2123, 0.4458: GTiff: Opened
> 4464x3603 overview.
> [Fri Sep 13 18:00:15 2024].2124, 0.4459: GTiff: Opened
> 2232x1801 overview.
> [Fri Sep 13 18:00:15 2024].2125, 0.4459: GTiff: Opened
> 1116x900 overview.
> [Fri Sep 13 18:00:15 2024].2125, 0.4460: GTiff: Opened 558x450
> overview.
> [Fri Sep 13 18:00:15 2024].2126, 0.4460: GTiff: Opened 279x225
> overview.
> [Fri Sep 13 18:00:15 2024].2126, 0.4461: GTiff: Opened band
> mask for 8929x7207 overview.
> [Fri Sep 13 18:00:15 2024].2127, 0.4462: GTiff: Opened band
> mask for 4464x3603 overview.
> [Fri Sep 13 18:00:15 2024].2128, 0.4462: GTiff: Opened band
> mask for 2232x1801 overview.
> [Fri Sep 13 18:00:15 2024].2128, 0.4463: GTiff: Opened band
> mask for 1116x900 overview.
> [Fri Sep 13 18:00:15 2024].2129, 0.4463: GTiff: Opened band
> mask for 558x450 overview.
> [Fri Sep 13 18:00:15 2024].2129, 0.4464: GTiff: Opened band
> mask for 279x225 overview.
> [Fri Sep 13 18:00:15 2024].2130, 0.4465: S3: Downloading
> 16384-32767
> (https://prod-pix4d-cloud-eu-central-1.s3.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff)...
> [Fri Sep 13 18:00:15 2024].3454, 0.5789: S3: Got response_code=206
> [Fri Sep 13 18:00:15 2024].3456, 0.5791: S3: Downloading
> 180224-589823
> (https://prod-pix4d-cloud-eu-central-1.s3.amazonaws.com/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff)...
> [Fri Sep 13 18:00:15 2024].4272, 0.6606: S3: Got response_code=206
> [Fri Sep 13 18:00:15 2024].4276, 0.6610: GDAL: GDAL_CACHEMAX =
> 1591 MB
>
> What can it be?
>
> BTW, in both cases it says
> ERROR: Status 2: File
> /vsis3/prod-pix4d-cloud-eu-central-1/user-5b21721a-90d0-49c9-9d1c-d2a0c5929be5/project-1969386/ortho_display.tiff
> could not be found
> And later it opens it. I don't know why.
>
> Thank you very much
> Javier.
>
> On Mon, 9 Sept 2024 at 23:49, Mayer, David P
> <dpmayer at usgs.gov> wrote:
>
> Would ~200 GB be large enough?
>
> Here is a mosaic of a section of Mars at 6 meters/pixel in
> a public S3 bucket:
> s3://asc-pds-services/mosaic/Mars/Mars_MRO_CTX_Equi_Mosaics_Robbins/6mpp/MC16_6mpp.16bit.tif
>
> Cheers,
> David
>
> -----------
> David Mayer
> Geographer
> Astrogeology Science Center
> US Geological Survey
> dpmayer at usgs.gov
> ------------------------------------------------------------------------
> *From:* gdal-dev <gdal-dev-bounces at lists.osgeo.org> on
> behalf of Javier Jimenez Shaw via gdal-dev
> <gdal-dev at lists.osgeo.org>
> *Sent:* Monday, September 9, 2024 13:55
> *To:* gdal dev <gdal-dev at lists.osgeo.org>
> *Subject:* [EXTERNAL] [gdal-dev] s3 request example
>
> * This email has been received from outside of DOI - Use
> caution before clicking on links, opening attachments, or
> responding. *
>
>
>
> Hi
>
> I am experiencing a much slower load of a remote COG using
> vsis3 with the QGIS from conda than the QGIS in my Ubuntu
> (that uses GDAL 3.4.1).
>
> Because both are QGIS 3.38, I think the problem is in the
> version of GDAL.
> I would like to research it more.
>
> Is there any example of a (big) COG file in Amazon S3 I
> can use to compare them? Unfortunately the credentials for
> the files I am using expire after a few hours, so they are
> useless to report anything.
>
> Best regards,
> Javier
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
--
http://www.spatialys.com
My software is free, but my time generally not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240913/6d382677/attachment-0001.htm>
More information about the gdal-dev
mailing list