[gdal-dev] Failing to write to s3 using GTiff driver.

Tue Aug 16 04:16:15 PDT 2022

I've included the rest of the debug below. It starts hitting 404s (I guess
because the objects don't exist) almost straight away:

> S3: GetFileSize(
> https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp)=0
>  response_code=404
> S3: /vsis3/my-bucket/test_batch_output/test.tif.warped.tif.tmp is not a
> object
>

Complete debug output:

gdal_translate --config CPL_VSIL_USE_TEMP_FILE_FOR_RANDOM_WRITE YES
> --config CPL_TMPDIR /tmp -of COG -co TILING_SCHEME\=GoogleMapsCompatible
> -co COMPRESS\=DEFLATE -co NUM_THREADS\=ALL_CPUS --debug on
> /vsis3/my-bucket/dtm/test.tif /vsis3/my-bucket/test_batch_output/test.tif
> HTTP: libcurl/7.81.0 OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8
> libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib
> nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.12
> S3: Downloading 0-16383 (https://my-bucket.s3.amazonaws.com/dtm/test.tif).
> ..
> S3: Got response_code=206
> S3: Downloading 102776832-102793215 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> S3: Downloading 102907904-102911196 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> GDAL: GDALOpen(/vsis3/my-bucket/dtm/test.tif, this=0x55a2882e8570)
> succeeds as GTiff.
> Input file size is 8400, 16400
> S3: GetFileSize(
> https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif)=0
>  response_code=404
> GTiff: ScanDirectories()
> GDAL: GDALDefaultOverviews::OverviewScan()
> GDAL: Computing area of interest: -4.12725, 53.1714, -4.09403, 53.2088
> COG: Using ZOOM_LEVEL 19
> COG: Reprojecting source dataset: start
> S3: GetFileSize(
> https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp)=0
>  response_code=404
> S3: /vsis3/my-bucket/test_batch_output/test.tif.warped.tif.tmp is not a
> object
> GDAL: Computing area of interest: -4.12725, 53.1714, -4.09403, 53.2088
> GDAL:
> GDALDriver::Create(GTiff,/vsis3/my-bucket/test_batch_output/test.tif.warped.tif.tmp,12544,23552,4,Byte,0x55a2886c5a10)
> 0WARP: Copying metadata from first source to destination dataset
> GDALWARP: Defining SKIP_NOSOURCE=YES
> WARP: SetAlphaMax: AlphaMax not set.
> S3: Downloading 102842368-102858751 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> S3: Downloading 16384-10485759 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> GDAL: GDAL_CACHEMAX = 392 MB
> S3: Downloading 10485760-20955135 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> S3: Downloading 102842368-102858751 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> S3: Downloading 102776832-102793215 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> S3: Downloading 20955136-20971519 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> S3: Downloading 20971520-21004287 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> S3: Downloading 21004288-21069823 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> S3: Downloading 21069824-21200895 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> S3: Downloading 21200896-21463039 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=0,0,1045x1762 Dst=0,0,1568x2944
> WARP: Using 2 threads
> S3: Downloading 21463040-21987327 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=977,0,1191x1795 Dst=1568,0,1568x2944
> WARP: Using 2 threads
> S3: Downloading 21987328-23035903 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=2101,0,1190x1828 Dst=3136,0,1568x2944
> WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=3224,0,1191x1862 Dst=4704,0,1568x2944
> WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=4348,0,1190x1895 Dst=6272,0,1568x2944
> WARP: Using 2 threads
> S3: Downloading 23035904-25133055 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=5471,0,1190x1928 Dst=7840,0,1568x2944
> .WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=6595,0,1190x1961 Dst=9408,0,1568x2944
> WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=7718,0,682x1994 Dst=10976,0,1568x2944
> WARP: Using 2 threads
> S3: Downloading 25133056-34684927 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> S3: Downloading 34684928-44236799 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> S3: Downloading 102842368-102875135 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> S3: Downloading 102776832-102809599 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> S3: Downloading 44236800-44253183 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> S3: Downloading 44253184-44285951 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> S3: Downloading 44285952-44351487 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> S3: Downloading 44351488-44482559 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> S3: Downloading 44482560-44744703 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> S3: Downloading 44744704-45268991 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> S3: Downloading 45268992-46317567 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> S3: Downloading 46317568-48414719 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=0,1723,982x2143 Dst=0,2944,1568x2944
> WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=915,1757,1191x2143 Dst=1568,2944,1568x2944
> .WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=2038,1790,1191x2143 Dst=3136,2944,1568x2944
> WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=3162,1823,1191x2143 Dst=4704,2944,1568x2944
> WARP: Using 2 threads
> S3: Downloading 48414720-50511871 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=4285,1857,1191x2142 Dst=6272,2944,1568x2944
> WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=5409,1890,1191x2142 Dst=7840,2944,1568x2944
> .WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=6533,1923,1190x2142 Dst=9408,2944,1568x2944
> WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=7656,1956,744x2142 Dst=10976,2944,1568x2944
> WARP: Using 2 threads
> S3: Downloading 50511872-60030975 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> S3: Downloading 60030976-69550079 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> S3: Downloading 102858752-102875135 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> S3: Downloading 102793216-102809599 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=0,3828,920x2143 Dst=0,5888,1568x2944
> WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=852,3861,1191x2143 Dst=1568,5888,1568x2944
> WARP: Using 2 threads
> S3: Downloading 69550080-69926911 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=1976,3895,1191x2142 Dst=3136,5888,1568x2944
> 10WARP: Using 2 threads
> S3: Downloading 69926912-70680575 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=3100,3928,1190x2143 Dst=4704,5888,1568x2944
> WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=4223,3961,1191x2143 Dst=6272,5888,1568x2944
> WARP: Using 2 threads
> S3: Downloading 70680576-72187903 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=5347,3994,1191x2143 Dst=7840,5888,1568x2944
> WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=6471,4027,1190x2143 Dst=9408,5888,1568x2944
> .WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=7594,4060,806x2143 Dst=10976,5888,1568x2944
> WARP: Using 2 threads
> S3: Downloading 72187904-82362367 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> S3: Downloading 82362368-92536831 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> S3: Downloading 102858752-102891519 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> S3: Downloading 102793216-102825983 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=0,5932,857x2144 Dst=0,8832,1568x2944
> WARP: Using 2 threads
> S3: Downloading 92536832-92782591 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=790,5966,1191x2143 Dst=1568,8832,1568x2944
> WARP: Using 2 threads
> S3: Downloading 92782592-93274111 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=1913,5999,1192x2143 Dst=3136,8832,1568x2944
> .WARP: Using 2 threads
> S3: Downloading 93274112-94257151 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=3037,6032,1191x2143 Dst=4704,8832,1568x2944
> WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=4161,6066,1191x2143 Dst=6272,8832,1568x2944
> WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=5285,6099,1191x2143 Dst=7840,8832,1568x2944
> WARP: Using 2 threads
> S3: Downloading 94257152-96223231 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=6409,6132,1190x2143 Dst=9408,8832,1568x2944
> .WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=7533,6165,867x2143 Dst=10976,8832,1568x2944
> WARP: Using 2 threads
> S3: Downloading 96223232-101171199 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=0,8037,795x2144 Dst=0,11776,1568x2944
> WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=727,8071,1191x2143 Dst=1568,11776,1568x2944
> WARP: Using 2 threads
> S3: Downloading 101171200-102793215 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=1851,8104,1191x2143 Dst=3136,11776,1568x2944
> WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=2975,8137,1191x2144 Dst=4704,11776,1568x2944
> 20WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=4099,8170,1191x2144 Dst=6272,11776,1568x2944
> WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=5223,8204,1191x2143 Dst=7840,11776,1568x2944
> WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=6347,8237,1191x2143 Dst=9408,11776,1568x2944
> WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=7471,8270,929x2143 Dst=10976,11776,1568x2944
> .WARP: Using 2 threads
> S3: Downloading 102891520-102907903 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> S3: Downloading 102825984-102842367 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=0,10142,732x2144 Dst=0,14720,1568x2944
> WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=665,10176,1191x2143 Dst=1568,14720,1568x2944
> WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=1789,10209,1191x2144 Dst=3136,14720,1568x2944
> WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=2913,10242,1191x2144 Dst=4704,14720,1568x2944
> .WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=4037,10276,1191x2143 Dst=6272,14720,1568x2944
> WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=5161,10309,1191x2143 Dst=7840,14720,1568x2944
> WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=6285,10342,1191x2143 Dst=9408,14720,1568x2944
> WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=7409,10375,991x2143 Dst=10976,14720,1568x2944
> .WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=0,12248,670x2143 Dst=0,17664,1568x2944
> WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=602,12281,1192x2144 Dst=1568,17664,1568x2944
> WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=1726,12314,1192x2144 Dst=3136,17664,1568x2944
> WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=2850,12348,1192x2143 Dst=4704,17664,1568x2944
> WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=3975,12381,1191x2144 Dst=6272,17664,1568x2944
> 30WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=5099,12414,1191x2144 Dst=7840,17664,1568x2944
> WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=6223,12447,1191x2144 Dst=9408,17664,1568x2944
> WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=7347,12480,1053x2144 Dst=10976,17664,1568x2944
> WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=0,14353,607x1091 Dst=0,20608,1568x1472
> .WARP: Using 2 threads
> S3: Downloading 102907904-102911196 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> S3: Downloading 102842368-102858751 (
> https://my-bucket.s3.amazonaws.com/dtm/test.tif)...
> S3: Got response_code=206
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=540,14386,1191x2014 Dst=1568,20608,1568x2944
> WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=1664,14420,1191x1980 Dst=3136,20608,1568x2944
> WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=2788,14453,1192x1947 Dst=4704,20608,1568x2944
> WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=3912,14486,1192x1914 Dst=6272,20608,1568x2944
> WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=5037,14520,1191x1880 Dst=7840,20608,1568x2944
> .WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=6161,14553,1191x1847 Dst=9408,20608,1568x2944
> WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=7285,14586,1115x1814 Dst=10976,20608,1568x2944
> WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=0,15421,14x72 Dst=686,22080,98x92
> WARP: Using 1 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=0,15423,576x977 Dst=784,22080,784x1472
> WARP: Using 2 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=0,15486,12x73 Dst=686,22172,98x92
> WARP: Using 1 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=0,15552,10x73 Dst=686,22264,98x92
> WARP: Using 1 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=0,15618,8x73 Dst=686,22356,98x92
> WARP: Using 1 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=0,15684,6x73 Dst=686,22448,98x92
> WARP: Using 1 threads
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
> Src=0,15750,4x72 Dst=686,22540,98x92
> WARP: Using 1 threads
> .COG: Reprojecting source dataset: end
> GTiff: ScanDirectories()
> S3: GetFileSize(
> https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.msk)=0
>  response_code=404
> S3: GetFileSize(
> https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.MSK)=0
>  response_code=404
> GDAL: GDALDefaultOverviews::OverviewScan()
> S3: GetFileSize(
> https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.ovr)=0
>  response_code=404
> S3: GetFileSize(
> https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.OVR)=0
>  response_code=404
> S3: GetFileSize(
> https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.aux)=0
>  response_code=404
> S3: GetFileSize(
> https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.AUX)=0
>  response_code=404
> S3: GetFileSize(
> https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.aux)=0
>  response_code=404
> S3: GetFileSize(
> https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.AUX)=0
>  response_code=404
> COG: Generating overviews of the imagery: start
> S3: GetFileSize(
> https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.ovr.tmp)=0
>  response_code=404
> S3: /vsis3/my-bucket/test_batch_output/test.tif.ovr.tmp is not a object
> GTiff: File being created as a BigTIFF.
> ERROR 1: Only read-only mode is supported for /vsicurl
> COG: Generating overviews of the imagery: end
> S3: UploadId:
> Rok8InTcYwFMoHYA3kVAwlLDycIxreczdLSODBpLA5exAshfS46ZclK.3.2.a_EAAY3vhoJQnc7DsVk9X3pzsRetSAZ.oc39kDwRe755ftroj5rH9BRXLPaNCcnuTGZP
> S3: Etag for part 1 is "52647419376831771b40964bad80d86e"
> S3: Etag for part 2 is "4bea862d0878bbe1dbe717aa8cea48c8"
> S3: Etag for part 3 is "5ba3c9e4ea924f06cdc2ee614e5627c0"
> S3: Etag for part 4 is "07e53a2e07a2575d1b2d89cf8040ebe8"
> GDAL:
> GDALClose(/vsis3/my-bucket/test_batch_output/test.tif.warped.tif.tmp,
> this=0x55a288913060)
> S3: GetFileSize(
> https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp)=207439264
>  response_code=206
> GDAL: GDALClose(/vsis3/my-bucket/dtm/test.tif, this=0x55a2882e8570)
> GDAL: In GDALDestroy - unloading GDAL shared library.
>

On Tue, 16 Aug 2022 at 12:11, William Roper <roper.william at gmail.com> wrote:

> Thanks for the reply Even.
>
> I've run with "--debug on" and I think the interesting bit is:
>
> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
>> Src=0,15684,6x73 Dst=686,22448,98x92
>> WARP: Using 1 threads
>> GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte()
>> Src=0,15750,4x72 Dst=686,22540,98x92
>> WARP: Using 1 threads
>> .COG: Reprojecting source dataset: end
>> GTiff: ScanDirectories()
>> S3: GetFileSize(
>> https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.msk)=0
>>  response_code=404
>> S3: GetFileSize(
>> https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.MSK)=0
>>  response_code=404
>> GDAL: GDALDefaultOverviews::OverviewScan()
>> S3: GetFileSize(
>> https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.ovr)=0
>>  response_code=404
>> S3: GetFileSize(
>> https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.OVR)=0
>>  response_code=404
>> S3: GetFileSize(
>> https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.aux)=0
>>  response_code=404
>> S3: GetFileSize(
>> https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.AUX)=0
>>  response_code=404
>> S3: GetFileSize(
>> https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.aux)=0
>>  response_code=404
>> S3: GetFileSize(
>> https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.AUX)=0
>>  response_code=404
>> COG: Generating overviews of the imagery: start
>> S3: GetFileSize(
>> https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.ovr.tmp)=0
>>  response_code=404
>> S3: /vsis3/my-bucket/test_batch_output/test.tif.ovr.tmp is not a object
>> GTiff: File being created as a BigTIFF.
>> ERROR 1: Only read-only mode is supported for /vsicurl
>> COG: Generating overviews of the imagery: end
>> S3: UploadId:
>> Rok8InTcYwFMoHYA3kVAwlLDycIxreczdLSODBpLA5exAshfS46ZclK.3.2.a_EAAY3vhoJQnc7DsVk9X3pzsRetSAZ.oc39kDwRe755ftroj5rH9BRXLPaNCcnuTGZP
>> S3: Etag for part 1 is "52647419376831771b40964bad80d86e"
>> S3: Etag for part 2 is "4bea862d0878bbe1dbe717aa8cea48c8"
>> S3: Etag for part 3 is "5ba3c9e4ea924f06cdc2ee614e5627c0"
>> S3: Etag for part 4 is "07e53a2e07a2575d1b2d89cf8040ebe8"
>> GDAL:
>> GDALClose(/vsis3/my-bucket/test_batch_output/test.tif.warped.tif.tmp,
>> this=0x55a288913060)
>> S3: GetFileSize(
>> https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp)=207439264
>>  response_code=206
>> GDAL: GDALClose(/vsis3/my-bucket/dtm/test.tif, this=0x55a2882e8570)
>> GDAL: In GDALDestroy - unloading GDAL shared library.
>>
>
>  It looks to me like gdal attempted to write a lot of the tmp files to s3,
> rather than locally? the command I used is:
>
> gdal_translate --config AWS_SECRET_ACCESS_KEY <hidden> --config
>> AWS_ACCESS_KEY_ID <hidden> --config CPL_VSIL_USE_TEMP_FILE_FOR_RANDOM_WRITE
>> YES --config CPL_TMPDIR /tmp -of COG -co
>> TILING_SCHEME\=GoogleMapsCompatible -co COMPRESS\=DEFLATE -co
>> NUM_THREADS\=ALL_CPUS --debug on /vsis3/my-bucket/dtm/test.tif /vsis3/
>> my-bucket/test_batch_output/test.tif
>>
>
> This is on Ubuntu 22.04 using gdal 3.4.1
>
> The iam user has full control over the bucket, checked with the "aws s3api
> get-bucket-acl" command, and I can read and write from the bucket using the
> "aws s3" command, so if it's an s3 permissions issue, then I'm failing to
> give gdal the necessary info - I understood AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID
> were sufficient.
>
> Thanks again for taking a look!
> Will
>
>
>
> On Tue, 16 Aug 2022 at 08:18, Even Rouault <even.rouault at spatialys.com>
> wrote:
>
>> William,
>>
>> can you add "--debug on" and retry to see if more hints are emitted about
>> what's going on.
>>
>> Even
>> Le 15/08/2022 à 11:45, William Roper a écrit :
>>
>> Hi list,
>>
>> I'm trying to convert a GeoTiff on s3 into a COG, that will also live on
>> s3. I'm working on EC2 instances, or fargate containers, because local
>> download and upload speeds are a bit slow.
>>
>> gdalwarp \
>>   --config AWS_SECRET_ACCESS_KEY <my-secret-key> \
>>   --config AWS_ACCESS_KEY_ID <my-access-key> \
>>   --config CPL_VSIL_USE_TEMP_FILE_FOR_RANDOM_WRITE YES \
>>   --config CPL_TMPDIR /tmp \
>>   -of COG
>>   -co TILING_SCHEME=GoogleMapsCompatible
>>   -co COMPRESS=DEFLATE
>>   -co NUM_THREADS=ALL_CPUS
>>   /vsis3/my-bucket/original-old-geotif.tif
>>   /vsis3/my-bucket/shiny-new-cog.tif
>>
>>
>> And get the following error:
>>
>> 0...10...20...30...ERROR 1: Only read-only mode is supported for /vsicurl
>>
>>
>> I get the same issue running in a docker container (osgeo/gdal:latest) or
>> on an ubuntu EC2 instance with gdal-bin installed with apt  (version
>> 3.4.1).
>>
>> I can work around it by writing the COG to the machine, and then
>> uploading to s3, but was wondered what I was doing wrong/not understanding.
>>
>> Many thanks
>> Will
>>
>>
>>
>> _______________________________________________
>> gdal-dev mailing listgdal-dev at lists.osgeo.orghttps://lists.osgeo.org/mailman/listinfo/gdal-dev
>>
>> -- http://www.spatialys.com
>> My software is free, but my time generally not.
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20220816/f2ffa88a/attachment-0001.htm>