<div dir="ltr"><div>I've included the rest of the debug below. It starts hitting 404s (I guess because the objects don't exist) almost straight away:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><span style="font-family:monospace">S3: GetFileSize(<a href="https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp)=0">https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp)=0</a> response_code=404<br>S3: /vsis3/my-bucket/test_batch_output/test.tif.warped.tif.tmp is not a object</span></div></blockquote><div><br></div><div>Complete debug output:</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><span style="font-family:monospace">gdal_translate --config CPL_VSIL_USE_TEMP_FILE_FOR_RANDOM_WRITE YES --config CPL_TMPDIR /tmp -of COG -co TILING_SCHEME\=GoogleMapsCompatible -co COMPRESS\=DEFLATE -co NUM_THREADS\=ALL_CPUS --debug on /vsis3/my-bucket/dtm/test.tif /vsis3/my-bucket/test_batch_output/test.tif<br>HTTP: libcurl/7.81.0 OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.12<br>S3: Downloading 0-16383 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>S3: Downloading 102776832-102793215 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>S3: Downloading 102907904-102911196 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>GDAL: GDALOpen(/vsis3/my-bucket/dtm/test.tif, this=0x55a2882e8570) succeeds as GTiff.<br>Input file size is 8400, 16400<br>S3: GetFileSize(<a href="https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif)=0">https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif)=0</a> response_code=404<br>GTiff: ScanDirectories()<br>GDAL: GDALDefaultOverviews::OverviewScan()<br>GDAL: Computing area of interest: -4.12725, 53.1714, -4.09403, 53.2088<br>COG: Using ZOOM_LEVEL 19<br>COG: Reprojecting source dataset: start<br>S3: GetFileSize(<a href="https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp)=0">https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp)=0</a> response_code=404<br>S3: /vsis3/my-bucket/test_batch_output/test.tif.warped.tif.tmp is not a object<br>GDAL: Computing area of interest: -4.12725, 53.1714, -4.09403, 53.2088<br>GDAL: GDALDriver::Create(GTiff,/vsis3/my-bucket/test_batch_output/test.tif.warped.tif.tmp,12544,23552,4,Byte,0x55a2886c5a10)<br>0WARP: Copying metadata from first source to destination dataset<br>GDALWARP: Defining SKIP_NOSOURCE=YES<br>WARP: SetAlphaMax: AlphaMax not set.<br>S3: Downloading 102842368-102858751 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>S3: Downloading 16384-10485759 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>GDAL: GDAL_CACHEMAX = 392 MB<br>S3: Downloading 10485760-20955135 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>S3: Downloading 102842368-102858751 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>S3: Downloading 102776832-102793215 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>S3: Downloading 20955136-20971519 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>S3: Downloading 20971520-21004287 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>S3: Downloading 21004288-21069823 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>S3: Downloading 21069824-21200895 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>S3: Downloading 21200896-21463039 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=0,0,1045x1762 Dst=0,0,1568x2944<br>WARP: Using 2 threads<br>S3: Downloading 21463040-21987327 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=977,0,1191x1795 Dst=1568,0,1568x2944<br>WARP: Using 2 threads<br>S3: Downloading 21987328-23035903 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=2101,0,1190x1828 Dst=3136,0,1568x2944<br>WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=3224,0,1191x1862 Dst=4704,0,1568x2944<br>WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=4348,0,1190x1895 Dst=6272,0,1568x2944<br>WARP: Using 2 threads<br>S3: Downloading 23035904-25133055 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=5471,0,1190x1928 Dst=7840,0,1568x2944<br>.WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=6595,0,1190x1961 Dst=9408,0,1568x2944<br>WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=7718,0,682x1994 Dst=10976,0,1568x2944<br>WARP: Using 2 threads<br>S3: Downloading 25133056-34684927 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>S3: Downloading 34684928-44236799 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>S3: Downloading 102842368-102875135 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>S3: Downloading 102776832-102809599 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>S3: Downloading 44236800-44253183 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>S3: Downloading 44253184-44285951 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>S3: Downloading 44285952-44351487 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>S3: Downloading 44351488-44482559 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>S3: Downloading 44482560-44744703 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>S3: Downloading 44744704-45268991 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>S3: Downloading 45268992-46317567 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>S3: Downloading 46317568-48414719 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=0,1723,982x2143 Dst=0,2944,1568x2944<br>WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=915,1757,1191x2143 Dst=1568,2944,1568x2944<br>.WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=2038,1790,1191x2143 Dst=3136,2944,1568x2944<br>WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=3162,1823,1191x2143 Dst=4704,2944,1568x2944<br>WARP: Using 2 threads<br>S3: Downloading 48414720-50511871 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=4285,1857,1191x2142 Dst=6272,2944,1568x2944<br>WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=5409,1890,1191x2142 Dst=7840,2944,1568x2944<br>.WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=6533,1923,1190x2142 Dst=9408,2944,1568x2944<br>WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=7656,1956,744x2142 Dst=10976,2944,1568x2944<br>WARP: Using 2 threads<br>S3: Downloading 50511872-60030975 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>S3: Downloading 60030976-69550079 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>S3: Downloading 102858752-102875135 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>S3: Downloading 102793216-102809599 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=0,3828,920x2143 Dst=0,5888,1568x2944<br>WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=852,3861,1191x2143 Dst=1568,5888,1568x2944<br>WARP: Using 2 threads<br>S3: Downloading 69550080-69926911 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=1976,3895,1191x2142 Dst=3136,5888,1568x2944<br>10WARP: Using 2 threads<br>S3: Downloading 69926912-70680575 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=3100,3928,1190x2143 Dst=4704,5888,1568x2944<br>WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=4223,3961,1191x2143 Dst=6272,5888,1568x2944<br>WARP: Using 2 threads<br>S3: Downloading 70680576-72187903 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=5347,3994,1191x2143 Dst=7840,5888,1568x2944<br>WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=6471,4027,1190x2143 Dst=9408,5888,1568x2944<br>.WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=7594,4060,806x2143 Dst=10976,5888,1568x2944<br>WARP: Using 2 threads<br>S3: Downloading 72187904-82362367 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>S3: Downloading 82362368-92536831 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>S3: Downloading 102858752-102891519 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>S3: Downloading 102793216-102825983 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=0,5932,857x2144 Dst=0,8832,1568x2944<br>WARP: Using 2 threads<br>S3: Downloading 92536832-92782591 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=790,5966,1191x2143 Dst=1568,8832,1568x2944<br>WARP: Using 2 threads<br>S3: Downloading 92782592-93274111 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=1913,5999,1192x2143 Dst=3136,8832,1568x2944<br>.WARP: Using 2 threads<br>S3: Downloading 93274112-94257151 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=3037,6032,1191x2143 Dst=4704,8832,1568x2944<br>WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=4161,6066,1191x2143 Dst=6272,8832,1568x2944<br>WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=5285,6099,1191x2143 Dst=7840,8832,1568x2944<br>WARP: Using 2 threads<br>S3: Downloading 94257152-96223231 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=6409,6132,1190x2143 Dst=9408,8832,1568x2944<br>.WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=7533,6165,867x2143 Dst=10976,8832,1568x2944<br>WARP: Using 2 threads<br>S3: Downloading 96223232-101171199 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=0,8037,795x2144 Dst=0,11776,1568x2944<br>WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=727,8071,1191x2143 Dst=1568,11776,1568x2944<br>WARP: Using 2 threads<br>S3: Downloading 101171200-102793215 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=1851,8104,1191x2143 Dst=3136,11776,1568x2944<br>WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=2975,8137,1191x2144 Dst=4704,11776,1568x2944<br>20WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=4099,8170,1191x2144 Dst=6272,11776,1568x2944<br>WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=5223,8204,1191x2143 Dst=7840,11776,1568x2944<br>WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=6347,8237,1191x2143 Dst=9408,11776,1568x2944<br>WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=7471,8270,929x2143 Dst=10976,11776,1568x2944<br>.WARP: Using 2 threads<br>S3: Downloading 102891520-102907903 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>S3: Downloading 102825984-102842367 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=0,10142,732x2144 Dst=0,14720,1568x2944<br>WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=665,10176,1191x2143 Dst=1568,14720,1568x2944<br>WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=1789,10209,1191x2144 Dst=3136,14720,1568x2944<br>WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=2913,10242,1191x2144 Dst=4704,14720,1568x2944<br>.WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=4037,10276,1191x2143 Dst=6272,14720,1568x2944<br>WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=5161,10309,1191x2143 Dst=7840,14720,1568x2944<br>WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=6285,10342,1191x2143 Dst=9408,14720,1568x2944<br>WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=7409,10375,991x2143 Dst=10976,14720,1568x2944<br>.WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=0,12248,670x2143 Dst=0,17664,1568x2944<br>WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=602,12281,1192x2144 Dst=1568,17664,1568x2944<br>WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=1726,12314,1192x2144 Dst=3136,17664,1568x2944<br>WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=2850,12348,1192x2143 Dst=4704,17664,1568x2944<br>WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=3975,12381,1191x2144 Dst=6272,17664,1568x2944<br>30WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=5099,12414,1191x2144 Dst=7840,17664,1568x2944<br>WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=6223,12447,1191x2144 Dst=9408,17664,1568x2944<br>WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=7347,12480,1053x2144 Dst=10976,17664,1568x2944<br>WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=0,14353,607x1091 Dst=0,20608,1568x1472<br>.WARP: Using 2 threads<br>S3: Downloading 102907904-102911196 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>S3: Downloading 102842368-102858751 (<a href="https://my-bucket.s3.amazonaws.com/dtm/test.tif).">https://my-bucket.s3.amazonaws.com/dtm/test.tif).</a>..<br>S3: Got response_code=206<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=540,14386,1191x2014 Dst=1568,20608,1568x2944<br>WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=1664,14420,1191x1980 Dst=3136,20608,1568x2944<br>WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=2788,14453,1192x1947 Dst=4704,20608,1568x2944<br>WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=3912,14486,1192x1914 Dst=6272,20608,1568x2944<br>WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=5037,14520,1191x1880 Dst=7840,20608,1568x2944<br>.WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=6161,14553,1191x1847 Dst=9408,20608,1568x2944<br>WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=7285,14586,1115x1814 Dst=10976,20608,1568x2944<br>WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=0,15421,14x72 Dst=686,22080,98x92<br>WARP: Using 1 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=0,15423,576x977 Dst=784,22080,784x1472<br>WARP: Using 2 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=0,15486,12x73 Dst=686,22172,98x92<br>WARP: Using 1 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=0,15552,10x73 Dst=686,22264,98x92<br>WARP: Using 1 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=0,15618,8x73 Dst=686,22356,98x92<br>WARP: Using 1 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=0,15684,6x73 Dst=686,22448,98x92<br>WARP: Using 1 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=0,15750,4x72 Dst=686,22540,98x92<br>WARP: Using 1 threads<br>.COG: Reprojecting source dataset: end<br>GTiff: ScanDirectories()<br>S3: GetFileSize(<a href="https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.msk)=0">https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.msk)=0</a> response_code=404<br>S3: GetFileSize(<a href="https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.MSK)=0">https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.MSK)=0</a> response_code=404<br>GDAL: GDALDefaultOverviews::OverviewScan()<br>S3: GetFileSize(<a href="https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.ovr)=0">https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.ovr)=0</a> response_code=404<br>S3: GetFileSize(<a href="https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.OVR)=0">https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.OVR)=0</a> response_code=404<br>S3: GetFileSize(<a href="https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.aux)=0">https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.aux)=0</a> response_code=404<br>S3: GetFileSize(<a href="https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.AUX)=0">https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.AUX)=0</a> response_code=404<br>S3: GetFileSize(<a href="https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.aux)=0">https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.aux)=0</a> response_code=404<br>S3: GetFileSize(<a href="https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.AUX)=0">https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.AUX)=0</a> response_code=404<br>COG: Generating overviews of the imagery: start<br>S3: GetFileSize(<a href="https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.ovr.tmp)=0">https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.ovr.tmp)=0</a> response_code=404<br>S3: /vsis3/my-bucket/test_batch_output/test.tif.ovr.tmp is not a object<br>GTiff: File being created as a BigTIFF.<br>ERROR 1: Only read-only mode is supported for /vsicurl<br>COG: Generating overviews of the imagery: end<br>S3: UploadId: Rok8InTcYwFMoHYA3kVAwlLDycIxreczdLSODBpLA5exAshfS46ZclK.3.2.a_EAAY3vhoJQnc7DsVk9X3pzsRetSAZ.oc39kDwRe755ftroj5rH9BRXLPaNCcnuTGZP<br>S3: Etag for part 1 is "52647419376831771b40964bad80d86e"<br>S3: Etag for part 2 is "4bea862d0878bbe1dbe717aa8cea48c8"<br>S3: Etag for part 3 is "5ba3c9e4ea924f06cdc2ee614e5627c0"<br>S3: Etag for part 4 is "07e53a2e07a2575d1b2d89cf8040ebe8"<br>GDAL: GDALClose(/vsis3/my-bucket/test_batch_output/test.tif.warped.tif.tmp, this=0x55a288913060)<br>S3: GetFileSize(<a href="https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp)=207439264">https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp)=207439264</a> response_code=206<br>GDAL: GDALClose(/vsis3/my-bucket/dtm/test.tif, this=0x55a2882e8570)<br>GDAL: In GDALDestroy - unloading GDAL shared library.</span><br></div></blockquote></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, 16 Aug 2022 at 12:11, William Roper <<a href="mailto:roper.william@gmail.com">roper.william@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div>Thanks for the reply Even.</div><div><br></div><div>I've run with "--debug on" and I think the interesting bit is:</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><span style="font-family:monospace">GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=0,15684,6x73 Dst=686,22448,98x92<br>WARP: Using 1 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=0,15750,4x72 Dst=686,22540,98x92<br>WARP: Using 1 threads<br>.COG: Reprojecting source dataset: end<br>GTiff: ScanDirectories()<br>S3: GetFileSize(<a href="https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.msk)=0" target="_blank">https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.msk)=0</a> response_code=404<br>S3: GetFileSize(<a href="https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.MSK)=0" target="_blank">https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.MSK)=0</a> response_code=404<br>GDAL: GDALDefaultOverviews::OverviewScan()<br>S3: GetFileSize(<a href="https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.ovr)=0" target="_blank">https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.ovr)=0</a> response_code=404<br>S3: GetFileSize(<a href="https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.OVR)=0" target="_blank">https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.OVR)=0</a> response_code=404<br>S3: GetFileSize(<a href="https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.aux)=0" target="_blank">https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.aux)=0</a> response_code=404<br>S3: GetFileSize(<a href="https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.AUX)=0" target="_blank">https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.AUX)=0</a> response_code=404<br>S3: GetFileSize(<a href="https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.aux)=0" target="_blank">https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.aux)=0</a> response_code=404<br>S3: GetFileSize(<a href="https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.AUX)=0" target="_blank">https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.AUX)=0</a> response_code=404<br>COG: Generating overviews of the imagery: start<br>S3: GetFileSize(<a href="https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.ovr.tmp)=0" target="_blank">https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.ovr.tmp)=0</a> response_code=404<br>S3: /vsis3/my-bucket/test_batch_output/test.tif.ovr.tmp is not a object<br>GTiff: File being created as a BigTIFF.<br>ERROR 1: Only read-only mode is supported for /vsicurl<br>COG: Generating overviews of the imagery: end<br>S3: UploadId: Rok8InTcYwFMoHYA3kVAwlLDycIxreczdLSODBpLA5exAshfS46ZclK.3.2.a_EAAY3vhoJQnc7DsVk9X3pzsRetSAZ.oc39kDwRe755ftroj5rH9BRXLPaNCcnuTGZP<br>S3: Etag for part 1 is "52647419376831771b40964bad80d86e"<br>S3: Etag for part 2 is "4bea862d0878bbe1dbe717aa8cea48c8"<br>S3: Etag for part 3 is "5ba3c9e4ea924f06cdc2ee614e5627c0"<br>S3: Etag for part 4 is "07e53a2e07a2575d1b2d89cf8040ebe8"<br>GDAL: GDALClose(/vsis3/my-bucket/test_batch_output/test.tif.warped.tif.tmp, this=0x55a288913060)<br>S3: GetFileSize(<a href="https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp)=207439264" target="_blank">https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp)=207439264</a> response_code=206<br>GDAL: GDALClose(/vsis3/my-bucket/dtm/test.tif, this=0x55a2882e8570)<br>GDAL: In GDALDestroy - unloading GDAL shared library.</span><br></div></blockquote><div><br></div><div> It looks to me like gdal attempted to write a lot of the tmp files to s3, rather than locally? the command I used is:</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><span style="font-family:monospace">gdal_translate --config AWS_SECRET_ACCESS_KEY <hidden> --config AWS_ACCESS_KEY_ID <hidden> --config CPL_VSIL_USE_TEMP_FILE_FOR_RANDOM_WRITE YES --config CPL_TMPDIR /tmp -of COG -co TILING_SCHEME\=GoogleMapsCompatible -co COMPRESS\=DEFLATE -co NUM_THREADS\=ALL_CPUS --debug on /vsis3/<span style="font-family:monospace">my-bucket</span>/<span style="font-family:monospace">dtm/test</span>.tif /vsis3/<span style="font-family:monospace">my-bucket</span>/test_batch_output/test.tif</span><br></div></blockquote><div><br></div><div>This is on Ubuntu 22.04 using gdal 3.4.1</div><div><br></div><div>The iam user has full control over the bucket, checked with the "aws s3api get-bucket-acl" command, and I can read and write from the bucket using the "aws s3" command, so if it's an s3 permissions issue, then I'm failing to give gdal the necessary info - I understood <span style="font-family:monospace">AWS_SECRET_ACCESS_KEY <span style="font-family:arial,sans-serif">and</span> <span style="font-family:monospace">AWS_ACCESS_KEY_ID <font face="arial,sans-serif">were sufficient.</font></span></span></div><div><span style="font-family:monospace"><span style="font-family:monospace"><br></span></span></div>Thanks again for taking a look! <br></div><div dir="ltr"><div><span style="font-family:monospace"><span style="font-family:monospace"></span></span></div><div><span style="font-family:monospace"><span style="font-family:monospace"><font face="arial,sans-serif">Will</font><br></span></span></div><div> </div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, 16 Aug 2022 at 08:18, Even Rouault <<a href="mailto:even.rouault@spatialys.com" target="_blank">even.rouault@spatialys.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>William,</p>
<p>can you add "--debug on" and retry to see if more hints are
emitted about what's going on.</p>
<p>Even<br>
</p>
<div>Le 15/08/2022 à 11:45, William Roper a
écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Hi list, <br>
</div>
<div><br>
</div>
<div>I'm trying to convert a GeoTiff on s3 into a COG, that will
also live on s3. I'm working on EC2 instances, or fargate
containers, because local download and upload speeds are a bit
slow. <br>
</div>
<div><br>
</div>
<div>
<pre>gdalwarp \
--config AWS_SECRET_ACCESS_KEY <my-secret-key> \
--config AWS_ACCESS_KEY_ID <my-access-key> \
--config CPL_VSIL_USE_TEMP_FILE_FOR_RANDOM_WRITE YES \
--config CPL_TMPDIR /tmp \
-of COG
-co TILING_SCHEME=GoogleMapsCompatible
-co COMPRESS=DEFLATE
-co NUM_THREADS=ALL_CPUS
/vsis3/my-bucket/original-old-geotif.tif
/vsis3/my-bucket/shiny-new-cog.tif</pre>
</div>
<div><br>
</div>
<div>And get the following error: <br>
</div>
<div>
<pre>0...10...20...30...ERROR 1: Only read-only mode is supported for /vsicurl</pre>
</div>
<div><br>
</div>
<div>I get the same issue running in a docker container
(osgeo/gdal:latest) or on an ubuntu EC2 instance with gdal-bin
installed with apt (version 3.4.1). <br>
</div>
<div><br>
</div>
<div>I can work around it by writing the COG to the machine, and
then uploading to s3, but was wondered what I was doing
wrong/not understanding.<br>
</div>
<div><br>
</div>
<div>Many thanks</div>
<div>Will<br>
</div>
<div><br>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
gdal-dev mailing list
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a>
</pre>
</blockquote>
<pre cols="72">--
<a href="http://www.spatialys.com" target="_blank">http://www.spatialys.com</a>
My software is free, but my time generally not.</pre>
</div>
</blockquote></div></div>
</blockquote></div>