<div dir="ltr"><div dir="ltr"><div>Thanks for the reply Even.</div><div><br></div><div>I've run with "--debug on" and I think the interesting bit is:</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><span style="font-family:monospace">GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=0,15684,6x73 Dst=686,22448,98x92<br>WARP: Using 1 threads<br>GDAL: GDALWarpKernel()::GWKCubicNoMasksOrDstDensityOnlyByte() Src=0,15750,4x72 Dst=686,22540,98x92<br>WARP: Using 1 threads<br>.COG: Reprojecting source dataset: end<br>GTiff: ScanDirectories()<br>S3: GetFileSize(<a href="https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.msk)=0">https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.msk)=0</a> response_code=404<br>S3: GetFileSize(<a href="https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.MSK)=0">https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.MSK)=0</a> response_code=404<br>GDAL: GDALDefaultOverviews::OverviewScan()<br>S3: GetFileSize(<a href="https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.ovr)=0">https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.ovr)=0</a> response_code=404<br>S3: GetFileSize(<a href="https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.OVR)=0">https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.OVR)=0</a> response_code=404<br>S3: GetFileSize(<a href="https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.aux)=0">https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.aux)=0</a> response_code=404<br>S3: GetFileSize(<a href="https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.AUX)=0">https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.AUX)=0</a> response_code=404<br>S3: GetFileSize(<a href="https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.aux)=0">https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.aux)=0</a> response_code=404<br>S3: GetFileSize(<a href="https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.AUX)=0">https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp.AUX)=0</a> response_code=404<br>COG: Generating overviews of the imagery: start<br>S3: GetFileSize(<a href="https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.ovr.tmp)=0">https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.ovr.tmp)=0</a> response_code=404<br>S3: /vsis3/my-bucket/test_batch_output/test.tif.ovr.tmp is not a object<br>GTiff: File being created as a BigTIFF.<br>ERROR 1: Only read-only mode is supported for /vsicurl<br>COG: Generating overviews of the imagery: end<br>S3: UploadId: Rok8InTcYwFMoHYA3kVAwlLDycIxreczdLSODBpLA5exAshfS46ZclK.3.2.a_EAAY3vhoJQnc7DsVk9X3pzsRetSAZ.oc39kDwRe755ftroj5rH9BRXLPaNCcnuTGZP<br>S3: Etag for part 1 is "52647419376831771b40964bad80d86e"<br>S3: Etag for part 2 is "4bea862d0878bbe1dbe717aa8cea48c8"<br>S3: Etag for part 3 is "5ba3c9e4ea924f06cdc2ee614e5627c0"<br>S3: Etag for part 4 is "07e53a2e07a2575d1b2d89cf8040ebe8"<br>GDAL: GDALClose(/vsis3/my-bucket/test_batch_output/test.tif.warped.tif.tmp, this=0x55a288913060)<br>S3: GetFileSize(<a href="https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp)=207439264">https://my-bucket.s3.amazonaws.com/test_batch_output/test.tif.warped.tif.tmp)=207439264</a> response_code=206<br>GDAL: GDALClose(/vsis3/my-bucket/dtm/test.tif, this=0x55a2882e8570)<br>GDAL: In GDALDestroy - unloading GDAL shared library.</span><br></div></blockquote><div><br></div><div> It looks to me like gdal attempted to write a lot of the tmp files to s3, rather than locally? the command I used is:</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><span style="font-family:monospace">gdal_translate --config AWS_SECRET_ACCESS_KEY <hidden> --config AWS_ACCESS_KEY_ID <hidden> --config CPL_VSIL_USE_TEMP_FILE_FOR_RANDOM_WRITE YES --config CPL_TMPDIR /tmp -of COG -co TILING_SCHEME\=GoogleMapsCompatible -co COMPRESS\=DEFLATE -co NUM_THREADS\=ALL_CPUS --debug on /vsis3/<span style="font-family:monospace">my-bucket</span>/<span style="font-family:monospace">dtm/test</span>.tif /vsis3/<span style="font-family:monospace">my-bucket</span>/test_batch_output/test.tif</span><br></div></blockquote><div><br></div><div>This is on Ubuntu 22.04 using gdal 3.4.1</div><div><br></div><div>The iam user has full control over the bucket, checked with the "aws s3api get-bucket-acl" command, and I can read and write from the bucket using the "aws s3" command, so if it's an s3 permissions issue, then I'm failing to give gdal the necessary info - I understood <span style="font-family:monospace">AWS_SECRET_ACCESS_KEY <span style="font-family:arial,sans-serif">and</span> <span style="font-family:monospace">AWS_ACCESS_KEY_ID <font face="arial,sans-serif">were sufficient.</font></span></span></div><div><span style="font-family:monospace"><span style="font-family:monospace"><br></span></span></div>Thanks again for taking a look! <br></div><div dir="ltr"><div><span style="font-family:monospace"><span style="font-family:monospace"></span></span></div><div><span style="font-family:monospace"><span style="font-family:monospace"><font face="arial,sans-serif">Will</font><br></span></span></div><div> </div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, 16 Aug 2022 at 08:18, Even Rouault <<a href="mailto:even.rouault@spatialys.com">even.rouault@spatialys.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>William,</p>
<p>can you add "--debug on" and retry to see if more hints are
emitted about what's going on.</p>
<p>Even<br>
</p>
<div>Le 15/08/2022 à 11:45, William Roper a
écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Hi list, <br>
</div>
<div><br>
</div>
<div>I'm trying to convert a GeoTiff on s3 into a COG, that will
also live on s3. I'm working on EC2 instances, or fargate
containers, because local download and upload speeds are a bit
slow. <br>
</div>
<div><br>
</div>
<div>
<pre>gdalwarp \
--config AWS_SECRET_ACCESS_KEY <my-secret-key> \
--config AWS_ACCESS_KEY_ID <my-access-key> \
--config CPL_VSIL_USE_TEMP_FILE_FOR_RANDOM_WRITE YES \
--config CPL_TMPDIR /tmp \
-of COG
-co TILING_SCHEME=GoogleMapsCompatible
-co COMPRESS=DEFLATE
-co NUM_THREADS=ALL_CPUS
/vsis3/my-bucket/original-old-geotif.tif
/vsis3/my-bucket/shiny-new-cog.tif</pre>
</div>
<div><br>
</div>
<div>And get the following error: <br>
</div>
<div>
<pre>0...10...20...30...ERROR 1: Only read-only mode is supported for /vsicurl</pre>
</div>
<div><br>
</div>
<div>I get the same issue running in a docker container
(osgeo/gdal:latest) or on an ubuntu EC2 instance with gdal-bin
installed with apt (version 3.4.1). <br>
</div>
<div><br>
</div>
<div>I can work around it by writing the COG to the machine, and
then uploading to s3, but was wondered what I was doing
wrong/not understanding.<br>
</div>
<div><br>
</div>
<div>Many thanks</div>
<div>Will<br>
</div>
<div><br>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
gdal-dev mailing list
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a>
</pre>
</blockquote>
<pre cols="72">--
<a href="http://www.spatialys.com" target="_blank">http://www.spatialys.com</a>
My software is free, but my time generally not.</pre>
</div>
</blockquote></div></div>