<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Tim,</p>
<p>wil be solved (in 3.11 only) per
<a class="moz-txt-link-freetext" href="https://github.com/OSGeo/gdal/pull/11915">https://github.com/OSGeo/gdal/pull/11915</a></p>
<p>In the meantime, check for gdal.Warp() return value: if None, an
error occurred</p>
<p>Even<br>
</p>
<div class="moz-cite-prefix">Le 01/03/2025 à 01:00, Tim Harris via
gdal-dev a écrit :<br>
</div>
<blockquote type="cite"
cite="mid:CAK3fJgVS+rmzya-NjxsMUguCSneBdHsjsoZNqQx=Lj-kpR=51A@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div>Hi, I'm having some trouble warping a VRT to a TIF where
the VRT and its constituent rasters are in S3. It stems from a
random read error, but the real problem is that the gdal.Warp
call didn't throw an exception when it was supposed to.</div>
<div><br>
</div>
<div>I saw there's this pretty old GitHub issue that may be
related, but I'm not sure:</div>
<div><a href="https://github.com/OSGeo/gdal/issues/3372"
moz-do-not-send="true" class="moz-txt-link-freetext">https://github.com/OSGeo/gdal/issues/3372</a></div>
<div><br>
</div>
<div>In my situation what really happened was a random /vsicurl
read failed... sort of like an issue I reported in December
that was fixed in GDAL 3.10.1:</div>
<div><a href="https://github.com/OSGeo/gdal/issues/11552"
moz-do-not-send="true" class="moz-txt-link-freetext">https://github.com/OSGeo/gdal/issues/11552</a></div>
<div><br>
</div>
<div>Here's a section of my log file (filename sanitized):</div>
<div><span style="font-family:monospace">ERROR 11: CURL error:
Empty reply from server<br>
0...10...20...30...40...50...60...70...80...90...100 - done.<br>
ERROR 4: `/vsis3/path/to/raster.tif' does not exist in the
file system, and is not recognized as a supported dataset
name.</span><br>
</div>
<div><br>
</div>
<div>Notice the CURL error but the progress still goes to 100%.
The warp completed successfully despite the read error.</div>
<div><br>
</div>
<div>I have a hopefully simple reproduce case here. The first
few shell commands will make two TIFs, put them in S3, then
make a VRT of those TIFs and also put that in S3. Then to
simulate the CURL error above, it just deletes one of the TIFs
so that the VRT references a non-existent file:</div>
<div><span style="font-family:monospace">gdal_create -of GTiff
-outsize 1000 1000 -bands 1 -a_srs EPSG:4326 -a_ullr 0 1 1 0
input1.tif<br>
gdal_create -of GTiff -outsize 1000 1000 -bands 1 -a_srs
EPSG:4326 -a_ullr 1 1 2 0 input2.tif<br>
aws s3 cp input1.tif s3://your-bucket/test/input1.tif<br>
aws s3 cp input2.tif s3://your-bucket/test/input2.tif<br>
gdalbuildvrt input.vrt /vsis3/your-bucket/test/input1.tif
/vsis3/your-bucket/test/input2.tif<br>
aws s3 cp input.vrt s3://your-bucket/test/input.vrt<br>
aws s3 rm s3://your-bucket/test/input2.tif</span></div>
<div><br>
</div>
<div>Then in Python:</div>
<div><span style="font-family:monospace">from osgeo import gdal<br>
gdal.UseExceptions()<br>
gdal.Warp("output.tif",
"/vsis3/tharris-bucket/test/input.vrt",
callback=gdal.TermProgress)</span></div>
<div><br>
</div>
<div>This throws a RuntimeError, as expected. But delete the
partial file and try with multithread=True:</div>
<div><span style="font-family:monospace">rm output.tif</span></div>
<div><span style="font-family:monospace">gdal.Warp("output.tif",
"/vsis3/tharris-bucket/test/input.vrt",
callback=gdal.TermProgress, multithread=True)</span></div>
<div><br>
</div>
<div>This logs the error, but doesn't throw an exception. The
end result is that I have a TIF that is missing the contents
of the file that suffered from that CURL read error.</div>
<div><br>
</div>
<div>For what it's worth this also happens with the command line
utility, it's just not as obvious. Without -multi, it only
prints the "0" for the progress before the error halts
execution:</div>
<div><span style="font-family:monospace">gdalwarp
/vsis3/your-bucket/test/input.vrt output.tif<br>
Processing /vsis3/your-bucket/test/input.vrt [1/1] : 0ERROR
4: `/vsis3/your-bucket/test/input2.tif' does not exist in
the file system, and is not recognized as a supported
dataset name.</span></div>
<div><br>
</div>
<div>But with -multi, notice the progress goes to 100%:</div>
<div><span style="font-family:monospace">gdalwarp
/vsis3/your-bucket/test/input.vrt output.tif -multi<br>
Processing /vsis3/your-bucket/test/input.vrt [1/1] : 0ERROR
4: `/vsis3/your-bucket/test/input2.tif' does not exist in
the file system, and is not recognized as a supported
dataset name.<br>
...10...20...30...40...50...60...70...80...90...100 - done.</span></div>
<div><br>
</div>
<div>I guess my workaround is to just not use multithread=True.
But is this something that could be fixed?</div>
<div><br>
</div>
<div>Thanks</div>
</div>
<br>
<fieldset class="moz-mime-attachment-header"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
gdal-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a>
<a class="moz-txt-link-freetext" href="https://lists.osgeo.org/mailman/listinfo/gdal-dev">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a>
</pre>
</blockquote>
<pre class="moz-signature" cols="72">--
<a class="moz-txt-link-freetext" href="http://www.spatialys.com">http://www.spatialys.com</a>
My software is free, but my time generally not.</pre>
</body>
</html>