<div dir="ltr">Thanks Even!</div><br><div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">On Tue, Mar 4, 2025 at 7:51 AM Even Rouault <<a href="mailto:even.rouault@spatialys.com">even.rouault@spatialys.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><u></u>
<div>
<p>Tim,</p>
<p>wil be solved (in 3.11 only) per
<a href="https://github.com/OSGeo/gdal/pull/11915" target="_blank">https://github.com/OSGeo/gdal/pull/11915</a></p>
<p>In the meantime, check for gdal.Warp() return value: if None, an
error occurred</p>
<p>Even<br>
</p>
<div>Le 01/03/2025 à 01:00, Tim Harris via
gdal-dev a écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Hi, I'm having some trouble warping a VRT to a TIF where
the VRT and its constituent rasters are in S3. It stems from a
random read error, but the real problem is that the gdal.Warp
call didn't throw an exception when it was supposed to.</div>
<div><br>
</div>
<div>I saw there's this pretty old GitHub issue that may be
related, but I'm not sure:</div>
<div><a href="https://github.com/OSGeo/gdal/issues/3372" target="_blank">https://github.com/OSGeo/gdal/issues/3372</a></div>
<div><br>
</div>
<div>In my situation what really happened was a random /vsicurl
read failed... sort of like an issue I reported in December
that was fixed in GDAL 3.10.1:</div>
<div><a href="https://github.com/OSGeo/gdal/issues/11552" target="_blank">https://github.com/OSGeo/gdal/issues/11552</a></div>
<div><br>
</div>
<div>Here's a section of my log file (filename sanitized):</div>
<div><span style="font-family:monospace">ERROR 11: CURL error:
Empty reply from server<br>
0...10...20...30...40...50...60...70...80...90...100 - done.<br>
ERROR 4: `/vsis3/path/to/raster.tif' does not exist in the
file system, and is not recognized as a supported dataset
name.</span><br>
</div>
<div><br>
</div>
<div>Notice the CURL error but the progress still goes to 100%.
The warp completed successfully despite the read error.</div>
<div><br>
</div>
<div>I have a hopefully simple reproduce case here. The first
few shell commands will make two TIFs, put them in S3, then
make a VRT of those TIFs and also put that in S3. Then to
simulate the CURL error above, it just deletes one of the TIFs
so that the VRT references a non-existent file:</div>
<div><span style="font-family:monospace">gdal_create -of GTiff
-outsize 1000 1000 -bands 1 -a_srs EPSG:4326 -a_ullr 0 1 1 0
input1.tif<br>
gdal_create -of GTiff -outsize 1000 1000 -bands 1 -a_srs
EPSG:4326 -a_ullr 1 1 2 0 input2.tif<br>
aws s3 cp input1.tif s3://your-bucket/test/input1.tif<br>
aws s3 cp input2.tif s3://your-bucket/test/input2.tif<br>
gdalbuildvrt input.vrt /vsis3/your-bucket/test/input1.tif
/vsis3/your-bucket/test/input2.tif<br>
aws s3 cp input.vrt s3://your-bucket/test/input.vrt<br>
aws s3 rm s3://your-bucket/test/input2.tif</span></div>
<div><br>
</div>
<div>Then in Python:</div>
<div><span style="font-family:monospace">from osgeo import gdal<br>
gdal.UseExceptions()<br>
gdal.Warp("output.tif",
"/vsis3/tharris-bucket/test/input.vrt",
callback=gdal.TermProgress)</span></div>
<div><br>
</div>
<div>This throws a RuntimeError, as expected. But delete the
partial file and try with multithread=True:</div>
<div><span style="font-family:monospace">rm output.tif</span></div>
<div><span style="font-family:monospace">gdal.Warp("output.tif",
"/vsis3/tharris-bucket/test/input.vrt",
callback=gdal.TermProgress, multithread=True)</span></div>
<div><br>
</div>
<div>This logs the error, but doesn't throw an exception. The
end result is that I have a TIF that is missing the contents
of the file that suffered from that CURL read error.</div>
<div><br>
</div>
<div>For what it's worth this also happens with the command line
utility, it's just not as obvious. Without -multi, it only
prints the "0" for the progress before the error halts
execution:</div>
<div><span style="font-family:monospace">gdalwarp
/vsis3/your-bucket/test/input.vrt output.tif<br>
Processing /vsis3/your-bucket/test/input.vrt [1/1] : 0ERROR
4: `/vsis3/your-bucket/test/input2.tif' does not exist in
the file system, and is not recognized as a supported
dataset name.</span></div>
<div><br>
</div>
<div>But with -multi, notice the progress goes to 100%:</div>
<div><span style="font-family:monospace">gdalwarp
/vsis3/your-bucket/test/input.vrt output.tif -multi<br>
Processing /vsis3/your-bucket/test/input.vrt [1/1] : 0ERROR
4: `/vsis3/your-bucket/test/input2.tif' does not exist in
the file system, and is not recognized as a supported
dataset name.<br>
...10...20...30...40...50...60...70...80...90...100 - done.</span></div>
<div><br>
</div>
<div>I guess my workaround is to just not use multithread=True.
But is this something that could be fixed?</div>
<div><br>
</div>
<div>Thanks</div>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
gdal-dev mailing list
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a>
</pre>
</blockquote>
<pre cols="72">--
<a href="http://www.spatialys.com" target="_blank">http://www.spatialys.com</a>
My software is free, but my time generally not.</pre>
</div>
</blockquote></div>