[gdal-dev] Missing exception when using multithread=True for gdal.Warp

[Even Rouault]

Tim,

wil be solved (in 3.11 only) per https://github.com/OSGeo/gdal/pull/11915

In the meantime, check for gdal.Warp() return value: if None, an error 
occurred

Even

Le 01/03/2025 à 01:00, Tim Harris via gdal-dev a écrit :
> Hi, I'm having some trouble warping a VRT to a TIF where the VRT and 
> its constituent rasters are in S3. It stems from a random read error, 
> but the real problem is that the gdal.Warp call didn't throw an 
> exception when it was supposed to.
>
> I saw there's this pretty old GitHub issue that may be related, but 
> I'm not sure:
> https://github.com/OSGeo/gdal/issues/3372
>
> In my situation what really happened was a random /vsicurl read 
> failed... sort of like an issue I reported in December that was fixed 
> in GDAL 3.10.1:
> https://github.com/OSGeo/gdal/issues/11552
>
> Here's a section of my log file (filename sanitized):
> ERROR 11: CURL error: Empty reply from server
> 0...10...20...30...40...50...60...70...80...90...100 - done.
> ERROR 4: `/vsis3/path/to/raster.tif' does not exist in the file 
> system, and is not recognized as a supported dataset name.
>
> Notice the CURL error but the progress still goes to 100%. The warp 
> completed successfully despite the read error.
>
> I have a hopefully simple reproduce case here. The first few shell 
> commands will make two TIFs, put them in S3, then make a VRT of those 
> TIFs and also put that in S3. Then to simulate the CURL error above, 
> it just deletes one of the TIFs so that the VRT references a 
> non-existent file:
> gdal_create -of GTiff -outsize 1000 1000 -bands 1 -a_srs EPSG:4326 
> -a_ullr 0 1 1 0 input1.tif
> gdal_create -of GTiff -outsize 1000 1000 -bands 1 -a_srs EPSG:4326 
> -a_ullr 1 1 2 0 input2.tif
> aws s3 cp input1.tif s3://your-bucket/test/input1.tif
> aws s3 cp input2.tif s3://your-bucket/test/input2.tif
> gdalbuildvrt input.vrt /vsis3/your-bucket/test/input1.tif 
> /vsis3/your-bucket/test/input2.tif
> aws s3 cp input.vrt s3://your-bucket/test/input.vrt
> aws s3 rm s3://your-bucket/test/input2.tif
>
> Then in Python:
> from osgeo import gdal
> gdal.UseExceptions()
> gdal.Warp("output.tif", "/vsis3/tharris-bucket/test/input.vrt", 
> callback=gdal.TermProgress)
>
> This throws a RuntimeError, as expected. But delete the partial file 
> and try with multithread=True:
> rm output.tif
> gdal.Warp("output.tif", "/vsis3/tharris-bucket/test/input.vrt", 
> callback=gdal.TermProgress, multithread=True)
>
> This logs the error, but doesn't throw an exception. The end result is 
> that I have a TIF that is missing the contents of the file that 
> suffered from that CURL read error.
>
> For what it's worth this also happens with the command line utility, 
> it's just not as obvious. Without -multi, it only prints the "0" for 
> the progress before the error halts execution:
> gdalwarp /vsis3/your-bucket/test/input.vrt output.tif
> Processing /vsis3/your-bucket/test/input.vrt [1/1] : 0ERROR 4: 
> `/vsis3/your-bucket/test/input2.tif' does not exist in the file 
> system, and is not recognized as a supported dataset name.
>
> But with -multi, notice the progress goes to 100%:
> gdalwarp /vsis3/your-bucket/test/input.vrt output.tif -multi
> Processing /vsis3/your-bucket/test/input.vrt [1/1] : 0ERROR 4: 
> `/vsis3/your-bucket/test/input2.tif' does not exist in the file 
> system, and is not recognized as a supported dataset name.
> ...10...20...30...40...50...60...70...80...90...100 - done.
>
> I guess my workaround is to just not use multithread=True. But is this 
> something that could be fixed?
>
> Thanks
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev

-- 
http://www.spatialys.com
My software is free, but my time generally not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20250304/5981820e/attachment.htm>