[gdal-dev] Missing exception when using multithread=True for gdal.Warp

[Javier Jimenez Shaw]

I'm trying to debug it in my machine. But after building in debug mode
enabling the environment with
. ../scripts/setdevenv.sh
the exceptions are not thrown any more in python (neither with
multithread=False)

On Sat, 1 Mar 2025 at 19:56, Sean Gillies via gdal-dev <
gdal-dev at lists.osgeo.org> wrote:

> Hi Tim,
>
> UseExceptions() might only affect your main thread. You may want to check
> in GDAL's code to see if this is the case. At any rate, you won't be able
> to handle Python exceptions that you might see raised from the warp threads.
>
> Hope this helps,
>
> On Sat, Mar 1, 2025 at 11:30 AM Tim Harris via gdal-dev <
> gdal-dev at lists.osgeo.org> wrote:
>
>> Aha, yes, it does appear to happen just with local files. In my attempts
>> to create a reproduce case yesterday, I overlooked that part. I did try it
>> early on but just with a single TIF in the VRT, in which case it does the
>> right thing and raises an exception. But with two TIFs and one missing, it
>> behaves as I explained with /vsis3.
>>
>> So create two TIFs and a VRT, then remove one TIF:
>> gdal_create -of GTiff -outsize 1000 1000 -bands 1 -a_srs EPSG:4326
>> -a_ullr 0 1 1 0 input1.tif
>> gdal_create -of GTiff -outsize 1000 1000 -bands 1 -a_srs EPSG:4326
>> -a_ullr 1 1 2 0 input2.tif
>> gdalbuildvrt input.vrt input1.tif input2.tif
>> rm input2.tif
>>
>> Then in Python:
>> from osgeo import gdal
>> gdal.UseExceptions()
>>
>> This raises an exception
>> gdal.Warp("output.tif", "input.vrt", callback=gdal.TermProgress)
>>
>> But this does not:
>> gdal.Warp("output.tif", "input.vrt", callback=gdal.TermProgress,
>> multithread=True)
>>
>>
>> On Sat, Mar 1, 2025 at 7:58 AM Javier Jimenez Shaw <j1 at jimenezshaw.com>
>> wrote:
>>
>>> Does it also happen without S3, just with local files?
>>> It would simplify the analysis
>>>
>>> On Sat, 1 Mar 2025, 01:01 Tim Harris via gdal-dev, <
>>> gdal-dev at lists.osgeo.org> wrote:
>>>
>>>> Hi, I'm having some trouble warping a VRT to a TIF where the VRT and
>>>> its constituent rasters are in S3. It stems from a random read error, but
>>>> the real problem is that the gdal.Warp call didn't throw an exception when
>>>> it was supposed to.
>>>>
>>>> I saw there's this pretty old GitHub issue that may be related, but I'm
>>>> not sure:
>>>> https://github.com/OSGeo/gdal/issues/3372
>>>>
>>>> In my situation what really happened was a random /vsicurl read
>>>> failed... sort of like an issue I reported in December that was fixed in
>>>> GDAL 3.10.1:
>>>> https://github.com/OSGeo/gdal/issues/11552
>>>>
>>>> Here's a section of my log file (filename sanitized):
>>>> ERROR 11: CURL error: Empty reply from server
>>>> 0...10...20...30...40...50...60...70...80...90...100 - done.
>>>> ERROR 4: `/vsis3/path/to/raster.tif' does not exist in the file system,
>>>> and is not recognized as a supported dataset name.
>>>>
>>>> Notice the CURL error but the progress still goes to 100%. The warp
>>>> completed successfully despite the read error.
>>>>
>>>> I have a hopefully simple reproduce case here. The first few shell
>>>> commands will make two TIFs, put them in S3, then make a VRT of those TIFs
>>>> and also put that in S3. Then to simulate the CURL error above, it just
>>>> deletes one of the TIFs so that the VRT references a non-existent file:
>>>> gdal_create -of GTiff -outsize 1000 1000 -bands 1 -a_srs EPSG:4326
>>>> -a_ullr 0 1 1 0 input1.tif
>>>> gdal_create -of GTiff -outsize 1000 1000 -bands 1 -a_srs EPSG:4326
>>>> -a_ullr 1 1 2 0 input2.tif
>>>> aws s3 cp input1.tif s3://your-bucket/test/input1.tif
>>>> aws s3 cp input2.tif s3://your-bucket/test/input2.tif
>>>> gdalbuildvrt input.vrt /vsis3/your-bucket/test/input1.tif
>>>> /vsis3/your-bucket/test/input2.tif
>>>> aws s3 cp input.vrt s3://your-bucket/test/input.vrt
>>>> aws s3 rm s3://your-bucket/test/input2.tif
>>>>
>>>> Then in Python:
>>>> from osgeo import gdal
>>>> gdal.UseExceptions()
>>>> gdal.Warp("output.tif", "/vsis3/tharris-bucket/test/input.vrt",
>>>> callback=gdal.TermProgress)
>>>>
>>>> This throws a RuntimeError, as expected. But delete the partial file
>>>> and try with multithread=True:
>>>> rm output.tif
>>>> gdal.Warp("output.tif", "/vsis3/tharris-bucket/test/input.vrt",
>>>> callback=gdal.TermProgress, multithread=True)
>>>>
>>>> This logs the error, but doesn't throw an exception. The end result is
>>>> that I have a TIF that is missing the contents of the file that suffered
>>>> from that CURL read error.
>>>>
>>>> For what it's worth this also happens with the command line utility,
>>>> it's just not as obvious. Without -multi, it only prints the "0" for the
>>>> progress before the error halts execution:
>>>> gdalwarp /vsis3/your-bucket/test/input.vrt output.tif
>>>> Processing /vsis3/your-bucket/test/input.vrt [1/1] : 0ERROR 4:
>>>> `/vsis3/your-bucket/test/input2.tif' does not exist in the file system, and
>>>> is not recognized as a supported dataset name.
>>>>
>>>> But with -multi, notice the progress goes to 100%:
>>>> gdalwarp /vsis3/your-bucket/test/input.vrt output.tif -multi
>>>> Processing /vsis3/your-bucket/test/input.vrt [1/1] : 0ERROR 4:
>>>> `/vsis3/your-bucket/test/input2.tif' does not exist in the file system, and
>>>> is not recognized as a supported dataset name.
>>>> ...10...20...30...40...50...60...70...80...90...100 - done.
>>>>
>>>> I guess my workaround is to just not use multithread=True. But is this
>>>> something that could be fixed?
>>>>
>>>> Thanks
>>>> _______________________________________________
>>>> gdal-dev mailing list
>>>> gdal-dev at lists.osgeo.org
>>>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>>>
>>> _______________________________________________
>> gdal-dev mailing list
>> gdal-dev at lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>
>
>
> --
> Sean Gillies
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20250302/90ef170e/attachment.htm>