[gdal-dev] Notice: issue about multi-threaded GTiff compression+decompression

[Sean Gillies]

Thanks for the announcement, Even! I wonder if we should track such issues
in a list? Or maybe give them a unique GitHub label?

We don't plan to release a 3.6.5, correct?

I'm going to make a Rasterio post release that patches 3.6.4 by tomorrow:
https://github.com/rasterio/rasterio/issues/2943.

On Mon, Oct 16, 2023 at 9:26 AM Even Rouault via gdal-dev <
gdal-dev at lists.osgeo.org> wrote:

>
> Le 16/10/2023 à 17:14, Kurt Schwehr a écrit :
>
> Thanks for the heads up!
>
> Can you share that SHAs of the fix and cause?
>
> master: d083af1ec9c38e79bfb0532885570de6e5b8a410
>
> 3.7 branch (should apply to 3.6 as well):
> b5858ed5bc5004c97f7cd6000674015bdc70b33b
>
> cause: a worker thread of the multithreaded decoding could, in a situation
> where the block cache is full, cause a "dirty" (ie modified but not yet
> serialized to disk) block to be written, resulting in either a deadlock
> between the lock of the multithreaded decoder and the lock of the job queue
> mechanism, or other decoding threads could see their file handle being
> seek() by another thread.  In short: multithreading is hard.
>
>
> On Mon, Oct 16, 2023, 7:44 AM Even Rouault via gdal-dev <
> gdal-dev at lists.osgeo.org> wrote:
>
>> Hi,
>>
>> For GDAL 3.6.0 to 3.7.2, use of multi-threaded GTiff
>> compression+decompression, in particular within the same file, as for
>> example in a "gdalwarp -co COMPRESS=... -co NUM_THREADS=..." workflow
>> can lead to deadlocks (process stalled forever) and/or data corruption
>> (sometimes without errors at generation). Cf
>> https://github.com/OSGeo/gdal/issues/8470 for a reproducer. The fix is
>> in https://github.com/OSGeo/gdal/pull/8561
>>
>> The issue is particularly visible on Windows, or more generally any
>> operating system (or file system where the output file is located) which
>> has no VSIVirtualHandle::PRead() implementation, but it can also be
>> occasionally reproduced on Linux (at least as a deadlock).
>>
>> Even
>>
>>
-- 
Sean Gillies
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20231016/e36bc903/attachment.htm>