[gdal-dev] Notice: issue about multi-threaded GTiff compression+decompression

[Even Rouault]

Le 16/10/2023 à 17:14, Kurt Schwehr a écrit :
> Thanks for the heads up!
>
> Can you share that SHAs of the fix and cause?

master: d083af1ec9c38e79bfb0532885570de6e5b8a410

3.7 branch (should apply to 3.6 as well): 
b5858ed5bc5004c97f7cd6000674015bdc70b33b

cause: a worker thread of the multithreaded decoding could, in a 
situation where the block cache is full, cause a "dirty" (ie modified 
but not yet serialized to disk) block to be written, resulting in either 
a deadlock between the lock of the multithreaded decoder and the lock of 
the job queue mechanism, or other decoding threads could see their file 
handle being seek() by another thread.  In short: multithreading is hard.

>
> On Mon, Oct 16, 2023, 7:44 AM Even Rouault via gdal-dev 
> <gdal-dev at lists.osgeo.org> wrote:
>
>     Hi,
>
>     For GDAL 3.6.0 to 3.7.2, use of multi-threaded GTiff
>     compression+decompression, in particular within the same file, as for
>     example in a "gdalwarp -co COMPRESS=... -co NUM_THREADS=..." workflow
>     can lead to deadlocks (process stalled forever) and/or data
>     corruption
>     (sometimes without errors at generation). Cf
>     https://github.com/OSGeo/gdal/issues/8470 for a reproducer. The
>     fix is
>     in https://github.com/OSGeo/gdal/pull/8561
>
>     The issue is particularly visible on Windows, or more generally any
>     operating system (or file system where the output file is located)
>     which
>     has no VSIVirtualHandle::PRead() implementation, but it can also be
>     occasionally reproduced on Linux (at least as a deadlock).
>
>     Even
>
>     -- 
>     http://www.spatialys.com
>     My software is free, but my time generally not.
>
>     _______________________________________________
>     gdal-dev mailing list
>     gdal-dev at lists.osgeo.org
>     https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
-- 
http://www.spatialys.com
My software is free, but my time generally not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20231016/6f8e8e3d/attachment.htm>