[gdal-dev] Race condition between forked processes with opened Tiff dataset on Linux

[Jiri Drbalek]

Even,

that was really helpful. Finally, I solved it without any need to modify
GDAL.

I do the mmap() myself in my application. Then I create VSILFILE out of the
mmapped block by VSIFileFromMemBuffer().

I can then GDALOpen() the /vsimem/ file and fork freely.

The GTIFF_USE_MMAP is a cherry on the cake.

I still have to evaluate performance impact, but it seems promising so far.

Thanks for the advice.

Jiri




2017-12-15 11:29 GMT+01:00 Even Rouault <even.rouault at spatialys.com>:

> On jeudi 14 décembre 2017 23:31:48 CET Jiri Drbalek wrote:
>
> > Dear Even,
>
> >
>
> > Thank you for your helpful answer.
>
> >
>
> > I forgot to mention that when I was testing the fork() situation with
>
> > libtiff alone, it was working fine when the tiff file was memory
>
> > mapped. Unfortunately, for some reason, GDAL doesn't support memory
>
> > mapping of compressed tiffs. Libtiff can read them, at least those
>
> > with deflate compression I've tested. What is the reason for that
>
> > restriction?
>
>
>
> mmap is platform specific, so there was a need for a more general
> mechanism. And at the time the GeoTIFF driver was created, 32 bit processes
> were still common but large files already existed, so even on Linux, mmap
> wasn't really always usable. Another potential issue is that the OS might
> not behave appropriately if you mmap() a file larger than the available RAM
> and read it entirely. At least that was my experience with some older
> kernels where the OS wouldn't unload cached pages aggressively enough,
> making it irresponsive due to heavy cache swapping.
>
>
>
> Another option at the /vsi level I looked yesterday was the use of pread()
> that takes both a file offset and size to read, and makes it possible to
> use the same underlying file descriptor from multiple threads/processes.
> But the caveat is that this translates directly as a system call, bypassing
> file stream buffering.
>
>
>
> > Should I try to enable mmaping of compressed tiffs?
>
>
>
> That could be a solution worth investigating. Perhaps restricted to 64bit
> posix platforms, and with a configuration option, such as the existing
> GTIFF_USE_MMAP that you'll see if you look at frmts/gtiff/tifvsi.cpp. The
> name is a bit misleading since that's currently only available for a
> pseudo-mmap emulation for /vsimem/ files, that was added per
> https://trac.osgeo.org/gdal/changeset/39555. As you may wonder why this
> was done, the aim was to be able to test the code paths in libtiff that are
> mmap() specific, when GDAL is tortured by oss-fuzz.
>
>
>
> Even
>
>
>
> --
>
> Spatialys - Geospatial professional services
>
> http://www.spatialys.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20171220/5c108d57/attachment.html>