[gdal-dev] Call for review on RFC 101: Raster dataset read-only thread-safety

[Frank Warmerdam]

Even,

I'm actually surprised that it is considered impractical to make some/most
drivers thread safe (read-only) natively and I thought key ones already
were mostly there in the past though I'm not so confident in my memory.
 Looking through the GTiff code I find myself not able to work out where
locking is done and how we are managing the directory loading/switching as
block reads are done.

I would certainly like to work towards read-only thread safety for some
drivers with:
 - a shared block cache for all threads reading from a dataset
 - no duplication of file handles per-thread
 - a way of checking that a dataset is considered to be thread safe.

I'm curious if you can talk a bit more about the challenges of
accomplishing the above.   I had been hoping to take advantage of my
(assumption) that this already mostly existed in GDAL for our new high
performance environment at Planet so it is of more than academic interest
to me.  Also, doesn't gdalwarp's multithreading support already depend on
read-only thread safety for source files?

The above said, I can also see some utility in RFC 101 and have no
objection to it.

Best regards,
Frank

On Fri, Sep 6, 2024 at 12:45 PM Kurt Schwehr via gdal-dev <
gdal-dev at lists.osgeo.org> wrote:

> My take would be to not offer the force option unless there is someone who
> states they really need it.
>
> On Fri, Sep 6, 2024 at 6:52 AM Even Rouault via gdal-dev <
> gdal-dev at lists.osgeo.org> wrote:
>
>> Hi,
>>
>> No other reactions? In particular w.r.t. the open question whether it is
>> worth to offer a "force" mode in the situations where a dataset cannot
>> be re-opened (or for a band whose belonging to a dataset cannot be
>> determined) where the thread-safe wrapper would just put a mutex around
>> the calls to that dataset/band (so obviously this wouldn't scale with
>> the number of threads). I'm probably leaning on not offering this option
>> for now, if there's no obvious need it is useful. It can always been
>> added afterwards.
>>
>> Even
>>
>> Le 29/08/2024 à 14:26, Even Rouault via gdal-dev a écrit :
>> > Hi,
>> >
>> > I've worked on a new RFC 101: Raster dataset read-only thread-safety:
>> > https://github.com/OSGeo/gdal/pull/10676
>> >
>> > Summary: This RFC enables users to get instances of GDALDataset (and
>> > their related objects such as GDALRasterBand) that are thread-safe for
>> > read-only raster operations, that is such instances can be safely used
>> > from multiple threads without locking.
>> >
>> > It is backed by a preliminary implementation (fairly complete, apart
>> > from writing new tests...). There's at least one open question raised
>> > in the RFC.
>> >
>> > PS: you can see one of the benefits of the migration of the doc to
>> > ReadTheDocs hosting, with the preview of the updated documentation in
>> > pull requests. So this is a way to mention that the move to RtD
>> > hosting has now been completed, with the default version ("latest")
>> > reflecting the state of the master branch, as well as the version for
>> > the release/3.9 branch. Thanks to Dan Baston (and other people
>> > involved) for this work! There might be a few adjustments needed, so
>> > if you notice something unusual w.r.t. to the previous state, please
>> > file a ticket.
>> >
>> > Even
>> >
>> --
>> http://www.spatialys.com
>> My software is free, but my time generally not.
>>
>> _______________________________________________
>> gdal-dev mailing list
>> gdal-dev at lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>


-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam,
warmerdam at pobox.com
light and sound - activate the windows | USA: +1 650-701-7823
<http://voice.google.com/calls?a=nc,%2B16507017823>
<http://voice.google.com/calls?a=nc,%2B16507017823>
and watch the world go round - Rush    | CAN: +1 343-550-9984
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240906/b466db77/attachment.htm>