[gdal-dev] "libertiff" (was Re: Call for review on RFC 101: Raster dataset read-only thread-safety)

[Even Rouault]

Hi,

Somewhat related to some of the discussions in that thread, to be able 
to read "uncommon" (to remain polite) GeoTIFF products generated by the 
Sentinel Application Toolbox (https://github.com/OSGeo/gdal/pull/10867), 
I found that using libtiff, libgeotiff or the GDAL GTiff driver wasn't 
going to be practical, so I wrote my own TIFF library, dubbed 
"libertiff": https://github.com/libertiff/libertiff

C++11, no variable argument brain damage like in libtiff, header only, 
read-only, no codec support for now (no actual pixel reading capability 
in fact, just telling you the offset&size of tiles&strips. yes, that's a 
big limitation :-)), thread-safe by design.

Fits in 1,500 lines of code or so: 
https://github.com/libertiff/libertiff/blob/master/libertiff.hpp. Could 
maybe serve as a foundation for a potential future thread-safe read-only 
GDAL GeoTIFF driver.

Even

Le 06/09/2024 à 21:57, Frank Warmerdam a écrit :
> Even,
>
> I'm actually surprised that it is considered impractical to make 
> some/most drivers thread safe (read-only) natively and I thought key 
> ones already were mostly there in the past though I'm not so confident 
> in my memory.   Looking through the GTiff code I find myself not able 
> to work out where locking is done and how we are managing the 
> directory loading/switching as block reads are done.
>
> I would certainly like to work towards read-only thread safety for 
> some drivers with:
>  - a shared block cache for all threads reading from a dataset
>  - no duplication of file handles per-thread
>  - a way of checking that a dataset is considered to be thread safe.
>
> I'm curious if you can talk a bit more about the challenges of 
> accomplishing the above.   I had been hoping to take advantage of my 
> (assumption) that this already mostly existed in GDAL for our new high 
> performance environment at Planet so it is of more than academic 
> interest to me.  Also, doesn't gdalwarp's multithreading support 
> already depend on read-only thread safety for source files?
>
> The above said, I can also see some utility in RFC 101 and have no 
> objection to it.
>
> Best regards,
> Frank
>
> On Fri, Sep 6, 2024 at 12:45 PM Kurt Schwehr via gdal-dev 
> <gdal-dev at lists.osgeo.org> wrote:
>
>     My take would be to not offer the force option unless there is
>     someone who states they really need it.
>
>     On Fri, Sep 6, 2024 at 6:52 AM Even Rouault via gdal-dev
>     <gdal-dev at lists.osgeo.org> wrote:
>
>         Hi,
>
>         No other reactions? In particular w.r.t. the open question
>         whether it is
>         worth to offer a "force" mode in the situations where a
>         dataset cannot
>         be re-opened (or for a band whose belonging to a dataset
>         cannot be
>         determined) where the thread-safe wrapper would just put a
>         mutex around
>         the calls to that dataset/band (so obviously this wouldn't
>         scale with
>         the number of threads). I'm probably leaning on not offering
>         this option
>         for now, if there's no obvious need it is useful. It can
>         always been
>         added afterwards.
>
>         Even
>
>         Le 29/08/2024 à 14:26, Even Rouault via gdal-dev a écrit :
>         > Hi,
>         >
>         > I've worked on a new RFC 101: Raster dataset read-only
>         thread-safety:
>         > https://github.com/OSGeo/gdal/pull/10676
>         >
>         > Summary: This RFC enables users to get instances of
>         GDALDataset (and
>         > their related objects such as GDALRasterBand) that are
>         thread-safe for
>         > read-only raster operations, that is such instances can be
>         safely used
>         > from multiple threads without locking.
>         >
>         > It is backed by a preliminary implementation (fairly
>         complete, apart
>         > from writing new tests...). There's at least one open
>         question raised
>         > in the RFC.
>         >
>         > PS: you can see one of the benefits of the migration of the
>         doc to
>         > ReadTheDocs hosting, with the preview of the updated
>         documentation in
>         > pull requests. So this is a way to mention that the move to RtD
>         > hosting has now been completed, with the default version
>         ("latest")
>         > reflecting the state of the master branch, as well as the
>         version for
>         > the release/3.9 branch. Thanks to Dan Baston (and other people
>         > involved) for this work! There might be a few adjustments
>         needed, so
>         > if you notice something unusual w.r.t. to the previous
>         state, please
>         > file a ticket.
>         >
>         > Even
>         >
>         -- 
>         http://www.spatialys.com
>         My software is free, but my time generally not.
>
>         _______________________________________________
>         gdal-dev mailing list
>         gdal-dev at lists.osgeo.org
>         https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
>     _______________________________________________
>     gdal-dev mailing list
>     gdal-dev at lists.osgeo.org
>     https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
>
>
> -- 
> ---------------------------------------+--------------------------------------
> I set the clouds in motion - turn up   | Frank Warmerdam, 
> warmerdam at pobox.com
> light and sound - activate the windows | USA: +1 650-701-7823
> and watch the world go round - Rush    | CAN: +1 343-550-9984

-- 
http://www.spatialys.com
My software is free, but my time generally not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240924/842d9a62/attachment-0001.htm>