<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hi,</p>
<p>Somewhat related to some of the discussions in that thread, to be
able to read "uncommon" (to remain polite) GeoTIFF products
generated by the Sentinel Application Toolbox
(<a class="moz-txt-link-freetext" href="https://github.com/OSGeo/gdal/pull/10867">https://github.com/OSGeo/gdal/pull/10867</a>), I found that using
libtiff, libgeotiff or the GDAL GTiff driver wasn't going to be
practical, so I wrote my own TIFF library, dubbed "libertiff": <a
class="moz-txt-link-freetext"
href="https://github.com/libertiff/libertiff">https://github.com/libertiff/libertiff</a><br>
</p>
<p>C++11, no variable argument brain damage like in libtiff, header
only, read-only, no codec support for now (no actual pixel reading
capability in fact, just telling you the offset&size of
tiles&strips. yes, that's a big limitation :-)), thread-safe
by design.<br>
</p>
<p>Fits in 1,500 lines of code or so: <a
class="moz-txt-link-freetext"
href="https://github.com/libertiff/libertiff/blob/master/libertiff.hpp">https://github.com/libertiff/libertiff/blob/master/libertiff.hpp</a>.
Could maybe serve as a foundation for a potential future
thread-safe read-only GDAL GeoTIFF driver.<br>
</p>
<p>Even<br>
</p>
<div class="moz-cite-prefix">Le 06/09/2024 à 21:57, Frank Warmerdam
a écrit :<br>
</div>
<blockquote type="cite"
cite="mid:CA+YzLBfUPuFEjs3Bjck6fgPQn3uGvn9-KFh_ihOk8Giz8CzkaQ@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div dir="ltr">
<div class="gmail_default"
style="font-family:arial,sans-serif">Even,</div>
<div class="gmail_default"
style="font-family:arial,sans-serif"><br>
</div>
<div class="gmail_default"
style="font-family:arial,sans-serif">I'm actually surprised
that it is considered impractical to make some/most drivers
thread safe (read-only) natively and I thought key ones
already were mostly there in the past though I'm not so
confident in my memory. Looking through the GTiff code I
find myself not able to work out where locking is done and
how we are managing the directory loading/switching as block
reads are done. </div>
<div class="gmail_default"
style="font-family:arial,sans-serif"><br>
</div>
<div class="gmail_default"
style="font-family:arial,sans-serif">I would certainly like
to work towards read-only thread safety for some drivers
with:</div>
<div class="gmail_default"
style="font-family:arial,sans-serif"> - a shared block cache
for all threads reading from a dataset</div>
<div class="gmail_default"
style="font-family:arial,sans-serif"> - no duplication of
file handles per-thread</div>
<div class="gmail_default"
style="font-family:arial,sans-serif"> - a way of checking
that a dataset is considered to be thread safe. </div>
<div class="gmail_default"
style="font-family:arial,sans-serif"><br>
</div>
<div class="gmail_default"
style="font-family:arial,sans-serif">I'm curious if you can
talk a bit more about the challenges of accomplishing the
above. I had been hoping to take advantage of my
(assumption) that this already mostly existed in GDAL for
our new high performance environment at Planet so it is of
more than academic interest to me. Also, doesn't gdalwarp's
multithreading support already depend on read-only thread
safety for source files?</div>
<div class="gmail_default"
style="font-family:arial,sans-serif"><br>
</div>
<div class="gmail_default"
style="font-family:arial,sans-serif">The above said, I can
also see some utility in RFC 101 and have no objection to
it. </div>
<div class="gmail_default"
style="font-family:arial,sans-serif"><br>
</div>
<div class="gmail_default"
style="font-family:arial,sans-serif">Best regards,</div>
<div class="gmail_default"
style="font-family:arial,sans-serif">Frank</div>
<div class="gmail_default"
style="font-family:arial,sans-serif"><br>
</div>
</div>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Sep 6, 2024 at
12:45 PM Kurt Schwehr via gdal-dev <<a
href="mailto:gdal-dev@lists.osgeo.org"
moz-do-not-send="true" class="moz-txt-link-freetext">gdal-dev@lists.osgeo.org</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">My take would be to not offer the force
option unless there is someone who states they really need
it.</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Sep 6, 2024 at
6:52 AM Even Rouault via gdal-dev <<a
href="mailto:gdal-dev@lists.osgeo.org" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">gdal-dev@lists.osgeo.org</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
No other reactions? In particular w.r.t. the open
question whether it is <br>
worth to offer a "force" mode in the situations where a
dataset cannot <br>
be re-opened (or for a band whose belonging to a dataset
cannot be <br>
determined) where the thread-safe wrapper would just put
a mutex around <br>
the calls to that dataset/band (so obviously this
wouldn't scale with <br>
the number of threads). I'm probably leaning on not
offering this option <br>
for now, if there's no obvious need it is useful. It can
always been <br>
added afterwards.<br>
<br>
Even<br>
<br>
Le 29/08/2024 à 14:26, Even Rouault via gdal-dev a
écrit :<br>
> Hi,<br>
><br>
> I've worked on a new RFC 101: Raster dataset
read-only thread-safety: <br>
> <a href="https://github.com/OSGeo/gdal/pull/10676"
rel="noreferrer" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">https://github.com/OSGeo/gdal/pull/10676</a><br>
><br>
> Summary: This RFC enables users to get instances of
GDALDataset (and <br>
> their related objects such as GDALRasterBand) that
are thread-safe for <br>
> read-only raster operations, that is such instances
can be safely used <br>
> from multiple threads without locking.<br>
><br>
> It is backed by a preliminary implementation
(fairly complete, apart <br>
> from writing new tests...). There's at least one
open question raised <br>
> in the RFC.<br>
><br>
> PS: you can see one of the benefits of the
migration of the doc to <br>
> ReadTheDocs hosting, with the preview of the
updated documentation in <br>
> pull requests. So this is a way to mention that the
move to RtD <br>
> hosting has now been completed, with the default
version ("latest") <br>
> reflecting the state of the master branch, as well
as the version for <br>
> the release/3.9 branch. Thanks to Dan Baston (and
other people <br>
> involved) for this work! There might be a few
adjustments needed, so <br>
> if you notice something unusual w.r.t. to the
previous state, please <br>
> file a ticket.<br>
><br>
> Even<br>
><br>
-- <br>
<a href="http://www.spatialys.com" rel="noreferrer"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">http://www.spatialys.com</a><br>
My software is free, but my time generally not.<br>
<br>
_______________________________________________<br>
gdal-dev mailing list<br>
<a href="mailto:gdal-dev@lists.osgeo.org"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">gdal-dev@lists.osgeo.org</a><br>
<a
href="https://lists.osgeo.org/mailman/listinfo/gdal-dev"
rel="noreferrer" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a><br>
</blockquote>
</div>
_______________________________________________<br>
gdal-dev mailing list<br>
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">gdal-dev@lists.osgeo.org</a><br>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev"
rel="noreferrer" target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a><br>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
<span class="gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div><font face="monospace">---------------------------------------+--------------------------------------<br>
I set the clouds in motion - turn up | Frank
Warmerdam, <a href="mailto:warmerdam@pobox.com"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">warmerdam@pobox.com</a><br>
light and sound - activate the windows | USA: +1
650-701-7823<br>
and watch the world go round - Rush | CAN: +1
343-550-9984</font></div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
<a class="moz-txt-link-freetext" href="http://www.spatialys.com">http://www.spatialys.com</a>
My software is free, but my time generally not.</pre>
</body>
</html>