<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<title></title>
</head>
<body text="#000000" bgcolor="#ffffff">
Thanks Frank, that's what I have been doing until now. I was hoping
that GDAL could do some of the bookkeeping for me.<br>
<br>
Bu the multithreaded issue remains important. Anything that could
speed up the warping process, for whatever configuration, would be
welcome and I would be happy to assist.<br>
<br>
Jan <br>
<br>
On 14-1-2013 15:32, Frank Warmerdam wrote:
<blockquote
cite="mid:CA+YzLBc0-pOvEZuDURB+nAwG7Ux8jy9FC6xqPuVBOgj=Ct+W3w@mail.gmail.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=ISO-8859-1">
<p dir="ltr">Jan,</p>
<p dir="ltr">While it is desirable to resolve the multithreaded
write issue in gdal I don't think it is critical for your use
case. </p>
<p dir="ltr">I think your case would be handled more appropriately
by multiple *processes*. You would still run into issues with
write consistency but of a different nature. The easiest
approach is to assign different spatial regions to each worker
server and run these linearly. </p>
<p dir="ltr">Best regards,</p>
<div class="gmail_quote">On Jan 12, 2013 8:05 AM, "Jan Hartmann"
<<a moz-do-not-send="true"
href="mailto:j.l.h.hartmann@uva.nl">j.l.h.hartmann@uva.nl</a>>
wrote:<br type="attribution">
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt
0.8ex; border-left: 1px solid rgb(204, 204, 204);
padding-left: 1ex;">
<div text="#000000" bgcolor="#FFFFFF"> <font face="Times New
Roman, Times, serif">I would be interested in an
implementation. I'm preparing a proposal to georeference
the complete cadastral map of the Netherlands in 1832 at a
10 cm/pixel scale with Cloud facilities. Gdalwarp is the
central piece of software, and distributed processing
capabilities would be very important. Could you please
think about the possibilities, and what kind of funding
would be required, so I can take that in during the next
few months?<br>
<br>
Jan<br>
<br>
</font>
<div>On 01/12/2013 04:35 PM, Even Rouault wrote:<br>
</div>
<blockquote type="cite">
<blockquote type="cite">
<pre>ex. convert
multiple datasets to different output datasets in a parallel way.
</pre>
</blockquote>
<pre>As Frank underlined, there's currently an issue with the global block cache
regarding write support.
Imagine that you have 2 threads A and B.
Thread A deal with dataset A, and thread B deal with dataset .
Thread A is in the middle of writing some tile/line of dataset A.
Thread B is trying to fill a new entry in the block cache (with new read data,
or new data to write). But the block cache is full. So the last recently used
entry must be discarded. If that entry is a dirty block of dataset A, then it
must be flushed to disk, in the context of thread B, but at that time thread A
is also writing data... Which might be an issue since drivers are re-entrant
(can be invoked by multiple threads, if each thread deal with different
datasets) not thread-safe.
This specific case here could be fixed in different ways :
A) Making drivers thread-safe (or accessing them through a thread-safe layer),
that is to say add a dataset level mutex
B) or having a per-dataset block cache instead of a global block cache
C) deal differently with dirty blocks. Only flush them if the operation that
need to discard the dirty block is initiated by an operation on the same
dataset as the dirty block.
</pre>
<blockquote type="cite">
<pre>Would
those parallel operations not be affected by GDAL caching for bot read and
write.Since the cache is set to a limit. Is Accessing the current used
cache value concurrent safeto increase it/decrease it ?
</pre>
</blockquote>
<pre>Hum, I see that GDALGetCacheMax() and GDALSetCacheMax() are not thread safe
currently. We would need to protect them by the raster block mutex, with a
leading call to CPLMutexHolderD( &hRBMutex );
_______________________________________________
gdal-dev mailing list
<a moz-do-not-send="true" href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a>
<a moz-do-not-send="true" href="http://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">http://lists.osgeo.org/mailman/listinfo/gdal-dev</a>
</pre>
</blockquote>
<br>
</div>
<br>
_______________________________________________<br>
gdal-dev mailing list<br>
<a moz-do-not-send="true"
href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a><br>
<a moz-do-not-send="true"
href="http://lists.osgeo.org/mailman/listinfo/gdal-dev"
target="_blank">http://lists.osgeo.org/mailman/listinfo/gdal-dev</a><br>
</blockquote>
</div>
</blockquote>
</body>
</html>