<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 12pt;
font-family:Calibri
}
--></style></head>
<body class='hmmessage'><div dir='ltr'>Thanks Even for your response. That clarifies more the current capabilities of GDAL on parallel processing.<div>I think that as parallel processing gain more momentum, GDAL and multiple GDAL based applications would benefit for adopting it.</div><div>In Windows world, parallel processing was added in Visual Studio 2010 (PPL for C++ and TPL for .net) and re-enforced a bit more</div><div>in Visual Studio 2012 by C++ AMP: GPU based processing.</div><div><br></div><div>I do not know enough the low level GDAL caching but it seems to me that option B) could work (dataset based caching).</div><div>I agree that exposing capability level per driver about concurrent safety is a good one. As some drivers depends</div><div>on 3rd party libraries that might not be concurrent safe. Tagging those drivers as not concurrent safe would be a first step,</div><div>before trying to address their underlying concurrency safe issues.</div><div><br></div><div>As far as reading datasets, I tested HDF4 or 5 dataset to read its subdatasets in a parallel way and encountered an error.</div><div>So some drivers may have the parallel reading issue as well. May be reading sub-datasets is simply not concurrent safe even</div><div>if each sub-dataset is opened as its own dataset ? </div><div><br></div><div>If there is interest to work on parallel writing (and may be reading), I can help at least by testing.</div><div><br></div><div>Thanks</div><div>Noureddine Farah<br><br><div><div id="SkyDrivePlaceholder"></div>> From: even.rouault@mines-paris.org<br>> To: gdal-dev@lists.osgeo.org<br>> Subject: Re: [gdal-dev] does gdal support multiple simultaneous writers to raster<br>> Date: Sat, 12 Jan 2013 16:35:28 +0100<br>> CC: nfarah@hotmail.com<br>> <br>> > ex. convert<br>> > multiple datasets to different output datasets in a parallel way.<br>> <br>> As Frank underlined, there's currently an issue with the global block cache <br>> regarding write support.<br>> <br>> Imagine that you have 2 threads A and B.<br>> Thread A deal with dataset A, and thread B deal with dataset .<br>> Thread A is in the middle of writing some tile/line of dataset A.<br>> Thread B is trying to fill a new entry in the block cache (with new read data, <br>> or new data to write). But the block cache is full. So the last recently used <br>> entry must be discarded. If that entry is a dirty block of dataset A, then it <br>> must be flushed to disk, in the context of thread B, but at that time thread A <br>> is also writing data... Which might be an issue since drivers are re-entrant <br>> (can be invoked by multiple threads, if each thread deal with different <br>> datasets) not thread-safe.<br>> <br>> This specific case here could be fixed in different ways :<br>> A) Making drivers thread-safe (or accessing them through a thread-safe layer), <br>> that is to say add a dataset level mutex<br>> B) or having a per-dataset block cache instead of a global block cache<br>> C) deal differently with dirty blocks. Only flush them if the operation that <br>> need to discard the dirty block is initiated by an operation on the same <br>> dataset as the dirty block.<br>> <br>> <br>> > Would<br>> > those parallel operations not be affected by GDAL caching for bot read and<br>> > write.Since the cache is set to a limit. Is Accessing the current used<br>> > cache value concurrent safeto increase it/decrease it ?<br>> <br>> Hum, I see that GDALGetCacheMax() and GDALSetCacheMax() are not thread safe <br>> currently. We would need to protect them by the raster block mutex, with a <br>> leading call to CPLMutexHolderD( &hRBMutex );<br></div></div> </div></body>
</html>