Thanks for the feedback Antonio and Even, it is very helpful.<br><br>I probably should have specified that I want to do message passing across a cluster using MPI, rather than shared memory multithreading. I'll do some performance testing to see what the relative speeds of having workers send their data to a single writing master vs concurrent writes on NFS is. I suspect it might be highly dependant on the specific hardware involved.<br>
<br>-Francis<br><br><div class="gmail_quote">On 10 February 2011 06:29, Even Rouault <span dir="ltr"><<a href="mailto:even.rouault@mines-paris.org">even.rouault@mines-paris.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
Le mercredi 09 février 2011 09:59:02, Antonio Valentino a écrit :<br><div><div class="h5">
> Are you sure it is the best way to parallelize your problem?<br>
> If you write tho the disk operation are in some way serialized.<br>
> It is a common schema to have several worker processes/thread for CPU<br>
> bound computations and a single IO processes/thread that collects all<br>
> results and write them to disk.<br>
><br>
> The IO process/thread int this case can also be designed to implement<br>
> some optimization or caching mechanism that you could not apply at<br>
> worker process level.<br>
><br>
> With this schema you can bypass completely the problem of concurrent<br>
> writes without loss of performance IMHO.<br>
<br>
</div></div>Yes I completely agree with the approach you suggest. Genuine concurrent<br>
writing is likely not possible on the whole chain and will result in queing at<br>
some point, for example at the OS level. (Unless you use some RAID, in which<br>
case if the file is spread over multiple disks, it is perhaps possible to have<br>
genuinge parallel writing of blocks of data of the same file.)<br>
<br></blockquote><br></div>