[gdal-dev] Best practices for concurrent writes with GDAL (was RasterIO in paralel)

[Francis Markham]

Thanks for the feedback Antonio and Even, it is very helpful.

I probably should have specified that I want to do message passing across a
cluster using MPI, rather than shared memory multithreading.  I'll do some
performance testing to see what the relative speeds of having workers send
their data to a single writing master vs concurrent writes on NFS is.  I
suspect it might be highly dependant on the specific hardware involved.

-Francis

On 10 February 2011 06:29, Even Rouault <even.rouault at mines-paris.org>wrote:

> Le mercredi 09 février 2011 09:59:02, Antonio Valentino a écrit :
> > Are you sure it is the best way to parallelize your problem?
> > If you write tho the disk operation are in some way serialized.
> > It is a common schema to have several worker processes/thread for CPU
> > bound computations and a single IO processes/thread that collects all
> > results and write them to disk.
> >
> > The IO process/thread int this case can also be designed to implement
> > some optimization or caching mechanism that you could not apply at
> > worker process level.
> >
> > With this schema you can bypass completely the problem of concurrent
> > writes without loss of performance IMHO.
>
> Yes I completely agree with the approach you suggest. Genuine concurrent
> writing is likely not possible on the whole chain and will result in queing
> at
> some point, for example at the OS level. (Unless you use some RAID, in
> which
> case if the file is spread over multiple disks, it is perhaps possible to
> have
> genuinge parallel writing of blocks of data of the same file.)
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/gdal-dev/attachments/20110210/59c483c2/attachment.html