[gdal-dev] Best practices for concurrent writes with GDAL (was RasterIO in paralel)

[Even Rouault]

Le mardi 08 février 2011 23:38:49, Francis Markham a écrit :
> Is there a document anywhere specifying the best practices for parallel
> writes to a GDAL raster?  I have an embarrasingly parallel problem that I
> would like to parallelise with MPI, but I'm not sure what I am allowed to
> do in parallel.  I would like to assign blocks exclusively to worker
> threads to read and write concurrently, but I am unsure how I might do
> this safely.

There's 
http://trac.osgeo.org/gdal/wiki/FAQMiscellaneous#IstheGDALlibrarythread-safe 
and http://gdal.org/classGDALDataset.html#6764788806a1785c97036d1dba064497

which sums up to :
* you can generally read concurrently provided you use as many GDALDataset 
objects as you have concurrent threads
* you cannot write concurrently into the same file (and using several 
GDALDataset won't help)

> 
> Any suggestions would be greatly appreciated.
> 
> Kind Regards,
> 
> Francis Markham
> 
> On 9 February 2011 09:14, Even Rouault <even.rouault at mines-paris.org> wrote:
> > Le mardi 08 février 2011 23:07:36, Stefano Moratto a écrit :
> > > Thanks for the suggestion,
> > > 
> > > I have to read png from openstreetmap. I'm trying to render an A4 paper
> > > size sheet so I have a lot of data to move.
> > > It may be better to divide the area to be read in blocks and to give to
> > > each thread a block. Foreach  block a thread should read all the bands.
> > > Probabilly I should allign the block to the tile's boundary
> > 
> > Unfortunately the PNG format is such that it is not possible to seek to
> > an arbitrary line without decoding the data of all previous scanlines,
> > so this approach won't work. The same would be true for JPEG or GIF. If
> > you can fetch
> > the data as TIFF, you could however try that approach.
> > 
> > > Stefano
> > > 
> > > > Le mardi 08 février 2011 22:41:50, Stefano Moratto a écrit :
> > > > > Hello,
> > > > > 
> > > > >           given an opened GDAL dataset, is it possibile to perform
> > > > > 
> > > > > gdalrasterio calls for each band in a dedicated thread per band in
> > > > > a thead-safe way?
> > > > > E.g. I have a raster image with 3 band (r,g,b). Since I have 4 core
> > > > > I
> > > > 
> > > > would
> > > > 
> > > > > like to I run a thread for a band r, one for band g and one for
> > > > > band
> > 
> > g.
> > 
> > > > This has been discussed recently. See
> > > > http://lists.osgeo.org/pipermail/gdal-
> > > > dev/2011-January/027567.html and the following messages.
> > > > 
> > > > (I'd note that if you process a pixel interleaved image, there are
> > > > optimizations in the GTiff driver to make the fetching of other bands
> > > > almost
> > > > 'free' if you read block by block, so there's probably little to
> > > > gain, but the
> > > > results of your experiments are welcome)
> > > > 
> > > > > Regards,
> > > > > Stefano
> > 
> > _______________________________________________
> > gdal-dev mailing list
> > gdal-dev at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/gdal-dev