[gdal-dev] parallel processing

[James McManus]

Frank Warmerdam wrote:
> James McManus wrote:
>> Has implementing parallel process for GDAL and OGR been considered?
>
> James,
>
> There has not been much work on parallel processing within GDAL but
> some applications do use GDAL in multiple threads at a time and there
> has been considerable work to make aspects of GDAL thread safe.  In
> particular modern GDAL should support a "many readers" situation fairly
> well now as long as each of the readers is operating on distinct
> GDALDataset objects.
>
> For various reasons you can't have multiple readers active on a common
> gdal dataset, and when GDAL is writing there should really only be
> one thread active at a time.
>
> Best regards,
Frank,
/
/That is basically what I am looking for. I would like to take advantage 
of my additional cores and CPU when I import data using v.in.ogr.  As 
you know large vector files take a long time to read, so it would be 
great if I could employ additional threads.  I searched the web to see 
if any other IO software was doing this. I found that netcdf has a 
library which uses MPI-IO:

http://www-unix.mcs.anl.gov/parallel-netcdf/

Reading spaghetti shape files into GRASS requires building of topology 
among other things. Multi-thread processing would speed this up 
dramatically. With the proliferation of multi-thread processors for PC's 
it seems like something that would eventually have to be incorporated 
into GDAL/OGR.

Jim