[gdal-dev] Concurrent and efficient access to features in OGRLayer

[Even Rouault]

Hi Julien,

> 
> I am writing some code that processes features from a Layer, and I am
> facing two distinct problems :
> 
> 1) Since the processing of features might be heavy, I would like to do
> some parallel processing, but I did not find any way to iterate on
> features in parallel, even for read-only / in-memory datasets. Do you
> have any idea on how I could achieve that ?

Some drivers might have an efficient implementation of SetNextByIndex(), so 
you could use that to open the same dataset as many times as needed, position
the reading cursor with SetNextByIndex() at different indexes in the different 
threads, and then iterate with GetNextFeature()

Another approach would be to have a single reader (as most of the time 
iterating over source features is really neglectable comparing to the 
processing done with them), and then push them to a queue that is consumed by 
several processing threads. That's more or less what I've done recently in the 
writer side of the MVT driver: the CreateFeature() method pushes the features 
to a queue, and different worker threads consume this queue and process them.

> 
> 2) I am using spatial filtering to retrieve a subset of the features
> corresponding to a given area, and then iterate on the area to process.
> My code therefore uses spatial filters quite intensively, which might be
> very slow without spatial indexing and a large number of features. Is
> there a way to add spatial indexing for in memory dataset ?

If you are thinking about the MEM driver, there's no implementation of spatial 
indexing in it currently.
If you are using GeoPackage or Shapefiles stored in /vsimem/, then spatial 
indexes if they exist will be used.

Even


-- 
Spatialys - Geospatial professional services
http://www.spatialys.com