[gdal-dev] RFC 47 and Threading

[Even Rouault]

Hi,

I guess the silence in thread is due to people being impressed by the 
austerity of the topic...

For what is worth, I've had a few opportunities to discuss directly with Blake 
about his work. I'm very impressed with his energy and enthousiasm to hack in 
that difficult core area of GDAL, and his understanding of the current code base 
! But I must confess I'm concerned by the amount of changes and the difficulty 
to ensure their correctness. Especially with three different mutexes...

I'm not sure the ratio gain/effort to make dataset methods "a bit more" thread 
safe is high enough. Very few drivers can be made fully thread-safe, and fully 
parallelized (i.e. without a big mutex). Basically all file-based drivers or 
database-based drivers cannot. The former ones because the file handle cannot 
be used simultaneously by different threads. The latter ones for the same 
reason with the database connection handles. So basically that would just 
address the case of the MEM driver (which is an important use case for Blake). 
And in that instance, it would already be possible to use several MEM dataset 
handles that would point to the same memory buffer (and with a tiny per-dataset 
cache size since it is not really needed for a fast datasource such as MEM)

On the other side, I would be very pleased with having "just" the preliminary 
step of Blake's work, i.e. the possibility to choose a per-band block cache 
strategy instead of the global block cache. That should already address most 
scaling issues.

Even


> I wanted to call to everyone's attention the work I have been doing in an
> attempt to make it possible for more portions of GDAL to be thread safe and
> improve speed in multi-threaded environments. I have put a RFC up here:
> 
> http://trac.osgeo.org/gdal/wiki/rfc47_dataset_caching
> 
> Originally my work focused on simply changing the way the LRU cache
> operated, but I ended up doing more in an attempt to make it much more
> efficient while still being thread safe.
> 
> Being that this is a core part of GDAL, I wanted to get everyone's
> thoughts/questions/concerns on the different ways I have gone about
> attempting to solve this problem.
> 
> Thanks,
> 
> Blake

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com