[Gdal-dev] Multi-threading Support

[Frank Warmerdam]

Folks,

I have committed a variety of changes to GDAL in support of thread
safety.  On win32 these capabilities should now be utilized by default.
On unix you would need to configure --with-pthread and then hand edit
a definition for CPL_THREADLOCAL into cpl_config.h.  I intend to make
the use on unix smoother when I return.

I have done some fairly intensive tests with multireadtest.exe (in 
gdal/apps) and I have reasonably confidence in multi-threaded
use within the the following restrictions:

 o Any given GDALDataset is used only from one thread.
 o The GDAL driver list should be initialized only once (ie. a
    call to GDALAllRegister()). 
 o Currently error handlers are per-thread, so if you want to install
    a custom error handler, you will need to do this for each thread
    depending on it. 
 o The block cache is shared amoung all threads. 
 o My testing has been mostly in a read-only environment.  I don't
    see any obvious problem with writing, but it hasn't been tested
    in a multithreaded environment. 
  o I have not reviewed the OGRSpatialReference related code for
     threadsafety yet.  

The code is incorporated in CVS now.  Some changes went in this
morning, so last night's cvs snapshot will still be buggy.  

For background, I have made a number of additions to cpl_multiproc.cpp/h
including the addition of a mutex holder class.  

I have also modified the assumptions with regard to CPLAcquireMutex().  
It is now expected to support recursive mutex acquisition.  That is, a 
thread can safely acquire an already held mutex again.  Normal reference 
counting rules apply.  I am a bit concerned about how portable this will
be on different pthreads implementations.  Currently I have tested it on
Linux and I had to use the apparently non-standard
PTHREAD_RECURSIVE_MUTEX_INITIALIZER_NP mutex initialization.
It may be that additional work will be needed for multi-threaded mutex
support on other platforms. 

The other key change was introduction of the CPL_THREADLOCAL
macro.  This attribute for a variable marks it as being thread-local.
That is, a separate instance will exist for each thread.  I have applied
this to a variety of static buffers in gdal/port such as the "last error
message" buffer.   On win32 this evaluates to __declspec(thread). 
On other platforms it resolves to nothing by default, but if you have
an appropriately modern version of gcc you can have it expand
to __thread.  I have tested this with gcc 3.3 on Linux successfully.
I still need to develop the appropriate configuration control around 
this definition.

Those with a strong interest in GDAL multi-threaded support are
encouraged to test it over the next two weeks while I am away.
However, as it is all quite new some teething problems are to be
expected.  So take care with the latest CVS version of GDAL.

Best regards,
-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | Geospatial Programmer for Rent