[gdal-dev] Multithreading issue on pixel-based intensive access (GDAL 2.1.0)

Thu Jul 28 06:11:13 PDT 2016

Thanks Even,

I don't want to waste your time, so just before sending any code to be tested I would like to be sure it is not a bug from my side. Although I usually work with GDAL compiled in 64bits, I had to swap to x86 for these tests. When trying to compile in 32bits with Visual Studio 2015 I got the following unresolved symbols:

-----------------------------------------------
        link /nologo /dll /INCLUDE:OSRValidate  /INCLUDE:OPTGetProjectionMethods  /INCLUDE:OGR_G_GetPointCount  /INCLUDE:OGRRegisterAll /INCLUDE:GDALSimpleImageWarp  /INCLUDE:GDALReprojectImage  /INCLUDE:GDALComputeMedianCutPCT  /INCLUDE:GDALDitherRGB2PCT  /INCLUDE:OCTNewCoordinateTransformation port\*.obj gcore\*.obj alg\*.obj frmts\o\*.obj ogr\ogrsf_frmts\ogrsf_frmts.lib ogr\ogr.lib gnm\*.obj gnm\gnm_frmts\o\*.obj   apps\commonutils.obj apps\gdalinfo_lib.obj apps\gdal_translate_lib.obj apps\gdalwarp_lib.obj apps\ogr2ogr_lib.obj  apps\gdaldem_lib.obj apps\nearblack_lib.obj apps\gdal_grid_lib.obj apps\gdal_rasterize_lib.obj apps\gdalbuildvrt_lib.obj .\..\expat-2.2.0/lib/libexpat.lib                 .\..\sqlite-3.12.2\sqlite3.lib   .\..\geos-3.5.0\src\geos_c_i.lib legacy_stdio_definitions.lib odbc32.lib odbccp32.lib user32.lib ws2_32.lib gcore\Version.res  /out:gdal201.dll /implib:gdal_i.lib   /debug
   Creating library gdal_i.lib and object gdal_i.exp
LINK : error LNK2001: unresolved external symbol OCTNewCoordinateTransformation
LINK : error LNK2001: unresolved external symbol GDALDitherRGB2PCT
LINK : error LNK2001: unresolved external symbol GDALComputeMedianCutPCT
LINK : error LNK2001: unresolved external symbol GDALReprojectImage
LINK : error LNK2001: unresolved external symbol GDALSimpleImageWarp
LINK : error LNK2001: unresolved external symbol OGRRegisterAll
LINK : error LNK2001: unresolved external symbol OGR_G_GetPointCount
LINK : error LNK2001: unresolved external symbol OPTGetProjectionMethods
LINK : error LNK2001: unresolved external symbol OSRValidate
gdal201.dll : fatal error LNK1120: 9 unresolved externals
NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\link.EXE"' : return code '0x460'
Stop.
--------------------------------------

So, in order to get the DLL, I manually linked it just by invoking the previous link command without all the "/INCLUDES" that were failing:

--------------------------------------

        link /nologo /dll port\*.obj gcore\*.obj alg\*.obj frmts\o\*.obj ogr\ogrsf_frmts\ogrsf_frmts.lib ogr\ogr.lib gnm\*.obj gnm\gnm_frmts\o\*.obj   apps\commonutils.obj apps\gdalinfo_lib.obj apps\gdal_translate_lib.obj apps\gdalwarp_lib.obj apps\ogr2ogr_lib.obj  apps\gdaldem_lib.obj apps\nearblack_lib.obj apps\gdal_grid_lib.obj apps\gdal_rasterize_lib.obj apps\gdalbuildvrt_lib.obj .\..\expat-2.2.0/lib/libexpat.lib .\..\sqlite-3.12.2\sqlite3.lib   .\..\geos-3.5.0\src\geos_c_i.lib legacy_stdio_definitions.lib odbc32.lib odbccp32.lib user32.lib ws2_32.lib gcore\Version.res  /out:gdal201.dll /implib:gdal_i.lib   /debug

--------------------------------------

Although I do not call any of the missing symbols from my code, do you think this could be the reason of the problem? Why is it failing with VS2015 in x86 (I can build GDAL in x64 with no problem)?

Javier,

Your use case looks like a legit use case of the API. I'm not aware of bugs regarding to locking of the raster block cache, but that doesn't mean that they aren't any. Could you prepare a self contained source code + dataset that would reproduce the issue ?

As far as I can see, GDALRasterBlock::Touch_unlocked() should only be invoked inside a mutex, so I wouldn't expect races.

We have a test program that stress tests the block cache, but at a higher level (RasterIO) : 
https://svn.osgeo.org/gdal/trunk/autotest/cpp/testblockcache.cpp
It is run in our continuous integration and works fine.

Even

> Hi guys, I'm experiencing an issue when running some simple tests 
> involving multithreading. I would like to know if it is something I am 
> not doing correctly with GDAL or it is a GDAL business dealing with 
> threads and block cache.
> 
> To make long story short, just imagine two threads accessing different 
> raster dataset (different files) and looping through all of its pixels 
> to read its value. So, for each pixel I do something like:
> 
> // Lock.
> GDALRasterBlock* poBlock = m_poRB->GetLockedBlockRef(blockXIndex,
> blockYIndex);
> void* pData = poBlock->GetDataRef();
> 
> // Access value.
> T value = ((T*)pData)[blockOffset];
> 
> // Unlock.
> poBlock->DropLock();
> 
> 
> I know this is a very aggressive way to access the pixel value as I'm 
> locking/unlocking in a pixel basis instead of a block basis (I mean 
> here, locking the block just once and reading all of its pixels, and 
> then unlocking), and also it may decrease performance. But I'm limited 
> by some requirements in my application legacy code and there is not 
> much I can do.
> 
> The problem with this implementation is that I get an assertion failed 
> in gdalrasterblock.cpp line 800, which is in summary, the moment where 
> a 'touched' block is moved on top of the LRU block cache list. It 
> seems that a race condition may be happening between threads.
> 
> So my question is: this way of accessing pixels with so many block 
> locks/unlocks is fully supported by GDAL as thread-safe? Or is it a 
> bug in GDAL's block cache management?
> 
> Thank you so much for your time and help.
> 
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev

--
Spatialys - Geospatial professional services http://www.spatialys.com