[gdal-dev] Multithread deadlock

[Francisco Javier Calzado]

Hi Even,

Bingo! I've just tested with an ENVI dataset and is failing, same issue with a few random blocks missing in the resulting rasters. Find below a link with the code for a simple test reproducing the issue (also included the ENVI dataset used). Just bear in mind that sometimes may generate the final raster properly, although, based on my testing, it is highly probable to get missing blocks with just a few runs:

https://drive.google.com/open?id=0B-OCl1FjBi0YUkN4WWZmRjRfM1k

I'm not sure if it is related with the RawRasterBand, as my custom-driver does not inherit from that class. However, both drivers, ENVI and mine, inherit its custom raster band from GDALPamRasterBand. Oh .... now I see, RawRasterBand/RawDataset inherit from GDALPam corresponding objects .... Something weird its happening in the common part of these drivers...


P.D.: Yes, I call FlushCache() on dataset destructor.




-----Original Message-----
From: Even Rouault [mailto:even.rouault at spatialys.com] 
Sent: 27 September, 2016 19:16
To: Francisco Javier Calzado <francisco.javier.calzado at ericsson.com>
Cc: gdal-dev at lists.osgeo.org
Subject: Re: [gdal-dev] Multithread deadlock

Le mardi 27 septembre 2016 16:35:26, Francisco Javier Calzado a écrit :
> Hi Even,
> 
> Your changes have fixed the issue with multiple threads, now they can 
> write simultaneously without getting locked. I've done some test in 
> GTiff format and everything went OK.

Did you try with other drivers, of the raw type, like ENVI since they look similar to your driver ?

> However, using a custom-made driver for a new format (called Wizard), 
> a new problem has arisen :( .
> 
> In that case, although threads finish its task and generate its 
> corresponding output raster, 2 weird things happen randomly and 
> (again) only when the block cache gets full during writing: - Some 
> blocks (lines) are missing, so they appear in the file with no data 
> value. - Resulting file format is corrupted.
> 
> It seems that 2 or more threads writing simultaneously (althought 
> different
> threads) mess up the block cache to each other when trying to flush 
> dirty blocks. I have tried to manually flush dirty blocks (either 
> calling
> block->FlushDirtyBlocks() or block->Write() ) everytime a block is
> completely written in the cache, but it doesn't work.
> 
> My driver is simple, nothing too complex, actually was mostly taken 
> from other drivers already implemented. Check below the IWriteMethod, 
> just in case I made a mistake.
> 
> Do you have any clue what is happening here?

Do you call FlushCache() in the destructor of your dataset class ? But if you didn't do, you'd also have issues in non threaded scenarios.

--
Spatialys - Geospatial professional services http://www.spatialys.com