[gdal-dev] Python Bindings and Closed Datasets

[Even Rouault]

On mercredi 30 octobre 2019 12:05:01 CET Patrick Young wrote:
> Hi all,
> 
> I've been experiencing some behavior using the GDAL python bindings where I
> am occasionally seeing what appears to be random blocks of the tiff being
> unwritten in geotiffs I've pushed to S3.  a small block(s) in one of the
> bands will be all zeros while everywhere else is good.
> 
> My setup is that I have a thread pool crunching through some gdal.Warp
> calls.  The main thread is polling for completed jobs and then uploading
> the file to s3.  My theory is that Python's garbage collector hasn't
> destroyed the dataset I've set to None before I start uploading.  Is this
> plausible?  

That would be surprising. As far as I know, the Python garbage collector is 
synchronous, so at the point where the last reference to an object is dropped, 
it is destroyed. Your below try / finally could be simplified as just:

gdal.Warp(f_out, f_in, options=gdal_warp_opts)

But that should behave exactly as your current code, and thus exhibit the same 
unexplained issue.
(however you might test the return of gdal.Warp() to detect errors, or enable 
gdal.UseExceptions())

However looking at your code I'm not sure if you're not warping several input 
files in the same output file. If so, the behaviour you see is expected as 
multi-threaded/multi-processing updates of a single target file are 
unspecified behaviour (but I'd expect such a scenario to always result in 
wrong results, not randomly)

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com