<div dir="ltr">After some more investigation, I've managed to put together the script below that manifests the issue on my machine (8 core, Ubuntu 19.04, GDAL 2.4.0, Python 3.7). Where I first observed this was in using 8 band WV2 imagery, so that's what the complicated input image is modeled after. Here's an example output:<div><br></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div>/tmp/output-15.tif generated an unexpected checksum (5888, 11776, 50432, 23552, 62208, 35328, 55347, 28467)</div><div>/tmp/output-26.tif generated an unexpected checksum (55040, 44544, 50432, 23552, 62208, 35328, 55347, 28467)</div><div>/tmp/output-32.tif generated an unexpected checksum (55040, 11776, 50432, 23552, 62208, 35328, 55347, 28467)</div><div>/tmp/output-56.tif generated an unexpected checksum (38656, 44544, 34048, 23552, 13056, 2560, 55347, 28467)</div><div>/tmp/output-59.tif generated an unexpected checksum (55040, 44544, 50432, 23552, 62208, 35328, 55347, 28467)</div><div>generated 5 checksums during concurrent warping of same tile</div></blockquote><div><br></div><div>The number different checksums you get back varies from run to run. Rather than calling gdal.Warp, I tried shelling out directly to the gdalwarp executable with the same arguments and didn't see the issue, so that is interesting. Also, a threadpool of size 1 seems to not have the same issue. I'll keep poking at it and see if i can distill it down to something more precise, but here's the script I've got at the moment:</div><div><br></div><div><font face="monospace">import os<br>from concurrent.futures import ThreadPoolExecutor, as_completed<br><br>from osgeo import gdal, osr<br><br>def warp_tile(f_in, f_out, warp_opts):<br> gdal_warp_opts = gdal.WarpOptions(**warp_opts, creationOptions=["TILED=YES", "COMPRESS=DEFLATE"])<br> warp_ds = gdal.Warp(f_out, f_in, options=gdal_warp_opts)<br><br>def create_input(f_in):<br> dvr = gdal.GetDriverByName('GTiff')<br> ds = dvr.Create(f_in, 8456, 7558, 8, gdal.GDT_UInt16)<br><br> ds.SetGeoTransform( (533949.5611715792, 2.12, 0.0, 4183240.981122436, 0.0, -2.12) )<br><br> srs = osr.SpatialReference()<br> srs.SetUTM(10, 1)<br> srs.SetWellKnownGeogCS('WGS84')<br> ds.SetProjection(srs.ExportToWkt())<br><br> for b in range(8):<br> ds.GetRasterBand(b+1).Fill(b+1)<br><br>if __name__ == "__main__":<br> gdal.UseExceptions()<br><br> f_in = '/tmp/input.tif'<br> create_input(f_in)<br><br> warp_opts = {'outputBounds': (539750.0, 4179750.0, 550250.0, 4190250.0), 'width': 5376, 'height': 5376, 'dstSRS': 'EPSG:32610', 'resampleAlg': 'bilinear'}<br> with ThreadPoolExecutor(max_workers=8) as executor:<br> job_d = {}<br> for i in range(100):<br> f_out = f'/tmp/output-{i}.tif'<br> job_d[executor.submit(warp_tile, f_in, f_out, warp_opts)] = f_out<br><br> check_sums = set()<br> for future in as_completed(job_d):<br> f_out = job_d[future]<br> try:<br> future.result()<br> except Exception as e:<br> print(f'failure warping {f_out}, {e}')<br> else:<br> ds = gdal.Open(f_out)<br> cs = tuple(ds.GetRasterBand(b+1).Checksum() for b in range(ds.RasterCount))<br> if cs != (38656, 11776, 50432, 23552, 62208, 35328, 55347, 28467):<br> print(f'{f_out} generated an unexpected checksum {cs}')<br> ds = None<br> check_sums.add(cs)<br> finally:<br> os.remove(f_out)<br> os.remove(f_in)<br> print(f'generated {len(check_sums)} checksums during concurrent warping of same tile')</font><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Oct 30, 2019 at 1:32 PM Even Rouault <<a href="mailto:even.rouault@spatialys.com">even.rouault@spatialys.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I've, unsuccessfully, tried to reproduce your issue with the following script:<br>
<br>
-----<br>
<br>
from osgeo import gdal<br>
import concurrent.futures<br>
<br>
def worker(in_f, out_f):<br>
gdal.Unlink(out_f)<br>
gdal.Warp(out_f, in_f, options = '-co COMPRESS=DEFLATE -co TILED=YES -ts 2048 2048')<br>
<br>
jobs = []<br>
<br>
for i in range(1000):<br>
jobs.append( ['byte.tif', '/tmp/byte%d.tif' % i] )<br>
<br>
with concurrent.futures.ThreadPoolExecutor(max_workers=8) as executor:<br>
future_map = {executor.submit(worker, in_f, out_f): out_f for in_f, out_f in jobs}<br>
for future in concurrent.futures.as_completed(future_map):<br>
out_f = future_map[future]<br>
try:<br>
future.result()<br>
except Exception as exc:<br>
print('%r generated an exception: %s' % (out_f, exc))<br>
<br>
print('Finished ' + out_f)<br>
ds = gdal.Open(out_f)<br>
if ds.GetRasterBand(1).Checksum() != 58226:<br>
print('wrong checksum for ' + out_f)<br>
<br>
------<br>
<br>
where the input byte.tif file is<br>
<a href="https://github.com/OSGeo/gdal/blob/master/autotest/gcore/data/byte.tif?raw=true" rel="noreferrer" target="_blank">https://github.com/OSGeo/gdal/blob/master/autotest/gcore/data/byte.tif?raw=true</a><br>
<br>
<br>
-- <br>
Spatialys - Geospatial professional services<br>
<a href="http://www.spatialys.com" rel="noreferrer" target="_blank">http://www.spatialys.com</a><br>
_______________________________________________<br>
gdal-dev mailing list<br>
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a><br>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" rel="noreferrer" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a></blockquote></div>