[gdal-dev] Re: Strange things with gdalwarp ...

[Hermann Peifer]

Adam Nowacki wrote:
> You can test with http://pastebin.com/m45e46f53
> Also remember to use -wo SKIP_NOSOURCE=YES
> 

Now the input images are read in many chunks, rather than 1 (or max 2) 
before. Nevertheless, the performance is really great. Now, 100 tiles 
are warped in 5 minutes! That's 4-5 times faster than anything else I 
tried before.

Hermann


Before applying the patch, with intentionally small memory
(-wm 40, which was fastest, as far as I could see)

GDAL: GDALOpen(unzipped/ASTGTM_N30E039_dem.tif, this=0x9711470) succeeds 
as GTiff.
GDAL: GDALWarpKernel()::GWKGeneralCase()
Src=0,0,3601x2185 Dst=42437,38906,3031x2594
GDAL: Potential thrashing on band 1 of unzipped/ASTGTM_N30E039_dem.tif.
GDAL: GDALWarpKernel()::GWKGeneralCase()
Src=636,0,2965x3601 Dst=45468,38906,3032x2594
GDAL: GDALClose(unzipped/ASTGTM_N30E039_dem.tif, this=0x9711470)


After applying the patch, with -wm 300 --config GDAL_CACHEMAX 300
(and -wo SKIP_NOSOURCE=YES, which I always used)

GDAL: GDALOpen(unzipped/ASTGTM_N30E039_dem.tif, this=0x86f5dd0) succeeds 
as GTiff.
GDAL: GDALWarpKernel()::GWKGeneralCase()
Src=704,0,435x91 Dst=45278,40770,95x81
GDAL: GDALWarpKernel()::GWKGeneralCase()
Src=1042,0,435x207 Dst=45373,40770,95x81
GDAL: GDALWarpKernel()::GWKGeneralCase()
Src=0,0,1383x2185 Dst=45089,40851,379x649
GDAL: GDALWarpKernel()::GWKGeneralCase()
Src=1380,0,863x439 Dst=45468,40689,189x162
GDAL: GDALWarpKernel()::GWKGeneralCase()
Src=2051,0,1058x673 Dst=45657,40527,190x324
GDAL: GDALWarpKernel()::GWKGeneralCase()
Src=2725,0,876x908 Dst=45847,40527,189x324
GDAL: GDALWarpKernel()::GWKGeneralCase()
Src=3585,167,16x368 Dst=46036,40608,95x81
GDAL: GDALWarpKernel()::GWKGeneralCase()
Src=3490,413,111x367 Dst=46036,40689,95x81
GDAL: GDALWarpKernel()::GWKGeneralCase()
Src=3395,659,206x367 Dst=46036,40770,95x81
GDAL: GDALWarpKernel()::GWKGeneralCase()
Src=636,204,2965x2911 Dst=45468,40851,758x649
GDAL: GDALWarpKernel()::GWKGeneralCase()
Src=3592,2125,9x366 Dst=46226,41175,94x81
GDAL: GDALWarpKernel()::GWKGeneralCase()
Src=3498,2371,103x366 Dst=46226,41256,94x81
GDAL: GDALWarpKernel()::GWKGeneralCase()
Src=3309,2617,292x615 Dst=46226,41337,94x163
GDAL: GDALClose(unzipped/ASTGTM_N30E039_dem.tif, this=0x86f5dd0)























> Hermann Peifer wrote:
>> Adam Nowacki wrote:
>>> Some rather counterintuitive gdalwarp behavior: the bigger 
>>> dfWarpMemoryLimit (-wm setting) the more cpu time will be wasted on 
>>> warping not existing pixels. Why? Warping begins with the destination 
>>> window size of entire output image size. If this size is larger than 
>>> dfWarpMemoryLimit it is split in half along the longest axis and any 
>>> half that doesn't contain the currently processed source file is 
>>> discarded. With large dfWarpMemoryLimit this subdivision process will 
>>> stop early with still large portions of out of source image pixels.
>>>
>>
>> Given Adam's explanations above, could someone tell me if my below 
>> assumptions are correct? Thanks.
>>
>> Just to repeat, the input files are ASTER GTiffs in WGS84, 3601x3601 
>> pixels each. The output is a single mosaic of all tiles, a 100m GTiff, 
>> in LAEA projection
>>
>>
>> gdalwarp -wm 300 --config GDAL_CACHEMAX 300 --debug on ... reports:
>>
>> Src=0,0,3601x3601 Dst=36375,31125,12125x10375
>>
>> I understand this as follows: the source image of 3601x3601 pixels is 
>> read as one chunk (which after reprojection and resampling should be 
>> around 800x1100 pixels). However, 12125x10375 pixels are actually 
>> written to the output file. So 99.5% of the destination window must be 
>> completely unrelated to my input image.
>>
>> If I counter-intuitively reduce the memory limit to, say: 40 MB and 
>> leave anything else unchanged, then gdalwarp behaviour changes to
>>
>> Src=0,0,3601x3601 Dst=45468,38906,1516x2594
>>
>> This tells me that the destination window is much smaller and gdalwarp 
>> will not waste time to write out so many irrelevant pixels, correct?
>>
>> Perhaps gdalwarp should not only test if the destination window fits 
>> into memory, but also check what would be the minimum destination 
>> window for warping the input image. This could speed up the mosaicing 
>> of small tiles into a bigger output file.
>>
>> Hermann
>> _______________________________________________
>> gdal-dev mailing list
>> gdal-dev at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>>
>>