[gdal-dev] Performance Issue with VRT Pixel Function and Large Number of Source Rasters

[Abdul Raheem Siddiqui]

Dear GDAL Community,

I am encountering a performance issue when using a VRT consisting of a
large number of source rasters and built-in C++ pixel function ("max"). I
would appreciate any guidance on whether some GDAL config option can
improve this, or I am doing something wrong, or this is a potential
optimization opportunity.

I have A VRT file referencing ~750 individual rasters (Byte data type, avg
size ~1000x1000 pixels, untiled, and same CRS for all source rasters). The
VRT uses the built-in “max” pixel function.

Running gdal_translate to convert the VRT to GTiff takes ~1.5 hours and
consumes ~4GB RAM.
gdal_translate -of GTiff merged.vrt OUTPUT.tif

When tripling the number of source rasters (to ~2250) by duplicating
entries in the VRT, processing time increases to ~4.5 hours, with RAM usage
rising to ~11.5GB.
gdal_translate -of GTiff merged_3x.vrt OUTPUT.tif

Extracting a small subset of the VRT via -projwin does not improve
performance (still slow, ~14GB RAM used).
gdal_translate -projwin -146955.241 797044.4497 -138766.1444 789656.0648
-of GTiff merged_3x.vrt OUTPUT.tif

*Removing the pixel function makes processing instantaneous, even with 2250
rasters.*

Performance is unaffected by --config GDAL_CACHEMAX or -co
NUM_THREADS=ALL_CPUS.

The issue persists across other datasets that I have. In fact, it gets
really worse when source rasters are relatively larger or of Float32 data
type.

Gdalinfo on one of the source rasters: https://pastebin.com/gjHDA2Wd
*Data:*
https://drive.google.com/file/d/1LGlzGGZvkPyXvKKgkPVzQGbBw55p5dRd/view?

System: Windows, 8 CPUs, 32GB RAM (GDAL 3.9.2 via OSGeo4W).

Thank you for your time and insights. Please reply if you are aware of how
performance can be improved.

Regards,

*Abdul Siddiqui, PE*

*abdul.siddiqui at ertcorp.com <abdul.siddiqui at ertcorp.com>*


*ERT  | *Earth Resources Technology, Inc.

14401 Sweitzer Ln. Ste 300

Laurel, MD 20707

https://www.ertcorp.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20250414/5115d1ac/attachment.htm>