<!DOCTYPE html><html><head><title></title></head><body><div style="font-family:Arial;">Hi Abdul,</div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">Can you bump <a href="https://gdal.org/en/stable/user/configoptions.html#config-GDAL_MAX_DATASET_POOL_SIZE">https://gdal.org/en/stable/user/configoptions.html#config-GDAL_MAX_DATASET_POOL_SIZE</a> and see if it helps? And do all inputs have the same block size of 1048x7? Ideally, your inputs and output will all have the same block size or a multiple of it, so you could also add -co BLOCKXSIZE=1048 -co BLOCKYSIZE=7.</div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">Note that these should apply just as well when you don't have a pixel function, so it's probably not it.</div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;"><a href="https://github.com/OSGeo/gdal/pull/11850">https://github.com/OSGeo/gdal/pull/11850</a> might also help in the future, but you'll need to change your workflow for it.</div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">Laurentiu</div><div style="font-family:Arial;"><br></div><div>On Mon, Apr 14, 2025, at 07:52, Abdul Raheem Siddiqui via gdal-dev wrote:</div><blockquote type="cite" id="qt" style=""><div dir="ltr"><div><div>Dear GDAL Community,</div><div><br></div><div>I am encountering a performance issue when using a VRT consisting of a large number of source rasters and built-in C++ pixel function ("max"). I would appreciate any guidance on whether some GDAL config option can improve this, or I am doing something wrong, or this is a potential optimization opportunity.</div><div><br></div><div>I have A VRT file referencing ~750 individual rasters (Byte data type, avg size ~1000x1000 pixels, untiled, and same CRS for all source rasters). The VRT uses the built-in “max” pixel function.</div><div><br></div><div>Running gdal_translate to convert the VRT to GTiff takes ~1.5 hours and consumes ~4GB RAM.</div><div><span class="font" style="font-family:monospace;">gdal_translate -of GTiff merged.vrt OUTPUT.tif</span></div><div><br></div><div>When tripling the number of source rasters (to ~2250) by duplicating entries in the VRT, processing time increases to ~4.5 hours, with RAM usage rising to ~11.5GB.</div><div><span class="font" style="font-family:monospace;">gdal_translate -of GTiff merged_3x.vrt OUTPUT.tif</span></div><div><br></div><div>Extracting a small subset of the VRT via -projwin does not improve performance (still slow, ~14GB RAM used).</div><div><span class="font" style="font-family:monospace;">gdal_translate -projwin -146955.241 797044.4497 -138766.1444 789656.0648 -of GTiff merged_3x.vrt OUTPUT.tif</span></div><div><br></div><div><b>Removing the pixel function makes processing instantaneous, even with 2250 rasters.</b></div><div><br></div><div>Performance is unaffected by --config GDAL_CACHEMAX or -co NUM_THREADS=ALL_CPUS.</div><div><br></div><div>The issue persists across other datasets that I have. In fact, it gets really worse when source rasters are relatively larger or of Float32 data type.</div><div><br></div><div>Gdalinfo on one of the source rasters: <a href="https://pastebin.com/gjHDA2Wd" target="_blank">https://pastebin.com/gjHDA2Wd</a></div><div><b>Data:</b> <a href="https://drive.google.com/file/d/1LGlzGGZvkPyXvKKgkPVzQGbBw55p5dRd/view" target="_blank">https://drive.google.com/file/d/1LGlzGGZvkPyXvKKgkPVzQGbBw55p5dRd/view</a>?</div><div><br></div><div>System: Windows, 8 CPUs, 32GB RAM (GDAL 3.9.2 via OSGeo4W).</div><div><br></div><div>Thank you for your time and insights. Please reply if you are aware of how performance can be improved.</div><div><br></div><div>Regards,</div><div><div dir="ltr" class="qt-gmail_signature"><div dir="ltr"><p class="qt-MsoNormal" style="color:rgb(80, 0, 80);"><span style="color:black;"><b>Abdul Siddiqui, PE</b></span></p><p class="qt-MsoNormal" style="color:rgb(80, 0, 80);"><u style="color:rgb(17, 85, 204);"><a href="mailto:abdul.siddiqui@ertcorp.com" target="_blank">abdul.siddiqui@ertcorp.com</a></u></p><p class="qt-MsoNormal"><br></p><p class="qt-MsoNormal"><b><span style="color:rgb(153, 0, 0);"><span class="size" style="font-size:12pt;">ERT | </span></span></b><span style="color:rgb(128, 6, 37);"><span class="font" style="font-family:Arial, sans-serif;"><span class="size" style="font-size:12pt;">Earth Resources Technology, Inc.</span></span></span><span style="color:rgb(136, 136, 136);"><span class="size" style="font-size:9.5pt;"></span></span></p><p class="qt-MsoNormal"><span style="color:black;"><span class="size" style="font-size:9.5pt;">14401 Sweitzer Ln. Ste 300</span></span><span class="size" style="font-size:9.5pt;"></span></p><p class="qt-MsoNormal"><span style="color:black;"><span class="size" style="font-size:9.5pt;">Laurel, MD 20707</span></span><span class="size" style="font-size:9.5pt;"></span></p><p class="qt-MsoNormal"><a href="https://www.ertcorp.com/" target="_blank">https://www.ertcorp.com</a></p></div></div></div></div><div><div dir="ltr" class="qt-gmail_signature"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><p style="color:rgb(62, 80, 97);margin-top:0px;margin-right:0px;margin-bottom:10px;margin-left:0px;font-size:12px;line-height:14px;"><br></p></div></div></div></div></div></div></div><div>_______________________________________________</div><div>gdal-dev mailing list</div><div><a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a></div><div><a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a></div><div><br></div></blockquote><div style="font-family:Arial;"><br></div></body></html>