<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Abdul,</p>
<p>if you add
<SkipNonContributingSources>true</SkipNonContributingSources>
as a child element of the <VRTRasterBand> element, and apply
patch
<a class="moz-txt-link-freetext" href="https://github.com/OSGeo/gdal/commit/3dbc60b334ee022f2993dca476b08d5fed01698c">https://github.com/OSGeo/gdal/commit/3dbc60b334ee022f2993dca476b08d5fed01698c</a>
, "gdal_translate -of GTiff merged.vrt OUTPUT.tif" completes in a
few minutes</p>
<p>Cf
<a class="moz-txt-link-freetext" href="https://gdal.org/en/stable/drivers/raster/vrt.html#using-derived-bands-with-pixel-functions-in-python">https://gdal.org/en/stable/drivers/raster/vrt.html#using-derived-bands-with-pixel-functions-in-python</a>
for the doc of SkipNonContributingSources</p>
<p>Even<br>
</p>
<div class="moz-cite-prefix">Le 14/04/2025 à 06:52, Abdul Raheem
Siddiqui via gdal-dev a écrit :<br>
</div>
<blockquote type="cite"
cite="mid:CACh_PwOdQa4gkKKAAkd61O4OfoqVTkdU569OP3D3qmh9DZnDTA@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div>Dear GDAL Community,<br>
<br>
I am encountering a performance issue when using a VRT
consisting of a large number of source rasters and built-in
C++ pixel function ("max"). I would appreciate any guidance on
whether some GDAL config option can improve this, or I am
doing something wrong, or this is a potential optimization
opportunity.<br>
<br>
I have A VRT file referencing ~750 individual rasters (Byte
data type, avg size ~1000x1000 pixels, untiled, and same CRS
for all source rasters). The VRT uses the built-in “max” pixel
function.<br>
<br>
Running gdal_translate to convert the VRT to GTiff takes ~1.5
hours and consumes ~4GB RAM.<br>
<font face="monospace">gdal_translate -of GTiff merged.vrt
OUTPUT.tif</font><br>
<br>
When tripling the number of source rasters (to ~2250) by
duplicating entries in the VRT, processing time increases to
~4.5 hours, with RAM usage rising to ~11.5GB.<br>
<font face="monospace">gdal_translate -of GTiff merged_3x.vrt
OUTPUT.tif</font><br>
<br>
Extracting a small subset of the VRT via -projwin does not
improve performance (still slow, ~14GB RAM used).<br>
<font face="monospace">gdal_translate -projwin -146955.241
797044.4497 -138766.1444 789656.0648 -of GTiff merged_3x.vrt
OUTPUT.tif</font><br>
<br>
<b>Removing the pixel function makes processing instantaneous,
even with 2250 rasters.</b><br>
<br>
Performance is unaffected by --config GDAL_CACHEMAX or -co
NUM_THREADS=ALL_CPUS.<br>
<br>
The issue persists across other datasets that I have. In fact,
it gets really worse when source rasters are relatively larger
or of Float32 data type.<br>
<br>
Gdalinfo on one of the source rasters: <a
href="https://pastebin.com/gjHDA2Wd" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">https://pastebin.com/gjHDA2Wd</a><br>
<b>Data:</b> <a
href="https://drive.google.com/file/d/1LGlzGGZvkPyXvKKgkPVzQGbBw55p5dRd/view"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://drive.google.com/file/d/1LGlzGGZvkPyXvKKgkPVzQGbBw55p5dRd/view</a>?<br>
<br>
System: Windows, 8 CPUs, 32GB RAM (GDAL 3.9.2 via OSGeo4W).<br>
<br>
Thank you for your time and insights. Please reply if you are
aware of how performance can be improved.<br>
<br>
Regards,
<div>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<p class="MsoNormal" style="color:rgb(80,0,80)"><span
style="color:black"><b>Abdul Siddiqui, PE</b></span></p>
<p class="MsoNormal" style="color:rgb(80,0,80)"><u
style="color:rgb(17,85,204)"><a
href="mailto:abdul.siddiqui@ertcorp.com"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">abdul.siddiqui@ertcorp.com</a></u></p>
<p class="MsoNormal"><br>
</p>
<p class="MsoNormal"><b><span
style="font-size:12pt;color:rgb(153,0,0)">ERT | </span></b><span
style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(128,6,37)">Earth
Resources Technology, Inc.</span><span
style="font-size:9.5pt;color:rgb(136,136,136)"></span></p>
<p class="MsoNormal"><span
style="font-size:9.5pt;color:black">14401 Sweitzer
Ln. Ste 300</span><span style="font-size:9.5pt"></span></p>
<p class="MsoNormal"><span
style="font-size:9.5pt;color:black">Laurel, MD 20707</span><span
style="font-size:9.5pt"></span></p>
<p class="MsoNormal"><a href="https://www.ertcorp.com/"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://www.ertcorp.com</a></p>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset class="moz-mime-attachment-header"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
gdal-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a>
<a class="moz-txt-link-freetext" href="https://lists.osgeo.org/mailman/listinfo/gdal-dev">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a>
</pre>
</blockquote>
<pre class="moz-signature" cols="72">--
<a class="moz-txt-link-freetext" href="http://www.spatialys.com">http://www.spatialys.com</a>
My software is free, but my time generally not.</pre>
</body>
</html>