<!DOCTYPE html><html><head><title></title><style type="text/css">p.MsoNormal,p.MsoNoSpacing{margin:0}</style></head><body><div style="font-family:Arial;">Hi,<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">Since it's not exactly clear from your description, what operations are you running, just the equivalent of <span class="font" style="font-family:menlo, consolas, monospace, sans-serif;">gdal.Translate()</span>? <span class="font" style="font-family:menlo, consolas, monospace, sans-serif;">gdal.Warp()</span>? GDAL can use threading in a couple of places:<br></div><ul><li style="font-family:Arial;">to compress the output before writing it, e.g. the <span class="font" style="font-family:menlo, consolas, monospace, sans-serif;">NUM_THREADS</span> creation option of GTiff <br></li><li style="font-family:Arial;">to decompress the input when reading a region larger than one block or strip, e.g. the <span class="font" style="font-family:menlo, consolas, monospace, sans-serif;">NUM_THREADS</span> open option of GTiff<br></li><li style="font-family:Arial;">for pipelining the I/O and warping in <span class="font" style="font-family:menlo, consolas, monospace, sans-serif;">gdalwarp</span> (<span class="font" style="font-family:menlo, consolas, monospace, sans-serif;">-multi</span>)<br></li><li style="font-family:Arial;">to parallelize warping itself in gdalwarp (<span class="font" style="font-family:menlo, consolas, monospace, sans-serif;">-wo NUM_THREADS</span>)<br></li></ul><div style="font-family:Arial;">And of course, there might be others I'm not aware of.<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">I'm not sure about the effects you see when setting the cache, but note that the default cache <span class="font" style="font-family:menlo, consolas, monospace, sans-serif;">GDAL_CACHEMAX</span> is "5% of the usable physical RAM, [...] consulted the first
time the cache size is requested". To disable the cache you can use <span class="font" style="font-family:menlo, consolas, monospace, sans-serif;">GDAL_CACHEMAX=0</span>, which can reduce the memory usage and speed up the program in very specific cases (e.g. when processing one block at a time without reading parts of the input twice), but becomes a lot less useful when you do any kind of warping or resampling.<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">Laurentiu</div><div style="font-family:Arial;"><br></div><div>On Tue, Apr 1, 2025, at 10:19, Varisht Ghedia via gdal-dev wrote:<br></div><blockquote type="cite" id="qt" style=""><div dir="ltr"><div class="qt-gmail_quote qt-gmail_quote_container"><div dir="ltr"><p>Dear GDAL Developers,<br></p><p>I am working on optimizing the processing times for MODIS datasets (LST_1Km and QC Day tile) using <code>pymodis</code> with some modifications. Specifically, I have added flags for:<br></p><ul><li><p>Running on all available CPU cores (<code>ALL_CORES</code>)<br></p></li><li><p>Adjusting GDAL cache size (<code>GDAL_CACHEMAX</code>)<br></p></li></ul><p>However, I am observing unexpected performance variations. In some cases, increasing the cache size degrades performance instead of improving it. Below are my test results for two different datasets from the same tile. Tile used: MOD11A1.A2025073.h10v10.061.2025074095514.hdf<br></p><p>EPSG:32618, Resampled to 30m<br></p><h3><b>QC_tile.tif</b><br></h3><pre><code>ALL_CORES + 2G
real 0m24.199s
user 0m53.352s
sys 0m9.998s
STANDARD RUN (No Cache, No Multi-Threading)
real 0m32.133s
user 0m30.581s
sys 0m2.299s
ALL_CORES + 512M
real 0m13.830s
user 0m51.083s
sys 0m1.911s
</code><br></pre><p>With 512M cache, performance improves significantly, but with larger caches (1G, 2G, 4G), execution time increases.<br></p><h3><b>LST_Day_1km.tif</b><br></h3><pre><code>ALL_CORES + 512M
real 0m42.863s
user 0m44.105s
sys 0m3.583s
STANDARD RUN (No Cache, No Multi-Threading)
real 0m45.121s
user 0m26.477s
sys 0m3.712s
ALL_CORES + 2G
real 0m37.548s
user 0m48.302s
sys 0m8.113s
ALL_CORES + 4G
real 0m51.845s
user 0m48.213s
sys 0m7.988s
</code><br></pre><p>For this dataset, using a 2G cache improves performance, but increasing it to 4G makes processing slower.<br></p><h3><b>Questions:</b><br></h3><ol><li><p>How does GDAL’s caching mechanism impact performance in these scenarios?<br></p></li><li><p>Why does increasing cache size sometimes degrade performance?<br></p></li><li><p>Is there a recommended way to tune cache settings for MODIS HDF processing, considering that some layers (like QC) behave differently from others (like LST_1Km)?<br></p></li></ol><p>Any insights into how GDAL handles multi-threading and caching internally would be greatly appreciated.<br></p><p>Thanks in advance for your help!<br></p><p>Best regards,<br></p><p>Varisht Ghedia<br></p></div></div></div><div>_______________________________________________<br></div><div>gdal-dev mailing list<br></div><div><a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a><br></div><div><a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a><br></div><div><br></div></blockquote><div style="font-family:Arial;"><br></div></body></html>