[gdal-dev] Fwd: Performance Variability with GDAL Caching and Multi-Threading for MODIS Data

[Varisht Ghedia]

Dear GDAL Developers,

I am working on optimizing the processing times for MODIS datasets (LST_1Km
and QC Day tile) using pymodis with some modifications. Specifically, I
have added flags for:

   -

   Running on all available CPU cores (ALL_CORES)
   -

   Adjusting GDAL cache size (GDAL_CACHEMAX)

However, I am observing unexpected performance variations. In some cases,
increasing the cache size degrades performance instead of improving it.
Below are my test results for two different datasets from the same tile.
Tile used: MOD11A1.A2025073.h10v10.061.2025074095514.hdf

EPSG:32618, Resampled to 30m
*QC_tile.tif*

ALL_CORES + 2G
real    0m24.199s
user    0m53.352s
sys     0m9.998s

STANDARD RUN (No Cache, No Multi-Threading)
real    0m32.133s
user    0m30.581s
sys     0m2.299s

ALL_CORES + 512M
real    0m13.830s
user    0m51.083s
sys     0m1.911s

With 512M cache, performance improves significantly, but with larger caches
(1G, 2G, 4G), execution time increases.
*LST_Day_1km.tif*

ALL_CORES + 512M
real    0m42.863s
user    0m44.105s
sys     0m3.583s

STANDARD RUN (No Cache, No Multi-Threading)
real    0m45.121s
user    0m26.477s
sys     0m3.712s

ALL_CORES + 2G
real    0m37.548s
user    0m48.302s
sys     0m8.113s

ALL_CORES + 4G
real    0m51.845s
user    0m48.213s
sys     0m7.988s

For this dataset, using a 2G cache improves performance, but increasing it
to 4G makes processing slower.
*Questions:*

   1.

   How does GDAL’s caching mechanism impact performance in these scenarios?
   2.

   Why does increasing cache size sometimes degrade performance?
   3.

   Is there a recommended way to tune cache settings for MODIS HDF
   processing, considering that some layers (like QC) behave differently from
   others (like LST_1Km)?

Any insights into how GDAL handles multi-threading and caching internally
would be greatly appreciated.

Thanks in advance for your help!

Best regards,

Varisht Ghedia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20250401/4b0b268d/attachment-0001.htm>