[gdal-dev] Fwd: Performance Variability with GDAL Caching and Multi-Threading for MODIS Data

[Laurențiu Nicola]

Hi,

Since it's not exactly clear from your description, what operations are you running, just the equivalent of gdal.Translate()? gdal.Warp()? GDAL can use threading in a couple of places:
 • to compress the output before writing it, e.g. the NUM_THREADS creation option of GTiff 
 • to decompress the input when reading a region larger than one block or strip, e.g. the NUM_THREADS open option of GTiff
 • for pipelining the I/O and warping in gdalwarp (-multi)
 • to parallelize warping itself in gdalwarp (-wo NUM_THREADS)
And of course, there might be others I'm not aware of.

I'm not sure about the effects you see when setting the cache, but note that the default cache GDAL_CACHEMAX is "5% of the  usable physical RAM, [...] consulted the first time the cache size is requested". To disable the cache you can use GDAL_CACHEMAX=0, which can reduce the memory usage and speed up the program in very specific cases (e.g. when processing one block at a time without reading parts of the input twice), but becomes a lot less useful when you do any kind of warping or resampling.

Laurentiu

On Tue, Apr 1, 2025, at 10:19, Varisht Ghedia via gdal-dev wrote:
> Dear GDAL Developers,
> 
> I am working on optimizing the processing times for MODIS datasets (LST_1Km and QC Day tile) using `pymodis` with some modifications. Specifically, I have added flags for:
> 
>  • Running on all available CPU cores (`ALL_CORES`)
> 
>  • Adjusting GDAL cache size (`GDAL_CACHEMAX`)
> 
> However, I am observing unexpected performance variations. In some cases, increasing the cache size degrades performance instead of improving it. Below are my test results for two different datasets from the same tile. Tile used: MOD11A1.A2025073.h10v10.061.2025074095514.hdf
> 
> EPSG:32618, Resampled to 30m
> 
> *QC_tile.tif*
> 
> `ALL_CORES + 2G  
> real    0m24.199s  
> user    0m53.352s  
> sys     0m9.998s  
> 
> STANDARD RUN (No Cache, No Multi-Threading)  
> real    0m32.133s  
> user    0m30.581s  
> sys     0m2.299s  
> 
> ALL_CORES + 512M  
> real    0m13.830s  
> user    0m51.083s  
> sys     0m1.911s  
`
> With 512M cache, performance improves significantly, but with larger caches (1G, 2G, 4G), execution time increases.
> 
> *LST_Day_1km.tif*
> 
> `ALL_CORES + 512M  
> real    0m42.863s  
> user    0m44.105s  
> sys     0m3.583s  
> 
> STANDARD RUN (No Cache, No Multi-Threading)  
> real    0m45.121s  
> user    0m26.477s  
> sys     0m3.712s  
> 
> ALL_CORES + 2G  
> real    0m37.548s  
> user    0m48.302s  
> sys     0m8.113s  
> 
> ALL_CORES + 4G  
> real    0m51.845s  
> user    0m48.213s  
> sys     0m7.988s  
`
> For this dataset, using a 2G cache improves performance, but increasing it to 4G makes processing slower.
> 
> *Questions:*
> 
>  1. How does GDAL’s caching mechanism impact performance in these scenarios?
> 
>  2. Why does increasing cache size sometimes degrade performance?
> 
>  3. Is there a recommended way to tune cache settings for MODIS HDF processing, considering that some layers (like QC) behave differently from others (like LST_1Km)?
> 
> Any insights into how GDAL handles multi-threading and caching internally would be greatly appreciated.
> 
> Thanks in advance for your help!
> 
> Best regards,
> 
> Varisht Ghedia
> 
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20250401/266ea99f/attachment.htm>