[gdal-dev] GDAL using tcmalloc accumulates memory

[Javier Jimenez Shaw]

I don't see the point of setting a default value to an option without
knowing what is it doing, and if it is really solving a problem.
At least now people can find your comment and try it if they want to.

On Wed, 4 Nov 2026 at 10:20, Stelios Lagaras via gdal-dev <
gdal-dev at lists.osgeo.org> wrote:

> Hi everyone,
>
> That's my first mail on gdal-dev list, thanks in advance for your
> feedback/replies.
>
> I have been using the GDAL's ubuntu images, which sets
> LD_PRELOAD=libtcmalloc_minimal.so.4 using tcmalloc as per docs hint Multi-threading
> — GDAL documentation
> <https://gdal.org/en/latest/user/multithreading.html#ram-fragmentation-and-multi-threading>
>
> Now, when I deploy an intensive IO (/vsis3) workflow in a Kubernetes+Spark
> Cluster,
> I have observed memory accumulation behavior, which apparently leads to
> OOM breakdowns. The workaround I've found is using a clean spark python
> process with:
> spark.python.worker.reuse=false, though I was curious to dive a bit
> further and find the root-cause of the memory stacking.
>
> After letting Opus 5.5 work in a long session, it came back with the
> suggestion of opting for TCMALLOC_AGGRESSIVE_DECOMMIT=true which is
> expected to address the memory issue.
> Now my humble question, is that something that could be set as a default
> env inside GDAL images? Could we add a note in the docs for awareness that
> the current default behavior
> together with Spark may raise this phenomenon?
>
> Thank you,
> Stelios
>
> On Sat, Oct 31, 2026 at 4:45 PM Stelios Lagaras <stelios.lagaras at gmail.com>
> wrote:
>
>> Hi everyone,
>>
>> That's my first mail on gdal-dev list, thanks in advance for your
>> feedback/replies.
>>
>> I have been using the GDAL's ubuntu images, which sets
>> LD_PRELOAD=libtcmalloc_minimal.so.4 using tcmalloc as per docs hint Multi-threading
>> — GDAL documentation
>> <https://gdal.org/en/latest/user/multithreading.html#ram-fragmentation-and-multi-threading>
>>
>> Now, when I deploy an intensive IO (/vsis3) workflow in a
>> Kubernetes+Spark Cluster,
>> I have observed memory accumulation behavior, which apparently leads to
>> OOM breakdowns. The workaround I've found is using a clean spark python
>> process with:
>> spark.python.worker.reuse=false, though I was curious to dive a bit
>> further and find the root-cause of the memory stacking.
>>
>> After letting Opus 5.5 work in a long session, it came back with the
>> suggestion of opting for TCMALLOC_AGGRESSIVE_DECOMMIT=true which is
>> expected to address the memory issue.
>> Now my humble question, is that something that could be set as a default
>> env inside GDAL images? Could we add a note in the docs for awareness that
>> the current default behavior
>> together with Spark may raise this phenomenon?
>>
>> Thank you,
>> Stelios
>>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20261104/f4091a5a/attachment.htm>