[gdal-dev] GDAL using tcmalloc accumulates memory

[Stelios Lagaras]

Hi everyone,

That's my first mail on gdal-dev list, thanks in advance for your
feedback/replies.

I have been using the GDAL's ubuntu images, which sets
LD_PRELOAD=libtcmalloc_minimal.so.4 using tcmalloc as per docs hint
Multi-threading
— GDAL documentation
<https://gdal.org/en/latest/user/multithreading.html#ram-fragmentation-and-multi-threading>

Now, when I deploy an intensive IO (/vsis3) workflow in a Kubernetes+Spark
Cluster,
I have observed memory accumulation behavior, which apparently leads to OOM
breakdowns. The workaround I've found is using a clean spark python process
with:
spark.python.worker.reuse=false, though I was curious to dive a bit further
and find the root-cause of the memory stacking.

After letting Opus 5.5 work in a long session, it came back with the
suggestion of opting for TCMALLOC_AGGRESSIVE_DECOMMIT=true which is
expected to address the memory issue.
Now my humble question, is that something that could be set as a default
env inside GDAL images? Could we add a note in the docs for awareness that
the current default behavior
together with Spark may raise this phenomenon?

Thank you,
Stelios

On Sat, Oct 31, 2026 at 4:45 PM Stelios Lagaras <stelios.lagaras at gmail.com>
wrote:

> Hi everyone,
>
> That's my first mail on gdal-dev list, thanks in advance for your
> feedback/replies.
>
> I have been using the GDAL's ubuntu images, which sets
> LD_PRELOAD=libtcmalloc_minimal.so.4 using tcmalloc as per docs hint Multi-threading
> — GDAL documentation
> <https://gdal.org/en/latest/user/multithreading.html#ram-fragmentation-and-multi-threading>
>
> Now, when I deploy an intensive IO (/vsis3) workflow in a Kubernetes+Spark
> Cluster,
> I have observed memory accumulation behavior, which apparently leads to
> OOM breakdowns. The workaround I've found is using a clean spark python
> process with:
> spark.python.worker.reuse=false, though I was curious to dive a bit
> further and find the root-cause of the memory stacking.
>
> After letting Opus 5.5 work in a long session, it came back with the
> suggestion of opting for TCMALLOC_AGGRESSIVE_DECOMMIT=true which is
> expected to address the memory issue.
> Now my humble question, is that something that could be set as a default
> env inside GDAL images? Could we add a note in the docs for awareness that
> the current default behavior
> together with Spark may raise this phenomenon?
>
> Thank you,
> Stelios
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20261104/a9ad6901/attachment.htm>