From lnicola at dend.ro Wed Nov 4 00:18:58 2026 From: lnicola at dend.ro (=?UTF-8?Q?Lauren=C8=9Biu_Nicola?=) Date: Wed, 04 Nov 2026 10:18:58 +0200 Subject: [gdal-dev] New GitHub setting for Copilot reviews Message-ID: <6437b75b-e49b-420e-925c-44792e2ce6cf@app.fastmail.com> Hello, Just a quick FYI, GitHub has a new option under Copilot / Code review / "Block reviews requested with an external Copilot license". Any project that not appreciate LLM usage might want to enable it. (cross-posted to GDAL/PROJ/QGIS) Laurentiu -------------- next part -------------- An HTML attachment was scrubbed... URL: From stelios.lagaras at gmail.com Wed Nov 4 01:19:55 2026 From: stelios.lagaras at gmail.com (Stelios Lagaras) Date: Wed, 4 Nov 2026 10:19:55 +0100 Subject: [gdal-dev] GDAL using tcmalloc accumulates memory In-Reply-To: References: Message-ID: Hi everyone, That's my first mail on gdal-dev list, thanks in advance for your feedback/replies. I have been using the GDAL's ubuntu images, which sets LD_PRELOAD=libtcmalloc_minimal.so.4 using tcmalloc as per docs hint Multi-threading ? GDAL documentation Now, when I deploy an intensive IO (/vsis3) workflow in a Kubernetes+Spark Cluster, I have observed memory accumulation behavior, which apparently leads to OOM breakdowns. The workaround I've found is using a clean spark python process with: spark.python.worker.reuse=false, though I was curious to dive a bit further and find the root-cause of the memory stacking. After letting Opus 5.5 work in a long session, it came back with the suggestion of opting for TCMALLOC_AGGRESSIVE_DECOMMIT=true which is expected to address the memory issue. Now my humble question, is that something that could be set as a default env inside GDAL images? Could we add a note in the docs for awareness that the current default behavior together with Spark may raise this phenomenon? Thank you, Stelios On Sat, Oct 31, 2026 at 4:45?PM Stelios Lagaras wrote: > Hi everyone, > > That's my first mail on gdal-dev list, thanks in advance for your > feedback/replies. > > I have been using the GDAL's ubuntu images, which sets > LD_PRELOAD=libtcmalloc_minimal.so.4 using tcmalloc as per docs hint Multi-threading > ? GDAL documentation > > > Now, when I deploy an intensive IO (/vsis3) workflow in a Kubernetes+Spark > Cluster, > I have observed memory accumulation behavior, which apparently leads to > OOM breakdowns. The workaround I've found is using a clean spark python > process with: > spark.python.worker.reuse=false, though I was curious to dive a bit > further and find the root-cause of the memory stacking. > > After letting Opus 5.5 work in a long session, it came back with the > suggestion of opting for TCMALLOC_AGGRESSIVE_DECOMMIT=true which is > expected to address the memory issue. > Now my humble question, is that something that could be set as a default > env inside GDAL images? Could we add a note in the docs for awareness that > the current default behavior > together with Spark may raise this phenomenon? > > Thank you, > Stelios > -------------- next part -------------- An HTML attachment was scrubbed... URL: From even.rouault at spatialys.com Wed Nov 4 03:36:52 2026 From: even.rouault at spatialys.com (Even Rouault) Date: Wed, 4 Nov 2026 12:36:52 +0100 Subject: [gdal-dev] Stepping out from GDAL PSC Message-ID: Hi, I'm stepping out from my position of member and chair of the GDAL PSC. Those years in this role have been great and I'm proud what the team has accomplished and want to thank all the people that have participated and will make the project still shine. I will only look from a distance but happy to try helping in case it would be needed. Best regards, Even -- http://www.spatialys.com My software is free, but my time generally not. LLMs contribute to global warming and brain rot. Let's guillotine them! "Ah ! ?a ira, ?a ira, ?a ira !" Initiator of https://fuckaifoundation.org From j1 at jimenezshaw.com Wed Nov 4 05:07:35 2026 From: j1 at jimenezshaw.com (Javier Jimenez Shaw) Date: Wed, 4 Nov 2026 14:07:35 +0100 Subject: [gdal-dev] GDAL using tcmalloc accumulates memory In-Reply-To: References: Message-ID: I don't see the point of setting a default value to an option without knowing what is it doing, and if it is really solving a problem. At least now people can find your comment and try it if they want to. On Wed, 4 Nov 2026 at 10:20, Stelios Lagaras via gdal-dev < gdal-dev at lists.osgeo.org> wrote: > Hi everyone, > > That's my first mail on gdal-dev list, thanks in advance for your > feedback/replies. > > I have been using the GDAL's ubuntu images, which sets > LD_PRELOAD=libtcmalloc_minimal.so.4 using tcmalloc as per docs hint Multi-threading > ? GDAL documentation > > > Now, when I deploy an intensive IO (/vsis3) workflow in a Kubernetes+Spark > Cluster, > I have observed memory accumulation behavior, which apparently leads to > OOM breakdowns. The workaround I've found is using a clean spark python > process with: > spark.python.worker.reuse=false, though I was curious to dive a bit > further and find the root-cause of the memory stacking. > > After letting Opus 5.5 work in a long session, it came back with the > suggestion of opting for TCMALLOC_AGGRESSIVE_DECOMMIT=true which is > expected to address the memory issue. > Now my humble question, is that something that could be set as a default > env inside GDAL images? Could we add a note in the docs for awareness that > the current default behavior > together with Spark may raise this phenomenon? > > Thank you, > Stelios > > On Sat, Oct 31, 2026 at 4:45?PM Stelios Lagaras > wrote: > >> Hi everyone, >> >> That's my first mail on gdal-dev list, thanks in advance for your >> feedback/replies. >> >> I have been using the GDAL's ubuntu images, which sets >> LD_PRELOAD=libtcmalloc_minimal.so.4 using tcmalloc as per docs hint Multi-threading >> ? GDAL documentation >> >> >> Now, when I deploy an intensive IO (/vsis3) workflow in a >> Kubernetes+Spark Cluster, >> I have observed memory accumulation behavior, which apparently leads to >> OOM breakdowns. The workaround I've found is using a clean spark python >> process with: >> spark.python.worker.reuse=false, though I was curious to dive a bit >> further and find the root-cause of the memory stacking. >> >> After letting Opus 5.5 work in a long session, it came back with the >> suggestion of opting for TCMALLOC_AGGRESSIVE_DECOMMIT=true which is >> expected to address the memory issue. >> Now my humble question, is that something that could be set as a default >> env inside GDAL images? Could we add a note in the docs for awareness that >> the current default behavior >> together with Spark may raise this phenomenon? >> >> Thank you, >> Stelios >> > _______________________________________________ > gdal-dev mailing list > gdal-dev at lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gdt at lexort.com Wed Nov 4 05:28:46 2026 From: gdt at lexort.com (Greg Troxel) Date: Wed, 04 Nov 2026 08:28:46 -0500 Subject: [gdal-dev] GDAL using tcmalloc accumulates memory In-Reply-To: (Javier Jimenez Shaw via gdal-dev's message of "Wed, 4 Nov 2026 14:07:35 +0100") References: Message-ID: Javier Jimenez Shaw via gdal-dev writes: > I don't see the point of setting a default value to an option without > knowing what is it doing, and if it is really solving a problem. > At least now people can find your comment and try it if they want to. Agreed. The compiled defaults should be reasonable for the broad class of users. I find environment variables being set in an "image", at all, quite surprising. And Stelios: I personally do not want to even indirectly read LLM output. If that can help you read the docs or find something reliable, that's up to you but an extrusion from the soup of the internet is not trustworthy. It's certainly not a basis for changing anything in gdal. I think it will take someone who really understands the layers of memory allocation, and caching of allocations. Allocators can try to retain freed objects to speed future allocations at the expense of continuing to use memory, instead of returning it to the next-higher-level allocator. If a massive amount of memory is retained in this way, that sounds like a bug. Separately, garbage collection is hard to do and hard to really understand. From tmoore at spatial.ca Wed Nov 4 06:43:47 2026 From: tmoore at spatial.ca (Tom Moore) Date: Wed, 04 Nov 2026 09:43:47 -0500 Subject: [gdal-dev] Stepping out from GDAL PSC In-Reply-To: References: Message-ID: Hi Even Thank you for all that you have done for this project. I was an infrequent participant in the community, but I appreciated your generous assistance when I had questions or was submitting PR's, and I am greatly impressed by your productivity and contributions to the project as evidenced by all of PR's attributed to you. You have been a wonderful steward of the project who will be difficult to replace. Tom On Wed, Nov 4, 2026, at 6:36 AM, Even Rouault via gdal-dev wrote: > Hi, > > I'm stepping out from my position of member and chair of the GDAL PSC. > Those years in this role have been great and I'm proud what the team has > accomplished and want to thank all the people that have participated and > will make the project still shine. I will only look from a distance but > happy to try helping in case it would be needed. > > Best regards, > > Even > > -- > http://www.spatialys.com > My software is free, but my time generally not. > LLMs contribute to global warming and brain rot. > Let's guillotine them! "Ah ! ?a ira, ?a ira, ?a ira !" > Initiator of https://fuckaifoundation.org > > _______________________________________________ > gdal-dev mailing list > gdal-dev at lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nelson.gerald.c at gmail.com Wed Nov 4 08:26:25 2026 From: nelson.gerald.c at gmail.com (Gerald Nelson) Date: Wed, 4 Nov 2026 16:26:25 +0000 Subject: [gdal-dev] Stepping out from GDAL PSC In-Reply-To: References: Message-ID: I am a lurking follower of the gdal-dev list. I have enjoyed reading the exchanges between people who really know what they are doing and benefitting from it indirectly. I agree with Tom about how Even?s leadership has been an impressive, key element to its success. And that he will be difficult to replace. Jerry -- Gerald C. Nelson Professor Emeritus, University of Illinois, Urbana-Champaign Telephone: +1-217-390-7888 Email: nelson.gerald.c at gmail.com http://bit.ly/1arho7d From: gdal-dev on behalf of Tom Moore via gdal-dev Date: Wednesday, November 4, 2026 at 7:53?AM To: Even Rouault Cc: gdal-dev at lists.osgeo.org Subject: Re: [gdal-dev] Stepping out from GDAL PSC Hi Even Thank you for all that you have done for this project. I was an infrequent participant in the community, but I appreciated your generous assistance when I had questions or was submitting PR's, and I am greatly impressed by your productivity and contributions to the project as evidenced by all of PR's attributed to you. You have been a wonderful steward of the project who will be difficult to replace. Tom On Wed, Nov 4, 2026, at 6:36 AM, Even Rouault via gdal-dev wrote: Hi, I'm stepping out from my position of member and chair of the GDAL PSC. Those years in this role have been great and I'm proud what the team has accomplished and want to thank all the people that have participated and will make the project still shine. I will only look from a distance but happy to try helping in case it would be needed. Best regards, Even -- http://www.spatialys.com My software is free, but my time generally not. LLMs contribute to global warming and brain rot. Let's guillotine them! "Ah ! ?a ira, ?a ira, ?a ira !" Initiator of https://fuckaifoundation.org _______________________________________________ gdal-dev mailing list gdal-dev at lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: