[postgis-devel] Proj Setup Costs

[Julien Rouhaud]

On Fri, May 14, 2021 at 7:36 AM Justin Pryzby <pryzby at telsasoft.com> wrote:
>
> On Thu, May 13, 2021 at 04:07:56PM -0700, Paul Ramsey wrote:
> > So, as discussed in an earlier thread, the high set-up penalty of the proj_create_crs_to_crs() function in the new proj API can be exercised a lot if a PL/PgSQL function that calls ST_Transform() is used repeatedly, since each call to ST_Transform() inside the PL/PgSQL function will get a fresh FunctionCallInfo and thus a fresh cache, and have to re-lookup transformations that, in the ordinary course of events, would be waiting in cache.
> >
> > Solutions? The only thing to do is take the Proj cache information out of the FunctionCallInfo extra slot (aka fcinfo->fn_extra) and move it to a global so it ends up with a livespan as long as the connection. This means the cache itself might need to be smarter, since it has to deal with potentially a backend lifespan worth of cache entries rather than as many as it might get during a single function lifetime.
> >
> > I'm concerned that we may be mucking around and filling up system memory contexts with things that perhaps we shouldn't, though.
> >
> > For example, we're dumping all our constants into CacheMemoryContext, and we're doing it once per backend. I'm not sure that we're not slowly adding and then orphaning that constant information each time a backend runs through a lifecycle (connection->fork->work->die) since the cache is, perhaps, not per-backend but per cluster? Scary if true.
>
> I don't think it's per cluster/instance - it's just an elaborate wrapper around
> malloc/free.
>
> src/backend/utils/mmgr/README

Yes, all memory contexts are backend local, at least until someone
implements a shared memory context.

> I suggest to create a child context, at least for accounting purposes, even if
> it's never deleted or reset.

+1, it's usually a good practice to create a child of a long lived
memory context (CacheMemoryContext or TopMemoryContext could be used
for that) if a non trivial amount of memory is used.  It can later be
used to diagnose possible memory leak using MemoryContextStats, either
explicitly or not.