[postgis-devel] [PATCH] Memory bug in geography btree functions

[Edmund Horner]

Hi Postgis devs,

When creating a B-tree index on a geography column, PostgreSQL will
use continue to rapidly allocate memory until the command finishes.
For even a moderate size table, it will exhaust all memory on the
server.

For instance, a table consisting of one million random points:

CREATE EXTENSION postgis;
CREATE TABLE t (shape geography NOT NULL);
INSERT INTO t SELECT st_geogfromtext('POINT('||(random()*360 - 180)||'
'||(random()*180-90)||')') FROM generate_series(1, 1000000) AS s(x);
CREATE INDEX ON t(shape);

This reproduces the problem with an error like:

ERROR:  out of memory
DETAIL:  Failed on request of size 32.

We first found it in on a 64 bit Centos VM with 64 GB of memory;
rather than getting this error, the backend process was eventually
killed by the kernel's OOM killer.

This happens on Postgresql 10.1, Postgis 2.4.1 (and older versions
that I tried).

SELECT postgis_full_version(), version();

                          postgis_full_version
    |                                                 version
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------
 POSTGIS="2.4.1 r16012" PGSQL="96" (procs need upgrade for use with
"100") GEOS="3.5.0-CAPI-1.9.0 r4084" PROJ="Rel. 4.8.0, 6 March 2012"
GDAL="GDAL 1.11.4, released 2016/01/25" LIBXML="2.9.1" LIBJSON="0.11"
RASTER | PostgreSQL 10.1 on x86_64-pc-linux-gnu, compiled by gcc (GCC)
4.8.5 20150623 (Red Hat 4.8.5-16), 64-bit

I discussed this with RhodiumToad on #postgresql who pointed out the
lack of PG_FREE_IF_COPY calls in the geography btree functions:
https://github.com/postgis/postgis/blob/svn-trunk/postgis/geography_btree.c

It seems that btree functions are supposed to explicitly free
allocated memory because they may be called repeatedly (such as when
an index is built) in a context that isn't cleared after each call.

The geometry btree functions were fixed earlier in this commit:
https://github.com/postgis/postgis/commit/97fd8b3b0f40b47a87d6d793fe3bf6590855ebc6#diff-94a8c7e68ca5f7092fe7d2280e2124c5

I've attached a patch that does the same for geography.

You could just about use the same C functions for both types, but I
notice lwgeom_le only checks for equality (which could be another
little bug):

PG_FUNCTION_INFO_V1(lwgeom_le);
Datum lwgeom_le(PG_FUNCTION_ARGS)
{
    GSERIALIZED *g1 = PG_GETARG_GSERIALIZED_P(0);
    GSERIALIZED *g2 = PG_GETARG_GSERIALIZED_P(1);
    int cmp = gserialized_cmp(g1, g2);
    PG_FREE_IF_COPY(g1, 0);
    PG_FREE_IF_COPY(g2, 1);
    if (cmp == 0)
      /* should be cmp <= 0? */
        PG_RETURN_BOOL(true);
    else
        PG_RETURN_BOOL(false);
}

Anyway, I hope this is useful.

Cheers,
Edmund
-------------- next part --------------
A non-text attachment was scrubbed...
Name: geog_btree_free_if_copy.patch
Type: application/octet-stream
Size: 1919 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/postgis-devel/attachments/20180126/d5714b8a/attachment.obj>