[postgis-devel] [PATCH] Memory bug in geography btree functions

[Paul Ramsey]

Thank you for the patch, that’s a good fix to have, I’ll have a look and apply back across our many versions ASAP.
P

> On Jan 25, 2018, at 10:47 PM, Edmund Horner <ejrh00 at gmail.com> wrote:
> 
> Hi Postgis devs,
> 
> When creating a B-tree index on a geography column, PostgreSQL will
> use continue to rapidly allocate memory until the command finishes.
> For even a moderate size table, it will exhaust all memory on the
> server.
> 
> For instance, a table consisting of one million random points:
> 
> CREATE EXTENSION postgis;
> CREATE TABLE t (shape geography NOT NULL);
> INSERT INTO t SELECT st_geogfromtext('POINT('||(random()*360 - 180)||'
> '||(random()*180-90)||')') FROM generate_series(1, 1000000) AS s(x);
> CREATE INDEX ON t(shape);
> 
> This reproduces the problem with an error like:
> 
> ERROR:  out of memory
> DETAIL:  Failed on request of size 32.
> 
> We first found it in on a 64 bit Centos VM with 64 GB of memory;
> rather than getting this error, the backend process was eventually
> killed by the kernel's OOM killer.
> 
> This happens on Postgresql 10.1, Postgis 2.4.1 (and older versions
> that I tried).
> 
> SELECT postgis_full_version(), version();
> 
>                          postgis_full_version
>    |                                                 version
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------
> POSTGIS="2.4.1 r16012" PGSQL="96" (procs need upgrade for use with
> "100") GEOS="3.5.0-CAPI-1.9.0 r4084" PROJ="Rel. 4.8.0, 6 March 2012"
> GDAL="GDAL 1.11.4, released 2016/01/25" LIBXML="2.9.1" LIBJSON="0.11"
> RASTER | PostgreSQL 10.1 on x86_64-pc-linux-gnu, compiled by gcc (GCC)
> 4.8.5 20150623 (Red Hat 4.8.5-16), 64-bit
> 
> I discussed this with RhodiumToad on #postgresql who pointed out the
> lack of PG_FREE_IF_COPY calls in the geography btree functions:
> https://github.com/postgis/postgis/blob/svn-trunk/postgis/geography_btree.c
> 
> It seems that btree functions are supposed to explicitly free
> allocated memory because they may be called repeatedly (such as when
> an index is built) in a context that isn't cleared after each call.
> 
> The geometry btree functions were fixed earlier in this commit:
> https://github.com/postgis/postgis/commit/97fd8b3b0f40b47a87d6d793fe3bf6590855ebc6#diff-94a8c7e68ca5f7092fe7d2280e2124c5
> 
> I've attached a patch that does the same for geography.
> 
> You could just about use the same C functions for both types, but I
> notice lwgeom_le only checks for equality (which could be another
> little bug):
> 
> PG_FUNCTION_INFO_V1(lwgeom_le);
> Datum lwgeom_le(PG_FUNCTION_ARGS)
> {
>    GSERIALIZED *g1 = PG_GETARG_GSERIALIZED_P(0);
>    GSERIALIZED *g2 = PG_GETARG_GSERIALIZED_P(1);
>    int cmp = gserialized_cmp(g1, g2);
>    PG_FREE_IF_COPY(g1, 0);
>    PG_FREE_IF_COPY(g2, 1);
>    if (cmp == 0)
>      /* should be cmp <= 0? */
>        PG_RETURN_BOOL(true);
>    else
>        PG_RETURN_BOOL(false);
> }
> 
> Anyway, I hope this is useful.
> 
> Cheers,
> Edmund
> <geog_btree_free_if_copy.patch>_______________________________________________
> postgis-devel mailing list
> postgis-devel at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/postgis-devel