[postgis-devel] running PostGIS on valgrind-enabled build /identifying memory issues

[Brian M Hamlin]

Hi Tomas - 

   Using a current PostGIS 2.4 / Pg 10.2 / GEOS 3.6.2 on Ubuntu 1604
I built PostGIS using the CFLAGS you indicated, installed, then ran 
valgrind make test. 
The log is not showing a lot of errors. I admit I have not looked at 
your sample log. 
 
  Am I missing a step here?  best regards  --Brian
 
--
pgs_work$ ./configure --enable-debug --enable-cassert CFLAGS='-Og 
-ggdb3 -fno-omit-frame-pointer -DUSE_VALGRIND 
-DRANDOMIZE_ALLOCATED_MEMORY'
--
pgs_work$ valgrind --log-file=/tmp/pg1.log make check
 
--
==16117== Memcheck, a memory error detector
==16117== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al. 
==16117== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==16117== Command: make check
==16117== Parent PID: 23666
==16117== 
==16117== 
==16117== HEAP SUMMARY:
==16117==     in use at exit: 153,104 bytes in 1,408 blocks
==16117==   total heap usage: 2,274 allocs, 866 frees, 373,775 bytes allocated
==16117== 
==16117== LEAK SUMMARY:
==16117==    definitely lost: 50 bytes in 5 blocks
==16117==    indirectly lost: 0 bytes in 0 blocks
==16117==      possibly lost: 0 bytes in 0 blocks
==16117==    still reachable: 153,054 bytes in 1,403 blocks
==16117==         suppressed: 0 bytes in 0 blocks
==16117== Rerun with --leak-check=full to see details of leaked memory
==16117== 
==16117== For counts of detected and suppressed errors, rerun with: -v
==16117== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
--
 
(the leaks seem to be from make, and very small)
 

On Sat, 10 Feb 2018 02:30:27 0100, Tomas Vondra 
<tomas.vondra at 2ndquadrant.com> wrote:

Hi,

We have a bunch of customers running PostGIS, and from time to time they
report us some sort of crash. The reports are rare and not reproducible,
so it's difficult to identify the actual root cause. But the symptoms
are usually consistent with memory corruption - invalid pointers in core
dumps, segfaults, etc. 

For PostgreSQL, valgrind turned out to be an invaluable tool to identify
various memory issues, so after after yet another crash report I had an
idea to try running valgrind-enabled PostGIS build. 

So built current PostgreSQL and PostGIS master branches (f069c91a57 and
d613598e67) with valgrind support:

./configure --enable-debug --enable-cassert
--prefix=/home/tomas/pg-debug
CFLAGS="-O0 -ggdb3 -fno-omit-frame-pointer
-DUSE_VALGRIND -DRANDOMIZE_ALLOCATED_MEMORY"

Disabling optimizations makes valgrind reports more accurate in my
experience, ggdb3 makes them easier to analyze. 

Running "make check" (in PostGIS source tree) against such build
identifies a fairly high number of possible issues. The report is
usually 0.5-1MB, but many of the reports follow the same pattern. 

See the attached example report - pretty much all the initial reports
are "jumps on uninitialised value" in do_analyze_rel. I didn't get to
the bottom of this yet, but my understanding is the GBOX in
compute_gserialized_stats_mode may not be fully initialized in some
cases, and some of the uninitialised fields are accessed later. 

There are a few more such patterns in the report, including for example
"invalid read" issues which in my experience usually means out-of-bounds
access (buffer overflows and such). 

I'm wondering if those are known issues - for example, while I find
valgrind quite reliable, it's still possible those are false positives. 

I plan to continue working on this, but I'm not very familiar with the
PostGIS code, so I'd appreciate help from the more experienced people on
this list. It's likely you'll immediately spot where the issue is. 
Anyone willing to look at the valgrind report?

The attached pg_ctl.patch makes it easier to start PostgreSQL under
valgrind. You may need to modify the hard-coded paths, though. 

regards

-- 
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

-------------------------

_______________________________________________
postgis-devel mailing list
postgis-devel at lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/postgis-devel

--
Brian M Hamlin
OSGeo California
blog.light42.com