[pgrouting-dev] SSL SYSCALL error: EOF detected

Mon Feb 23 04:52:42 PST 2015

Dear Stephen et al.

This seems to be a memory leak in pgr_dijkstra; RAM slowly swells up
until malloc fails. It happens after some 180 000 paths have been
calculated, meaning that each call is leaving behind about 10 Kb of
memory.

A stack trace is attached. I do not have the debug symbols for libc,
but for postgres it seems to be all there.

I'll keep you posted if I find something more.

Regards,

Luís

On 20 February 2015 at 15:16, Stephen Woodbridge
<woodbri at swoodbridge.com> wrote:
> On 2/20/2015 5:58 AM, Luís de Sousa wrote:
>>
>> Dear all,
>>
>> I am presently implementing the Optimal Meeting Point (OMP) [0]
>> algorithm with PyWPS. I have a draft brute force algorithm that in
>> smaller networks computes the OMP correctly by testing all nodes of
>> the network with pgr_dijkstra (I am not yet using heuristics since
>> costs are not only distance dependent).
>>
>> I am now testing this algorithm in a larger network, of some 50 000
>> nodes. I previously reported that for about 7%-8% of the nodes
>> pgr_dijkstra fails, returning an empty tuple [1]. After patching these
>> exceptions, I am stumping into another issue: somewhere between an
>> half and two thirds of the network have been processed, Postgres
>> crashes and I get the error: "SSL SYSCALL error: EOF detected" [2, 3].
>> This crash seems to happen randomly, with pgr_dijkstra invoked on
>> different nodes.
>
>
> Hi Luís,
>
> The error: "SSL SYSCALL error: EOF detected" is because the postgresql
> backend has crashed and you connection in python has terminated.
>
>> This issue should also be easy to circumvent, but it seems to be
>> symptom of something a bit more serious with pgrouting.
>
>
> There are a number of things that can crash the database, like running out
> of memory, or running into a bug in the code.
>
> google: gdb postgres stack trace
>
> There are a couple of ways to get a stack trace of a crash.
> 1. is to enable core files
> 2. is to run the database under gdb
>
> HTH,
>   -Steve
>
>> Regards,
>>
>> Luís
>>
>> [0] https://github.com/pgRouting/pgrouting/issues/289
>>
>> [1]
>> http://pgrouting-users.974093.n3.nabble.com/pgrouting-users-Empty-tuple-from-pgr-dijkstra-tp4025706.html
>>
>> [2]
>> http://stackoverflow.com/questions/20201711/pg-internalerror-ssl-syscall-error-eof-detected
>>
>> [3]
>> http://stackoverflow.com/questions/20217571/psycopg2-interfaceerror-connection-already-closed-pgr-astar
>> _______________________________________________
>> pgrouting-dev mailing list
>> pgrouting-dev at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/pgrouting-dev
>>
>
> _______________________________________________
> pgrouting-dev mailing list
> pgrouting-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/pgrouting-dev
-------------- next part --------------
GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.launchpad.net/gdb-linaro/>.
(gdb) handle SIG33 pass nostop noprint
Signal        Stop	Print	Pass to program	Description
SIG33         No	No	Yes		Real-time event 33
(gdb) set pagination 0
(gdb) attach 19546
Attaching to process 19546
Reading symbols from /usr/lib/postgresql/9.1/bin/postgres...Reading symbols from /usr/lib/debug/usr/lib/postgresql/9.1/bin/postgres...done.
done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libxml2.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libxml2.so.2
Reading symbols from /lib/x86_64-linux-gnu/libpam.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libpam.so.0
Reading symbols from /lib/x86_64-linux-gnu/libssl.so.1.0.0...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libssl.so.1.0.0
Reading symbols from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
Reading symbols from /usr/lib/x86_64-linux-gnu/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libkrb5.so.3
Reading symbols from /lib/x86_64-linux-gnu/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libcom_err.so.2
Reading symbols from /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2
Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libdl.so.2
Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libm.so.6
Reading symbols from /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libc.so.6
Reading symbols from /lib/x86_64-linux-gnu/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libz.so.1
Reading symbols from /usr/lib/x86_64-linux-gnu/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libk5crypto.so.3
Reading symbols from /usr/lib/x86_64-linux-gnu/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libkrb5support.so.0
Reading symbols from /lib/x86_64-linux-gnu/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libkeyutils.so.1
Reading symbols from /lib/x86_64-linux-gnu/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libresolv.so.2
Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Loaded symbols for /lib/x86_64-linux-gnu/libpthread.so.0
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2
Reading symbols from /usr/lib/x86_64-linux-gnu/libsasl2.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libsasl2.so.2
Reading symbols from /usr/lib/x86_64-linux-gnu/libgssapi.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libgssapi.so.3
Reading symbols from /usr/lib/x86_64-linux-gnu/libgnutls.so.26...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libgnutls.so.26
Reading symbols from /lib/x86_64-linux-gnu/libgcrypt.so.11...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libgcrypt.so.11
Reading symbols from /usr/lib/x86_64-linux-gnu/libheimntlm.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libheimntlm.so.0
Reading symbols from /usr/lib/x86_64-linux-gnu/libkrb5.so.26...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libkrb5.so.26
Reading symbols from /usr/lib/x86_64-linux-gnu/libasn1.so.8...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libasn1.so.8
Reading symbols from /usr/lib/x86_64-linux-gnu/libhcrypto.so.4...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libhcrypto.so.4
Reading symbols from /usr/lib/x86_64-linux-gnu/libroken.so.18...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libroken.so.18
Reading symbols from /usr/lib/x86_64-linux-gnu/libtasn1.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libtasn1.so.3
Reading symbols from /usr/lib/x86_64-linux-gnu/libp11-kit.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libp11-kit.so.0
Reading symbols from /lib/x86_64-linux-gnu/libgpg-error.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libgpg-error.so.0
Reading symbols from /usr/lib/x86_64-linux-gnu/libwind.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libwind.so.0
Reading symbols from /usr/lib/x86_64-linux-gnu/libheimbase.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libheimbase.so.1
Reading symbols from /usr/lib/x86_64-linux-gnu/libhx509.so.5...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libhx509.so.5
Reading symbols from /usr/lib/x86_64-linux-gnu/libsqlite3.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libsqlite3.so.0
Reading symbols from /lib/x86_64-linux-gnu/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libcrypt.so.1
Reading symbols from /lib/x86_64-linux-gnu/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_files.so.2
Reading symbols from /usr/lib/postgresql/9.1/lib/postgis-2.0.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/postgresql/9.1/lib/postgis-2.0.so
Reading symbols from /usr/lib/libgeos_c.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libgeos_c.so.1
Reading symbols from /usr/lib/libproj.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libproj.so.0
Reading symbols from /usr/lib/libgeos-3.3.8.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libgeos-3.3.8.so
Reading symbols from /usr/lib/x86_64-linux-gnu/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libstdc++.so.6
Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libgcc_s.so.1
Reading symbols from /usr/lib/postgresql/9.1/lib/librouting.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/postgresql/9.1/lib/librouting.so
0x00007fa9bc676feb in ?? () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) 
(gdb) 
(gdb) 
(gdb) 
(gdb) 
(gdb) 
(gdb) 
(gdb) bt
#0  0x00007fa9bc676feb in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007fa9bc5f9f61 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007fa9bc5f7c87 in malloc () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007fa9bc5a83d3 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#4  0x00007fa9bc5a60c5 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#5  0x00007fa9bc5a5943 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#6  0x00000000007078d9 in errhint (fmt=0x84d7e8 "In a moment you should be able to reconnect to the database and repeat your command.") at /build/buildd/postgresql-9.1-9.1.15/build/../src/backend/utils/error/elog.c:965
#7  0x0000000000641956 in quickdie (postgres_signal_arg=<optimized out>) at /build/buildd/postgresql-9.1-9.1.15/build/../src/backend/tcop/postgres.c:2607
#8  <signal handler called>
#9  0x00007fa9bc5f5c31 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#10 0x00007fa9bc5f7c95 in malloc () from /lib/x86_64-linux-gnu/libc.so.6
#11 0x00007fa9b5fe1ded in operator new(unsigned long) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#12 0x00007fa9b5af02cf in std::pair<std::list<boost::detail::sep_<unsigned long, boost::property<boost::edge_bundle_t, Vertex, boost::no_property> >, std::allocator<boost::detail::sep_<unsigned long, boost::property<boost::edge_bundle_t, Vertex, boost::no_property> > > >::iterator, bool> boost::graph_detail::push<std::list<boost::detail::sep_<unsigned long, boost::property<boost::edge_bundle_t, Vertex, boost::no_property> >, std::allocator<boost::detail::sep_<unsigned long, boost::property<boost::edge_bundle_t, Vertex, boost::no_property> > > >, boost::detail::sep_<unsigned long, boost::property<boost::edge_bundle_t, Vertex, boost::no_property> > >(std::list<boost::detail::sep_<unsigned long, boost::property<boost::edge_bundle_t, Vertex, boost::no_property> >, std::allocator<boost::detail::sep_<unsigned long, boost::property<boost::edge_bundle_t, Vertex, boost::no_property> > > >&, boost::detail::sep_<unsigned long, boost::property<boost::edge_bundle_t, Vertex, boost::no_property> > const&) () from /usr/lib/postgresql/9.1/lib/librouting.so
#13 0x00007fa9b5aef0c7 in boost_dijkstra () from /usr/lib/postgresql/9.1/lib/librouting.so
#14 0x00007fa9b5aeebf0 in shortest_path () from /usr/lib/postgresql/9.1/lib/librouting.so
#15 0x0000000000578edd in ExecMakeTableFunctionResult (funcexpr=0x7fa9affe91f8, econtext=0x7fa9affe8550, argContext=<optimized out>, expectedDesc=0x7fa9affe8da0, randomAccess=0 '\000') at /build/buildd/postgresql-9.1-9.1.15/build/../src/backend/executor/execQual.c:2141
#16 0x000000000058abca in FunctionNext (node=0x7fa9affe8440) at /build/buildd/postgresql-9.1-9.1.15/build/../src/backend/executor/nodeFunctionscan.c:66
#17 0x000000000057b0d7 in ExecScanFetch (recheckMtd=0x58ab40 <FunctionRecheck>, accessMtd=0x58ab50 <FunctionNext>, node=0x7fa9affe8440) at /build/buildd/postgresql-9.1-9.1.15/build/../src/backend/executor/execScan.c:82
#18 ExecScan (node=0x7fa9affe8440, accessMtd=0x58ab50 <FunctionNext>, recheckMtd=0x58ab40 <FunctionRecheck>) at /build/buildd/postgresql-9.1-9.1.15/build/../src/backend/executor/execScan.c:132
#19 0x0000000000573a18 in ExecProcNode (node=0x7fa9affe8440) at /build/buildd/postgresql-9.1-9.1.15/build/../src/backend/executor/execProcnode.c:416
#20 0x0000000000583c5b in MultiExecHash (node=0x7fa9affe7fa0) at /build/buildd/postgresql-9.1-9.1.15/build/../src/backend/executor/nodeHash.c:103
#21 0x000000000058461b in ExecHashJoin (node=0x7fa9ac04f170) at /build/buildd/postgresql-9.1-9.1.15/build/../src/backend/executor/nodeHashjoin.c:177
#22 0x00000000005739a8 in ExecProcNode (node=0x7fa9ac04f170) at /build/buildd/postgresql-9.1-9.1.15/build/../src/backend/executor/execProcnode.c:447
#23 0x0000000000570cd2 in ExecutePlan (dest=0x7fa9ac0152b8, direction=<optimized out>, numberTuples=0, sendTuples=1 '\001', operation=CMD_SELECT, planstate=0x7fa9ac04f170, estate=0x7fa9ac04f010) at /build/buildd/postgresql-9.1-9.1.15/build/../src/backend/executor/execMain.c:1451
#24 standard_ExecutorRun (queryDesc=0x7fa9ac022970, direction=<optimized out>, count=0) at /build/buildd/postgresql-9.1-9.1.15/build/../src/backend/executor/execMain.c:325
#25 0x00000000006471c7 in PortalRunSelect (portal=0x1d08150, forward=<optimized out>, count=0, dest=0x7fa9ac0152b8) at /build/buildd/postgresql-9.1-9.1.15/build/../src/backend/tcop/pquery.c:945
#26 0x00000000006486b0 in PortalRun (portal=0x1d08150, count=9223372036854775807, isTopLevel=1 '\001', dest=0x7fa9ac0152b8, altdest=0x7fa9ac0152b8, completionTag=0x7fffcbddeb30 "") at /build/buildd/postgresql-9.1-9.1.15/build/../src/backend/tcop/pquery.c:789
#27 0x000000000064452a in exec_simple_query (query_string=0x1d99c10 "SELECT seq, id1 AS node, id2 AS edge, cost, ST_AsGeoJSON(b.the_geom) FROM pgr_dijkstra('  SELECT gid AS id,     source::integer,     target::integer,     to_cost::double precision AS cost,     reverse"...) at /build/buildd/postgresql-9.1-9.1.15/build/../src/backend/tcop/postgres.c:1034
#28 PostgresMain (argc=<optimized out>, argv=<optimized out>, dbname=0x1cc87c8 "lamilo_routing", username=<optimized out>) at /build/buildd/postgresql-9.1-9.1.15/build/../src/backend/tcop/postgres.c:4021
#29 0x0000000000605943 in BackendRun (port=0x1d0c970) at /build/buildd/postgresql-9.1-9.1.15/build/../src/backend/postmaster/postmaster.c:3648
#30 BackendStartup (port=0x1d0c970) at /build/buildd/postgresql-9.1-9.1.15/build/../src/backend/postmaster/postmaster.c:3330
#31 ServerLoop () at /build/buildd/postgresql-9.1-9.1.15/build/../src/backend/postmaster/postmaster.c:1483
#32 0x00000000006063bc in PostmasterMain (argc=<optimized out>, argv=<optimized out>) at /build/buildd/postgresql-9.1-9.1.15/build/../src/backend/postmaster/postmaster.c:1144
#33 0x000000000045ac26 in main (argc=5, argv=0x1cc7180) at /build/buildd/postgresql-9.1-9.1.15/build/../src/backend/main/main.c:210
(gdb) exit
Undefined command: "exit".  Try "help".
(gdb) quit
A debugging session is active.

	Inferior 1 [process 19546] will be detached.

Quit anyway? (y or n) Detaching from program: /usr/lib/postgresql/9.1/bin/postgres, process 19546