[postgis-devel] [pgrouting] need help with std::bad_alloc issue

[Sandro Santilli]

On Tue, Jun 11, 2013 at 11:32:47AM -0400, Stephen Woodbridge wrote:
> On 6/11/2013 10:58 AM, Bborie Park wrote:
> >Steve,
> >
> >On what platform? Windows? Linux?
> 
> Linux, pg 9.2.4
> >
> >By the looks of it (I'm not very good at C++), std::bad_alloc comes
> >from a failed new allocation. Run gdb or valgrind yet?
> 
> Yes, gdb is not vary useful because the postgresql is compiled with
> -PIE and gdb does not support that well. valgrind is my friend I
> have run it before but it does not report anything useful in this
> case.
> 
> I'm not great with C++ either, but I'm stuck on that fact that this
> seems like C++ does not know how much memory is available so it
> fails but after connecting to the database it seems to have a better
> idea.

Did you look at the system memory state while testing ?
I suspect you're just not releasing memory associated with results
of queries run in previous connection, so that re-connection releases
them all for you, or something like that. Alternatively there might be
a memory leak in some database-side functions so that re-connecting
quits the old backend and releases all leaked memory with it.
Valgrind doesn't help because the memory isn't really lost, but rather
hold by the postgresql backend pool (released on reconnect).

As you said, bad_alloc doesn't come from palloc, but who's keeping 
all the system memory busy is still not known at this stage, so it
could be either sides. To be frank I think it's more likely for it
to be in pgrouting code itself, what do you think ?

--strk;

> 
> We use a lot of std::vector structures, these are arrays that get
> dynamically extended and as a result have a lot of realloc
> equovalents on them the causes memory fragmentation, can where
> possible I have changed the code to reserve a minimum size which has
> helped a lot, but this kind of issue is consistently reproduced
> regardless of reconnecting.
> 
> I'll take another run at it with valgrind, but I'm pretty sure this
> will not show anything new.
> 
> Thanks,
>   -Steve
> 
> >-bborie
> >
> >On Tue, Jun 11, 2013 at 7:33 AM, Stephen Woodbridge
> ><woodbri at swoodbridge.com> wrote:
> >>Hi devs,
> >>
> >>I have run into a strange problem with some pgrouting functions that you
> >>guys might have already seen in postgis.
> >>
> >>We have stored procedures that are C and C++ and in general they work fine,
> >>but if I create a database, connect to it, create extensions and run some
> >>commands I get std::bad_alloc error. If I simply reconnect to the database,
> >>the same command does not generate an error, and, in fact, if I reconnect
> >>after installing the extension, I never get this error.
> >>
> >>I have traced this down to the particular statement that is throwing the
> >>error, but there is nothing unique or particular about it. And we have seen
> >>this behavior in multiple commands.
> >>
> >>So I have to conclude that:
> >>
> >>1. we use the same pattern for most of our commands so that might be flawed
> >>in some basic way regarding memory
> >>
> >>2. that there  is something strange about create extension in that the
> >>libraries are not getting initialized correctly (or completely?) until a
> >>connection is made. We are trying to verify this is or is not unique to pg
> >>9.2.
> >>
> >>3. pgrouting installs multiple shared libraries in our extension and maybe
> >>postgresql assumes there is only going to be one shared library
> >>
> >>4. or something else totally different that we are missing
> >>
> >>So has anyone seen anything like this with postgis code?
> >>Any thoughts on what this might be? or how to run it down?
> >>
> >>I did post post a inquiry to the postgresql list and Tom responded with not
> >>enough information and to compile the server with --enable-cassert which I
> >>did (assuming my Debian recompile worked correctly), but since this is a C++
> >>error and not a postgresql palloc issue we have not seen and cassert errors.
> >>
> >>Thoughts?
> >>
> >>-Steve