<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
<title></title>
</head>
<body bgcolor="#ffffff" text="#000000">
Thanks strk....what's the best way to upgrade to the head? Do I have
to do a dump and load, or can I use the upgrade script? Just wondering
because the database is about 50 gigs.<br>
<br>
Charlie<br>
<br>
<a class="moz-txt-link-abbreviated" href="mailto:strk@refractions.net">strk@refractions.net</a> wrote:
<blockquote cite="mid20051209120517.GA15283@keybit.net" type="cite">
<pre wrap="">Charlie, thanks for the report.
The leak is plugged in CVS (both 1.0 and HEAD branches).
Also, I've added a couple of tests for polygonize in the
regress_ogc files.
--strk;
On Fri, Dec 09, 2005 at 01:26:55AM -0700, Charlie Savage wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi everyone,
I've seem to have discovered some pretty big memory leaks in polygonize.
Setup:
postgis 1.0.6
geos head (also seen on 2.1.4)
opensuse 10
Dell Dimension, 1Gb memory, 400 Gb Sata hard drive
Postgresql 8.1
Test case:
Running polygonize on approximately 3.6 million line chains, forming
approximately ?? polygons. I can run the query once, sometimes twice
(in the same client session), but then linux will kill the postgres
process sending it a kill -9 due to out of memory condition. Note that
the memory is not given back until the postmaster process serving the
query ends.
Example Mem Usage before query:
MemTotal: 1032980 kB
MemFree: 108124 kB
SwapTotal: 2032140 kB
SwapFree: 1979156 kB
Example Mem Usage after:
MemTotal: 1032980 kB
MemFree: 14032 kB
SwapTotal: 2032140 kB
SwapFree: 994252 kB
After the postgresql process grabs all free memory, then it starts to
eat away at the swap memory. You can see that the swap memory has been
reduced by half, roughly 1 megabyte.
To see what was causing the problem, I ran postmaster using valgrind.
This showed lots of little leaks, a few mid-size leaks, and one big one.
1. This is by far the biggest leak.
==28187== 8,905,045 (959,160 direct, 7,945,885 indirect) bytes in 39,965
blocks are definitely lost in loss record 68 of 70
==28187== at 0x401B7D6: operator new(unsigned) (vg_replace_malloc.c:164)
==28187== by 0x49A169B:
geos::GeometryFactory::createLineString(geos::CoordinateSequence*) const
(GeometryFactory.cpp:615)
==28187== by 0x4827BBA: PostGIS2GEOS_linestring
(lwgeom_geos_wrapper.cpp:336)==28187== by 0x48349F1: LWGEOM2GEOS
(lwgeom_geos.c:2321)
==28187== by 0x4834AC2: POSTGIS2GEOS (lwgeom_geos.c:2364)
==28187== by 0x48373D0: GEOS_polygonize_garray (lwgeom_geos.c:2452)
==28187== by 0x812D5D4: finalize_aggregate (in /usr/bin/postgres)
==28187== by 0x812DF19: ExecAgg (in /usr/bin/postgres)
==28187== by 0x8124B80: ExecProcNode (in /usr/bin/postgres)
==28187== by 0x81238C2: ExecutorRun (in /usr/bin/postgres)
==28187== by 0x819D57B: ProcessQuery (in /usr/bin/postgres)
==28187== by 0x819E9A9: PortalRun (in /usr/bin/postgres)
Looking at GEOS_polygonize_garray, line 2471, I see:
//pfree(vgeoms)
I'd guess this is the problem, all the line strings in the vgeoms area
that are allocated by geos are not destroyed. This is the first time
I've looked at the PostGIS code, but it seems to me that pfree is not
the correct call since vgeoms is created by GEOS and is not under
postgresql memory management.
2. This is the second biggest leak:
==28187== 713,741 (91,308 direct, 622,433 indirect) bytes in 7,609
blocks are definitely lost in loss record 65 of 70
==28187== at 0x401B7D6: operator new(unsigned) (vg_replace_malloc.c:164)
==28187== by 0x49A3FEB: geos::LinearRing::validateConstruction()
(LinearRing.cpp:109)
==28187== by 0x49A41A8:
geos::LinearRing::LinearRing(geos::CoordinateSequence*,
geos::GeometryFactory const*) (LinearRing.cpp:100)
==28187== by 0x49A14BB:
geos::GeometryFactory::createLinearRing(geos::CoordinateSequence const&)
const (GeometryFactory.cpp:439)
==28187== by 0x4A022E8: geos::polygonizeEdgeRing::getRingInternal()
(polygonizeEdgeRing.cpp:258)
==28187== by 0x4A0235C: geos::polygonizeEdgeRing::isValid()
(polygonizeEdgeRing.cpp:206)
==28187== by 0x4A0156E:
geos::Polygonizer::findValidRings(std::vector<geos::polygonizeEdgeRing*,
std::allocator<geos::polygonizeEdgeRing*> >*,
std::vector<geos::polygonizeEdgeRing*,
std::allocator<geos::polygonizeEdgeRing*> >*,
std::vector<geos::LineString*, std::allocator<geos::LineString*> >*)
(Polygonizer.cpp:208)
==28187== by 0x4A01C1B: geos::Polygonizer::polygonize()
(Polygonizer.cpp:181)==28187== by 0x4A01DDC:
geos::Polygonizer::getPolygons() (Polygonizer.cpp:118)
==28187== by 0x482B31D: GEOSpolygonize (lwgeom_geos_wrapper.cpp:1632)
==28187== by 0x4837406: GEOS_polygonize_garray (lwgeom_geos.c:2467)
==28187== by 0x812D5D4: finalize_aggregate (in /usr/bin/postgres)
3. And this is the third:
==28187== 105,744 bytes in 182 blocks are possibly lost in loss record
61 of 70
==28187== at 0x401B7D6: operator new(unsigned) (vg_replace_malloc.c:164)
==28187== by 0x482B7AA: std::vector<geos::Coordinate,
std::allocator<geos::Coordinate> >::vector(unsigned) (new_allocator.h:90)
==28187== by 0x4827B87: PostGIS2GEOS_linestring
(lwgeom_geos_wrapper.cpp:295)==28187== by 0x48349F1: LWGEOM2GEOS
(lwgeom_geos.c:2321)
==28187== by 0x4834AC2: POSTGIS2GEOS (lwgeom_geos.c:2364)
==28187== by 0x48373D0: GEOS_polygonize_garray (lwgeom_geos.c:2452)
==28187== by 0x812D5D4: finalize_aggregate (in /usr/bin/postgres)
==28187== by 0x812DF19: ExecAgg (in /usr/bin/postgres)
==28187== by 0x8124B80: ExecProcNode (in /usr/bin/postgres)
==28187== by 0x81238C2: ExecutorRun (in /usr/bin/postgres)
==28187== by 0x819D57B: ProcessQuery (in /usr/bin/postgres)
==28187== by 0x819E9A9: PortalRun (in /usr/bin/postgres)
I've included the rest of the valgrind info for completenesses sake below.
Thanks,
Charlie
==28187== 16 bytes in 1 blocks are definitely lost in loss record 10 of 70
==28187== at 0x401B7D6: operator new(unsigned) (vg_replace_malloc.c:164)
==28187== by 0x4827549: initGEOS (lwgeom_geos_wrapper.cpp:173)
==28187== by 0x4837366: GEOS_polygonize_garray (lwgeom_geos.c:2443)
==28187== by 0x812D5D4: finalize_aggregate (in /usr/bin/postgres)
==28187== by 0x812DF19: ExecAgg (in /usr/bin/postgres)
==28187== by 0x8124B80: ExecProcNode (in /usr/bin/postgres)
==28187== by 0x81238C2: ExecutorRun (in /usr/bin/postgres)
==28187== by 0x819D57B: ProcessQuery (in /usr/bin/postgres)
==28187== by 0x819E9A9: PortalRun (in /usr/bin/postgres)
==28187== by 0x819A80E: exec_simple_query (in /usr/bin/postgres)
==28187== by 0x819BDF2: PostgresMain (in /usr/bin/postgres)
==28187== by 0x81769E1: ServerLoop (in /usr/bin/postgres)
==28187==
==28187==
==28187== 36 bytes in 3 blocks are possibly lost in loss record 18 of 70
==28187== at 0x401B7D6: operator new(unsigned) (vg_replace_malloc.c:164)
==28187== by 0x49A3FEB: geos::LinearRing::validateConstruction()
(LinearRing.cpp:109)
==28187== by 0x49A41A8:
geos::LinearRing::LinearRing(geos::CoordinateSequence*,
geos::GeometryFactory const*) (LinearRing.cpp:100)
==28187== by 0x49A14BB:
geos::GeometryFactory::createLinearRing(geos::CoordinateSequence const&)
const (GeometryFactory.cpp:439)
==28187== by 0x4A022E8: geos::polygonizeEdgeRing::getRingInternal()
(polygonizeEdgeRing.cpp:258)
==28187== by 0x4A0235C: geos::polygonizeEdgeRing::isValid()
(polygonizeEdgeRing.cpp:206)
==28187== by 0x4A0156E:
geos::Polygonizer::findValidRings(std::vector<geos::polygonizeEdgeRing*,
std::allocator<geos::polygonizeEdgeRing*> >*,
std::vector<geos::polygonizeEdgeRing*,
std::allocator<geos::polygonizeEdgeRing*> >*,
std::vector<geos::LineString*, std::allocator<geos::LineString*> >*)
(Polygonizer.cpp:208)
==28187== by 0x4A01C1B: geos::Polygonizer::polygonize()
(Polygonizer.cpp:181)==28187== by 0x4A01DDC:
geos::Polygonizer::getPolygons() (Polygonizer.cpp:118)
==28187== by 0x482B31D: GEOSpolygonize (lwgeom_geos_wrapper.cpp:1632)
==28187== by 0x4837406: GEOS_polygonize_garray (lwgeom_geos.c:2467)
==28187== by 0x812D5D4: finalize_aggregate (in /usr/bin/postgres)
==28187==
==28187==
==28187== 40 bytes in 5 blocks are possibly lost in loss record 20 of 70
==28187== at 0x401B7D6: operator new(unsigned) (vg_replace_malloc.c:164)
==28187== by 0x499A9FB:
geos::DefaultCoordinateSequenceFactory::create(std::vector<geos::Coordinate,
std::allocator<geos::Coordinate> >*) const (geom.h:898)
==28187== by 0x4827BA9: PostGIS2GEOS_linestring
(lwgeom_geos_wrapper.cpp:334)==28187== by 0x48349F1: LWGEOM2GEOS
(lwgeom_geos.c:2321)
==28187== by 0x4834AC2: POSTGIS2GEOS (lwgeom_geos.c:2364)
==28187== by 0x48373D0: GEOS_polygonize_garray (lwgeom_geos.c:2452)
==28187== by 0x812D5D4: finalize_aggregate (in /usr/bin/postgres)
==28187== by 0x812DF19: ExecAgg (in /usr/bin/postgres)
==28187== by 0x8124B80: ExecProcNode (in /usr/bin/postgres)
==28187== by 0x81238C2: ExecutorRun (in /usr/bin/postgres)
==28187== by 0x819D57B: ProcessQuery (in /usr/bin/postgres)
==28187== by 0x819E9A9: PortalRun (in /usr/bin/postgres)
==28187==
==28187==
==28187== 72 bytes in 3 blocks are definitely lost in loss record 26 of 70
==28187== at 0x401B43A: malloc (vg_replace_malloc.c:149)
==28187== by 0x4166ADF: strdup (in /lib/tls/libc-2.3.5.so)
==28187== by 0x8227722: set_pglocale_pgservice (in /usr/bin/postgres)
==28187== by 0x813F2CD: main (in /usr/bin/postgres)
==28187==
==28187==
==28187== 119 bytes in 1 blocks are possibly lost in loss record 33 of 70
==28187== at 0x401B43A: malloc (vg_replace_malloc.c:149)
==28187== by 0x4166ADF: strdup (in /lib/tls/libc-2.3.5.so)
==28187== by 0x821AD70: save_ps_display_args (in /usr/bin/postgres)
==28187== by 0x813F2BB: main (in /usr/bin/postgres)
==28187==
==28187==
==28187== 156 bytes in 13 blocks are possibly lost in loss record 37 of 70
==28187== at 0x401B7D6: operator new(unsigned) (vg_replace_malloc.c:164)
==28187== by 0x4827B78: PostGIS2GEOS_linestring
(lwgeom_geos_wrapper.cpp:295)==28187== by 0x48349F1: LWGEOM2GEOS
(lwgeom_geos.c:2321)
==28187== by 0x4834AC2: POSTGIS2GEOS (lwgeom_geos.c:2364)
==28187== by 0x48373D0: GEOS_polygonize_garray (lwgeom_geos.c:2452)
==28187== by 0x812D5D4: finalize_aggregate (in /usr/bin/postgres)
==28187== by 0x812DF19: ExecAgg (in /usr/bin/postgres)
==28187== by 0x8124B80: ExecProcNode (in /usr/bin/postgres)
==28187== by 0x81238C2: ExecutorRun (in /usr/bin/postgres)
==28187== by 0x819D57B: ProcessQuery (in /usr/bin/postgres)
==28187== by 0x819E9A9: PortalRun (in /usr/bin/postgres)
==28187== by 0x819A80E: exec_simple_query (in /usr/bin/postgres)
==28187==
==28187==
==28187== 3,683 (304 direct, 3,379 indirect) bytes in 1 blocks are
definitely lost in loss record 41 of 70
==28187== at 0x401B43A: malloc (vg_replace_malloc.c:149)
==28187== by 0x821AD52: save_ps_display_args (in /usr/bin/postgres)
==28187== by 0x813F2BB: main (in /usr/bin/postgres)
==28187==
==28187==
==28187== 600 bytes in 25 blocks are possibly lost in loss record 44 of 70
==28187== at 0x401B7D6: operator new(unsigned) (vg_replace_malloc.c:164)
==28187== by 0x49A169B:
geos::GeometryFactory::createLineString(geos::CoordinateSequence*) const
(GeometryFactory.cpp:615)
==28187== by 0x4827BBA: PostGIS2GEOS_linestring
(lwgeom_geos_wrapper.cpp:336)==28187== by 0x48349F1: LWGEOM2GEOS
(lwgeom_geos.c:2321)
==28187== by 0x4834AC2: POSTGIS2GEOS (lwgeom_geos.c:2364)
==28187== by 0x48373D0: GEOS_polygonize_garray (lwgeom_geos.c:2452)
==28187== by 0x812D5D4: finalize_aggregate (in /usr/bin/postgres)
==28187== by 0x812DF19: ExecAgg (in /usr/bin/postgres)
==28187== by 0x8124B80: ExecProcNode (in /usr/bin/postgres)
==28187== by 0x81238C2: ExecutorRun (in /usr/bin/postgres)
==28187== by 0x819D57B: ProcessQuery (in /usr/bin/postgres)
==28187== by 0x819E9A9: PortalRun (in /usr/bin/postgres)
==28187==
==28187==
==28187== 992 bytes in 24 blocks are possibly lost in loss record 47 of 70
==28187== at 0x401B7D6: operator new(unsigned) (vg_replace_malloc.c:164)
==28187== by 0x48E7460: std::string::_Rep::_S_create(unsigned,
unsigned, std::allocator<char> const&) (in /usr/lib/libstdc++.so.6.0.6)
==28187== by 0x48E7942: (within /usr/lib/libstdc++.so.6.0.6)
==28187== by 0x48E7B18: std::string::string(char const*,
std::allocator<char> const&) (in /usr/lib/libstdc++.so.6.0.6)
==28187== by 0x4A0C9B8:
geos::IllegalArgumentException::IllegalArgumentException(std::string)
(IllegalArgumentException.cpp:38)
==28187== by 0x49A3FF6: geos::LinearRing::validateConstruction()
(LinearRing.cpp:109)
==28187== by 0x49A41A8:
geos::LinearRing::LinearRing(geos::CoordinateSequence*,
geos::GeometryFactory const*) (LinearRing.cpp:100)
==28187== by 0x49A14BB:
geos::GeometryFactory::createLinearRing(geos::CoordinateSequence const&)
const (GeometryFactory.cpp:439)
==28187== by 0x4A022E8: geos::polygonizeEdgeRing::getRingInternal()
(polygonizeEdgeRing.cpp:258)
==28187== by 0x4A0235C: geos::polygonizeEdgeRing::isValid()
(polygonizeEdgeRing.cpp:206)
==28187== by 0x4A0156E:
geos::Polygonizer::findValidRings(std::vector<geos::polygonizeEdgeRing*,
std::allocator<geos::polygonizeEdgeRing*> >*,
std::vector<geos::polygonizeEdgeRing*,
std::allocator<geos::polygonizeEdgeRing*> >*,
std::vector<geos::LineString*, std::allocator<geos::LineString*> >*)
(Polygonizer.cpp:208)
==28187== by 0x4A01C1B: geos::Polygonizer::polygonize()
(Polygonizer.cpp:181)==28187==
==28187==
==28187== 68,385 (9,648 direct, 58,737 indirect) bytes in 804 blocks are
definitely lost in loss record 57 of 70
==28187== at 0x401B7D6: operator new(unsigned) (vg_replace_malloc.c:164)
==28187== by 0x4A01B80: geos::Polygonizer::polygonize()
(Polygonizer.cpp:166)==28187== by 0x4A01DDC:
geos::Polygonizer::getPolygons() (Polygonizer.cpp:118)
==28187== by 0x482B31D: GEOSpolygonize (lwgeom_geos_wrapper.cpp:1632)
==28187== by 0x4837406: GEOS_polygonize_garray (lwgeom_geos.c:2467)
==28187== by 0x812D5D4: finalize_aggregate (in /usr/bin/postgres)
==28187== by 0x812DF19: ExecAgg (in /usr/bin/postgres)
==28187== by 0x8124B80: ExecProcNode (in /usr/bin/postgres)
==28187== by 0x81238C2: ExecutorRun (in /usr/bin/postgres)
==28187== by 0x819D57B: ProcessQuery (in /usr/bin/postgres)
==28187== by 0x819E9A9: PortalRun (in /usr/bin/postgres)
==28187== by 0x819A80E: exec_simple_query (in /usr/bin/postgres)
==28187==
==28187==
==28187== 105,744 bytes in 182 blocks are possibly lost in loss record
61 of 70
==28187== at 0x401B7D6: operator new(unsigned) (vg_replace_malloc.c:164)
==28187== by 0x482B7AA: std::vector<geos::Coordinate,
std::allocator<geos::Coordinate> >::vector(unsigned) (new_allocator.h:90)
==28187== by 0x4827B87: PostGIS2GEOS_linestring
(lwgeom_geos_wrapper.cpp:295)==28187== by 0x48349F1: LWGEOM2GEOS
(lwgeom_geos.c:2321)
==28187== by 0x4834AC2: POSTGIS2GEOS (lwgeom_geos.c:2364)
==28187== by 0x48373D0: GEOS_polygonize_garray (lwgeom_geos.c:2452)
==28187== by 0x812D5D4: finalize_aggregate (in /usr/bin/postgres)
==28187== by 0x812DF19: ExecAgg (in /usr/bin/postgres)
==28187== by 0x8124B80: ExecProcNode (in /usr/bin/postgres)
==28187== by 0x81238C2: ExecutorRun (in /usr/bin/postgres)
==28187== by 0x819D57B: ProcessQuery (in /usr/bin/postgres)
==28187== by 0x819E9A9: PortalRun (in /usr/bin/postgres)
==28187==
==28187==
==28187== 713,741 (91,308 direct, 622,433 indirect) bytes in 7,609
blocks are definitely lost in loss record 65 of 70
==28187== at 0x401B7D6: operator new(unsigned) (vg_replace_malloc.c:164)
==28187== by 0x49A3FEB: geos::LinearRing::validateConstruction()
(LinearRing.cpp:109)
==28187== by 0x49A41A8:
geos::LinearRing::LinearRing(geos::CoordinateSequence*,
geos::GeometryFactory const*) (LinearRing.cpp:100)
==28187== by 0x49A14BB:
geos::GeometryFactory::createLinearRing(geos::CoordinateSequence const&)
const (GeometryFactory.cpp:439)
==28187== by 0x4A022E8: geos::polygonizeEdgeRing::getRingInternal()
(polygonizeEdgeRing.cpp:258)
==28187== by 0x4A0235C: geos::polygonizeEdgeRing::isValid()
(polygonizeEdgeRing.cpp:206)
==28187== by 0x4A0156E:
geos::Polygonizer::findValidRings(std::vector<geos::polygonizeEdgeRing*,
std::allocator<geos::polygonizeEdgeRing*> >*,
std::vector<geos::polygonizeEdgeRing*,
std::allocator<geos::polygonizeEdgeRing*> >*,
std::vector<geos::LineString*, std::allocator<geos::LineString*> >*)
(Polygonizer.cpp:208)
==28187== by 0x4A01C1B: geos::Polygonizer::polygonize()
(Polygonizer.cpp:181)==28187== by 0x4A01DDC:
geos::Polygonizer::getPolygons() (Polygonizer.cpp:118)
==28187== by 0x482B31D: GEOSpolygonize (lwgeom_geos_wrapper.cpp:1632)
==28187== by 0x4837406: GEOS_polygonize_garray (lwgeom_geos.c:2467)
==28187== by 0x812D5D4: finalize_aggregate (in /usr/bin/postgres)
==28187==
==28187==
==28187== 8,905,045 (959,160 direct, 7,945,885 indirect) bytes in 39,965
blocks are definitely lost in loss record 68 of 70
==28187== at 0x401B7D6: operator new(unsigned) (vg_replace_malloc.c:164)
==28187== by 0x49A169B:
geos::GeometryFactory::createLineString(geos::CoordinateSequence*) const
(GeometryFactory.cpp:615)
==28187== by 0x4827BBA: PostGIS2GEOS_linestring
(lwgeom_geos_wrapper.cpp:336)==28187== by 0x48349F1: LWGEOM2GEOS
(lwgeom_geos.c:2321)
==28187== by 0x4834AC2: POSTGIS2GEOS (lwgeom_geos.c:2364)
==28187== by 0x48373D0: GEOS_polygonize_garray (lwgeom_geos.c:2452)
==28187== by 0x812D5D4: finalize_aggregate (in /usr/bin/postgres)
==28187== by 0x812DF19: ExecAgg (in /usr/bin/postgres)
==28187== by 0x8124B80: ExecProcNode (in /usr/bin/postgres)
==28187== by 0x81238C2: ExecutorRun (in /usr/bin/postgres)
==28187== by 0x819D57B: ProcessQuery (in /usr/bin/postgres)
==28187== by 0x819E9A9: PortalRun (in /usr/bin/postgres)
==28187==
==28187== LEAK SUMMARY:
==28187== definitely lost: 1,060,508 bytes in 48,383 blocks.
==28187== indirectly lost: 8,630,434 bytes in 136,236 blocks.
==28187== possibly lost: 107,687 bytes in 253 blocks.
==28187== still reachable: 1,938,129 bytes in 408 blocks.
==28187== suppressed: 0 bytes in 0 blocks.
==28187== Reachable blocks (those to which a pointer was found) are not
shown.
==28187== To see them, rerun with: --show-reachable=yes
</pre>
</blockquote>
<pre wrap=""><!---->
</pre>
<blockquote type="cite">
<pre wrap="">_______________________________________________
postgis-devel mailing list
<a class="moz-txt-link-abbreviated" href="mailto:postgis-devel@postgis.refractions.net">postgis-devel@postgis.refractions.net</a>
<a class="moz-txt-link-freetext" href="http://postgis.refractions.net/mailman/listinfo/postgis-devel">http://postgis.refractions.net/mailman/listinfo/postgis-devel</a>
</pre>
</blockquote>
<pre wrap=""><!---->
</pre>
</blockquote>
</body>
</html>