[postgis-devel] Polygonize Memory Leaks
strk at refractions.net
strk at refractions.net
Fri Dec 9 04:05:17 PST 2005
Charlie, thanks for the report.
The leak is plugged in CVS (both 1.0 and HEAD branches).
Also, I've added a couple of tests for polygonize in the
regress_ogc files.
--strk;
On Fri, Dec 09, 2005 at 01:26:55AM -0700, Charlie Savage wrote:
> Hi everyone,
>
> I've seem to have discovered some pretty big memory leaks in polygonize.
>
> Setup:
> postgis 1.0.6
> geos head (also seen on 2.1.4)
> opensuse 10
> Dell Dimension, 1Gb memory, 400 Gb Sata hard drive
> Postgresql 8.1
>
> Test case:
> Running polygonize on approximately 3.6 million line chains, forming
> approximately ?? polygons. I can run the query once, sometimes twice
> (in the same client session), but then linux will kill the postgres
> process sending it a kill -9 due to out of memory condition. Note that
> the memory is not given back until the postmaster process serving the
> query ends.
>
> Example Mem Usage before query:
> MemTotal: 1032980 kB
> MemFree: 108124 kB
> SwapTotal: 2032140 kB
> SwapFree: 1979156 kB
>
> Example Mem Usage after:
> MemTotal: 1032980 kB
> MemFree: 14032 kB
> SwapTotal: 2032140 kB
> SwapFree: 994252 kB
>
> After the postgresql process grabs all free memory, then it starts to
> eat away at the swap memory. You can see that the swap memory has been
> reduced by half, roughly 1 megabyte.
>
> To see what was causing the problem, I ran postmaster using valgrind.
> This showed lots of little leaks, a few mid-size leaks, and one big one.
>
>
> 1. This is by far the biggest leak.
>
> ==28187== 8,905,045 (959,160 direct, 7,945,885 indirect) bytes in 39,965
> blocks are definitely lost in loss record 68 of 70
> ==28187== at 0x401B7D6: operator new(unsigned) (vg_replace_malloc.c:164)
> ==28187== by 0x49A169B:
> geos::GeometryFactory::createLineString(geos::CoordinateSequence*) const
> (GeometryFactory.cpp:615)
> ==28187== by 0x4827BBA: PostGIS2GEOS_linestring
> (lwgeom_geos_wrapper.cpp:336)==28187== by 0x48349F1: LWGEOM2GEOS
> (lwgeom_geos.c:2321)
> ==28187== by 0x4834AC2: POSTGIS2GEOS (lwgeom_geos.c:2364)
> ==28187== by 0x48373D0: GEOS_polygonize_garray (lwgeom_geos.c:2452)
> ==28187== by 0x812D5D4: finalize_aggregate (in /usr/bin/postgres)
> ==28187== by 0x812DF19: ExecAgg (in /usr/bin/postgres)
> ==28187== by 0x8124B80: ExecProcNode (in /usr/bin/postgres)
> ==28187== by 0x81238C2: ExecutorRun (in /usr/bin/postgres)
> ==28187== by 0x819D57B: ProcessQuery (in /usr/bin/postgres)
> ==28187== by 0x819E9A9: PortalRun (in /usr/bin/postgres)
>
> Looking at GEOS_polygonize_garray, line 2471, I see:
>
> //pfree(vgeoms)
>
> I'd guess this is the problem, all the line strings in the vgeoms area
> that are allocated by geos are not destroyed. This is the first time
> I've looked at the PostGIS code, but it seems to me that pfree is not
> the correct call since vgeoms is created by GEOS and is not under
> postgresql memory management.
>
> 2. This is the second biggest leak:
>
> ==28187== 713,741 (91,308 direct, 622,433 indirect) bytes in 7,609
> blocks are definitely lost in loss record 65 of 70
> ==28187== at 0x401B7D6: operator new(unsigned) (vg_replace_malloc.c:164)
> ==28187== by 0x49A3FEB: geos::LinearRing::validateConstruction()
> (LinearRing.cpp:109)
> ==28187== by 0x49A41A8:
> geos::LinearRing::LinearRing(geos::CoordinateSequence*,
> geos::GeometryFactory const*) (LinearRing.cpp:100)
> ==28187== by 0x49A14BB:
> geos::GeometryFactory::createLinearRing(geos::CoordinateSequence const&)
> const (GeometryFactory.cpp:439)
> ==28187== by 0x4A022E8: geos::polygonizeEdgeRing::getRingInternal()
> (polygonizeEdgeRing.cpp:258)
> ==28187== by 0x4A0235C: geos::polygonizeEdgeRing::isValid()
> (polygonizeEdgeRing.cpp:206)
> ==28187== by 0x4A0156E:
> geos::Polygonizer::findValidRings(std::vector<geos::polygonizeEdgeRing*,
> std::allocator<geos::polygonizeEdgeRing*> >*,
> std::vector<geos::polygonizeEdgeRing*,
> std::allocator<geos::polygonizeEdgeRing*> >*,
> std::vector<geos::LineString*, std::allocator<geos::LineString*> >*)
> (Polygonizer.cpp:208)
> ==28187== by 0x4A01C1B: geos::Polygonizer::polygonize()
> (Polygonizer.cpp:181)==28187== by 0x4A01DDC:
> geos::Polygonizer::getPolygons() (Polygonizer.cpp:118)
> ==28187== by 0x482B31D: GEOSpolygonize (lwgeom_geos_wrapper.cpp:1632)
> ==28187== by 0x4837406: GEOS_polygonize_garray (lwgeom_geos.c:2467)
> ==28187== by 0x812D5D4: finalize_aggregate (in /usr/bin/postgres)
>
> 3. And this is the third:
>
> ==28187== 105,744 bytes in 182 blocks are possibly lost in loss record
> 61 of 70
> ==28187== at 0x401B7D6: operator new(unsigned) (vg_replace_malloc.c:164)
> ==28187== by 0x482B7AA: std::vector<geos::Coordinate,
> std::allocator<geos::Coordinate> >::vector(unsigned) (new_allocator.h:90)
> ==28187== by 0x4827B87: PostGIS2GEOS_linestring
> (lwgeom_geos_wrapper.cpp:295)==28187== by 0x48349F1: LWGEOM2GEOS
> (lwgeom_geos.c:2321)
> ==28187== by 0x4834AC2: POSTGIS2GEOS (lwgeom_geos.c:2364)
> ==28187== by 0x48373D0: GEOS_polygonize_garray (lwgeom_geos.c:2452)
> ==28187== by 0x812D5D4: finalize_aggregate (in /usr/bin/postgres)
> ==28187== by 0x812DF19: ExecAgg (in /usr/bin/postgres)
> ==28187== by 0x8124B80: ExecProcNode (in /usr/bin/postgres)
> ==28187== by 0x81238C2: ExecutorRun (in /usr/bin/postgres)
> ==28187== by 0x819D57B: ProcessQuery (in /usr/bin/postgres)
> ==28187== by 0x819E9A9: PortalRun (in /usr/bin/postgres)
>
> I've included the rest of the valgrind info for completenesses sake below.
>
> Thanks,
>
> Charlie
>
> ==28187== 16 bytes in 1 blocks are definitely lost in loss record 10 of 70
> ==28187== at 0x401B7D6: operator new(unsigned) (vg_replace_malloc.c:164)
> ==28187== by 0x4827549: initGEOS (lwgeom_geos_wrapper.cpp:173)
> ==28187== by 0x4837366: GEOS_polygonize_garray (lwgeom_geos.c:2443)
> ==28187== by 0x812D5D4: finalize_aggregate (in /usr/bin/postgres)
> ==28187== by 0x812DF19: ExecAgg (in /usr/bin/postgres)
> ==28187== by 0x8124B80: ExecProcNode (in /usr/bin/postgres)
> ==28187== by 0x81238C2: ExecutorRun (in /usr/bin/postgres)
> ==28187== by 0x819D57B: ProcessQuery (in /usr/bin/postgres)
> ==28187== by 0x819E9A9: PortalRun (in /usr/bin/postgres)
> ==28187== by 0x819A80E: exec_simple_query (in /usr/bin/postgres)
> ==28187== by 0x819BDF2: PostgresMain (in /usr/bin/postgres)
> ==28187== by 0x81769E1: ServerLoop (in /usr/bin/postgres)
> ==28187==
> ==28187==
> ==28187== 36 bytes in 3 blocks are possibly lost in loss record 18 of 70
> ==28187== at 0x401B7D6: operator new(unsigned) (vg_replace_malloc.c:164)
> ==28187== by 0x49A3FEB: geos::LinearRing::validateConstruction()
> (LinearRing.cpp:109)
> ==28187== by 0x49A41A8:
> geos::LinearRing::LinearRing(geos::CoordinateSequence*,
> geos::GeometryFactory const*) (LinearRing.cpp:100)
> ==28187== by 0x49A14BB:
> geos::GeometryFactory::createLinearRing(geos::CoordinateSequence const&)
> const (GeometryFactory.cpp:439)
> ==28187== by 0x4A022E8: geos::polygonizeEdgeRing::getRingInternal()
> (polygonizeEdgeRing.cpp:258)
> ==28187== by 0x4A0235C: geos::polygonizeEdgeRing::isValid()
> (polygonizeEdgeRing.cpp:206)
> ==28187== by 0x4A0156E:
> geos::Polygonizer::findValidRings(std::vector<geos::polygonizeEdgeRing*,
> std::allocator<geos::polygonizeEdgeRing*> >*,
> std::vector<geos::polygonizeEdgeRing*,
> std::allocator<geos::polygonizeEdgeRing*> >*,
> std::vector<geos::LineString*, std::allocator<geos::LineString*> >*)
> (Polygonizer.cpp:208)
> ==28187== by 0x4A01C1B: geos::Polygonizer::polygonize()
> (Polygonizer.cpp:181)==28187== by 0x4A01DDC:
> geos::Polygonizer::getPolygons() (Polygonizer.cpp:118)
> ==28187== by 0x482B31D: GEOSpolygonize (lwgeom_geos_wrapper.cpp:1632)
> ==28187== by 0x4837406: GEOS_polygonize_garray (lwgeom_geos.c:2467)
> ==28187== by 0x812D5D4: finalize_aggregate (in /usr/bin/postgres)
> ==28187==
> ==28187==
> ==28187== 40 bytes in 5 blocks are possibly lost in loss record 20 of 70
> ==28187== at 0x401B7D6: operator new(unsigned) (vg_replace_malloc.c:164)
> ==28187== by 0x499A9FB:
> geos::DefaultCoordinateSequenceFactory::create(std::vector<geos::Coordinate,
> std::allocator<geos::Coordinate> >*) const (geom.h:898)
> ==28187== by 0x4827BA9: PostGIS2GEOS_linestring
> (lwgeom_geos_wrapper.cpp:334)==28187== by 0x48349F1: LWGEOM2GEOS
> (lwgeom_geos.c:2321)
> ==28187== by 0x4834AC2: POSTGIS2GEOS (lwgeom_geos.c:2364)
> ==28187== by 0x48373D0: GEOS_polygonize_garray (lwgeom_geos.c:2452)
> ==28187== by 0x812D5D4: finalize_aggregate (in /usr/bin/postgres)
> ==28187== by 0x812DF19: ExecAgg (in /usr/bin/postgres)
> ==28187== by 0x8124B80: ExecProcNode (in /usr/bin/postgres)
> ==28187== by 0x81238C2: ExecutorRun (in /usr/bin/postgres)
> ==28187== by 0x819D57B: ProcessQuery (in /usr/bin/postgres)
> ==28187== by 0x819E9A9: PortalRun (in /usr/bin/postgres)
> ==28187==
> ==28187==
> ==28187== 72 bytes in 3 blocks are definitely lost in loss record 26 of 70
> ==28187== at 0x401B43A: malloc (vg_replace_malloc.c:149)
> ==28187== by 0x4166ADF: strdup (in /lib/tls/libc-2.3.5.so)
> ==28187== by 0x8227722: set_pglocale_pgservice (in /usr/bin/postgres)
> ==28187== by 0x813F2CD: main (in /usr/bin/postgres)
> ==28187==
> ==28187==
> ==28187== 119 bytes in 1 blocks are possibly lost in loss record 33 of 70
> ==28187== at 0x401B43A: malloc (vg_replace_malloc.c:149)
> ==28187== by 0x4166ADF: strdup (in /lib/tls/libc-2.3.5.so)
> ==28187== by 0x821AD70: save_ps_display_args (in /usr/bin/postgres)
> ==28187== by 0x813F2BB: main (in /usr/bin/postgres)
> ==28187==
> ==28187==
> ==28187== 156 bytes in 13 blocks are possibly lost in loss record 37 of 70
> ==28187== at 0x401B7D6: operator new(unsigned) (vg_replace_malloc.c:164)
> ==28187== by 0x4827B78: PostGIS2GEOS_linestring
> (lwgeom_geos_wrapper.cpp:295)==28187== by 0x48349F1: LWGEOM2GEOS
> (lwgeom_geos.c:2321)
> ==28187== by 0x4834AC2: POSTGIS2GEOS (lwgeom_geos.c:2364)
> ==28187== by 0x48373D0: GEOS_polygonize_garray (lwgeom_geos.c:2452)
> ==28187== by 0x812D5D4: finalize_aggregate (in /usr/bin/postgres)
> ==28187== by 0x812DF19: ExecAgg (in /usr/bin/postgres)
> ==28187== by 0x8124B80: ExecProcNode (in /usr/bin/postgres)
> ==28187== by 0x81238C2: ExecutorRun (in /usr/bin/postgres)
> ==28187== by 0x819D57B: ProcessQuery (in /usr/bin/postgres)
> ==28187== by 0x819E9A9: PortalRun (in /usr/bin/postgres)
> ==28187== by 0x819A80E: exec_simple_query (in /usr/bin/postgres)
> ==28187==
> ==28187==
> ==28187== 3,683 (304 direct, 3,379 indirect) bytes in 1 blocks are
> definitely lost in loss record 41 of 70
> ==28187== at 0x401B43A: malloc (vg_replace_malloc.c:149)
> ==28187== by 0x821AD52: save_ps_display_args (in /usr/bin/postgres)
> ==28187== by 0x813F2BB: main (in /usr/bin/postgres)
> ==28187==
> ==28187==
> ==28187== 600 bytes in 25 blocks are possibly lost in loss record 44 of 70
> ==28187== at 0x401B7D6: operator new(unsigned) (vg_replace_malloc.c:164)
> ==28187== by 0x49A169B:
> geos::GeometryFactory::createLineString(geos::CoordinateSequence*) const
> (GeometryFactory.cpp:615)
> ==28187== by 0x4827BBA: PostGIS2GEOS_linestring
> (lwgeom_geos_wrapper.cpp:336)==28187== by 0x48349F1: LWGEOM2GEOS
> (lwgeom_geos.c:2321)
> ==28187== by 0x4834AC2: POSTGIS2GEOS (lwgeom_geos.c:2364)
> ==28187== by 0x48373D0: GEOS_polygonize_garray (lwgeom_geos.c:2452)
> ==28187== by 0x812D5D4: finalize_aggregate (in /usr/bin/postgres)
> ==28187== by 0x812DF19: ExecAgg (in /usr/bin/postgres)
> ==28187== by 0x8124B80: ExecProcNode (in /usr/bin/postgres)
> ==28187== by 0x81238C2: ExecutorRun (in /usr/bin/postgres)
> ==28187== by 0x819D57B: ProcessQuery (in /usr/bin/postgres)
> ==28187== by 0x819E9A9: PortalRun (in /usr/bin/postgres)
> ==28187==
> ==28187==
> ==28187== 992 bytes in 24 blocks are possibly lost in loss record 47 of 70
> ==28187== at 0x401B7D6: operator new(unsigned) (vg_replace_malloc.c:164)
> ==28187== by 0x48E7460: std::string::_Rep::_S_create(unsigned,
> unsigned, std::allocator<char> const&) (in /usr/lib/libstdc++.so.6.0.6)
> ==28187== by 0x48E7942: (within /usr/lib/libstdc++.so.6.0.6)
> ==28187== by 0x48E7B18: std::string::string(char const*,
> std::allocator<char> const&) (in /usr/lib/libstdc++.so.6.0.6)
> ==28187== by 0x4A0C9B8:
> geos::IllegalArgumentException::IllegalArgumentException(std::string)
> (IllegalArgumentException.cpp:38)
> ==28187== by 0x49A3FF6: geos::LinearRing::validateConstruction()
> (LinearRing.cpp:109)
> ==28187== by 0x49A41A8:
> geos::LinearRing::LinearRing(geos::CoordinateSequence*,
> geos::GeometryFactory const*) (LinearRing.cpp:100)
> ==28187== by 0x49A14BB:
> geos::GeometryFactory::createLinearRing(geos::CoordinateSequence const&)
> const (GeometryFactory.cpp:439)
> ==28187== by 0x4A022E8: geos::polygonizeEdgeRing::getRingInternal()
> (polygonizeEdgeRing.cpp:258)
> ==28187== by 0x4A0235C: geos::polygonizeEdgeRing::isValid()
> (polygonizeEdgeRing.cpp:206)
> ==28187== by 0x4A0156E:
> geos::Polygonizer::findValidRings(std::vector<geos::polygonizeEdgeRing*,
> std::allocator<geos::polygonizeEdgeRing*> >*,
> std::vector<geos::polygonizeEdgeRing*,
> std::allocator<geos::polygonizeEdgeRing*> >*,
> std::vector<geos::LineString*, std::allocator<geos::LineString*> >*)
> (Polygonizer.cpp:208)
> ==28187== by 0x4A01C1B: geos::Polygonizer::polygonize()
> (Polygonizer.cpp:181)==28187==
> ==28187==
> ==28187== 68,385 (9,648 direct, 58,737 indirect) bytes in 804 blocks are
> definitely lost in loss record 57 of 70
> ==28187== at 0x401B7D6: operator new(unsigned) (vg_replace_malloc.c:164)
> ==28187== by 0x4A01B80: geos::Polygonizer::polygonize()
> (Polygonizer.cpp:166)==28187== by 0x4A01DDC:
> geos::Polygonizer::getPolygons() (Polygonizer.cpp:118)
> ==28187== by 0x482B31D: GEOSpolygonize (lwgeom_geos_wrapper.cpp:1632)
> ==28187== by 0x4837406: GEOS_polygonize_garray (lwgeom_geos.c:2467)
> ==28187== by 0x812D5D4: finalize_aggregate (in /usr/bin/postgres)
> ==28187== by 0x812DF19: ExecAgg (in /usr/bin/postgres)
> ==28187== by 0x8124B80: ExecProcNode (in /usr/bin/postgres)
> ==28187== by 0x81238C2: ExecutorRun (in /usr/bin/postgres)
> ==28187== by 0x819D57B: ProcessQuery (in /usr/bin/postgres)
> ==28187== by 0x819E9A9: PortalRun (in /usr/bin/postgres)
> ==28187== by 0x819A80E: exec_simple_query (in /usr/bin/postgres)
> ==28187==
> ==28187==
> ==28187== 105,744 bytes in 182 blocks are possibly lost in loss record
> 61 of 70
> ==28187== at 0x401B7D6: operator new(unsigned) (vg_replace_malloc.c:164)
> ==28187== by 0x482B7AA: std::vector<geos::Coordinate,
> std::allocator<geos::Coordinate> >::vector(unsigned) (new_allocator.h:90)
> ==28187== by 0x4827B87: PostGIS2GEOS_linestring
> (lwgeom_geos_wrapper.cpp:295)==28187== by 0x48349F1: LWGEOM2GEOS
> (lwgeom_geos.c:2321)
> ==28187== by 0x4834AC2: POSTGIS2GEOS (lwgeom_geos.c:2364)
> ==28187== by 0x48373D0: GEOS_polygonize_garray (lwgeom_geos.c:2452)
> ==28187== by 0x812D5D4: finalize_aggregate (in /usr/bin/postgres)
> ==28187== by 0x812DF19: ExecAgg (in /usr/bin/postgres)
> ==28187== by 0x8124B80: ExecProcNode (in /usr/bin/postgres)
> ==28187== by 0x81238C2: ExecutorRun (in /usr/bin/postgres)
> ==28187== by 0x819D57B: ProcessQuery (in /usr/bin/postgres)
> ==28187== by 0x819E9A9: PortalRun (in /usr/bin/postgres)
> ==28187==
> ==28187==
> ==28187== 713,741 (91,308 direct, 622,433 indirect) bytes in 7,609
> blocks are definitely lost in loss record 65 of 70
> ==28187== at 0x401B7D6: operator new(unsigned) (vg_replace_malloc.c:164)
> ==28187== by 0x49A3FEB: geos::LinearRing::validateConstruction()
> (LinearRing.cpp:109)
> ==28187== by 0x49A41A8:
> geos::LinearRing::LinearRing(geos::CoordinateSequence*,
> geos::GeometryFactory const*) (LinearRing.cpp:100)
> ==28187== by 0x49A14BB:
> geos::GeometryFactory::createLinearRing(geos::CoordinateSequence const&)
> const (GeometryFactory.cpp:439)
> ==28187== by 0x4A022E8: geos::polygonizeEdgeRing::getRingInternal()
> (polygonizeEdgeRing.cpp:258)
> ==28187== by 0x4A0235C: geos::polygonizeEdgeRing::isValid()
> (polygonizeEdgeRing.cpp:206)
> ==28187== by 0x4A0156E:
> geos::Polygonizer::findValidRings(std::vector<geos::polygonizeEdgeRing*,
> std::allocator<geos::polygonizeEdgeRing*> >*,
> std::vector<geos::polygonizeEdgeRing*,
> std::allocator<geos::polygonizeEdgeRing*> >*,
> std::vector<geos::LineString*, std::allocator<geos::LineString*> >*)
> (Polygonizer.cpp:208)
> ==28187== by 0x4A01C1B: geos::Polygonizer::polygonize()
> (Polygonizer.cpp:181)==28187== by 0x4A01DDC:
> geos::Polygonizer::getPolygons() (Polygonizer.cpp:118)
> ==28187== by 0x482B31D: GEOSpolygonize (lwgeom_geos_wrapper.cpp:1632)
> ==28187== by 0x4837406: GEOS_polygonize_garray (lwgeom_geos.c:2467)
> ==28187== by 0x812D5D4: finalize_aggregate (in /usr/bin/postgres)
> ==28187==
> ==28187==
> ==28187== 8,905,045 (959,160 direct, 7,945,885 indirect) bytes in 39,965
> blocks are definitely lost in loss record 68 of 70
> ==28187== at 0x401B7D6: operator new(unsigned) (vg_replace_malloc.c:164)
> ==28187== by 0x49A169B:
> geos::GeometryFactory::createLineString(geos::CoordinateSequence*) const
> (GeometryFactory.cpp:615)
> ==28187== by 0x4827BBA: PostGIS2GEOS_linestring
> (lwgeom_geos_wrapper.cpp:336)==28187== by 0x48349F1: LWGEOM2GEOS
> (lwgeom_geos.c:2321)
> ==28187== by 0x4834AC2: POSTGIS2GEOS (lwgeom_geos.c:2364)
> ==28187== by 0x48373D0: GEOS_polygonize_garray (lwgeom_geos.c:2452)
> ==28187== by 0x812D5D4: finalize_aggregate (in /usr/bin/postgres)
> ==28187== by 0x812DF19: ExecAgg (in /usr/bin/postgres)
> ==28187== by 0x8124B80: ExecProcNode (in /usr/bin/postgres)
> ==28187== by 0x81238C2: ExecutorRun (in /usr/bin/postgres)
> ==28187== by 0x819D57B: ProcessQuery (in /usr/bin/postgres)
> ==28187== by 0x819E9A9: PortalRun (in /usr/bin/postgres)
> ==28187==
> ==28187== LEAK SUMMARY:
> ==28187== definitely lost: 1,060,508 bytes in 48,383 blocks.
> ==28187== indirectly lost: 8,630,434 bytes in 136,236 blocks.
> ==28187== possibly lost: 107,687 bytes in 253 blocks.
> ==28187== still reachable: 1,938,129 bytes in 408 blocks.
> ==28187== suppressed: 0 bytes in 0 blocks.
> ==28187== Reachable blocks (those to which a pointer was found) are not
> shown.
> ==28187== To see them, rerun with: --show-reachable=yes
>
>
>
> _______________________________________________
> postgis-devel mailing list
> postgis-devel at postgis.refractions.net
> http://postgis.refractions.net/mailman/listinfo/postgis-devel
--
/"\ ASCII Ribbon Campaign
\ / Respect for low technology.
X Keep e-mail messages readable by any computer system.
/ \ Keep it ASCII.
More information about the postgis-devel
mailing list