[GRASS-dev] v.select: increasing memory consumption
Stefan Blumentrath
Stefan.Blumentrath at nina.no
Thu Feb 21 06:58:53 PST 2019
Thanks again, Marus,
Here the results from the new run:
valgrind --leak-check=yes --show-reachable=yes v.select ainput=grid100 binput=landNorway<mailto:binput=landNorway at u_torkild.tveraa> output=testNorwayGrid
==5173== by 0x4E79461: Vect__open_old (in /usr/lib/grass76/lib/libgrass_vector.7.6.0.so)
==5173== by 0x4E79DE0: Vect_open_old2 (in /usr/lib/grass76/lib/libgrass_vector.7.6.0.so)
==5173== by 0x402B96: main (in /usr/lib/grass76/bin/v.select)
==5173==
==5173== 923,048 bytes in 78,794 blocks are still reachable in loss record 213 of 219
==5173== at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==5173== by 0x52C6BCE: G__malloc (in /usr/lib/grass76/lib/libgrass_gis.7.6.0.so)
==5173== by 0x5F33897: dig_Rd_P_line (in /usr/lib/grass76/lib/libgrass_dig2.7.6.0.so)
==5173== by 0x5F310ED: dig_load_plus (in /usr/lib/grass76/lib/libgrass_dig2.7.6.0.so)
==5173== by 0x4E781BD: Vect_open_topo (in /usr/lib/grass76/lib/libgrass_vector.7.6.0.so)
==5173== by 0x4E79443: Vect__open_old (in /usr/lib/grass76/lib/libgrass_vector.7.6.0.so)
==5173== by 0x4E79DE0: Vect_open_old2 (in /usr/lib/grass76/lib/libgrass_vector.7.6.0.so)
==5173== by 0x402B96: main (in /usr/lib/grass76/bin/v.select)
==5173==
==5173== 1,512,320 bytes in 37,808 blocks are still reachable in loss record 214 of 219
==5173== at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==5173== by 0x52C6BCE: G__malloc (in /usr/lib/grass76/lib/libgrass_gis.7.6.0.so)
==5173== by 0x5F3B436: dig_alloc_area (in /usr/lib/grass76/lib/libgrass_dig2.7.6.0.so)
==5173== by 0x5F33D16: dig_Rd_P_area (in /usr/lib/grass76/lib/libgrass_dig2.7.6.0.so)
==5173== by 0x5F3114D: dig_load_plus (in /usr/lib/grass76/lib/libgrass_dig2.7.6.0.so)
==5173== by 0x4E781BD: Vect_open_topo (in /usr/lib/grass76/lib/libgrass_vector.7.6.0.so)
==5173== by 0x4E79443: Vect__open_old (in /usr/lib/grass76/lib/libgrass_vector.7.6.0.so)
==5173== by 0x4E79DE0: Vect_open_old2 (in /usr/lib/grass76/lib/libgrass_vector.7.6.0.so)
==5173== by 0x402B96: main (in /usr/lib/grass76/bin/v.select)
==5173==
==5173== 1,621,440 bytes in 33,780 blocks are still reachable in loss record 215 of 219
==5173== at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==5173== by 0x52C6BCE: G__malloc (in /usr/lib/grass76/lib/libgrass_gis.7.6.0.so)
==5173== by 0x5F3B056: dig_alloc_node (in /usr/lib/grass76/lib/libgrass_dig2.7.6.0.so)
==5173== by 0x5F334ED: dig_Rd_P_node (in /usr/lib/grass76/lib/libgrass_dig2.7.6.0.so)
==5173== by 0x5F3108D: dig_load_plus (in /usr/lib/grass76/lib/libgrass_dig2.7.6.0.so)
==5173== by 0x4E781BD: Vect_open_topo (in /usr/lib/grass76/lib/libgrass_vector.7.6.0.so)
==5173== by 0x4E79443: Vect__open_old (in /usr/lib/grass76/lib/libgrass_vector.7.6.0.so)
==5173== by 0x4E79DE0: Vect_open_old2 (in /usr/lib/grass76/lib/libgrass_vector.7.6.0.so)
==5173== by 0x402B96: main (in /usr/lib/grass76/bin/v.select)
==5173==
==5173== 750,941,956 bytes in 1 blocks are still reachable in loss record 216 of 219
==5173== at 0x4C2FB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==5173== by 0x52C6CAD: G__calloc (in /usr/lib/grass76/lib/libgrass_gis.7.6.0.so)
==5173== by 0x402BF8: main (in /usr/lib/grass76/bin/v.select)
==5173==
==5173== 1,502,514,272 bytes in 2 blocks are still reachable in loss record 217 of 219
==5173== at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==5173== by 0x52C6DB7: G__realloc (in /usr/lib/grass76/lib/libgrass_gis.7.6.0.so)
==5173== by 0x5F3B321: dig_alloc_lines (in /usr/lib/grass76/lib/libgrass_dig2.7.6.0.so)
==5173== by 0x5F310D1: dig_load_plus (in /usr/lib/grass76/lib/libgrass_dig2.7.6.0.so)
==5173== by 0x4E781BD: Vect_open_topo (in /usr/lib/grass76/lib/libgrass_vector.7.6.0.so)
==5173== by 0x4E79443: Vect__open_old (in /usr/lib/grass76/lib/libgrass_vector.7.6.0.so)
==5173== by 0x4E79DE0: Vect_open_old2 (in /usr/lib/grass76/lib/libgrass_vector.7.6.0.so)
==5173== by 0x402B96: main (in /usr/lib/grass76/bin/v.select)
==5173==
==5173== 2,254,225,080 bytes in 3 blocks are still reachable in loss record 218 of 219
==5173== at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==5173== by 0x52C6BCE: G__malloc (in /usr/lib/grass76/lib/libgrass_gis.7.6.0.so)
==5173== by 0x5F2F4AD: dig_read_cidx (in /usr/lib/grass76/lib/libgrass_dig2.7.6.0.so)
==5173== by 0x4E5C917: Vect_cidx_open (in /usr/lib/grass76/lib/libgrass_vector.7.6.0.so)
==5173== by 0x4E7992F: Vect__open_old (in /usr/lib/grass76/lib/libgrass_vector.7.6.0.so)
==5173== by 0x4E79DE0: Vect_open_old2 (in /usr/lib/grass76/lib/libgrass_vector.7.6.0.so)
==5173== by 0x402B96: main (in /usr/lib/grass76/bin/v.select)
==5173==
==5173== 4,507,542,768 bytes in 187,814,282 blocks are still reachable in loss record 219 of 219
==5173== at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==5173== by 0x52C6BCE: G__malloc (in /usr/lib/grass76/lib/libgrass_gis.7.6.0.so)
==5173== by 0x5F3B1D6: dig_alloc_line (in /usr/lib/grass76/lib/libgrass_dig2.7.6.0.so)
==5173== by 0x5F33841: dig_Rd_P_line (in /usr/lib/grass76/lib/libgrass_dig2.7.6.0.so)
==5173== by 0x5F310ED: dig_load_plus (in /usr/lib/grass76/lib/libgrass_dig2.7.6.0.so)
==5173== by 0x4E781BD: Vect_open_topo (in /usr/lib/grass76/lib/libgrass_vector.7.6.0.so)
==5173== by 0x4E79443: Vect__open_old (in /usr/lib/grass76/lib/libgrass_vector.7.6.0.so)
==5173== by 0x4E79DE0: Vect_open_old2 (in /usr/lib/grass76/lib/libgrass_vector.7.6.0.so)
==5173== by 0x402B96: main (in /usr/lib/grass76/bin/v.select)
==5173==
==5173== LEAK SUMMARY:
==5173== definitely lost: 956 bytes in 39 blocks
==5173== indirectly lost: 72 bytes in 10 blocks
==5173== possibly lost: 79 bytes in 4 blocks
==5173== still reachable: 9,025,026,330 bytes in 188,201,312 blocks
==5173== of which reachable via heuristic:
==5173== newarray : 488,856 bytes in 20,369 blocks
==5173== suppressed: 0 bytes in 0 blocks
==5173==
==5173== For counts of detected and suppressed errors, rerun with: -v
==5173== ERROR SUMMARY: 27 errors from 27 contexts (suppressed: 0 from 0)
Cheers
Stefan
From: Markus Metz <markus.metz.giswork at gmail.com>
Sent: torsdag 21. februar 2019 15:23
To: Stefan Blumentrath <Stefan.Blumentrath at nina.no>
Cc: Markus Neteler <neteler at osgeo.org>; GRASS developers list (grass-dev at lists.osgeo.org) <grass-dev at lists.osgeo.org>
Subject: Re: [GRASS-dev] v.select: increasing memory consumption
Hi Stefan,
On Thu, Feb 21, 2019 at 2:20 PM Stefan Blumentrath <Stefan.Blumentrath at nina.no<mailto:Stefan.Blumentrath at nina.no>> wrote:
>
> Hi again,
>
> Please find below the output from a run of the problematic command with valgrind (note that ainput data (points) are stored in PostGIS (v.external)).
>
> Hope that helps to shed some light…
>
> Cheers
>
> Stefan
>
> valgrind --leak-check=yes v.select ainput=grid100 binput=landNorway output=testNorwayGrid
>
> [...]
>
> ==13217== LEAK SUMMARY:
> ==13217== definitely lost: 964 bytes in 39 blocks
> ==13217== indirectly lost: 72 bytes in 10 blocks
> ==13217== possibly lost: 79 bytes in 4 blocks
> ==13217== still reachable: 9,025,026,330 bytes in 188,201,312 blocks
this is the problem, please run valgrind again with
--show-reachable=yes
we need to find out where these 9,025,026,330 bytes have been allocated.
Markus M
>
> From: Markus Metz <markus.metz.giswork at gmail.com<mailto:markus.metz.giswork at gmail.com>>
> Sent: lørdag 19. januar 2019 22:17
> To: Stefan Blumentrath <Stefan.Blumentrath at nina.no<mailto:Stefan.Blumentrath at nina.no>>
> Cc: Markus Neteler <neteler at osgeo.org<mailto:neteler at osgeo.org>>; GRASS developers list (grass-dev at lists.osgeo.org<mailto:grass-dev at lists.osgeo.org>) <grass-dev at lists.osgeo.org<mailto:grass-dev at lists.osgeo.org>>
> Subject: Re: [GRASS-dev] v.select: increasing memory consumption
>
>
>
>
>
> On Fri, Jan 18, 2019 at 8:29 AM Stefan Blumentrath <Stefan.Blumentrath at nina.no<mailto:Stefan.Blumentrath at nina.no>> wrote:
> >
> > THanks, we`ll try and report back.
> >
> > -----Original Message-----
> > From: Markus Neteler <neteler at osgeo.org<mailto:neteler at osgeo.org>>
> > Sent: fredag 18. januar 2019 07:52
> > To: Stefan Blumentrath <Stefan.Blumentrath at nina.no<mailto:Stefan.Blumentrath at nina.no>>
> > Cc: GRASS developers list (grass-dev at lists.osgeo.org<mailto:grass-dev at lists.osgeo.org>) <grass-dev at lists.osgeo.org<mailto:grass-dev at lists.osgeo.org>>
> > Subject: Re: [GRASS-dev] v.select: increasing memory consumption
> >
> > Hi,
> >
> > On Thu, Jan 17, 2019 at 2:11 PM Stefan Blumentrath <Stefan.Blumentrath at nina.no<mailto:Stefan.Blumentrath at nina.no>> wrote:
> > >
> > > Dear all,
> > >
> > > A colleague of mine just experienced that v.select in GRASS 7.4.3 (from UbuntuGIS) did not stop allocating memory until the server (Ubuntu 16.04) almost died and he killed the process.
> > >
> > > Is that a bug or would v.select have stopped allocating memory at some point before the system gets unusable? Or is this something the user should take care of? If I remember correctly there is a “LOW_MEM option” for v.in.ogr would such a thing be needed here to.
> > >
> > > In a multi-user environment this can be a bit tricky…
> >
> > yeah!
> >
> > > Any hints? Should I open a ticket (probably hard to reproduce with small test-dataset)?
> >
> > Please re-run the job with "valgrind", see
> >
> > https://grasswiki.osgeo.org/wiki/GRASS_Debugging#Using_Valgrind
> >
>
> > That will give insights about potential memory leaks. He/you may CTRL-C it on the way to keep the server alive.
>
>
>
> A simple run fo v.select with valgrind does not show any memory leak (increase of memory consumption within a loop). Can you provide the test command and ideally also the data causing the problem? That would help with debugging.
>
>
>
> Markus M
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-dev/attachments/20190221/0a5a5951/attachment-0001.html>
More information about the grass-dev
mailing list