[GRASS-dev] v.select: increasing memory consumption
Markus Metz
markus.metz.giswork at gmail.com
Thu Feb 21 13:04:10 PST 2019
Hi Stefan,
On Thu, Feb 21, 2019 at 3:58 PM Stefan Blumentrath <
Stefan.Blumentrath at nina.no> wrote:
>
> Thanks again, Marus,
>
> Here the results from the new run:
>
> valgrind --leak-check=yes --show-reachable=yes v.select ainput=grid100
binput=landNorway output=testNorwayGrid
the below output looks to me like one of the input maps is really large,
memory is consumed by just loading the vector. You can try v.info -t on the
two input maps and try to rebuild topology on the input maps. I guess that
rebuilding topology with v.build would consume just as much memory.
Markus M
>
> ==5173== by 0x4E79461: Vect__open_old (in /usr/lib/grass76/lib/
libgrass_vector.7.6.0.so)
>
> ==5173== by 0x4E79DE0: Vect_open_old2 (in /usr/lib/grass76/lib/
libgrass_vector.7.6.0.so)
>
> ==5173== by 0x402B96: main (in /usr/lib/grass76/bin/v.select)
>
> ==5173==
>
> ==5173== 923,048 bytes in 78,794 blocks are still reachable in loss
record 213 of 219
>
> ==5173== at 0x4C2DB8F: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
>
> ==5173== by 0x52C6BCE: G__malloc (in /usr/lib/grass76/lib/
libgrass_gis.7.6.0.so)
>
> ==5173== by 0x5F33897: dig_Rd_P_line (in /usr/lib/grass76/lib/
libgrass_dig2.7.6.0.so)
>
> ==5173== by 0x5F310ED: dig_load_plus (in /usr/lib/grass76/lib/
libgrass_dig2.7.6.0.so)
>
> ==5173== by 0x4E781BD: Vect_open_topo (in /usr/lib/grass76/lib/
libgrass_vector.7.6.0.so)
>
> ==5173== by 0x4E79443: Vect__open_old (in /usr/lib/grass76/lib/
libgrass_vector.7.6.0.so)
>
> ==5173== by 0x4E79DE0: Vect_open_old2 (in /usr/lib/grass76/lib/
libgrass_vector.7.6.0.so)
>
> ==5173== by 0x402B96: main (in /usr/lib/grass76/bin/v.select)
>
> ==5173==
>
> ==5173== 1,512,320 bytes in 37,808 blocks are still reachable in loss
record 214 of 219
>
> ==5173== at 0x4C2DB8F: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
>
> ==5173== by 0x52C6BCE: G__malloc (in /usr/lib/grass76/lib/
libgrass_gis.7.6.0.so)
>
> ==5173== by 0x5F3B436: dig_alloc_area (in /usr/lib/grass76/lib/
libgrass_dig2.7.6.0.so)
>
> ==5173== by 0x5F33D16: dig_Rd_P_area (in /usr/lib/grass76/lib/
libgrass_dig2.7.6.0.so)
>
> ==5173== by 0x5F3114D: dig_load_plus (in /usr/lib/grass76/lib/
libgrass_dig2.7.6.0.so)
>
> ==5173== by 0x4E781BD: Vect_open_topo (in /usr/lib/grass76/lib/
libgrass_vector.7.6.0.so)
>
> ==5173== by 0x4E79443: Vect__open_old (in /usr/lib/grass76/lib/
libgrass_vector.7.6.0.so)
>
> ==5173== by 0x4E79DE0: Vect_open_old2 (in /usr/lib/grass76/lib/
libgrass_vector.7.6.0.so)
>
> ==5173== by 0x402B96: main (in /usr/lib/grass76/bin/v.select)
>
> ==5173==
>
> ==5173== 1,621,440 bytes in 33,780 blocks are still reachable in loss
record 215 of 219
>
> ==5173== at 0x4C2DB8F: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
>
> ==5173== by 0x52C6BCE: G__malloc (in /usr/lib/grass76/lib/
libgrass_gis.7.6.0.so)
>
> ==5173== by 0x5F3B056: dig_alloc_node (in /usr/lib/grass76/lib/
libgrass_dig2.7.6.0.so)
>
> ==5173== by 0x5F334ED: dig_Rd_P_node (in /usr/lib/grass76/lib/
libgrass_dig2.7.6.0.so)
>
> ==5173== by 0x5F3108D: dig_load_plus (in /usr/lib/grass76/lib/
libgrass_dig2.7.6.0.so)
>
> ==5173== by 0x4E781BD: Vect_open_topo (in /usr/lib/grass76/lib/
libgrass_vector.7.6.0.so)
>
> ==5173== by 0x4E79443: Vect__open_old (in /usr/lib/grass76/lib/
libgrass_vector.7.6.0.so)
>
> ==5173== by 0x4E79DE0: Vect_open_old2 (in /usr/lib/grass76/lib/
libgrass_vector.7.6.0.so)
>
> ==5173== by 0x402B96: main (in /usr/lib/grass76/bin/v.select)
>
> ==5173==
>
> ==5173== 750,941,956 bytes in 1 blocks are still reachable in loss record
216 of 219
>
> ==5173== at 0x4C2FB55: calloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
>
> ==5173== by 0x52C6CAD: G__calloc (in /usr/lib/grass76/lib/
libgrass_gis.7.6.0.so)
>
> ==5173== by 0x402BF8: main (in /usr/lib/grass76/bin/v.select)
>
> ==5173==
>
> ==5173== 1,502,514,272 bytes in 2 blocks are still reachable in loss
record 217 of 219
>
> ==5173== at 0x4C2DB8F: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
>
> ==5173== by 0x52C6DB7: G__realloc (in /usr/lib/grass76/lib/
libgrass_gis.7.6.0.so)
>
> ==5173== by 0x5F3B321: dig_alloc_lines (in /usr/lib/grass76/lib/
libgrass_dig2.7.6.0.so)
>
> ==5173== by 0x5F310D1: dig_load_plus (in /usr/lib/grass76/lib/
libgrass_dig2.7.6.0.so)
>
> ==5173== by 0x4E781BD: Vect_open_topo (in /usr/lib/grass76/lib/
libgrass_vector.7.6.0.so)
>
> ==5173== by 0x4E79443: Vect__open_old (in /usr/lib/grass76/lib/
libgrass_vector.7.6.0.so)
>
> ==5173== by 0x4E79DE0: Vect_open_old2 (in /usr/lib/grass76/lib/
libgrass_vector.7.6.0.so)
>
> ==5173== by 0x402B96: main (in /usr/lib/grass76/bin/v.select)
>
> ==5173==
>
> ==5173== 2,254,225,080 bytes in 3 blocks are still reachable in loss
record 218 of 219
>
> ==5173== at 0x4C2DB8F: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
>
> ==5173== by 0x52C6BCE: G__malloc (in /usr/lib/grass76/lib/
libgrass_gis.7.6.0.so)
>
> ==5173== by 0x5F2F4AD: dig_read_cidx (in /usr/lib/grass76/lib/
libgrass_dig2.7.6.0.so)
>
> ==5173== by 0x4E5C917: Vect_cidx_open (in /usr/lib/grass76/lib/
libgrass_vector.7.6.0.so)
>
> ==5173== by 0x4E7992F: Vect__open_old (in /usr/lib/grass76/lib/
libgrass_vector.7.6.0.so)
>
> ==5173== by 0x4E79DE0: Vect_open_old2 (in /usr/lib/grass76/lib/
libgrass_vector.7.6.0.so)
>
> ==5173== by 0x402B96: main (in /usr/lib/grass76/bin/v.select)
>
> ==5173==
>
> ==5173== 4,507,542,768 bytes in 187,814,282 blocks are still reachable in
loss record 219 of 219
>
> ==5173== at 0x4C2DB8F: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
>
> ==5173== by 0x52C6BCE: G__malloc (in /usr/lib/grass76/lib/
libgrass_gis.7.6.0.so)
>
> ==5173== by 0x5F3B1D6: dig_alloc_line (in /usr/lib/grass76/lib/
libgrass_dig2.7.6.0.so)
>
> ==5173== by 0x5F33841: dig_Rd_P_line (in /usr/lib/grass76/lib/
libgrass_dig2.7.6.0.so)
>
> ==5173== by 0x5F310ED: dig_load_plus (in /usr/lib/grass76/lib/
libgrass_dig2.7.6.0.so)
>
> ==5173== by 0x4E781BD: Vect_open_topo (in /usr/lib/grass76/lib/
libgrass_vector.7.6.0.so)
>
> ==5173== by 0x4E79443: Vect__open_old (in /usr/lib/grass76/lib/
libgrass_vector.7.6.0.so)
>
> ==5173== by 0x4E79DE0: Vect_open_old2 (in /usr/lib/grass76/lib/
libgrass_vector.7.6.0.so)
>
> ==5173== by 0x402B96: main (in /usr/lib/grass76/bin/v.select)
>
> ==5173==
>
> ==5173== LEAK SUMMARY:
>
> ==5173== definitely lost: 956 bytes in 39 blocks
>
> ==5173== indirectly lost: 72 bytes in 10 blocks
>
> ==5173== possibly lost: 79 bytes in 4 blocks
>
> ==5173== still reachable: 9,025,026,330 bytes in 188,201,312 blocks
>
> ==5173== of which reachable via heuristic:
>
> ==5173== newarray : 488,856 bytes in
20,369 blocks
>
> ==5173== suppressed: 0 bytes in 0 blocks
>
> ==5173==
>
> ==5173== For counts of detected and suppressed errors, rerun with: -v
>
> ==5173== ERROR SUMMARY: 27 errors from 27 contexts (suppressed: 0 from 0)
>
>
>
>
>
> Cheers
>
> Stefan
>
>
>
>
>
> From: Markus Metz <markus.metz.giswork at gmail.com>
> Sent: torsdag 21. februar 2019 15:23
> To: Stefan Blumentrath <Stefan.Blumentrath at nina.no>
> Cc: Markus Neteler <neteler at osgeo.org>; GRASS developers list (
grass-dev at lists.osgeo.org) <grass-dev at lists.osgeo.org>
> Subject: Re: [GRASS-dev] v.select: increasing memory consumption
>
>
>
> Hi Stefan,
>
> On Thu, Feb 21, 2019 at 2:20 PM Stefan Blumentrath <
Stefan.Blumentrath at nina.no> wrote:
> >
> > Hi again,
> >
> > Please find below the output from a run of the problematic command with
valgrind (note that ainput data (points) are stored in PostGIS
(v.external)).
> >
> > Hope that helps to shed some light…
> >
> > Cheers
> >
> > Stefan
> >
> > valgrind --leak-check=yes v.select ainput=grid100 binput=landNorway
output=testNorwayGrid
> >
> > [...]
> >
> > ==13217== LEAK SUMMARY:
> > ==13217== definitely lost: 964 bytes in 39 blocks
> > ==13217== indirectly lost: 72 bytes in 10 blocks
> > ==13217== possibly lost: 79 bytes in 4 blocks
> > ==13217== still reachable: 9,025,026,330 bytes in 188,201,312 blocks
>
> this is the problem, please run valgrind again with
> --show-reachable=yes
>
>
>
> we need to find out where these 9,025,026,330 bytes have been allocated.
>
>
>
> Markus M
>
>
>
> >
>
> > From: Markus Metz <markus.metz.giswork at gmail.com>
> > Sent: lørdag 19. januar 2019 22:17
> > To: Stefan Blumentrath <Stefan.Blumentrath at nina.no>
> > Cc: Markus Neteler <neteler at osgeo.org>; GRASS developers list (
grass-dev at lists.osgeo.org) <grass-dev at lists.osgeo.org>
> > Subject: Re: [GRASS-dev] v.select: increasing memory consumption
> >
> >
> >
> >
> >
> > On Fri, Jan 18, 2019 at 8:29 AM Stefan Blumentrath <
Stefan.Blumentrath at nina.no> wrote:
> > >
> > > THanks, we`ll try and report back.
> > >
> > > -----Original Message-----
> > > From: Markus Neteler <neteler at osgeo.org>
> > > Sent: fredag 18. januar 2019 07:52
> > > To: Stefan Blumentrath <Stefan.Blumentrath at nina.no>
> > > Cc: GRASS developers list (grass-dev at lists.osgeo.org) <
grass-dev at lists.osgeo.org>
> > > Subject: Re: [GRASS-dev] v.select: increasing memory consumption
> > >
> > > Hi,
> > >
> > > On Thu, Jan 17, 2019 at 2:11 PM Stefan Blumentrath <
Stefan.Blumentrath at nina.no> wrote:
> > > >
> > > > Dear all,
> > > >
> > > > A colleague of mine just experienced that v.select in GRASS 7.4.3
(from UbuntuGIS) did not stop allocating memory until the server (Ubuntu
16.04) almost died and he killed the process.
> > > >
> > > > Is that a bug or would v.select have stopped allocating memory at
some point before the system gets unusable? Or is this something the user
should take care of? If I remember correctly there is a “LOW_MEM option”
for v.in.ogr would such a thing be needed here to.
> > > >
> > > > In a multi-user environment this can be a bit tricky…
> > >
> > > yeah!
> > >
> > > > Any hints? Should I open a ticket (probably hard to reproduce with
small test-dataset)?
> > >
> > > Please re-run the job with "valgrind", see
> > >
> > > https://grasswiki.osgeo.org/wiki/GRASS_Debugging#Using_Valgrind
> > >
> >
> > > That will give insights about potential memory leaks. He/you may
CTRL-C it on the way to keep the server alive.
> >
> >
> >
> > A simple run fo v.select with valgrind does not show any memory leak
(increase of memory consumption within a loop). Can you provide the test
command and ideally also the data causing the problem? That would help with
debugging.
> >
> >
> >
> > Markus M
> >
> >
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-dev/attachments/20190221/a0249763/attachment-0001.html>
More information about the grass-dev
mailing list