[GRASS-dev] v.select: increasing memory consumption

Thu Feb 21 05:20:51 PST 2019

Hi again,

Please find below the output from a run of the problematic command with valgrind (note that ainput data (points) are stored in PostGIS (v.external)).

Hope that helps to shed some light…

Cheers
Stefan

valgrind --leak-check=yes v.select ainput=grid100 binput=landNorway<mailto:binput=landNorway at u_torkild.tveraa> output=testNorwayGrid

==13217==    by 0x52CE187: ??? (in /usr/lib/grass76/lib/libgrass_gis.7.6.0.so)
==13217==    by 0x52E02D4: ??? (in /usr/lib/grass76/lib/libgrass_gis.7.6.0.so)
==13217==    by 0x52E0537: G_fopen_old (in /usr/lib/grass76/lib/libgrass_gis.7.6.0.so)
==13217==    by 0x4E63A88: ??? (in /usr/lib/grass76/lib/libgrass_vector.7.6.0.so)
==13217==    by 0x4E64247: Vect_read_dblinks (in /usr/lib/grass76/lib/libgrass_vector.7.6.0.so)
==13217==    by 0x4E795CA: Vect__open_old (in /usr/lib/grass76/lib/libgrass_vector.7.6.0.so)
==13217==    by 0x4E79DE0: Vect_open_old2 (in /usr/lib/grass76/lib/libgrass_vector.7.6.0.so)
==13217==    by 0x402B96: main (in /usr/lib/grass76/bin/v.select)
==13217==
==13217== 35 bytes in 2 blocks are definitely lost in loss record 80 of 219
==13217==    at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==13217==    by 0x52C6BCE: G__malloc (in /usr/lib/grass76/lib/libgrass_gis.7.6.0.so)
==13217==    by 0x52F2222: G_store (in /usr/lib/grass76/lib/libgrass_gis.7.6.0.so)
==13217==    by 0x52CE187: ??? (in /usr/lib/grass76/lib/libgrass_gis.7.6.0.so)
==13217==    by 0x52E02D4: ??? (in /usr/lib/grass76/lib/libgrass_gis.7.6.0.so)
==13217==    by 0x52E0537: G_fopen_old (in /usr/lib/grass76/lib/libgrass_gis.7.6.0.so)
==13217==    by 0x4E799F5: Vect__open_old (in /usr/lib/grass76/lib/libgrass_vector.7.6.0.so)
==13217==    by 0x4E79DE0: Vect_open_old2 (in /usr/lib/grass76/lib/libgrass_vector.7.6.0.so)
==13217==    by 0x402B96: main (in /usr/lib/grass76/bin/v.select)
==13217==
==13217== 122 bytes in 2 blocks are definitely lost in loss record 137 of 219
==13217==    at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==13217==    by 0x52C6BCE: G__malloc (in /usr/lib/grass76/lib/libgrass_gis.7.6.0.so)
==13217==    by 0x52F2222: G_store (in /usr/lib/grass76/lib/libgrass_gis.7.6.0.so)
==13217==    by 0x52F295E: G_tempfile_pid (in /usr/lib/grass76/lib/libgrass_gis.7.6.0.so)
==13217==    by 0x5F37144: dig_spidx_init (in /usr/lib/grass76/lib/libgrass_dig2.7.6.0.so)
==13217==    by 0x4E78541: Vect_open_sidx (in /usr/lib/grass76/lib/libgrass_vector.7.6.0.so)
==13217==    by 0x4E79461: Vect__open_old (in /usr/lib/grass76/lib/libgrass_vector.7.6.0.so)
==13217==    by 0x4E79DE0: Vect_open_old2 (in /usr/lib/grass76/lib/libgrass_vector.7.6.0.so)
==13217==    by 0x402B96: main (in /usr/lib/grass76/bin/v.select)
==13217==
==13217== 122 bytes in 2 blocks are definitely lost in loss record 138 of 219
==13217==    at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==13217==    by 0x52C6BCE: G__malloc (in /usr/lib/grass76/lib/libgrass_gis.7.6.0.so)
==13217==    by 0x52F2222: G_store (in /usr/lib/grass76/lib/libgrass_gis.7.6.0.so)
==13217==    by 0x52F295E: G_tempfile_pid (in /usr/lib/grass76/lib/libgrass_gis.7.6.0.so)
==13217==    by 0x5F3717A: dig_spidx_init (in /usr/lib/grass76/lib/libgrass_dig2.7.6.0.so)
==13217==    by 0x4E78541: Vect_open_sidx (in /usr/lib/grass76/lib/libgrass_vector.7.6.0.so)
==13217==    by 0x4E79461: Vect__open_old (in /usr/lib/grass76/lib/libgrass_vector.7.6.0.so)
==13217==    by 0x4E79DE0: Vect_open_old2 (in /usr/lib/grass76/lib/libgrass_vector.7.6.0.so)
==13217==    by 0x402B96: main (in /usr/lib/grass76/bin/v.select)
==13217==
==13217== 122 bytes in 2 blocks are definitely lost in loss record 139 of 219
==13217==    at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==13217==    by 0x52C6BCE: G__malloc (in /usr/lib/grass76/lib/libgrass_gis.7.6.0.so)
==13217==    by 0x52F2222: G_store (in /usr/lib/grass76/lib/libgrass_gis.7.6.0.so)
==13217==    by 0x52F295E: G_tempfile_pid (in /usr/lib/grass76/lib/libgrass_gis.7.6.0.so)
==13217==    by 0x5F371B0: dig_spidx_init (in /usr/lib/grass76/lib/libgrass_dig2.7.6.0.so)
==13217==    by 0x4E78541: Vect_open_sidx (in /usr/lib/grass76/lib/libgrass_vector.7.6.0.so)
==13217==    by 0x4E79461: Vect__open_old (in /usr/lib/grass76/lib/libgrass_vector.7.6.0.so)
==13217==    by 0x4E79DE0: Vect_open_old2 (in /usr/lib/grass76/lib/libgrass_vector.7.6.0.so)
==13217==    by 0x402B96: main (in /usr/lib/grass76/bin/v.select)
==13217==
==13217== 122 bytes in 2 blocks are definitely lost in loss record 140 of 219
==13217==    at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==13217==    by 0x52C6BCE: G__malloc (in /usr/lib/grass76/lib/libgrass_gis.7.6.0.so)
==13217==    by 0x52F2222: G_store (in /usr/lib/grass76/lib/libgrass_gis.7.6.0.so)
==13217==    by 0x52F295E: G_tempfile_pid (in /usr/lib/grass76/lib/libgrass_gis.7.6.0.so)
==13217==    by 0x5F371E6: dig_spidx_init (in /usr/lib/grass76/lib/libgrass_dig2.7.6.0.so)
==13217==    by 0x4E78541: Vect_open_sidx (in /usr/lib/grass76/lib/libgrass_vector.7.6.0.so)
==13217==    by 0x4E79461: Vect__open_old (in /usr/lib/grass76/lib/libgrass_vector.7.6.0.so)
==13217==    by 0x4E79DE0: Vect_open_old2 (in /usr/lib/grass76/lib/libgrass_vector.7.6.0.so)
==13217==    by 0x402B96: main (in /usr/lib/grass76/bin/v.select)
==13217==
==13217== 168 (96 direct, 72 indirect) bytes in 2 blocks are definitely lost in loss record 150 of 219
==13217==    at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==13217==    by 0x52C6BCE: G__malloc (in /usr/lib/grass76/lib/libgrass_gis.7.6.0.so)
==13217==    by 0x4E64879: Vect_get_dblink (in /usr/lib/grass76/lib/libgrass_vector.7.6.0.so)
==13217==    by 0x4E64BC3: Vect_get_field_number (in /usr/lib/grass76/lib/libgrass_vector.7.6.0.so)
==13217==    by 0x402BB9: main (in /usr/lib/grass76/bin/v.select)
==13217==
==13217== LEAK SUMMARY:
==13217==    definitely lost: 964 bytes in 39 blocks
==13217==    indirectly lost: 72 bytes in 10 blocks
==13217==      possibly lost: 79 bytes in 4 blocks
==13217==    still reachable: 9,025,026,330 bytes in 188,201,312 blocks
==13217==                       of which reachable via heuristic:
==13217==                         newarray           : 488,856 bytes in 20,369 blocks
==13217==         suppressed: 0 bytes in 0 blocks

From: Markus Metz <markus.metz.giswork at gmail.com>
Sent: lørdag 19. januar 2019 22:17
To: Stefan Blumentrath <Stefan.Blumentrath at nina.no>
Cc: Markus Neteler <neteler at osgeo.org>; GRASS developers list (grass-dev at lists.osgeo.org) <grass-dev at lists.osgeo.org>
Subject: Re: [GRASS-dev] v.select: increasing memory consumption

On Fri, Jan 18, 2019 at 8:29 AM Stefan Blumentrath <Stefan.Blumentrath at nina.no<mailto:Stefan.Blumentrath at nina.no>> wrote:
>
> THanks, we`ll try and report back.
>
> -----Original Message-----
> From: Markus Neteler <neteler at osgeo.org<mailto:neteler at osgeo.org>>
> Sent: fredag 18. januar 2019 07:52
> To: Stefan Blumentrath <Stefan.Blumentrath at nina.no<mailto:Stefan.Blumentrath at nina.no>>
> Cc: GRASS developers list (grass-dev at lists.osgeo.org<mailto:grass-dev at lists.osgeo.org>) <grass-dev at lists.osgeo.org<mailto:grass-dev at lists.osgeo.org>>
> Subject: Re: [GRASS-dev] v.select: increasing memory consumption
>
> Hi,
>
> On Thu, Jan 17, 2019 at 2:11 PM Stefan Blumentrath <Stefan.Blumentrath at nina.no<mailto:Stefan.Blumentrath at nina.no>> wrote:
> >
> > Dear all,
> >
> > A colleague of mine just experienced that v.select in GRASS 7.4.3 (from UbuntuGIS) did not stop allocating memory until the server (Ubuntu 16.04) almost died and he killed the process.
> >
> > Is that a bug or would v.select have stopped allocating memory at some point before the system gets unusable? Or is this something the user should take care of? If I remember correctly there is a “LOW_MEM option” for v.in.ogr would such a thing be needed here to.
> >
> > In a multi-user environment this can be a bit tricky…
>
> yeah!
>
> > Any hints? Should I open a ticket (probably hard to reproduce with small test-dataset)?
>
> Please re-run the job with "valgrind", see
>
> https://grasswiki.osgeo.org/wiki/GRASS_Debugging#Using_Valgrind
>
> That will give insights about potential memory leaks. He/you may CTRL-C it on the way to keep the server alive.

A simple run fo v.select with valgrind does not show any memory leak (increase of memory consumption within a loop). Can you provide the test command and ideally also the data causing the problem? That would help with debugging.

Markus M

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-dev/attachments/20190221/c9fef8ce/attachment-0001.html>