[GRASS-dev] Memory consumption using pygrass.utils.get_raster_for_points

[Markus Neteler]

hi Steven

(yes, the mail came through)

On Sat, Feb 9, 2019 at 5:39 PM Steven Pawley <dr.stevenpawley at gmail.com> wrote:
>
> Hello devs,
>
> When running pygrass.utils.get_raster_for_points repeatedly, it appears
> that the python memory allocation continuously increases until all ram
> is consumed, even if the extracted values are not being collected (or
> are overwriting the same variable).
>
> I noticed this when extracting raster data at point locations, when
> using a large point dataset, even though I had pre-allocated a numpy
> array to receive the results.

Do you see a chance to run valgrind on this? (maybe there are more
appropriate tools in the Python world):

https://grasswiki.osgeo.org/wiki/GRASS_Debugging#Using_Valgrind

> Below is an example on the nc_spm_08_grass7 example data (in the landsat
> mapset), repeating the operation say 50 times on the same point vector
> dataset. I wouldn't have expected the memory consumption to continuously
> increase for this operation, because I'm overwriting the 'arr' variable
> each time. However, if you repeat this enough times, you will run out of
> system memory and the allocated memory does not appear to be released,
> i.e. even if you manually force garbage collection.
>
> Any suggestions?
>
>
> from grass.pygrass.vector import VectorTopo
> from grass.pygrass.raster import RasterRow
> from grass.pygrass.modules.shortcuts import raster as r
> from grass.pygrass.gis.region import Region
> from grass.pygrass.utils import get_raster_for_points
>
> reg = Region()
> reg.from_rast("landclass96")
> reg.write()
> reg.set_raster_region()
>
> r.random(input="landclass96", npoints=200000, vector="landclass96_roi",
>           overwrite=True)
>
> points = VectorTopo("landclass96_roi")
> points.open("r")
>
> # repeat spatial query of raster
> for i in range(50):
>      with RasterRow("lsat5_1987_10") as src:
>          arr = get_raster_for_points(points, src)

Hope someone here can support you!

Markus