[GRASS-user] i.segment.stats memory usage error
James Duffy
james.philip.duffy at gmail.com
Thu Oct 27 12:33:23 PDT 2016
On 27 October 2016 at 17:48, Moritz Lennert <mlennert at club.worldonline.be>
wrote:
> Le Thu, 27 Oct 2016 16:45:47 +0100,
> James Duffy <james.philip.duffy at gmail.com> a écrit :
>
> > On 27 October 2016 at 16:33, Moritz Lennert
> > <mlennert at club.worldonline.be> wrote:
> >
> > > On 27/10/16 16:15, Moritz Lennert wrote:
> > >
> > >> On 27/10/16 15:38, James Duffy wrote:
> > >>
> > >>>
> > >>>
> > >>> On 27 October 2016 at 13:53, Moritz Lennert
> > >>> <mlennert at club.worldonline.be
> > >>> <mailto:mlennert at club.worldonline.be>> wrote:
> > >>>
> > >>> On 27/10/16 12:51, James Duffy wrote:
> > >>>
> > >>>
> > >>>
> > >>> On 27 October 2016 at 11:45, Moritz Lennert
> > >>> <mlennert at club.worldonline.be
> > >>> <mailto:mlennert at club.worldonline.be>
> > >>> <mailto:mlennert at club.worldonline.be
> > >>> <mailto:mlennert at club.worldonline.be>>> wrote:
> > >>>
> > >>>
> > >>>
> > >>> Le 27 octobre 2016 12:35:14 GMT+02:00, James Duffy
> > >>> <james.philip.duffy at gmail.com
> > >>> <mailto:james.philip.duffy at gmail.com>
> > >>> <mailto:james.philip.duffy at gmail.com
> > >>> <mailto:james.philip.duffy at gmail.com>>>
> > >>> a écrit :
> > >>> >On 27 October 2016 at 11:08, Moritz Lennert
> > >>> ><mlennert at club.worldonline.be
> > >>> <mailto:mlennert at club.worldonline.be>
> > >>> <mailto:mlennert at club.worldonline.be
> > >>> <mailto:mlennert at club.worldonline.be>>>
> > >>> >wrote:
> > >>> >
> > >>> >>
> > >>> >>
> > >>> >> Le 27 octobre 2016 11:22:02 GMT+02:00, James Duffy
> > >>> >> < james.philip.duffy at gmail.com
> > >>> <mailto:james.philip.duffy at gmail.com>
> > >>> <mailto:james.philip.duffy at gmail.com
> > >>> <mailto:james.philip.duffy at gmail.com>>> a écrit :
> > >>>
> > >>> >> >Hello,
> > >>> >> >
> > >>> >> >I'm trying to use i.segment.stats with GRASS
> > >>> >> >7.0.4 in
> > >>> the osgeolive
> > >>> >> >32bit
> > >>> >> >operating system.
> > >>> >> >
> > >>> >> >Prior to using this tool I have successfully
> > >>> >> >created my
> > >>> segmented
> > >>> >map
> > >>> >> >using
> > >>> >> >i.segment.
> > >>> >> >
> > >>> >> >When I try to execute the following command:
> > >>> >> >
> > >>> >> >i.segment.stats --overwrite --verbose
> > >>> map=gp_seg_optimum at gp1 \
> > >>> >>
> > >>>
> > >>> >rasters=gp_ortho.1 at gp1,gp_ortho.2 at gp1,gp_ortho.3 at gp1,gp_ortho.4 at gp1
> > >>> >\
> > >>> >> >raster_statistics=min,max,mean,stddev,variance,sum
> > >>> >> >\
> > >>> >>
> > >>> >csvfile=/home/jpd205/Wales_GRASS/GarronPill/gp_seg_stats \
> > >>> >> >separator=comma
> > >>> >> >
> > >>> >> >I get this error:
> > >>> >> >
> > >>> >> >Calculating geometry statistics
> > >>> >> >ERROR: G_malloc: unable to allocate 4273800320
> > >>> >> >bytes of
> > >>> memory at
> > >>> >> > /tmp/tmpgzUtnA/r.object.geometry/main.c:129
> > >>> >>
> > >>> >> This is coming from r.object.geometry.
> > >>> >>
> > >>> >> What are your region settings (g.region -p) ? How
> > >>> >> many
> > >>> segments do
> > >>> >you
> > >>> >> have ?
> > >>> >>
> > >>> >
> > >>> >GRASS 7.0.4 (GarronPill):~ > g.region -p
> > >>> >projection: 99 (OSGB 1936 / British National Grid)
> > >>> >zone: 0
> > >>> >datum: osgb36
> > >>> >ellipsoid: airy
> > >>> >north: 208007.00931776
> > >>> >south: 207952.59780698
> > >>> >west: 200993.90853302
> > >>> >east: 201097.28911076
> > >>> >nsres: 0.00430914
> > >>> >ewres: 0.00430914
> > >>> >rows: 12627
> > >>> >cols: 23991
> > >>> >cells: 302934357
> > >>>
> > >>> Are you sure 4mm is correct for the resolution ?
> > >>>
> > >>>
> > >>> Yes. It's high resolution imagery from a drone.
> > >>>
> > >>>
> > >>>
> > >>> What does r.info <http://r.info> <http://r.info> on
> > >>>
> > >>>
> > >>> >
> > >>> >
> > >>> >>
> > >>> >> Try running I.segment.stats without requesting
> > >>> >> form
> > >>> statistics.
> > >>> >
> > >>> >
> > >>> >Running:
> > >>> >
> > >>> >i.segment.stats --overwrite --verbose
> > >>> map=gp_seg_optimum at gp1 \
> > >>> >csvfile=/home/jpd205/Wales_
> GRASS/GarronPill/gp_seg_stats
> > >>> >\ separator=comma
> > >>>
> > >>> Default is to do form statistics, so the above line
> > >>> also does. I
> > >>> have to admit that I don't know what happens when you
> > >>> give an empty
> > >>> parameter such as
> > >>>
> > >>> area_measures= or area_measures=""
> > >>>
> > >>> What does r.info <http://r.info> <http://r.info>
> > >>> gp_seg_optimum give you ?
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> +-----------------------------------------------------------
> > >>> -----------------+
> > >>>
> > >>> | Map: gp_seg_optimum Date: Thu Oct
> > >>> 27 06:45:57
> > >>> 2016 |
> > >>> | Mapset: gp1 Login of
> > >>> Creator: jpd205 |
> > >>> | Location:
> > >>>
> > >>> GarronPill |
> > >>> | DataBase:
> > >>>
> > >>> /home/jpd205/Wales_GRASS |
> > >>> | Title: ( gp_seg_optimum
> > >>> ) |
> > >>> | Timestamp:
> > >>> none
> > >>> |
> > >>>
> > >>> |-----------------------------------------------------------
> > >>> -----------------|
> > >>>
> > >>> |
> > >>> |
> > >>> | Type of Map: raster Number of
> > >>> Categories: 0 |
> > >>> | Data Type:
> > >>> CELL
> > >>> | | Rows:
> > >>> 12627
> > >>> | | Columns:
> > >>> 23991
> > >>> | | Total Cells:
> > >>> 302934357
> > >>> | | Projection: OSGB 1936 / British National
> > >>> Grid |
> > >>> | N: 208007.00931776 S: 207952.59780698
> > >>> Res: 0.00430914 |
> > >>> | E: 201097.28911076 W: 200993.90853302
> > >>> Res: 0.00430914 |
> > >>> | Range of data: min = 35 max =
> > >>> 133556294 |
> > >>> |
> > >>>
> > >>>
> > >>> Ok, I think I see at least part of the problem: in GRASS 7.0
> > >>> i.segment numbers segments non sequentially, i.e. whenever a
> > >>> new segment is created out of the merger of two existing
> > >>> segments, a new id is assigned. In GRASS trunk (the development
> > >>> version) i.segment is rewritten to, at the end, number all
> > >>> segments sequentially from 1 to the number of segments.
> > >>>
> > >>> r.object.geometry allocates memory according to the new
> > >>> system. As it says in the code:
> > >>>
> > >>> /* REMARK: The following is only true if object ids are
> > >>> continuously numbered */
> > >>> n_objects = max - min + 1;
> > >>> obj_geos = G_malloc(n_objects * sizeof(struct obj_geo));
> > >>>
> > >>> So, one solution would be to use the development version of
> > >>> GRASS.
> > >>>
> > >>>
> > >>>
> > >>> Another would be to run r.clump on the result of the
> > >>> segmentation in order to renumber the segments before running it
> > >>> through i.segment.stats.
> > >>>
> > >>>
> > >>> This option seemed the most straightforward given my current
> > >>> setup. I ran r.clump with diagnoal enabled, and the output looked
> > >>> good. It only took 2mins to run.
> > >>>
> > >>> I then proceeded with the original i.segment.stats command and
> > >>> got the following:
> > >>>
> > >>> Calculating geometry statistics
> > >>> Calculating statistics for raster gp_ortho.1 at gp1
> > >>> ERROR: G_realloc: unable to allocate 572000 bytes of memory at
> > >>> raster/r.univar/r.univar_main.c:324
> > >>>
> > >>> It looks like the tool got further, but is still getting stuck on
> > >>> something...
> > >>>
> > >>
> > >> Yes, now there a memory issue in the part where it uses r.univar to
> > >> calculate statistics per segment on one of the spectral bands.
> > >>
> > >
> > > Could you run
> > >
> > > r.univar -et gp_ortho.1 zones=ClumpedSegmentMap
> > > output=testoutput.csv
> > >
> > > to if r.univar by itself handles this (which would mean I could try
> > > to change the data handling in i.segment.stats to make it more
> > > memory efficient), or not ?
> > >
> > > Thanks !
> > >
> > > Moritz
> > >
> >
> > r.info on the clumped map gives:
> >
> > +-----------------------------------------------------------
> -----------------+
> > | Map: gp_seg_optimum_clump Date: Thu Oct 27 14:28:30
> > 2016 |
> > | Mapset: gp1 Login of Creator:
> > jpd205 |
> > | Location:
> > GarronPill |
> > | DataBase:
> > /home/jpd205/Wales_GRASS |
> > | Title: gp_seg_optimum_clump ( gp_seg_optimum_clump
> > ) |
> > | Timestamp:
> > none |
> > |-----------------------------------------------------------
> -----------------|
> > |
> > |
> > | Type of Map: raster Number of Categories:
> > 0 |
> > | Data Type:
> > CELL |
> > | Rows:
> > 12627 |
> > | Columns:
> > 23991 |
> > | Total Cells:
> > 302934357 |
> > | Projection: OSGB 1936 / British National
> > Grid |
> > | N: 208007.00931776 S: 207952.59780698 Res:
> > 0.00430914 |
> > | E: 201097.28911076 W: 200993.90853302 Res:
> > 0.00430914 |
> > | Range of data: min = 1 max =
> > 1264957 |
> > |
> > |
> > | Data
> > Source: |
> > | gp_seg_optimum at gp1
> > |
> > |
> > |
> > |
> > |
> > | Data
> > Description: |
> > | generated by
> > r.clump |
> > |
> > |
> > |
> > Comments:
> > | | r.clump --overwrite --verbose -d input="gp_seg_optimum at gp1"
> > output="\ |
> > | gp_seg_optimum_clump"
> > title="gp_seg_optimum_clump" |
> > |
> > |
> > +-----------------------------------------------------------
> -----------------+
> >
> > Running:
> >
> > r.univar -et gp_ortho.1 zones=gp_seg_optimum_clump
> > output=testoutput2.csv
> >
> > Returns:
> >
> > Current region rows: 12627, cols: 23991
> > ERROR: G_realloc: unable to allocate 224000 bytes of memory at
> > raster/r.univar/r.univar_main.c:324
>
>
> Ok, so the issue is with r.univar (although the issue has made me
> aware that i.segment.stats could be made way more memory efficient
> as well).
>
> How much RAM do you have on your machine ?
>
It's a virtual machine which I have dedicated about 10GB.
>
> This warrants a bug report. Would you be willing to file one on
> http://trac.osgeo.org/grass ?
>
I'm happy to submit it if you can guide me as to what exactly should go
into it please?
>
> Moritz
>
James
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-user/attachments/20161027/3c26cf2c/attachment-0001.html>
More information about the grass-user
mailing list