[GRASS-user] i.segment.stats memory usage error
Moritz Lennert
mlennert at club.worldonline.be
Thu Oct 27 09:48:08 PDT 2016
Le Thu, 27 Oct 2016 16:45:47 +0100,
James Duffy <james.philip.duffy at gmail.com> a écrit :
> On 27 October 2016 at 16:33, Moritz Lennert
> <mlennert at club.worldonline.be> wrote:
>
> > On 27/10/16 16:15, Moritz Lennert wrote:
> >
> >> On 27/10/16 15:38, James Duffy wrote:
> >>
> >>>
> >>>
> >>> On 27 October 2016 at 13:53, Moritz Lennert
> >>> <mlennert at club.worldonline.be
> >>> <mailto:mlennert at club.worldonline.be>> wrote:
> >>>
> >>> On 27/10/16 12:51, James Duffy wrote:
> >>>
> >>>
> >>>
> >>> On 27 October 2016 at 11:45, Moritz Lennert
> >>> <mlennert at club.worldonline.be
> >>> <mailto:mlennert at club.worldonline.be>
> >>> <mailto:mlennert at club.worldonline.be
> >>> <mailto:mlennert at club.worldonline.be>>> wrote:
> >>>
> >>>
> >>>
> >>> Le 27 octobre 2016 12:35:14 GMT+02:00, James Duffy
> >>> <james.philip.duffy at gmail.com
> >>> <mailto:james.philip.duffy at gmail.com>
> >>> <mailto:james.philip.duffy at gmail.com
> >>> <mailto:james.philip.duffy at gmail.com>>>
> >>> a écrit :
> >>> >On 27 October 2016 at 11:08, Moritz Lennert
> >>> ><mlennert at club.worldonline.be
> >>> <mailto:mlennert at club.worldonline.be>
> >>> <mailto:mlennert at club.worldonline.be
> >>> <mailto:mlennert at club.worldonline.be>>>
> >>> >wrote:
> >>> >
> >>> >>
> >>> >>
> >>> >> Le 27 octobre 2016 11:22:02 GMT+02:00, James Duffy
> >>> >> < james.philip.duffy at gmail.com
> >>> <mailto:james.philip.duffy at gmail.com>
> >>> <mailto:james.philip.duffy at gmail.com
> >>> <mailto:james.philip.duffy at gmail.com>>> a écrit :
> >>>
> >>> >> >Hello,
> >>> >> >
> >>> >> >I'm trying to use i.segment.stats with GRASS
> >>> >> >7.0.4 in
> >>> the osgeolive
> >>> >> >32bit
> >>> >> >operating system.
> >>> >> >
> >>> >> >Prior to using this tool I have successfully
> >>> >> >created my
> >>> segmented
> >>> >map
> >>> >> >using
> >>> >> >i.segment.
> >>> >> >
> >>> >> >When I try to execute the following command:
> >>> >> >
> >>> >> >i.segment.stats --overwrite --verbose
> >>> map=gp_seg_optimum at gp1 \
> >>> >>
> >>>
> >>> >rasters=gp_ortho.1 at gp1,gp_ortho.2 at gp1,gp_ortho.3 at gp1,gp_ortho.4 at gp1
> >>> >\
> >>> >> >raster_statistics=min,max,mean,stddev,variance,sum
> >>> >> >\
> >>> >>
> >>> >csvfile=/home/jpd205/Wales_GRASS/GarronPill/gp_seg_stats \
> >>> >> >separator=comma
> >>> >> >
> >>> >> >I get this error:
> >>> >> >
> >>> >> >Calculating geometry statistics
> >>> >> >ERROR: G_malloc: unable to allocate 4273800320
> >>> >> >bytes of
> >>> memory at
> >>> >> > /tmp/tmpgzUtnA/r.object.geometry/main.c:129
> >>> >>
> >>> >> This is coming from r.object.geometry.
> >>> >>
> >>> >> What are your region settings (g.region -p) ? How
> >>> >> many
> >>> segments do
> >>> >you
> >>> >> have ?
> >>> >>
> >>> >
> >>> >GRASS 7.0.4 (GarronPill):~ > g.region -p
> >>> >projection: 99 (OSGB 1936 / British National Grid)
> >>> >zone: 0
> >>> >datum: osgb36
> >>> >ellipsoid: airy
> >>> >north: 208007.00931776
> >>> >south: 207952.59780698
> >>> >west: 200993.90853302
> >>> >east: 201097.28911076
> >>> >nsres: 0.00430914
> >>> >ewres: 0.00430914
> >>> >rows: 12627
> >>> >cols: 23991
> >>> >cells: 302934357
> >>>
> >>> Are you sure 4mm is correct for the resolution ?
> >>>
> >>>
> >>> Yes. It's high resolution imagery from a drone.
> >>>
> >>>
> >>>
> >>> What does r.info <http://r.info> <http://r.info> on
> >>>
> >>>
> >>> >
> >>> >
> >>> >>
> >>> >> Try running I.segment.stats without requesting
> >>> >> form
> >>> statistics.
> >>> >
> >>> >
> >>> >Running:
> >>> >
> >>> >i.segment.stats --overwrite --verbose
> >>> map=gp_seg_optimum at gp1 \
> >>> >csvfile=/home/jpd205/Wales_GRASS/GarronPill/gp_seg_stats
> >>> >\ separator=comma
> >>>
> >>> Default is to do form statistics, so the above line
> >>> also does. I
> >>> have to admit that I don't know what happens when you
> >>> give an empty
> >>> parameter such as
> >>>
> >>> area_measures= or area_measures=""
> >>>
> >>> What does r.info <http://r.info> <http://r.info>
> >>> gp_seg_optimum give you ?
> >>>
> >>>
> >>>
> >>>
> >>> +-----------------------------------------------------------
> >>> -----------------+
> >>>
> >>> | Map: gp_seg_optimum Date: Thu Oct
> >>> 27 06:45:57
> >>> 2016 |
> >>> | Mapset: gp1 Login of
> >>> Creator: jpd205 |
> >>> | Location:
> >>>
> >>> GarronPill |
> >>> | DataBase:
> >>>
> >>> /home/jpd205/Wales_GRASS |
> >>> | Title: ( gp_seg_optimum
> >>> ) |
> >>> | Timestamp:
> >>> none
> >>> |
> >>>
> >>> |-----------------------------------------------------------
> >>> -----------------|
> >>>
> >>> |
> >>> |
> >>> | Type of Map: raster Number of
> >>> Categories: 0 |
> >>> | Data Type:
> >>> CELL
> >>> | | Rows:
> >>> 12627
> >>> | | Columns:
> >>> 23991
> >>> | | Total Cells:
> >>> 302934357
> >>> | | Projection: OSGB 1936 / British National
> >>> Grid |
> >>> | N: 208007.00931776 S: 207952.59780698
> >>> Res: 0.00430914 |
> >>> | E: 201097.28911076 W: 200993.90853302
> >>> Res: 0.00430914 |
> >>> | Range of data: min = 35 max =
> >>> 133556294 |
> >>> |
> >>>
> >>>
> >>> Ok, I think I see at least part of the problem: in GRASS 7.0
> >>> i.segment numbers segments non sequentially, i.e. whenever a
> >>> new segment is created out of the merger of two existing
> >>> segments, a new id is assigned. In GRASS trunk (the development
> >>> version) i.segment is rewritten to, at the end, number all
> >>> segments sequentially from 1 to the number of segments.
> >>>
> >>> r.object.geometry allocates memory according to the new
> >>> system. As it says in the code:
> >>>
> >>> /* REMARK: The following is only true if object ids are
> >>> continuously numbered */
> >>> n_objects = max - min + 1;
> >>> obj_geos = G_malloc(n_objects * sizeof(struct obj_geo));
> >>>
> >>> So, one solution would be to use the development version of
> >>> GRASS.
> >>>
> >>>
> >>>
> >>> Another would be to run r.clump on the result of the
> >>> segmentation in order to renumber the segments before running it
> >>> through i.segment.stats.
> >>>
> >>>
> >>> This option seemed the most straightforward given my current
> >>> setup. I ran r.clump with diagnoal enabled, and the output looked
> >>> good. It only took 2mins to run.
> >>>
> >>> I then proceeded with the original i.segment.stats command and
> >>> got the following:
> >>>
> >>> Calculating geometry statistics
> >>> Calculating statistics for raster gp_ortho.1 at gp1
> >>> ERROR: G_realloc: unable to allocate 572000 bytes of memory at
> >>> raster/r.univar/r.univar_main.c:324
> >>>
> >>> It looks like the tool got further, but is still getting stuck on
> >>> something...
> >>>
> >>
> >> Yes, now there a memory issue in the part where it uses r.univar to
> >> calculate statistics per segment on one of the spectral bands.
> >>
> >
> > Could you run
> >
> > r.univar -et gp_ortho.1 zones=ClumpedSegmentMap
> > output=testoutput.csv
> >
> > to if r.univar by itself handles this (which would mean I could try
> > to change the data handling in i.segment.stats to make it more
> > memory efficient), or not ?
> >
> > Thanks !
> >
> > Moritz
> >
>
> r.info on the clumped map gives:
>
> +----------------------------------------------------------------------------+
> | Map: gp_seg_optimum_clump Date: Thu Oct 27 14:28:30
> 2016 |
> | Mapset: gp1 Login of Creator:
> jpd205 |
> | Location:
> GarronPill |
> | DataBase:
> /home/jpd205/Wales_GRASS |
> | Title: gp_seg_optimum_clump ( gp_seg_optimum_clump
> ) |
> | Timestamp:
> none |
> |----------------------------------------------------------------------------|
> |
> |
> | Type of Map: raster Number of Categories:
> 0 |
> | Data Type:
> CELL |
> | Rows:
> 12627 |
> | Columns:
> 23991 |
> | Total Cells:
> 302934357 |
> | Projection: OSGB 1936 / British National
> Grid |
> | N: 208007.00931776 S: 207952.59780698 Res:
> 0.00430914 |
> | E: 201097.28911076 W: 200993.90853302 Res:
> 0.00430914 |
> | Range of data: min = 1 max =
> 1264957 |
> |
> |
> | Data
> Source: |
> | gp_seg_optimum at gp1
> |
> |
> |
> |
> |
> | Data
> Description: |
> | generated by
> r.clump |
> |
> |
> |
> Comments:
> | | r.clump --overwrite --verbose -d input="gp_seg_optimum at gp1"
> output="\ |
> | gp_seg_optimum_clump"
> title="gp_seg_optimum_clump" |
> |
> |
> +----------------------------------------------------------------------------+
>
> Running:
>
> r.univar -et gp_ortho.1 zones=gp_seg_optimum_clump
> output=testoutput2.csv
>
> Returns:
>
> Current region rows: 12627, cols: 23991
> ERROR: G_realloc: unable to allocate 224000 bytes of memory at
> raster/r.univar/r.univar_main.c:324
Ok, so the issue is with r.univar (although the issue has made me
aware that i.segment.stats could be made way more memory efficient
as well).
How much RAM do you have on your machine ?
This warrants a bug report. Would you be willing to file one on
http://trac.osgeo.org/grass ?
Moritz
More information about the grass-user
mailing list