[GRASS-dev] [GRASS GIS] #2045: r.to.vect: use less memory

[GRASS GIS]

#2045: r.to.vect: use less memory
------------------------------+---------------------------------------------
 Reporter:  mlennert          |       Owner:  grass-dev@…              
     Type:  enhancement       |      Status:  new                      
 Priority:  normal            |   Milestone:  7.0.0                    
Component:  Raster            |     Version:  svn-trunk                
 Keywords:  r.to.vect memory  |    Platform:  Unspecified              
      Cpu:  Unspecified       |  
------------------------------+---------------------------------------------

Comment(by mlennert):

 Replying to [comment:4 mmetz]:
 > Replying to [comment:3 mlennert]:
 > > Replying to [comment:2 mlennert]:
 > > > Replying to [comment:1 mmetz]:
 > > > > Replying to [ticket:2045 mlennert]:
 > > > > > Trying to convert a raster file to vector areas on a machine
 with 10GB, the process was killed after having used up all memory and
 swap.
 > > > >
 > > > > There was a memory leak in r.to.vect, fixed in trunk r57281.
 > > >
 > > > (and r57281)
 > > >
 > > > Thanks ! I now get through the "Extracting areas..." part without
 the process being killed. I still see continuous increase in memory usage,
 though, up to 88,5% at the end of the extraction stage. Is this normal ?
 > > >
 > > > Now it's busy writing areas...
 > > >
 > >
 > > And memory usage continued to increase. The process was killed during
 the "Registering primitives..." step.
 >
 > You can try to set the environment variable GRASS_VECTOR_LOWMEM before
 running r.to.vect. GRASS_VECTOR_LOWMEM reduces the amount of memory used
 by vector topology structures (the spatial index is built on disk).
 >
 > IOW, r.to.vect might use quite a bit of memory which is difficult to
 change, and the vector topology structures also need memory (in g7 much
 less than in g6).

 So you think the continuous increase up to 88.5% during the area
 extraction is normal ?
 I'll try the GRASS_VECTOR_LOWMEM option on Monday.

 Just brainstorming:

 I'm not sure I completely understand the module's workings, but IIUC, it
 systematically goes through rows an columns and checks whether pixel
 values change. But does that mean it reads all areas into memory before
 writing them ? Would it be feasible / of interest to write each area
 immediately after its boundaries have been identified and then free the
 memory again ?

 I could even imagine a solution where the module works with segments of
 the map, writes out the areas identified in a segment, then goes on to the
 next segment and then at the end use a v.dissolve equivalent to fusion all
 neighboring areas with identical values. Unless the v.dissolve process
 takes a lot of memory, I could imagine that at least the first part could
 work with a low memory consumption, or ?

-- 
Ticket URL: <https://trac.osgeo.org/grass/ticket/2045#comment:5>
GRASS GIS <http://grass.osgeo.org>