[GRASS5] Re: r.sunmask - memory leakage in GRASS lib

[Markus Neteler]

Hi Huidae,
hi all, (as this might be interesting for all)

thanks for looking into the memory leakage in
  r.sunmask - 
  src/libes/gis/get_row.c

On Tue, Jun 05, 2001 at 07:39:16PM +0900, GRASS wrote:
> Hi Markus,
> 
> Sorry for my late response.
> 
> From: Markus Neteler <neteler at geog.uni-hannover.de>
> > Hi Huidae, hi Justin,
> >
> > Justin: many thanks for below comments! I try to involve Huidae :-)
> >
> > Huidae: the r.sunmask still contains the memory leakage which appears
> > with bigger DEMs - it leads to a partially system crash. Justin has
> > been investigating, but didn't find the leakage yet. May I ask you
> > for assistance? I am willing to learn how to debug such problems,
> > but I am out of references/tutorials.
> > Perhaps this problem indicates a memory leakage in GRASS GIS library
> > which would also affect other modules.
> >
> > Here what Justin found out:
> > On Tue, May 29, 2001 at 05:32:10PM +0700, Justin Hickey wrote:
> > > Hi Markus
> > > 
> > > The latest update is not too good but I have a little bit of
> > > information. A debugger I have on our SGI can do a memory leak trace,
> > > unfortunately it dumps core trying to run r.sunmask. I don't know why.
> > > The error message doesn't make any sense to me.
> > > 
> > > So I played around a bit and determined the following. The memory is
> > > being allocated every so often inside the 
> > > 
> > > while (col1<window.cols)
> > > 
> > > loop (line 385). The "top" program indicates that it increments in 32K
> > > chunks every 40 - 50 iterations through that loop. The number of
> > > iterations is not accurate due to timing differences between top and the
> > > debugger. The only place I found an allocation was in a call to
> > > G_zlib_read() from src/libes/gis/flate.c The call stack is
> > > 
> > > G_get_raster_row()
> > > get_map_row_nomask()
> > > read_data()
> > > read_data_fp_compressed()
> > > G_zlib_read()
> > > 
> > > However, it appears that this memory is freed properly and thus should
> > > not be the cause of the leak. :-(
> > > 
> > > This doesn't mean that it is the only allocation function, in fact the
> > > evidence indicates that there must be another one somewhere. However,
> > > I'm not sure where it is and you would have to trace through the
> > > possible calls to find it.
> > > 
> > > One thing to keep in mind is that the figures I mentioned above could be
> > > dependent on the data. That is, the chunk size and number of iterations
> > > could be very different for different data.
> > > 
> > > I'm sorry, but I can't spend any more time on this. My suggestion would
> > > be to start with the G_get_raster_row() function since it seems to be
> > > the only thing in that loop which might do some memory allocation. Then
> > > you trace through each function call inside G_get_raster_row() and try
> > > to find memory allocated but not freed. You might want to look into
> > > Linux debugging tools that could help you with this.
> > > 
> > > Anyway, I hope this helps a little bit and good luck with the hunt!
> > > Sorry I couldn't be of much help.
> > > 
> > > -- 
> > > Sincerely,
> > > 
> > > Jazzman (a.k.a. Justin Hickey)  e-mail: jhickey at hpcc.nectec.or.th
> > [...]
> >
> > As I need the module for a project, I have to get it running. Therefore
> > I am bothering you... my apologies!
> >
> > If you run the module with on a 2000x2000 DEM (or more), you can see
> > the memory consumption.
> >
> 
> I've tested r.sunmask.
> 
> The memory leakage appears in
> 
> 	col = data_buf;
> 	
> 	pair = readamount / (n+1);
> 	while (pair-- > 0) {
> 	  repeat = *cmp++;
> 	  while(repeat--)
> 	    for (k = 0; k < n; k++)
> 	      *col++ = cmp[k];	/**********/
> 	  cmp += n;
> 	}
> 
> in read_data_compressed()
> in src/libes/gis/get_row.c
> 
> I've experienced the following case with 2000*2000 map:
> 
> 
> 	COMPRESSED = 1
> 	readamount = 18		// size of cmp when reading
> 	n = 1
> 	DATA_NCOLS = 2000
> 
> so
> 
> 	if (COMPRESSED < 0 || readamount < n * DATA_NCOLS)
> 
> is true,
> 
> 	
> 	cmp structure is..
> 
> 	pair1:	REPEAT | repeat1 | repeat2 | ... | repeatREPEAT
> 	pair2:	REPEAT | repeat1 | repeat2 | ... | repeatREPEAT
> 
> 		...
> 
> 	pairP:	REPEAT | repeat1 | repeat2 | ... | repeatREPEAT
> 
> 	repeatn structure is..
> 	
> 		cmp1 | cmp2 | ... | cmpn	one byte per each section
> 
> ...
> 
> Anyway, I met a memory leakage only if
> 
> 	*col++ = cmp[k];
> 
> is active.
> 
> I think a compressed data structure is somewhat different from an
> uncompressing algorithm. I'm not sure :-).
> 
> I don't have much idea on a compression algorithm.
> 
> I'll continue to investigate.
> 
> Yours,
> Huidae Cho

Perhaps Eric Miller can suggest as he worked on the compression?
It seems we are close the the problem now.

Thanks for investigating, Huidae,

 Markus