[GRASS-dev] Re: [GRASS GIS] #475: r.stats: last bin always has a single cell

[GRASS GIS]

#475: r.stats: last bin always has a single cell
-----------------------+----------------------------------------------------
  Reporter:  hamish    |       Owner:  grass-dev at lists.osgeo.org
      Type:  defect    |      Status:  new                      
  Priority:  critical  |   Milestone:  6.4.0                    
 Component:  Raster    |     Version:  svn-develbranch6         
Resolution:            |    Keywords:  r.stats                  
  Platform:  All       |         Cpu:  All                      
-----------------------+----------------------------------------------------
Comment (by glynn):

 Replying to [comment:3 hamish]:
 > request for help solving this critical bug: we are reporting bad
 statistical data from our core stats module. It is used by r.report,
 d.histogram, etc and on to users' work output.
 >
 > I'm horrified by this bug but unfamiliar with hash tables and don't know
 how to proceed with debugging.

 I think that you're looking in the wrong place.

 This appears to be a problem with the quantisation semantics.

 In your test case, r.stats calls
 {{{
 G_quant_add_rule(&q, 1061.064087, 1846.743408, 1, 10)
 }}}

 It then assigns the quantisation rules to the input map.

 Cell values are obtained using G_get_c_raster_row(), which converts the
 map's FP values to integer CELL values via G_quant_get_cell_value(). Given
 the above quantisation rule, it converts FP values as follows:
 {{{
 > print G_quant_get_cell_value(&q, 1846.743408)
 $15 = 10
 > print G_quant_get_cell_value(&q, 1846.743407)
 $17 = 9
 > print G_quant_get_cell_value(&q, 1846.743409)
 $18 = -2147483648
 }}}

 Essentially, G_quant_add_rule() treats the ranges as half-open, i.e. the
 values range from low (inclusive) to high (exclusive). While half-open
 ranges are a common concept (e.g. floor() behaves the same way), the range
 of a GRASS raster is closed, i.e. both the low and high values are
 inclusive.

 I suggest changing r.stats to use cHigh=nsteps+1, and truncate the
 quantised values into range (fudging dHigh may seem simpler, but the
 maximum value may be positive, negative or zero, and the range may be
 1e+300 or 1e-300, so choosing a suitable fudge factor isn't
 straightforward).

-- 
Ticket URL: <http://trac.osgeo.org/grass/ticket/475#comment:4>
GRASS GIS <http://grass.osgeo.org>