[GRASS-dev] r.quantile strange output at small percentiles intervals

[Glynn Clements]

Paulo van Breugel wrote:

> When running r.quantile with percentiles set at 0-100 at steps of 0.25, all
> runs fine

> When setting at smaller steps, e.g., 0.2, the output becomes strange

> 9.216583:0.000000:407
> 0.000000:0.000000:408
> 0.000000:0.000000:409

> Any ideas why this is?

It uses an 8-bit value to store the bin number for each slot, which
imposes a limit of 255 bins. Exceeding this number causes earlier bins
to be "recycled" while later bins aren't filled (leaving them
containing zeroes).

In r53951, I've enlarged this to 16 bits and added an error if there
are more than 65535 quantiles (the number of bins will never exceed
the number of quantiles).

-- 
Glynn Clements <glynn at gclements.plus.com>