[GRASS-dev] r.quantile strange output at small percentiles intervals

[Paulo van Breugel]

Great, thanks for the quick fix / improvement!



On Wed 21 Nov 2012 04:36:10 PM CET, Glynn Clements wrote:
>
> Paulo van Breugel wrote:
>
>> When running r.quantile with percentiles set at 0-100 at steps of 0.25, all
>> runs fine
>
>> When setting at smaller steps, e.g., 0.2, the output becomes strange
>
>> 9.216583:0.000000:407
>> 0.000000:0.000000:408
>> 0.000000:0.000000:409
>
>> Any ideas why this is?
>
> It uses an 8-bit value to store the bin number for each slot, which
> imposes a limit of 255 bins. Exceeding this number causes earlier bins
> to be "recycled" while later bins aren't filled (leaving them
> containing zeroes).
>
> In r53951, I've enlarged this to 16 bits and added an error if there
> are more than 65535 quantiles (the number of bins will never exceed
> the number of quantiles).
>