[GRASS-user] r.mapcalc comparisons between floating points and integers

[Glynn Clements]

Ricky Robertson wrote:

> stupid question alert....
> 
> has anyone run across this kind of behavior before in r.mapcalc?
> 
> i have these maps with absurdly large values and i am trying to pull out
> the pixels whose values are below some threshold. the original map
> has floating point values. i was/am using integer values for the theshold.
> for small thresholds (e.g., less than 2,000,000,000, i.e., 2 billion)
> everything seemed hunky-dory. however, when i moved up to 3 billion, it
> wouldn't pull any out. for some thresholds above that, it would generate
> strange results, others would just do the no-pixels thing.

The maximum value of a signed 32-bit integer is 2^31-1 =
2,147,483,647. If you try to convert any FP value larger than that to
an integer, the result is undefined (and likely platform-dependent; on
Linux/x86, I get nulls).

> i have written a little script to show and reproduce in excruciating
> detail what the strange behavior is. here is the basic summary. when
> i do a "less than" comparison with the large integer, i get no pixels
> even though my threshold is well below the maximum value. when i do
> a "greater than" comparison (which should give me the complement),
> i get some pixels. actually, i get all of the non-null pixels regardless of
> whether they are above or below the threshold.
> 
> when i do the "less than" using 3 billion point zero (3000000000.0)
> instead of 3 billion (3000000000), it pulls out some pixels. when i do
> the "greater than" using the float, it pulls out some other pixels.
> using the floating point versions seems to add up in that the number of
> greater-than pixels plus the number of less-than pixels adds up to the
> total number of non-null pixels.

Integer literals are parsed using the system's atoi() function. For
integers which exceed the 32-bit range, the result may be
platform-dependent. On Linux/x86, oversize values are clamped to
+/-2147483647.

-- 
Glynn Clements <glynn at gclements.plus.com>