[GRASS-user] r.mapcalc comparisons between floating points and integers

[Ricky Robertson]

stupid question alert....

has anyone run across this kind of behavior before in r.mapcalc?

i have these maps with absurdly large values and i am trying to pull out
the pixels whose values are below some threshold. the original map
has floating point values. i was/am using integer values for the theshold.
for small thresholds (e.g., less than 2,000,000,000, i.e., 2 billion)
everything seemed hunky-dory. however, when i moved up to 3 billion, it
wouldn't pull any out. for some thresholds above that, it would generate
strange results, others would just do the no-pixels thing.

i have written a little script to show and reproduce in excruciating
detail what the strange behavior is. here is the basic summary. when
i do a "less than" comparison with the large integer, i get no pixels
even though my threshold is well below the maximum value. when i do
a "greater than" comparison (which should give me the complement),
i get some pixels. actually, i get all of the non-null pixels regardless of
whether they are above or below the threshold.

when i do the "less than" using 3 billion point zero (3000000000.0)
instead of 3 billion (3000000000), it pulls out some pixels. when i do
the "greater than" using the float, it pulls out some other pixels.
using the floating point versions seems to add up in that the number of
greater-than pixels plus the number of less-than pixels adds up to the
total number of non-null pixels.

i'm guessing this has to do with some sort of overflow when i try to use
an integer larger than the memory representation is intended to handle...

one can observe the script that demonstrates this strange behavior at

https://netfiles.uiuc.edu/rdrobert/www/bug_demo.sh

the output can be seen at

https://netfiles.uiuc.edu/rdrobert/www/bug_output.txt

and if you are really a masochist, i could try to send copies of the
data by request, but i doubt anyone is that bored...

peace,

ricky robertson