[GRASS-user] r.mapcalc comparisons between floating points and integers

[Juan Miguel Garijo]

----- Mensaje original ----
De: Ricky Robertson <r2icky at gmail.com>
Para: grass-user at lists.osgeo.org
Enviado: viernes, 18 de enero, 2008 22:04:19
Asunto: [GRASS-user] r.mapcalc comparisons between floating points and integers

stupid question alert....

has anyone run across this kind of behavior before in r.mapcalc?

i have these maps with absurdly large values and i am trying to pull
 out
the pixels whose values are below some threshold. the original map
has floating point values. i was/am using integer values for the
 theshold.
for small thresholds (e.g., less than 2,000,000,000, i.e., 2 billion)
everything seemed hunky-dory. however, when i moved up to 3 billion, it
wouldn't pull any out. for some thresholds above that, it would
 generate
strange results, others would just do the no-pixels thing.

i have written a little script to show and reproduce in excruciating
detail what the strange behavior is. here is the basic summary. when
i do a "less than" comparison with the large integer, i get no pixels
even though my threshold is well below the maximum value. when i do
a "greater than" comparison (which should give me the complement),
i get some pixels. actually, i get all of the non-null pixels
 regardless of
whether they are above or below the threshold.

when i do the "less than" using 3 billion point zero (3000000000.0)
instead of 3 billion (3000000000), it pulls out some pixels. when i do
the "greater than" using the float, it pulls out some other pixels.
using the floating point versions seems to add up in that the number of
greater-than pixels plus the number of less-than pixels adds up to the
total number of non-null pixels.

i'm guessing this has to do with some sort of overflow when i try to
 use
an integer larger than the memory representation is intended to
 handle...

one can observe the script that demonstrates this strange behavior at

https://netfiles.uiuc.edu/rdrobert/www/bug_demo.sh

the output can be seen at

https://netfiles.uiuc.edu/rdrobert/www/bug_output.txt

and if you are really a masochist, i could try to send copies of the
data by request, but i doubt anyone is that bored...

peace,

ricky robertson



Maximum value for a variable of type int.

   
2147483647 


Juan Miguel Garijo





      ______________________________________________ 
Web Revelación Yahoo! 2007:
Premio Favorita del Público
http://es.promotions.yahoo.com/revelacion2007/favoritos/