[gdal-dev] Issue with gdalchksum on Fedora 21 ppc64le?

[Even Rouault]

Le jeudi 27 août 2015 06:20:41, Gawade P a écrit :
> Good Morning, Even.
> I debugged this more, focusing, as you suggested, on the  computation
> of the colormap and looks like the difference in the value of the
> checksum is due to the difference in calculation of the color map
> values for negative slopes. The following lines of code in
> frmts/northwood/northwood.cpp, createIP() function are exhibiting
> different behavior on intel and ppc:
>      for( i = wm + 1; i < index; i++)
>     {
>         map[i].r = map[wm].r + (unsigned char)(((i - wm) * rslope) + 0.5);
>         map[i].g = map[wm].g + (unsigned char)(((i - wm) * gslope) + 0.5);
>         map[i].b = map[wm].b + (unsigned char)(((i - wm) * bslope) + 0.5);
>     }
> primarily, because of the (unsigned char) casting of the second
> operand. Looks like the behavior could be undefined if the float value
> resulting from the computation of the expression (((i - wm) * bslope)
> + 0.5) is negative, as would be the case for negative slope.
> For ppc, the expression  (unsigned char)(((i - wm) * bslope) + 0.5) is
> evaluating to zero, whenever the result of (((i - wm) * bslope) + 0.5)
> is -ve i.e. for -ve slope

Interesting. It is likely that the result of the cast (unsigned 
char)negative_floating_point_value is undefined by the C standard and there 
might be differences in behaviour according to the CPU here (seems to be 
confirmed by http://stackoverflow.com/questions/2490600/iphone-floats-cast-to-
unsigned-ints-get-set-to-0-if-they-are-negative)

I think the following should be portable:

map[i].r = (unsigned char)(map[wm].r + (i - wm) * rslope + 0.5);
map[i].g = (unsigned char)(map[wm].g + (i - wm) * gslope + 0.5);
map[i].b = (unsigned char)(map[wm].b + (i - wm) * bslope + 0.5);

Could you confirm ?

W might want to explictly clamp to [0,255] before casting, but according to 
the formulas of the slope, I'm confident the result should already be in 
[0,255.X] range.

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com