[gdal-dev] Issue with gdalchksum on Fedora 21 ppc64le?

[Gawade P]

Good Morning, Even.
I debugged this more, focusing, as you suggested, on the  computation
of the colormap and looks like the difference in the value of the
checksum is due to the difference in calculation of the color map
values for negative slopes. The following lines of code in
frmts/northwood/northwood.cpp, createIP() function are exhibiting
different behavior on intel and ppc:
     for( i = wm + 1; i < index; i++)
    {
        map[i].r = map[wm].r + (unsigned char)(((i - wm) * rslope) + 0.5);
        map[i].g = map[wm].g + (unsigned char)(((i - wm) * gslope) + 0.5);
        map[i].b = map[wm].b + (unsigned char)(((i - wm) * bslope) + 0.5);
    }
primarily, because of the (unsigned char) casting of the second
operand. Looks like the behavior could be undefined if the float value
resulting from the computation of the expression (((i - wm) * bslope)
+ 0.5) is negative, as would be the case for negative slope.
For ppc, the expression  (unsigned char)(((i - wm) * bslope) + 0.5) is
evaluating to zero, whenever the result of (((i - wm) * bslope) + 0.5)
is -ve i.e. for -ve slope

I printed the values of the map in this loop and this is the output on
ppc when bslope is -ve.:
createIP: index = 1760,  wm = 1369
createIP: rslope = 0.652174, gslope = 0.000000, bslope = -0.652174
createIP: map[i].r = 1,  map[i].g = 255,  map[i].b = 255
createIP: map[i].r = 1,  map[i].g = 255,  map[i].b = 255
createIP: map[i].r = 2,  map[i].g = 255,  map[i].b = 255
createIP: map[i].r = 3,  map[i].g = 255,  map[i].b = 255
createIP: map[i].r = 3,  map[i].g = 255,  map[i].b = 255
createIP: map[i].r = 4,  map[i].g = 255,  map[i].b = 255
createIP: map[i].r = 5,  map[i].g = 255,  map[i].b = 255
createIP: map[i].r = 5,  map[i].g = 255,  map[i].b = 255
createIP: map[i].r = 6,  map[i].g = 255,  map[i].b = 255
createIP: map[i].r = 7,  map[i].g = 255,  map[i].b = 255

map[I].b remains constant.

The corresponding output on intel x86_64 is:
createIP: rslope = 0.652174, gslope = 0.000000, bslope = -0.652174
createIP: map[i].r = 1,  map[i].g = 255,  map[i].b = 255
createIP: map[i].r = 1,  map[i].g = 255,  map[i].b = 255
createIP: map[i].r = 2,  map[i].g = 255,  map[i].b = 254
createIP: map[i].r = 3,  map[i].g = 255,  map[i].b = 253
createIP: map[i].r = 3,  map[i].g = 255,  map[i].b = 253
createIP: map[i].r = 4,  map[i].g = 255,  map[i].b = 252
createIP: map[i].r = 5,  map[i].g = 255,  map[i].b = 251
createIP: map[i].r = 5,  map[i].g = 255,  map[i].b = 251
createIP: map[i].r = 6,  map[i].g = 255,  map[i].b = 250
createIP: map[i].r = 7,  map[i].g = 255,  map[i].b = 249

map[I].b shows a constant downward trend, as expected.

To confirm the analysis, I changed the type-casting from (unsigned
char) to (signed char) as follows on ppc and got the expected result:
        map[i].r = map[wm].r + (signed char)(((i - wm) * rslope) + 0.5);
        map[i].g = map[wm].g + (signed char)(((i - wm) * gslope) + 0.5);
        map[i].b = map[wm].b + (signed char)(((i - wm) * bslope) + 0.5);

bash-4.2# gdalchksum.py nwt_grd.grd
28093
33626
20260
25856

I am trying to work on a fix that could correct the behavior on ppc
without impacting the behavior on x86.
Please let me know if you have any thoughts on this.
Thanks!

On Tue, Aug 25, 2015 at 6:47 PM, Even Rouault
<even.rouault at spatialys.com> wrote:
>
>>
>> From a quick look it doesn't appear to be the same issue as for HF2. And
>> I've actually recompiled GDAL with -fsigned-char on x86_64 to simulate
>> ppc64le behaviour regarding char,
>
> I was obviously not well awoken this morning, since I should rather have used
> -funsigned-char to make char behave like unsigned char. Anyway after trying
> this switch, the conclusion is the same...
>
> --
> Spatialys - Geospatial professional services
> http://www.spatialys.com