[gdal-dev] problem with ArcSDE raster (negative values)

[bartvde at osgis.nl]

Hi Cor,

in ArcMap I see negative values, in GDAL all values seem to have a shift
of 256, so -21 becomes 235. Nodata values are also off with 256, so -256
in ArcMap becomes 0 in GDAL.

Best regards,
Bart

> Hi Bart,
>
> On Tuesday 08 December 2009 10:45:25 bartvde at osgis.nl wrote:
>> I've debugged the problem to the following part of
>> frmts/sde/sderasterband.cpp:
>>
>>     if (bits_per_pixel >= 8) {
>>         memcpy(pImage, pixels, block_size);
>>         GByte *p = reinterpret_cast<GByte*>(pImage);
>>     } else {
>>
>> The values in the pixels variable are still okay, and contain mostly
>> negative values as well as 0 (for example -42).
>>
>> The p variable only contains positive values and 0 values.
>
> What differences do you see when reading the data using ArcMap and
> reading it using gdal?
> - Is it only the no-data values that get truncated?
> - Do all negative values get converted to a positive value?
> - Something else?
>
> The piece of code you're showing only deals with copying bits and bytes
> from one buffer to another. These buffers have types that are convenient
> for doing that and have no relation with the real type of the values in
> the buffers. Using these pointers to test the values is not very useful,
> I think. Why do you think this piece of code must contain a bug?
>
> BTW, the current trunk version of this module looks different. Maybe
> upgrading gdal solves your problem already?
>
> Sorry, more questions than answers...
>
> Kor
>
>
>