[gdal-dev] problem with ArcSDE raster (negative values)

[bartvde at osgis.nl]

Sorry I meant Kor ofcourse :-)

Btw, the GByte *p = reinterpret_cast<GByte*>(pImage); line was a paste
error from my debugging, it's also not in 1.6.1 which I am using.

Best regards,
Bart

> Hi Cor,
>
> in ArcMap I see negative values, in GDAL all values seem to have a shift
> of 256, so -21 becomes 235. Nodata values are also off with 256, so -256
> in ArcMap becomes 0 in GDAL.
>
> Best regards,
> Bart
>
>> Hi Bart,
>>
>> On Tuesday 08 December 2009 10:45:25 bartvde at osgis.nl wrote:
>>> I've debugged the problem to the following part of
>>> frmts/sde/sderasterband.cpp:
>>>
>>>     if (bits_per_pixel >= 8) {
>>>         memcpy(pImage, pixels, block_size);
>>>         GByte *p = reinterpret_cast<GByte*>(pImage);
>>>     } else {
>>>
>>> The values in the pixels variable are still okay, and contain mostly
>>> negative values as well as 0 (for example -42).
>>>
>>> The p variable only contains positive values and 0 values.
>>
>> What differences do you see when reading the data using ArcMap and
>> reading it using gdal?
>> - Is it only the no-data values that get truncated?
>> - Do all negative values get converted to a positive value?
>> - Something else?
>>
>> The piece of code you're showing only deals with copying bits and bytes
>> from one buffer to another. These buffers have types that are convenient
>> for doing that and have no relation with the real type of the values in
>> the buffers. Using these pointers to test the values is not very useful,
>> I think. Why do you think this piece of code must contain a bug?
>>
>> BTW, the current trunk version of this module looks different. Maybe
>> upgrading gdal solves your problem already?
>>
>> Sorry, more questions than answers...
>>
>> Kor
>>
>>
>>
>
>