[postgis-devel] [wktraster] Core tests failure for r5841

[Pierre Racine]

So would everybody be happy if we just clamp the provided double value to the desired pixel type MAX or MIN when there is an overflow (e.g. for 16BSI -20000- is first clamped to -32768)?

So the code would go from:

            case PT_16BSI:
            {
                int16_t *ptr = mem;

                for (i = 0; i < numval; i++)
                    ptr[i] = (int16_t) initialvalue;
                checkvalint = ptr[0];
                break;
            }

to

            case PT_16BSI:
            {
                int16_t *ptr = mem;

                if (initialvalue < SHRT_MIN)
                    initialvalue = SHRT_MIN;
                if (initialvalue > SHRT_MAX)
                    initialvalue = SHRT_MAX;

                for (i = 0; i < numval; i++)
                    ptr[i] = (int16_t) initialvalue;
                checkvalint = ptr[0];
                break;
            }

for all int types. Is this "portable, robust and numerically stable". If not, why? Because those constants are different from one architecture to the other?

If the value do not overflow, do we also have to explicitly truncate double before converting to int? If we don't, should we still expect undefined results?

            case PT_16BSI:
            {
                int16_t *ptr = mem;

                initialvalue = trunc(initialvalue);

                if (initialvalue < SHRT_MIN)
                    initialvalue = SHRT_MIN;
                if (initialvalue > SHRT_MAX)
                    initialvalue = SHRT_MAX;

                for (i = 0; i < numval; i++)
                    ptr[i] = (int16_t) initialvalue;
                checkvalint = ptr[0];
                break;
            }

Pierre

>-----Original Message-----
>From: strk [mailto:strk at keybit.net]
>Sent: 13 janvier 2011 08:48
>To: PostGIS Development Discussion
>Cc: Pierre Racine
>Subject: Re: [postgis-devel] [wktraster] Core tests failure for r5841
>
>On Thu, Jan 13, 2011 at 01:33:02PM +0000, Mateusz Loskot wrote:
>
>> Once the truncation rules are defined for integral types, it should be
>> not a problem to implement them. For instance, what choices we have to
>> cope with -32769 value for 16BSI?
>> - cast to int16_t and document result as undefined
>> - truncate towards Zero by storing SHRT_MIN (SHRT_MAX for positive)
>> - wrap around
>> - store NODATA value
>> - ...
>
>To keep things simple I'd always clamp any number < -32768 to -32768
>and any number > 32767 to 32767.
>
>Makes sense from a sampling point of view too after all..
>
>--strk;
>
>  ()   Free GIS & Flash consultant/developer
>  /\   http://strk.keybit.net/services.html