[postgis-tickets] r15953 - Undefined behaviour in pointarray_to_encoded_polyline (2.2 branch)

[Paul Ramsey]

Author: pramsey
Date: 2017-10-10 11:04:34 -0700 (Tue, 10 Oct 2017)
New Revision: 15953

Modified:
   branches/2.2/NEWS
   branches/2.2/liblwgeom/lwout_encoded_polyline.c
Log:
Undefined behaviour in pointarray_to_encoded_polyline (2.2 branch)
(References #3891)



Modified: branches/2.2/NEWS
===================================================================
--- branches/2.2/NEWS	2017-10-10 17:06:08 UTC (rev 15952)
+++ branches/2.2/NEWS	2017-10-10 18:04:34 UTC (rev 15953)
@@ -15,6 +15,7 @@
   - #3875, Fix undefined behaviour in shift operation
   - #3874, lw_dist2d_pt_arc division by zero
   - #3882, undefined behaviour in zigzag with negative inputs
+  - #3891, undefined behaviour in pointarray_to_encoded_polyline
 
 
 PostGIS 2.2.5

Modified: branches/2.2/liblwgeom/lwout_encoded_polyline.c
===================================================================
--- branches/2.2/liblwgeom/lwout_encoded_polyline.c	2017-10-10 17:06:08 UTC (rev 15952)
+++ branches/2.2/liblwgeom/lwout_encoded_polyline.c	2017-10-10 18:04:34 UTC (rev 15953)
@@ -61,7 +61,7 @@
 	stringbuffer_t *sb;
 	double scale = pow(10,precision);
 
-	/* Take the double value and multiply it by 1x10^percision, rounding the result */
+	/* Take the double value and multiply it by 1x10^precision, rounding the result */
 	prevPoint = getPoint2d_cp(pa, 0);
 	delta[0] = round(prevPoint->y*scale);
 	delta[1] = round(prevPoint->x*scale);
@@ -78,8 +78,8 @@
 	/* value to binary: a negative value must be calculated using its two's complement */
 	for (i=0; i<pa->npoints*2; i++)
 	{
-		/* Left-shift the binary value one bit */
-		delta[i] <<= 1;
+		/* Multiply by 2 for a signed left shift */
+		delta[i] *= 2;
 		/* if value is negative, invert this encoding */
 		if (delta[i] < 0) {
 			delta[i] = ~(delta[i]);