[geos-devel] [GEOS] #869: CommonBits::zeroLowerBits undefined behavior

[GEOS]

#869: CommonBits::zeroLowerBits undefined behavior
------------------------+-------------------------------
 Reporter:  goatbar     |      Owner:  geos-devel@…
     Type:  defect      |     Status:  new
 Priority:  minor       |  Milestone:  3.6.3
Component:  Default     |    Version:  3.6.2
 Severity:  Unassigned  |   Keywords:  UndefinedBehavior
------------------------+-------------------------------
 It is possible to get nBits to be a shift larger than `1 << 63`

 I don't have a cleaned up test case for this issue.  I will try to add one
 to this issue soon.

 This is probably not the right fix...
 {{{#!C++
 /*static public*/
 int64
 CommonBits::zeroLowerBits(int64 bits, int nBits)
 {
         if (nBits >= sizeof(nBits) * 8) return 0;
         int64 invMask = (1<< nBits)-1;
         int64 mask = ~ invMask;
         int64 zeroed = bits & mask;
         return zeroed;
 }
 }}}

 Found with autofuzz.

-- 
Ticket URL: <https://trac.osgeo.org/geos/ticket/869>
GEOS <http://trac.osgeo.org/geos>
GEOS (Geometry Engine - Open Source) is a C++ port of the Java Topology Suite (JTS).