[postgis-devel] Bugs in gserialized_gist_compress_2d

[Alexander Korotkov]

Hi!

While investigating NaN handling in GiST indexes I found two buggy checks
gserialized_gist_compress_2d:

/* Is the bounding box valid (non-empty, non-infinite)? If not, return
input uncompressed. */
 if ( result == LW_FAILURE )
{
 POSTGIS_DEBUG(4, "[GIST] empty geometry!");
PG_RETURN_POINTER(entry_in);
 }

/* Check all the dimensions for finite values */
 if ( ! finite(bbox_out.xmax) || ! finite(bbox_out.xmin) ||
     ! finite(bbox_out.ymax) || ! finite(bbox_out.ymin) )
 {
POSTGIS_DEBUG(4, "[GIST] infinite geometry!");
 PG_RETURN_POINTER(entry_in);
}

On failure these checks return original GISTENTRY with geometry key. I
don't understand what is it supposed to do, but what it's actually do is to
interpret first 16 bytes of geometry as box3df. For sure, it leads to
broken index key and invalid answers from index scan.

------
With best regards,
Alexander Korotkov.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/postgis-devel/attachments/20130908/343e76cc/attachment.html>