[postgis-devel] Bugs in gserialized_gist_compress_2d

[Paul Ramsey]

Thoughts on what we *should* do with invalid inputs? Have invalid keys? Not sure what the implications of a nan key, or inf key are. 
P.

-- 
Paul Ramsey
http://cleverelephant.ca/
http://postgis.net/


On Sunday, 8 September, 2013 at 3:23 AM, Alexander Korotkov wrote:

> Hi!
> 
> While investigating NaN handling in GiST indexes I found two buggy checks gserialized_gist_compress_2d:
> 
> /* Is the bounding box valid (non-empty, non-infinite)? If not, return input uncompressed. */ 
> if ( result == LW_FAILURE )
> {
> POSTGIS_DEBUG(4, "[GIST] empty geometry!");
> PG_RETURN_POINTER(entry_in);
> }
> 
> 
> /* Check all the dimensions for finite values */ 
> if ( ! finite(bbox_out.xmax) || ! finite(bbox_out.xmin) ||
> ! finite(bbox_out.ymax) || ! finite(bbox_out.ymin) )
> {
> POSTGIS_DEBUG(4, "[GIST] infinite geometry!");
> PG_RETURN_POINTER(entry_in);
> }
> 
> 
> On failure these checks return original GISTENTRY with geometry key. I don't understand what is it supposed to do, but what it's actually do is to interpret first 16 bytes of geometry as box3df. For sure, it leads to broken index key and invalid answers from index scan. 
> ------
> With best regards,
> Alexander Korotkov.
> _______________________________________________
> postgis-devel mailing list
> postgis-devel at lists.osgeo.org (mailto:postgis-devel at lists.osgeo.org)
> http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel