[geos-devel] Assertion Failure with VC++ and STRtree:yComparator

[Charlie Savage]

There appears to be a fairly serious issue with the trunk version of 
GEOS with STRtree:yComparator(Boundable *a, Boundable *b).

There are various comments from mloskot and strk about how the 
comparison should work.  Choices seem to be:

	return STRtree::centreY(aEnv) < STRtree::centreY(bEnv);

Or

return std::fabs( STRtree::centreY(aEnv) - STRtree::centreY(bEnv) ) < 1e-30


Currently, the second option is used.  But on Windows, using a debug 
VC++ build, that is almost guaranteed to cause an assertion failure. 
VC++ has code in xutility that checks the comparison in both directions:

template<class _Pr, class _Ty1, class _Ty2> inline
	bool __CLRCALL_OR_CDECL _Debug_lt_pred(_Pr _Pred, _Ty1& _Left, _Ty2& 
_Right,
		const wchar_t *_Where, unsigned int _Line)
	{	// test if _Pred(_Left, _Right) and _Pred is strict weak ordering
	if (!_Pred(_Left, _Right))
		return (false);
	else if (_Pred(_Right, _Left))
		_DEBUG_ERROR2("invalid operator<", _Where, _Line);
	return (true);
	}

Assume the centers of the two envelopes are equivalent - that will 
trigger the 2nd part of the if statement causing an assertion failure.

Can this be fixed so that the comparison works as expected?  The first 
version does the trick, but strk wrote some scary comments that it 
corrupts memory in some cases (although with a quick glance I don't see 
how).

Thanks,

Charlie

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3237 bytes
Desc: S/MIME Cryptographic Signature
Url : http://lists.osgeo.org/pipermail/geos-devel/attachments/20070828/4947ddd8/smime.bin