[postgis-devel] ST_Contains(POLYGON, POINT) memory leak

Thu Sep 25 11:22:46 PDT 2008

The p-i-p shortcut is in contains, intersects, within, covers,
coveredby, I believe. Once (if?) I've got the backend code memory
clean, I'll go through and see what I can do to clean up the calling
code and perhaps turn on MULTI.

P.

On Thu, Sep 25, 2008 at 11:05 AM, Obe, Regina <robe.dnd at cityofboston.gov> wrote:
> Paul,
>
> Since you are sort of in that neck of the woods anyway - you feel brave
> enough to
> try to get the MULTIPOLY contains remarked out short-cut case to work in
> contains() function in lwgeom_geos_c.c.
>
> I think I had tried uncommenting the code and compiling and getting a
> compile
> error, but admittedly I am compile challenged so could be just something
> stupid I am doing.
>
> The short-cut I don't think exists in the other functions, but if you can
> get it to work in one, don't see why you wouldn't be able to in others.
>
> Thanks,
> Regina
>
> -----Original Message-----
> From: postgis-devel-bounces at postgis.refractions.net on behalf of Paul Ramsey
> Sent: Thu 9/25/2008 1:43 PM
> To: Kevin Neufeld; Chris Hodgson
> Cc: PostGIS Development Discussion
> Subject: [postgis-devel] ST_Contains(POLYGON, POINT) memory leak
>
> Kevin,
>
> I've committed two changes to SVN now that make this case leak a *lot*
> less. You'll see no particular performance gain on your test workload,
> since your workload is pathological with respect to caching indexed
> objects (you have only one test point per polygon, so the time spend
> indexing is wasted), but the memory use is much better.
>
> It's still not optimal however... there's memory coming out, so I
> might have to try the ugly hack to make the memory allocation visible
> to valgrind.
>
> FYI, in terms of the speed of the shortcut code, on a non-pathological
> case (about 100 points per test polygon), the shortcut version returns
> in 3s, the standard code in 100s.
>
> Paul
>
> On Tue, Sep 23, 2008 at 10:21 AM, Kevin Neufeld
> <kneufeld at refractions.net> wrote:
>> I'm not sure.  Is the PIP common to those functions?  Unfortunately, I
>> don't
>> have time to work on this now, but Martin said I could send you the
>> polygons
>> I'm using  (they're the really large wetlands in the peace region for CWB)
>> so you can see if you can reproduce the problem ... it's possible it's
>> just
>> a bogus install I have here.
>>
>> ftp://ftp.refractions.net/pub/refractions/ftp_postgis/test_pip.Fc.dmp
>> use pg_restore to extract
>> /opt/pgsql83/bin/pg_restore -Fc -O test_pip.Fc.dmp | psql postgis
>>
>> I put both of my test tables in a tmp schema for this dmp file.
>>
>> Cheers,
>> -- Kevin
>>
>> Paul Ramsey wrote:
>>>
>>> Is it limited to P-i-P?
>>>
>>> P
>>>
>>> On Tue, Sep 23, 2008 at 8:56 AM, Kevin Neufeld <kneufeld at refractions.net>
>>> wrote:
>>>>
>>>> By the way, this is not limited to ST_Contains.
>>>>
>>>> ST_Within         : 1.7GB - 16sec
>>>> ST_Intersects     : 1.7GB - 16sec
>>>> ST_CoveredBy      : 1.7GB - 16sec
>>>> ST_Distance = 0   : 26MB  - 5sec
>>>> ST_Relate         : 32MB  - 1:30sec
>>>>
>>>> -- Kevin
>>>>
>>>> Kevin Neufeld wrote:
>>>>>
>>>>> I'm still able to reproduce the memory leak.  It looks like you're
>>>>> right
>>>>> though, it's nothing to do with GEOS, but rather the PIP in PostGIS.
>>>>> I'm not sure what is going on yet, (I have yet to find a specific test
>>>>> case) but here are my findings.
>>>>>
>>>>> I made two little sh scripts to track memory usage (psql82.sh and
>>>>> psql83.sh), each producing a log file.  I happen to run the same GEOS
>>>>> version in both instances, but running pgis 1.1.6 on 8.2 and 1.3.3 on
>>>>> 8.3
>>>>>
>>>>> I have a sample table of ~120000 polygons and for testing, generated a
>>>>> second table of points using pointonsurface.  Running the same query on
>>>>> both
>>>>> instances (using contains() on a table join of the two tables using 150
>>>>> points:
>>>>> - pgis 1.1.6 finishes in 1:30secs using 30MB of memory. - pgis 1.3.3
>>>>> finishes in 12sec using 1.7GB of memory.
>>>>>
>>>>> If I query an extra 50 points, the 1.3.3 instance runs out of memory
>>>>> with
>>>>> the attached crash log.
>>>>>
>>>>> Next, I'll try pgis SVN head on the 8.3 instance instead of 1.3.3  (but
>>>>> not tonite :) ).
>>>>>
>>>>> Thoughts?
>>>>> -- Kevin
>>>>>
>>>>>
>>>>>
>>>>> Paul Ramsey wrote:
>>>>>>
>>>>>> OK, I ran Mark's P-i-P short circuit through valgrind and it doesn't
>>>>>> leak anything at all. It also is sort of hard to get to run, since
>>>>>> it's only strict POLYGON and POINT, MULTIPOLYGON need not apply.
>>>>>> P-i-mP didn't leak either, so I'm not sure what to make of Kevin's
>>>>>> result at this point.
>>>>>>
>>>>>> Only difference right now is I'm running postgis-trunk and geos-trunk
>>>>>> for my testing.
>>>>>>
>>>>>> Paul
>>>>>>
>>>>>> On Mon, Sep 22, 2008 at 12:53 PM, Paul Ramsey
>>>>>> <pramsey at cleverelephant.ca>
>>>>>> wrote:
>>>>>>
>>>>>>> Yes, it points that the problem might not be in GEOS after all, it
>>>>>>> might be in how the memory-context handling is happening.
>>>>>>>
>>>>>>> In PostGIS 1.3.3, the Contains and Intersects tests have a P-i-P
>>>>>>> short-circuit that uses a cached 1-d r-tree to do a version of
>>>>>>> prepared geometry testing (Mark Leslie implemented this). Ben then
>>>>>>> copied that code for his memory caching work that used the GEOS
>>>>>>> prepared geometry.
>>>>>>>
>>>>>>> Try going into the lwgeom_geos_c.c file and commenting out the
>>>>>>> mem-caching code path for one of your tests and see if the memory
>>>>>>> problem goes away,
>>>>>>>
>>>>>>> *or*
>>>>>>>
>>>>>>> Re-run your test, replacing the points with short two-point lines.
>>>>>>>
>>>>>>> Since Mark's code was in turn copied from the proj4 projection object
>>>>>>> caching routine (lwgeom_transform.c) that code might *also* leak,
>>>>>>> although since the number of cache transitions is low, it might not
>>>>>>> be
>>>>>>> so noticeable.
>>>>>>>
>>>>>>> Paul
>>>>>>>
> _______________________________________________
> postgis-devel mailing list
> postgis-devel at postgis.refractions.net
> http://postgis.refractions.net/mailman/listinfo/postgis-devel
>
> ________________________________
>
> The substance of this message, including any attachments, may be
> confidential, legally privileged and/or exempt from disclosure pursuant to
> Massachusetts law. It is intended solely for the addressee. If you received
> this in error, please contact the sender and delete the material from any
> computer.
>
> ________________________________
>
> Help make the earth a greener place. If at all possible resist printing this
> email and join us in saving paper.
>
> _______________________________________________
> postgis-devel mailing list
> postgis-devel at postgis.refractions.net
> http://postgis.refractions.net/mailman/listinfo/postgis-devel
>
>