[postgis-users] ERROR: array size exceeds themaximumallowed(134217727)

[Paul Ramsey]

This looks like a winner, at least insofar as it removes the max array
error condition for me in OS/X. Huge thanks to Greg Stark for digging
up the references to Tom Lane's changes in the array agg stuff...

On Mon, Mar 22, 2010 at 2:15 PM, Paul Ramsey <pramsey at cleverelephant.ca> wrote:
> Mike,
> See if this patch makes anything better. It matches tlane's changes to
> the 8.4 agg finalfunc.
> P.
>
> On Mon, Mar 22, 2010 at 1:29 PM, Greg Stark <gsstark at mit.edu> wrote:
>> On Mon, Mar 22, 2010 at 4:45 PM, Paul Ramsey <pramsey at cleverelephant.ca> wrote:
>>> Did you already try replacing your postgis functions with array_agg
>>> calls to see if we can push the problem back over the fence to pgsql
>>> land?
>>
>> On Sat, Mar 20, 2010 at 9:17 AM, Mike Leahy <mgleahy at alumni.uwaterloo.ca> wrote:
>>> Running this query on various data will produce one of two results.  One is
>>> the error mentioned in the subject (ERROR:  array size exceeds the maximum
>>> allowed (134217727)).  I can find very little information on this error.  The
>>> other outcome is that it often causes the PostgreSQL backend to segfault (see
>>> gdboutput.txt).
>>
>> gdboutput.txt:
>>
>> Program terminated with signal 11, Segmentation fault.
>> #0  0x00007fa4be23615b in pfree () from
>> /usr/lib/postgresql/8.4/bin/postgres
>> #1  0x00007fa4be18091b in makeMdArrayResult () from
>> /usr/lib/postgresql/8.4/bin/postgres
>> #2  0x00007fa4b7f038bc in pgis_accum_finalfn () from
>> /usr/lib/postgresql/8.4/lib/postgis-1.5.so
>> #3  0x00007fa4b7f039ee in pgis_geometry_union_finalfn () from
>> /usr/lib/postgresql/8.4/lib/postgis-1.5.so
>>
>> Any chance you can generate one of these seg faults from a build with
>> symbols and with assertions enabled? It might catch the problem
>> earlier and provide more info.
>>
>> IIRC there were some memory management changes which required changes
>> to array_agg() and which had some risk of causing problems for other
>> sites with similar coding. Is it possible your'e missing some of these
>> changes? I'm having trouble tracking them all down but at least
>> there's these:
>>
>>
>> commit 3d332de2eab8a01c0ef3f58ea69de2010fe8a1e1
>> Author: Tom Lane <tgl at sss.pgh.pa.us>
>> Date:   Thu Jul 23 20:45:27 2009 +0000
>>
>>    In a non-hashed Agg node, reset the "aggcontext" at group
>> boundaries, instead
>>    of individually pfree'ing pass-by-reference transition values.  This should
>>    be at least as fast as the prior coding, and it has the major advantage of
>>    clearing out any working data an aggregate function may have stored in or
>>    underneath the aggcontext.  This avoids memory leakage when an aggregate
>>    such as array_agg() is used in GROUP BY mode.  Per report from Chris Spotts.
>>
>>    Back-patch to 8.4.  In principle the problem could arise in prior versions,
>>    but since they didn't have array_agg the issue seems not critical.
>>
>> commit 7f83b61cc26eeac0c5a09add49f6cf899f87fc0b
>> Author: Tom Lane <tgl at sss.pgh.pa.us>
>> Date:   Sat Jun 20 18:45:28 2009 +0000
>>
>>    Fix things so that array_agg_finalfn does not modify or free its input
>>    ArrayBuildState, per trouble report from Merlin Moncure.  By adopting
>>    this fix, we are essentially deciding that aggregate final-functions
>>    should not modify their inputs ever.  Adjust documentation and comments
>>    to match that conclusion.
>>
>> --
>> greg
>> _______________________________________________
>> postgis-users mailing list
>> postgis-users at postgis.refractions.net
>> http://postgis.refractions.net/mailman/listinfo/postgis-users
>>
>