From trac at osgeo.org Mon Nov 3 02:21:43 2025 From: trac at osgeo.org (PostGIS) Date: Mon, 03 Nov 2025 10:21:43 -0000 Subject: [PostGIS] #5593: Fails to build with imagemagic 6.9.12.98 In-Reply-To: <050.d70c51fcb8f375a80f52b2069c0d9bcc@osgeo.org> References: <050.d70c51fcb8f375a80f52b2069c0d9bcc@osgeo.org> Message-ID: <065.78fd875e3853fa843541f0cf68690cc5@osgeo.org> #5593: Fails to build with imagemagic 6.9.12.98 -----------------------------+--------------------------- Reporter: Bas Couwenberg | Owner: strk Type: defect | Status: closed Priority: medium | Milestone: PostGIS 3.4.1 Component: build | Version: 3.4.x Resolution: wontfix | Keywords: -----------------------------+--------------------------- Comment (by Bas Couwenberg): Just for the record, the graphicsmagick-imagemagic-compat package was recently dropped because !ImageMagick and !GraphicsMagic are increasingly diverging ([https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=413954#97 Debian Bug #413954]). The Debian package build fails with both plain graphicsmagic & imagemagic, so we stopped building the postgis-doc package. -- Ticket URL: PostGIS The PostGIS Trac is used for bug, enhancement & task tracking, a user and developer wiki, and a view into the subversion code repository of PostGIS project. From trac at osgeo.org Mon Nov 3 08:02:46 2025 From: trac at osgeo.org (PostGIS) Date: Mon, 03 Nov 2025 16:02:46 -0000 Subject: [PostGIS] #5593: Fails to build with imagemagic 6.9.12.98 In-Reply-To: <050.d70c51fcb8f375a80f52b2069c0d9bcc@osgeo.org> References: <050.d70c51fcb8f375a80f52b2069c0d9bcc@osgeo.org> Message-ID: <065.d53affe0135645a4fcc0e062f94b3438@osgeo.org> #5593: Fails to build with imagemagic 6.9.12.98 -----------------------------+--------------------------- Reporter: Bas Couwenberg | Owner: strk Type: defect | Status: reopened Priority: medium | Milestone: PostGIS 3.4.1 Component: build | Version: 3.4.x Resolution: | Keywords: -----------------------------+--------------------------- Changes (by komzpa): * resolution: wontfix => * status: closed => reopened -- Ticket URL: PostGIS The PostGIS Trac is used for bug, enhancement & task tracking, a user and developer wiki, and a view into the subversion code repository of PostGIS project. From trac at osgeo.org Mon Nov 3 08:04:42 2025 From: trac at osgeo.org (PostGIS) Date: Mon, 03 Nov 2025 16:04:42 -0000 Subject: [PostGIS] #5593: Fails to build with imagemagic 6.9.12.98 In-Reply-To: <050.d70c51fcb8f375a80f52b2069c0d9bcc@osgeo.org> References: <050.d70c51fcb8f375a80f52b2069c0d9bcc@osgeo.org> Message-ID: <065.4950d2150a95a339c5cbdea21a959589@osgeo.org> #5593: Fails to build with imagemagic 6.9.12.98 -----------------------------+--------------------------- Reporter: Bas Couwenberg | Owner: komzpa Type: defect | Status: new Priority: medium | Milestone: PostGIS 3.4.1 Component: build | Version: 3.4.x Resolution: | Keywords: -----------------------------+--------------------------- Changes (by komzpa): * owner: strk => komzpa * status: reopened => new Comment: It looks like both imagemagick and graphicsmagick stopping providing convert command. Which is probably for the good as word `convert` can mean too many different conversions and now ogr2ogr can take it instead/s. Will take a look how to fix this. -- Ticket URL: PostGIS The PostGIS Trac is used for bug, enhancement & task tracking, a user and developer wiki, and a view into the subversion code repository of PostGIS project. From trac at osgeo.org Tue Nov 4 00:56:05 2025 From: trac at osgeo.org (PostGIS) Date: Tue, 04 Nov 2025 08:56:05 -0000 Subject: [PostGIS] #6012: Memory leak in function lwcircstring_from_lwpointarray Message-ID: <050.c313b633ffae4edc5682f59a3f451b19@osgeo.org> #6012: Memory leak in function lwcircstring_from_lwpointarray ----------------------+--------------------------- Reporter: ezimanyi | Owner: pramsey Type: defect | Status: new Priority: medium | Milestone: PostGIS 3.6.1 Component: postgis | Version: 3.5.x Keywords: | ----------------------+--------------------------- In function lwcircstring_construct the following line should be added {{{ FLAGS_SET_READONLY(result->points->flags, 0); }}} to get the ownership of the pointarray that was created with the read-only flag to 1 in function `ptarray_construct_reference_data` which has been called in function `lwcircstring_from_lwpointarray` -- Ticket URL: PostGIS The PostGIS Trac is used for bug, enhancement & task tracking, a user and developer wiki, and a view into the subversion code repository of PostGIS project. From trac at osgeo.org Wed Nov 5 04:50:37 2025 From: trac at osgeo.org (PostGIS) Date: Wed, 05 Nov 2025 12:50:37 -0000 Subject: [PostGIS] #6012: Memory leak in function lwcircstring_from_lwpointarray In-Reply-To: <050.c313b633ffae4edc5682f59a3f451b19@osgeo.org> References: <050.c313b633ffae4edc5682f59a3f451b19@osgeo.org> Message-ID: <065.77f27c73762677a19ed646485a7ddc98@osgeo.org> #6012: Memory leak in function lwcircstring_from_lwpointarray -----------------------+--------------------------- Reporter: ezimanyi | Owner: pramsey Type: defect | Status: new Priority: medium | Milestone: PostGIS 3.6.1 Component: postgis | Version: 3.5.x Resolution: | Keywords: -----------------------+--------------------------- Comment (by pramsey): The whole point of the function ptarray_construct_reference_data is to build a point array on top of an array that is not owned by the overarching struct, as it says in the comment atop the function, so turning off READONLY is explicitly opposite the contract, and it would make no sense to do that. Do you have a valgrind result showing this leak? -- Ticket URL: PostGIS The PostGIS Trac is used for bug, enhancement & task tracking, a user and developer wiki, and a view into the subversion code repository of PostGIS project. From trac at osgeo.org Thu Nov 6 00:57:30 2025 From: trac at osgeo.org (PostGIS) Date: Thu, 06 Nov 2025 08:57:30 -0000 Subject: [PostGIS] #6012: Memory leak in function lwcircstring_from_lwpointarray In-Reply-To: <050.c313b633ffae4edc5682f59a3f451b19@osgeo.org> References: <050.c313b633ffae4edc5682f59a3f451b19@osgeo.org> Message-ID: <065.fc633d0a4a4ce4644b08e05fe0345f60@osgeo.org> #6012: Memory leak in function lwcircstring_from_lwpointarray -----------------------+--------------------------- Reporter: ezimanyi | Owner: pramsey Type: defect | Status: new Priority: medium | Milestone: PostGIS 3.6.1 Component: postgis | Version: 3.5.x Resolution: | Keywords: -----------------------+--------------------------- Comment (by mschoema): I believe the issue is actually in lwcircstring_from_lwpointarray and comes from how the three functions interact. Essentially, lwcircstring_from_lwpointarray first allocates memory to write the points in, then calls ptarray_construct_reference_data, then calls lwcircstring_construct. But this means that at the end of the function, the resulting LWCIRCSTRING does not own its own pointarry. This will thus cause memory leaks when trying to free the circstring, because no-one else owns this pointarray. I believe lwcircstring_from_lwpointarray should either call ptarray_construct_copy_data (but that's an unnecessary memory allocation) or change the readonly flag of its pointarray before returning. -- Ticket URL: PostGIS The PostGIS Trac is used for bug, enhancement & task tracking, a user and developer wiki, and a view into the subversion code repository of PostGIS project. From trac at osgeo.org Thu Nov 6 01:29:28 2025 From: trac at osgeo.org (PostGIS) Date: Thu, 06 Nov 2025 09:29:28 -0000 Subject: [PostGIS] #6012: Memory leak in function lwcircstring_from_lwpointarray In-Reply-To: <050.c313b633ffae4edc5682f59a3f451b19@osgeo.org> References: <050.c313b633ffae4edc5682f59a3f451b19@osgeo.org> Message-ID: <065.34c422b0c59bace99f4fb20b70e6ed04@osgeo.org> #6012: Memory leak in function lwcircstring_from_lwpointarray -----------------------+--------------------------- Reporter: ezimanyi | Owner: pramsey Type: defect | Status: new Priority: medium | Milestone: PostGIS 3.6.1 Component: postgis | Version: 3.5.x Resolution: | Keywords: -----------------------+--------------------------- Comment (by mschoema): Note that this issue also happens in lwcircstring_from_lwmpoint, but these are the only two calls to ptarray_construct_reference_data other than the ones in the _from_gserialized functions. I also see that these functions (lwcircstring_from_lwpointarray and lwcircstring_from_lwmpoint, amongst a few others) are only defined in lwcircstring.c (though not static) and not exposed in liblwgeom.h Is there a reason for that? I see an equivalent lwline_from_ptarray function for lines, but it is marked as deprecated in liblwgeom.h. Why was it deprecated, and what should the alternative be to create a line (or a circular string) from a set of points? -- Ticket URL: PostGIS The PostGIS Trac is used for bug, enhancement & task tracking, a user and developer wiki, and a view into the subversion code repository of PostGIS project. From trac at osgeo.org Thu Nov 6 13:13:41 2025 From: trac at osgeo.org (PostGIS) Date: Thu, 06 Nov 2025 21:13:41 -0000 Subject: [PostGIS] #6012: Memory leak in function lwcircstring_from_lwpointarray In-Reply-To: <050.c313b633ffae4edc5682f59a3f451b19@osgeo.org> References: <050.c313b633ffae4edc5682f59a3f451b19@osgeo.org> Message-ID: <065.a08c43c5cb03ec6eec3a96fe7dad2b59@osgeo.org> #6012: Memory leak in function lwcircstring_from_lwpointarray -----------------------+--------------------------- Reporter: ezimanyi | Owner: pramsey Type: defect | Status: new Priority: medium | Milestone: PostGIS 3.6.1 Component: postgis | Version: 3.5.x Resolution: | Keywords: -----------------------+--------------------------- Comment (by pramsey): That explains why it's not showing up in valgrind, lwcircstring_from_lwpointarray is never used. Nor is lwcircstring_from_lwmpoint. The answer for why they are declared up top I think is because if they were just made static the compiler would complain about unused static functions. So it was just reactivity to a compiler complaint. It's not clear to me we should keep them around "just in case". -- Ticket URL: PostGIS The PostGIS Trac is used for bug, enhancement & task tracking, a user and developer wiki, and a view into the subversion code repository of PostGIS project. From trac at osgeo.org Thu Nov 6 14:15:48 2025 From: trac at osgeo.org (PostGIS) Date: Thu, 06 Nov 2025 22:15:48 -0000 Subject: [PostGIS] #6012: Memory leak in function lwcircstring_from_lwpointarray In-Reply-To: <050.c313b633ffae4edc5682f59a3f451b19@osgeo.org> References: <050.c313b633ffae4edc5682f59a3f451b19@osgeo.org> Message-ID: <065.ac3467ba8c39f6d1b056cd7196ed911e@osgeo.org> #6012: Memory leak in function lwcircstring_from_lwpointarray -----------------------+--------------------------- Reporter: ezimanyi | Owner: pramsey Type: defect | Status: new Priority: medium | Milestone: PostGIS 3.6.1 Component: postgis | Version: 3.5.x Resolution: | Keywords: -----------------------+--------------------------- Comment (by ezimanyi): The following MEOS program {{{ /** * @file * @brief A simple MEOS program that creates a circle and prints its * WKT representation using the liblwgeom library. * * The program can be build as follows * @code * gcc -Wall -g -I/usr/local/include -o geocircle_test geocircle_test.c -L/usr/local/lib -lmeos * @endcode */ #include #include #include #include extern GSERIALIZED *geocircle_make(double x, double y, double radius, int32_t srid); /* Main program */ int main(void) { /* Initialize MEOS */ meos_initialize(); GSERIALIZED *circle = geocircle_make(1, 1, 1, 5676); char *circle_out = geo_as_ewkt(circle, 6); printf("%s\n", circle_out); free(circle); free(circle_out); /* Finalize MEOS */ meos_finalize(); return 0; } }}} tested with Valgrind when commenting in/out the line FLAGS_SET_READONLY(result->points->flags, 0); https://github.com/estebanzimanyi/MobilityDB/blob/meos_test/postgis/liblwgeom/lwcircstring.c#L74C3-L74C21 is as follows ---------------------------------------------------------------------------------- {{{ $ valgrind -s --leak-check=full --show-leak-kinds=all ./geocircle_test ==3353068== Memcheck, a memory error detector ==3353068== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al. ==3353068== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info ==3353068== Command: ./geocircle_test ==3353068== SRID=5676;CURVEPOLYGON(CIRCULARSTRING(0 1,2 1,0 1)) ==3353068== ==3353068== HEAP SUMMARY: ==3353068== in use at exit: 0 bytes in 0 blocks ==3353068== total heap usage: 8,201 allocs, 8,201 frees, 967,211 bytes allocated ==3353068== ==3353068== All heap blocks were freed -- no leaks are possible ==3353068== ==3353068== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0) }}} ---------------------------------------------------------------------------------- {{{ $ valgrind -s --leak-check=full --show-leak-kinds=all ./geocircle_test ==3365033== Memcheck, a memory error detector ==3365033== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al. ==3365033== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info ==3365033== Command: ./geocircle_test ==3365033== SRID=5676;CURVEPOLYGON(CIRCULARSTRING(0 1,2 1,0 1)) ==3365033== ==3365033== HEAP SUMMARY: ==3365033== in use at exit: 48 bytes in 1 blocks ==3365033== total heap usage: 8,201 allocs, 8,200 frees, 967,211 bytes allocated ==3365033== ==3365033== 48 bytes in 1 blocks are definitely lost in loss record 1 of 1 ==3365033== at 0x4848899: malloc (in /usr/libexec/valgrind /vgpreload_memcheck-amd64-linux.so) ==3365033== by 0x4A25935: default_allocator (lwutil.c:91) ==3365033== by 0x4A25EFD: lwalloc (lwutil.c:229) ==3365033== by 0x49CF81B: lwcircstring_from_lwpointarray (lwcircstring.c:175) ==3365033== by 0x498CCFB: lwcircle_make (tcbuffer_spatialfuncs.c:111) ==3365033== by 0x498CDBB: geocircle_make (tcbuffer_spatialfuncs.c:126) ==3365033== by 0x109227: main (geocircle_test.c:26) ==3365033== ==3365033== LEAK SUMMARY: ==3365033== definitely lost: 48 bytes in 1 blocks ==3365033== indirectly lost: 0 bytes in 0 blocks ==3365033== possibly lost: 0 bytes in 0 blocks ==3365033== still reachable: 0 bytes in 0 blocks ==3365033== suppressed: 0 bytes in 0 blocks ==3365033== ==3365033== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0) $ }}} ---------------------------------------------------------------------------------- -- Ticket URL: PostGIS The PostGIS Trac is used for bug, enhancement & task tracking, a user and developer wiki, and a view into the subversion code repository of PostGIS project. From trac at osgeo.org Fri Nov 7 07:00:43 2025 From: trac at osgeo.org (PostGIS) Date: Fri, 07 Nov 2025 15:00:43 -0000 Subject: [PostGIS] #5984: PostGIS selectivity is screwing up queries and forcing it to choose a spatial index when it's inappropriate In-Reply-To: <046.75677ad63d74709c6ed20baed0392698@osgeo.org> References: <046.75677ad63d74709c6ed20baed0392698@osgeo.org> Message-ID: <061.ea3e3092e2f9615d95986e8060b5e7e8@osgeo.org> #5984: PostGIS selectivity is screwing up queries and forcing it to choose a spatial index when it's inappropriate -----------------------+--------------------------- Reporter: robe | Owner: komzpa Type: defect | Status: new Priority: critical | Milestone: PostGIS 3.4.5 Component: postgis | Version: 3.5.x Resolution: | Keywords: -----------------------+--------------------------- Changes (by robe): * owner: pramsey => komzpa -- Ticket URL: PostGIS The PostGIS Trac is used for bug, enhancement & task tracking, a user and developer wiki, and a view into the subversion code repository of PostGIS project. From git at osgeo.org Fri Nov 7 10:35:17 2025 From: git at osgeo.org (git at osgeo.org) Date: Fri, 7 Nov 2025 10:35:17 -0800 (PST) Subject: [SCM] PostGIS branch stable-3.6 updated. 3.6.0-27-g21517f662 Message-ID: <20251107183518.55B9419F171@trac.osgeo.org> This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "PostGIS". The branch, stable-3.6 has been updated via 21517f662368c90bbee0dfdebff3cb82059467fd (commit) via a518412e516d4c7b216fd2f2797b20495d095e44 (commit) via b950728171f5d26fc3bbdd2f1cdf023cbffb0169 (commit) from 58b4e2571588b4aace175431f2ff33445c230ee4 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 21517f662368c90bbee0dfdebff3cb82059467fd Author: Darafei Praliaskouski Date: Fri Nov 7 22:34:25 2025 +0400 Touch up NEWS Closes #5984 diff --git a/NEWS b/NEWS index b7fc26db9..7a06e124f 100644 --- a/NEWS +++ b/NEWS @@ -14,10 +14,7 @@ PostGIS 3.6.1 by extension are owned by extension authored: Andrey Borodin (Yandex), reported by Sergey Bobrov (Kaspersky) - #5754, ST_ForcePolygonCCW reverses lines (Paul Ramsey) - -* Bug Fixes * - - - #5959, Prevent histogram target overflow when analysing massive tables (Darafei Praliaskouski) + - #5959, #5984, Prevent histogram target overflow when analysing massive tables (Darafei Praliaskouski) PostGIS 3.6.0 commit a518412e516d4c7b216fd2f2797b20495d095e44 Author: Darafei Praliaskouski Date: Thu Oct 30 03:46:26 2025 +0400 Guard against histogram axis dimension underflow References #5959 References #5984 diff --git a/postgis/cunit/cu_tester.c b/postgis/cunit/cu_tester.c index b4dd46aa7..eb6ba5b0d 100644 --- a/postgis/cunit/cu_tester.c +++ b/postgis/cunit/cu_tester.c @@ -28,6 +28,7 @@ #include #include +#include #include #include "../gserialized_estimate_support.h" @@ -82,6 +83,23 @@ histogram_budget_clamps(void) CU_ASSERT_EQUAL(histogram_cell_budget((double)INT_MAX, 50000, INT_MAX), INT_MAX); } +static void +histogram_axis_allocation_guards(void) +{ + /* Baseline: evenly split a 10k target over two varying dimensions. */ + CU_ASSERT_EQUAL(histogram_axis_cells(10000, 2, 0.5), 100); + + /* Skewed axis ratios that collapse to tiny powers still return one cell. */ + CU_ASSERT_EQUAL(histogram_axis_cells(10000, 2, 1e-9), 1); + + /* Denormals, NaNs and negative ratios should not leak to the histogram. */ + CU_ASSERT_EQUAL(histogram_axis_cells(10000, 2, NAN), 1); + CU_ASSERT_EQUAL(histogram_axis_cells(10000, 2, -0.5), 1); + + /* Extremely aggressive ratios remain bounded by the square root of the budget. */ + CU_ASSERT_EQUAL(histogram_axis_cells(INT_MAX, 2, 1.0), (int)sqrt((double)INT_MAX * 2.0)); +} + static void nd_stats_indexing_behaviour(void) { @@ -138,6 +156,7 @@ main(void) goto cleanup; if (!CU_add_test(suite, "histogram budget clamps", histogram_budget_clamps) || + !CU_add_test(suite, "histogram axis guards", histogram_axis_allocation_guards) || !CU_add_test(suite, "nd_stats value index guards", nd_stats_indexing_behaviour) || !CU_add_test(suite, "nd_box ratio edge cases", nd_box_ratio_cases)) { diff --git a/postgis/gserialized_estimate.c b/postgis/gserialized_estimate.c index ea887ecb6..54adae679 100644 --- a/postgis/gserialized_estimate.c +++ b/postgis/gserialized_estimate.c @@ -1516,11 +1516,10 @@ compute_gserialized_stats_mode(VacAttrStats *stats, AnalyzeAttrFetchFunc fetchfu * Scale the target cells number by the # of dims and ratio, * then take the appropriate root to get the estimated number of cells * on this axis (eg, pow(0.5) for 2d, pow(0.333) for 3d, pow(0.25) for 4d) - */ - histo_size[d] = (int)pow((double)histo_cells_target * histo_ndims * edge_ratio, 1/(double)histo_ndims); - /* If something goes awry, just give this dim one slot */ - if ( ! histo_size[d] ) - histo_size[d] = 1; + * The dedicated helper clamps pathological floating point inputs so we + * do not resurrect the NaN propagation reported in #5959 on amd64. + */ + histo_size[d] = histogram_axis_cells(histo_cells_target, histo_ndims, edge_ratio); } histo_cells_new *= histo_size[d]; } diff --git a/postgis/gserialized_estimate_support.h b/postgis/gserialized_estimate_support.h index 0d3a23d75..6b372a43e 100644 --- a/postgis/gserialized_estimate_support.h +++ b/postgis/gserialized_estimate_support.h @@ -151,6 +151,46 @@ histogram_cell_budget(double total_rows, int ndims, int attstattarget) return (int)budget; } +/* + * Allocate histogram buckets along a single axis in proportion to the observed + * density variation. The caller passes in the global histogram target along + * with the number of axes that exhibited variation in the sampled data and the + * relative contribution of the current axis (edge_ratio). Earlier versions + * evaluated the pow() call directly in the caller, which exposed the planner to + * NaN propagation on some amd64 builds when the ratio was denormal or negative + * (see #5959). Keeping the calculation in one place allows us to clamp the + * inputs and provide a predictable fallback for problematic floating point + * combinations. + */ +static inline int +histogram_axis_cells(int histo_cells_target, int histo_ndims, double edge_ratio) +{ + double scaled; + double axis_cells; + + if (histo_cells_target <= 0 || histo_ndims <= 0) + return 1; + + if (!(edge_ratio > 0.0) || !isfinite(edge_ratio)) + return 1; + + scaled = (double)histo_cells_target * (double)histo_ndims * edge_ratio; + if (!(scaled > 0.0) || !isfinite(scaled)) + return 1; + + axis_cells = pow(scaled, 1.0 / (double)histo_ndims); + if (!(axis_cells > 0.0) || !isfinite(axis_cells)) + return 1; + + if (axis_cells >= (double)INT_MAX) + return INT_MAX; + + if (axis_cells <= 1.0) + return 1; + + return (int)axis_cells; +} + /* * Compute the portion of 'target' covered by 'cover'. The caller supplies the * dimensionality because ND_BOX always carries four slots. Degenerate volumes commit b950728171f5d26fc3bbdd2f1cdf023cbffb0169 Author: Darafei Praliaskouski Date: Thu Oct 30 02:55:59 2025 +0400 Prevent histogram target overflow when analysing massive tables Add CUnit tests for overflow scenarios Closes #5959 diff --git a/NEWS b/NEWS index 4dec87f2a..b7fc26db9 100644 --- a/NEWS +++ b/NEWS @@ -15,6 +15,10 @@ PostGIS 3.6.1 authored: Andrey Borodin (Yandex), reported by Sergey Bobrov (Kaspersky) - #5754, ST_ForcePolygonCCW reverses lines (Paul Ramsey) +* Bug Fixes * + + - #5959, Prevent histogram target overflow when analysing massive tables (Darafei Praliaskouski) + PostGIS 3.6.0 2025/09/01 diff --git a/configure.ac b/configure.ac index dbd4f59c3..12e26a06b 100644 --- a/configure.ac +++ b/configure.ac @@ -1927,6 +1927,7 @@ AC_CONFIG_FILES([GNUmakefile libpgcommon/Makefile libpgcommon/cunit/Makefile postgis/Makefile + postgis/cunit/Makefile postgis/sqldefines.h sfcgal/Makefile $SFCGAL_MAKEFILE_LIST diff --git a/postgis/cunit/Makefile.in b/postgis/cunit/Makefile.in new file mode 100644 index 000000000..483e4ca10 --- /dev/null +++ b/postgis/cunit/Makefile.in @@ -0,0 +1,43 @@ +# ********************************************************************** +# * +# * PostGIS - Spatial Types for PostgreSQL +# * http://postgis.net +# * +# * Copyright 2025 Darafei Praliaskouski +# * +# * This is free software; you can redistribute and/or modify it under +# * the terms of the GNU General Public Licence. See the COPYING file. +# * +# ********************************************************************** + +srcdir = @srcdir@ +top_builddir = @top_builddir@ + +CC=@CC@ +LIBTOOL=@LIBTOOL@ +CFLAGS = @CFLAGS@ @CPPFLAGS@ @PGSQL_BE_CPPFLAGS@ @CUNIT_CPPFLAGS@ -I.. -I$(top_builddir) -I at top_srcdir@/liblwgeom -I at top_builddir@/liblwgeom -I at top_srcdir@/libpgcommon -I at top_builddir@/libpgcommon +LDFLAGS = @CUNIT_LDFLAGS@ -lm + +VPATH = $(srcdir) + +OBJS = cu_tester.o + +# Build the standalone histogram helper tester. +all: cu_tester + +# Execute the suite directly; no installation step is required. +check: all + $(LIBTOOL) --mode=execute ./cu_tester + +# Link the tester with libtool; all helper code is header-only. +cu_tester: $(OBJS) + $(LIBTOOL) --mode=link $(CC) $(CFLAGS) -o $@ $(OBJS) $(LDFLAGS) + +%.o: %.c + $(CC) $(CFLAGS) -c -o $@ $< + +clean: + rm -f $(OBJS) cu_tester + +clobber distclean: clean + rm -f Makefile diff --git a/postgis/cunit/cu_tester.c b/postgis/cunit/cu_tester.c new file mode 100644 index 000000000..b4dd46aa7 --- /dev/null +++ b/postgis/cunit/cu_tester.c @@ -0,0 +1,154 @@ +/********************************************************************** + * + * PostGIS - Spatial Types for PostgreSQL + * http://postgis.net + * + * This file is part of PostGIS + * + * PostGIS is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 2 of the License, or + * (at your option) any later version. + * + * PostGIS is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with PostGIS. If not, see . + * + ********************************************************************** + * + * Copyright 2025 (C) Darafei Praliaskouski + * + **********************************************************************/ + +#include "postgres.h" + +#include +#include +#include + +#include "../gserialized_estimate_support.h" + +static ND_BOX +make_box(float minx, float miny, float minz, float minm, float maxx, float maxy, float maxz, float maxm) +{ + ND_BOX box; + + memset(&box, 0, sizeof(box)); + box.min[0] = minx; + box.min[1] = miny; + box.min[2] = minz; + box.min[3] = minm; + box.max[0] = maxx; + box.max[1] = maxy; + box.max[2] = maxz; + box.max[3] = maxm; + return box; +} + +static void +histogram_budget_clamps(void) +{ + /* Zero or negative row counts disable histogram construction. */ + CU_ASSERT_EQUAL(histogram_cell_budget(0.0, 2, 100), 0); + CU_ASSERT_EQUAL(histogram_cell_budget(-1.0, 4, 100), 0); + + /* Degenerate dimensionality cannot allocate histogram space. */ + CU_ASSERT_EQUAL(histogram_cell_budget(1000.0, 0, 100), 0); + + /* Matches the classic pow(attstattarget, ndims) path. */ + CU_ASSERT_EQUAL(histogram_cell_budget(1e6, 2, 100), 10000); + CU_ASSERT_EQUAL(histogram_cell_budget(1e6, 3, 50), 125000); + + /* attstattarget^ndims exceeds ndims * 100000 and must be clamped. */ + CU_ASSERT_EQUAL(histogram_cell_budget(1e6, 4, 50), 400000); + + /* attstattarget<=0 is normalised to the smallest viable target. */ + CU_ASSERT_EQUAL(histogram_cell_budget(1e6, 2, 0), 1); + + /* Row clamp shrinks the grid for small relations. */ + CU_ASSERT_EQUAL(histogram_cell_budget(1.0, 2, 100), 20); + + /* Large tables now preserve the dimensional cap instead of overflowing. */ + CU_ASSERT_EQUAL(histogram_cell_budget(1.5e8, 2, 100), 10000); + + /* Regression for #5984: huge attstat targets stabilise instead of wrapping. */ + CU_ASSERT_EQUAL(histogram_cell_budget(5e6, 2, 10000), 200000); + + /* Trigger the INT_MAX guard once both other caps exceed it. */ + CU_ASSERT_EQUAL(histogram_cell_budget((double)INT_MAX, 50000, INT_MAX), INT_MAX); +} + +static void +nd_stats_indexing_behaviour(void) +{ + ND_STATS stats; + const int good_index[ND_DIMS] = {1, 2, 0, 0}; + const int bad_index[ND_DIMS] = {1, 5, 0, 0}; + + memset(&stats, 0, sizeof(stats)); + stats.ndims = 3; + stats.size[0] = 4.0f; + stats.size[1] = 5.0f; + stats.size[2] = 3.0f; + + /* Three-dimensional index (x=1, y=2, z=0) collapses into 1 + 2 * 4. */ + CU_ASSERT_EQUAL(nd_stats_value_index(&stats, good_index), 1 + 2 * 4); + /* Any request outside the histogram bounds triggers a guard. */ + CU_ASSERT_EQUAL(nd_stats_value_index(&stats, bad_index), -1); + + /* Regression for #5959: ndims higher than populated sizes still honours guards. */ + stats.ndims = 4; + CU_ASSERT_EQUAL(nd_stats_value_index(&stats, good_index), -1); +} + +static void +nd_box_ratio_cases(void) +{ + ND_BOX covering = make_box(0.0f, 0.0f, 0.0f, 0.0f, 2.0f, 2.0f, 2.0f, 0.0f); + ND_BOX interior = make_box(0.5f, 0.5f, 0.5f, 0.0f, 1.5f, 1.5f, 1.5f, 0.0f); + ND_BOX partial = make_box(0.0f, 0.0f, 0.0f, 0.0f, 0.5f, 0.5f, 0.5f, 0.0f); + ND_BOX target = make_box(0.0f, 0.0f, 0.0f, 0.0f, 1.0f, 1.0f, 1.0f, 0.0f); + ND_BOX flat = make_box(0.0f, 0.0f, 0.0f, 0.0f, 1.0f, 1.0f, 0.0f, 0.0f); + ND_BOX touch = make_box(2.0f, 0.0f, 0.0f, 0.0f, 3.0f, 1.0f, 1.0f, 0.0f); + + /* Full coverage should evaluate to one regardless of the extra extent. */ + CU_ASSERT_DOUBLE_EQUAL(nd_box_ratio(&covering, &interior, 3), 1.0, 1e-12); + /* A shared octant carries one eighth of the reference volume. */ + CU_ASSERT_DOUBLE_EQUAL(nd_box_ratio(&partial, &target, 3), 0.125, 1e-12); + /* Degenerate slabs have zero volume in three dimensions. */ + CU_ASSERT_DOUBLE_EQUAL(nd_box_ratio(&covering, &flat, 3), 0.0, 1e-12); + /* Boxes that only touch along a face should not count as overlap. */ + CU_ASSERT_DOUBLE_EQUAL(nd_box_ratio(&covering, &touch, 3), 0.0, 1e-12); +} + +int +main(void) +{ + CU_pSuite suite; + unsigned int failures = 0; + if (CU_initialize_registry() != CUE_SUCCESS) + return CU_get_error(); + + suite = CU_add_suite("gserialized_histogram_helpers", NULL, NULL); + if (!suite) + goto cleanup; + + if (!CU_add_test(suite, "histogram budget clamps", histogram_budget_clamps) || + !CU_add_test(suite, "nd_stats value index guards", nd_stats_indexing_behaviour) || + !CU_add_test(suite, "nd_box ratio edge cases", nd_box_ratio_cases)) + { + goto cleanup; + } + + CU_basic_set_mode(CU_BRM_VERBOSE); + CU_basic_run_tests(); + +cleanup: + failures = CU_get_number_of_tests_failed(); + CU_cleanup_registry(); + return failures == 0 ? CUE_SUCCESS : 1; +} diff --git a/postgis/gserialized_estimate.c b/postgis/gserialized_estimate.c index 1e84228b0..ea887ecb6 100644 --- a/postgis/gserialized_estimate.c +++ b/postgis/gserialized_estimate.c @@ -19,11 +19,10 @@ ********************************************************************** * * Copyright 2012 (C) Paul Ramsey + * Copyright 2025 (C) Darafei Praliaskouski * **********************************************************************/ - - /********************************************************************** THEORY OF OPERATION @@ -112,10 +111,12 @@ dimensionality cases. (2D geometry) &&& (3D column), etc. #include "stringbuffer.h" #include "liblwgeom.h" #include "lwgeodetic.h" -#include "lwgeom_pg.h" /* For debugging macros. */ +#include "lwgeom_pg.h" /* For debugging macros. */ #include "gserialized_gist.h" /* For index common functions */ +#include "gserialized_estimate_support.h" #include +#include #if HAVE_IEEEFP_H #include #endif @@ -144,8 +145,7 @@ Datum _postgis_gserialized_stats(PG_FUNCTION_ARGS); /* Local prototypes */ static Oid table_get_spatial_index(Oid tbl_oid, int16 attnum, int *key_type, int16 *idx_attnum); -static GBOX * spatial_index_read_extent(Oid idx_oid, int idx_att_num, int key_type); - +static GBOX *spatial_index_read_extent(Oid idx_oid, int idx_att_num, int key_type); /* Other prototypes */ float8 gserialized_joinsel_internal(PlannerInfo *root, List *args, JoinType jointype, int mode); @@ -186,13 +186,6 @@ Datum geometry_estimated_extent(PG_FUNCTION_ARGS); */ #define SDFACTOR 3.25 -/** -* The maximum number of dimensions our code can handle. -* We'll use this to statically allocate a bunch of -* arrays below. -*/ -#define ND_DIMS 4 - /** * Minimum width of a dimension that we'll bother trying to * compute statistics on. Bearing in mind we have no control @@ -219,68 +212,6 @@ Datum geometry_estimated_extent(PG_FUNCTION_ARGS); #define FALLBACK_ND_SEL 0.2 #define FALLBACK_ND_JOINSEL 0.3 -/** -* N-dimensional box type for calculations, to avoid doing -* explicit axis conversions from GBOX in all calculations -* at every step. -*/ -typedef struct ND_BOX_T -{ - float4 min[ND_DIMS]; - float4 max[ND_DIMS]; -} ND_BOX; - -/** -* N-dimensional box index type -*/ -typedef struct ND_IBOX_T -{ - int min[ND_DIMS]; - int max[ND_DIMS]; -} ND_IBOX; - - -/** -* N-dimensional statistics structure. Well, actually -* four-dimensional, but set up to handle arbitrary dimensions -* if necessary (really, we just want to get the 2,3,4-d cases -* into one shared piece of code). -*/ -typedef struct ND_STATS_T -{ - /* Dimensionality of the histogram. */ - float4 ndims; - - /* Size of n-d histogram in each dimension. */ - float4 size[ND_DIMS]; - - /* Lower-left (min) and upper-right (max) spatial bounds of histogram. */ - ND_BOX extent; - - /* How many rows in the table itself? */ - float4 table_features; - - /* How many rows were in the sample that built this histogram? */ - float4 sample_features; - - /* How many not-Null/Empty features were in the sample? */ - float4 not_null_features; - - /* How many features actually got sampled in the histogram? */ - float4 histogram_features; - - /* How many cells in histogram? (sizex*sizey*sizez*sizem) */ - float4 histogram_cells; - - /* How many cells did those histogram features cover? */ - /* Since we are pro-rating coverage, this number should */ - /* now always equal histogram_features */ - float4 cells_covered; - - /* Variable length # of floats for histogram */ - float4 value[1]; -} ND_STATS; - typedef struct { /* Saved state from std_typanalyze() */ AnalyzeAttrComputeStatsFunc std_compute_stats; @@ -318,13 +249,12 @@ text_p_get_mode(const text *txt) char *modestr; if (VARSIZE_ANY_EXHDR(txt) <= 0) return mode; - modestr = (char*)VARDATA(txt); - if ( modestr[0] == 'N' ) + modestr = (char *)VARDATA(txt); + if (modestr[0] == 'N') mode = 0; return mode; } - /** * Integer comparison function for qsort */ @@ -372,7 +302,7 @@ total_double(const double *vals, int nvals) int i; float total = 0; /* Calculate total */ - for ( i = 0; i < nvals; i++ ) + for (i = 0; i < nvals; i++) total += vals[i]; return total; @@ -425,33 +355,6 @@ stddev(const int *vals, int nvals) } #endif /* POSTGIS_DEBUG_LEVEL >= 3 */ -/** -* Given a position in the n-d histogram (i,j,k) return the -* position in the 1-d values array. -*/ -static int -nd_stats_value_index(const ND_STATS *stats, int *indexes) -{ - int d; - int accum = 1, vdx = 0; - - /* Calculate the index into the 1-d values array that the (i,j,k,l) */ - /* n-d histogram coordinate implies. */ - /* index = x + y * sizex + z * sizex * sizey + m * sizex * sizey * sizez */ - for ( d = 0; d < (int)(stats->ndims); d++ ) - { - int size = (int)(stats->size[d]); - if ( indexes[d] < 0 || indexes[d] >= size ) - { - POSTGIS_DEBUGF(3, " bad index at (%d, %d)", indexes[0], indexes[1]); - return -1; - } - vdx += indexes[d] * accum; - accum *= size; - } - return vdx; -} - /** * Convert an #ND_BOX to a JSON string for printing */ @@ -722,50 +625,6 @@ nd_box_overlap(const ND_STATS *nd_stats, const ND_BOX *nd_box, ND_IBOX *nd_ibox) return true; } -/** -* Returns the proportion of b2 that is covered by b1. -*/ -static inline double -nd_box_ratio(const ND_BOX *b1, const ND_BOX *b2, int ndims) -{ - int d; - bool covered = true; - double ivol = 1.0; - double vol2 = 1.0; - - for ( d = 0 ; d < ndims; d++ ) - { - if ( b1->max[d] <= b2->min[d] || b1->min[d] >= b2->max[d] ) - return 0.0; /* Disjoint */ - - if ( b1->min[d] > b2->min[d] || b1->max[d] < b2->max[d] ) - covered = false; - } - - if ( covered ) - return 1.0; - - for ( d = 0; d < ndims; d++ ) - { - double width2 = b2->max[d] - b2->min[d]; - double imin, imax, iwidth; - - vol2 *= width2; - - imin = Max(b1->min[d], b2->min[d]); - imax = Min(b1->max[d], b2->max[d]); - iwidth = imax - imin; - iwidth = Max(0.0, iwidth); - - ivol *= iwidth; - } - - if ( vol2 == 0.0 ) - return vol2; - - return ivol / vol2; -} - /* How many bins shall we use in figuring out the distribution? */ #define MAX_NUM_BINS 50 #define BIN_MIN_SIZE 10 @@ -894,9 +753,9 @@ nd_increment(ND_IBOX *ibox, int ndims, int *counter) { int d = 0; - while ( d < ndims ) + while (d < ndims) { - if ( counter[d] < ibox->max[d] ) + if (counter[d] < ibox->max[d]) { counter[d] += 1; break; @@ -905,7 +764,7 @@ nd_increment(ND_IBOX *ibox, int ndims, int *counter) d++; } /* That's it, cannot increment any more! */ - if ( d == ndims ) + if (d == ndims) return false; /* Increment complete! */ @@ -1321,9 +1180,9 @@ gserialized_joinsel_internal(PlannerInfo *root, List *args, JoinType jointype, i PG_FUNCTION_INFO_V1(gserialized_gist_joinsel); Datum gserialized_gist_joinsel(PG_FUNCTION_ARGS) { - PlannerInfo *root = (PlannerInfo *) PG_GETARG_POINTER(0); + PlannerInfo *root = (PlannerInfo *)PG_GETARG_POINTER(0); /* Oid operator = PG_GETARG_OID(1); */ - List *args = (List *) PG_GETARG_POINTER(2); + List *args = (List *)PG_GETARG_POINTER(2); JoinType jointype = (JoinType) PG_GETARG_INT16(3); int mode = PG_GETARG_INT32(4); @@ -1512,22 +1371,13 @@ compute_gserialized_stats_mode(VacAttrStats *stats, AnalyzeAttrFetchFunc fetchfu #endif } - /* - * We'll build a histogram having stats->attr->attstattarget - * (default 100) cells on each side, within reason... - * we'll use ndims*100000 as the maximum number of cells. - * Also, if we're sampling a relatively small table, we'll try to ensure that - * we have a smaller grid. - */ #if POSTGIS_PGSQL_VERSION >= 170 - histo_cells_target = (int)pow((double)(stats->attstattarget), (double)ndims); POSTGIS_DEBUGF(3, " stats->attstattarget: %d", stats->attstattarget); + histo_cells_target = histogram_cell_budget(total_rows, ndims, stats->attstattarget); #else - histo_cells_target = (int)pow((double)(stats->attr->attstattarget), (double)ndims); POSTGIS_DEBUGF(3, " stats->attr->attstattarget: %d", stats->attr->attstattarget); + histo_cells_target = histogram_cell_budget(total_rows, ndims, stats->attr->attstattarget); #endif - histo_cells_target = Min(histo_cells_target, ndims * 100000); - histo_cells_target = Min(histo_cells_target, (int)(10 * ndims * total_rows)); POSTGIS_DEBUGF(3, " target # of histogram cells: %d", histo_cells_target); /* If there's no useful features, we can't work out stats */ @@ -1836,8 +1686,6 @@ compute_gserialized_stats_mode(VacAttrStats *stats, AnalyzeAttrFetchFunc fetchfu return; } - - /** * In order to do useful selectivity calculations in both 2-D and N-D * modes, we actually have to generate two stats objects, one for 2-D @@ -1875,7 +1723,6 @@ compute_gserialized_stats(VacAttrStats *stats, AnalyzeAttrFetchFunc fetchfunc, } } - /** * This function will be called when the ANALYZE command is run * on a column of the "geometry" or "geography" type. diff --git a/postgis/gserialized_estimate_support.h b/postgis/gserialized_estimate_support.h new file mode 100644 index 000000000..0d3a23d75 --- /dev/null +++ b/postgis/gserialized_estimate_support.h @@ -0,0 +1,197 @@ +/********************************************************************** + * + * PostGIS - Spatial Types for PostgreSQL + * http://postgis.net + * + * This file is part of PostGIS + * + * PostGIS is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 2 of the License, or + * (at your option) any later version. + * + * PostGIS is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with PostGIS. If not, see . + * + ********************************************************************** + * + * Internal helpers shared between the gserialized selectivity + * implementation and the unit tests. + * + * Keeping the routines header-only ensures the planner code and the + * harness evaluate the exact same floating-point flows without the + * cross-object plumbing that previously complicated maintenance. + * Nothing here is installed; the header is meant for + * gserialized_estimate.c and for the dedicated CUnit suite only. + * + ********************************************************************** + * + * Copyright 2012 (C) Paul Ramsey + * Copyright 2025 (C) Darafei Praliaskouski + * + **********************************************************************/ + +#ifndef POSTGIS_GSERIALIZED_ESTIMATE_SUPPORT_H +#define POSTGIS_GSERIALIZED_ESTIMATE_SUPPORT_H + +#include "postgres.h" + +#include +#include + +/* The maximum number of dimensions our statistics code supports. */ +#define ND_DIMS 4 + +/* Lightweight n-dimensional box representation for selectivity math. */ +typedef struct ND_BOX_T { + float4 min[ND_DIMS]; + float4 max[ND_DIMS]; +} ND_BOX; + +/* Integer counterpart used for histogram cell iteration. */ +typedef struct ND_IBOX_T { + int min[ND_DIMS]; + int max[ND_DIMS]; +} ND_IBOX; + +/* On-disk representation of the histogram emitted by ANALYZE. */ +typedef struct ND_STATS_T { + float4 ndims; + float4 size[ND_DIMS]; + ND_BOX extent; + float4 table_features; + float4 sample_features; + float4 not_null_features; + float4 histogram_features; + float4 histogram_cells; + float4 cells_covered; + float4 value[1]; +} ND_STATS; + +/* + * Return the flattened index for the histogram coordinate expressed by + * 'indexes'. A negative result signals that one of the axes fell outside + * the histogram definition. + */ +static inline int +nd_stats_value_index(const ND_STATS *stats, const int *indexes) +{ + int d; + int accum = 1; + int vdx = 0; + + for (d = 0; d < (int)(stats->ndims); d++) + { + int size = (int)(stats->size[d]); + if (indexes[d] < 0 || indexes[d] >= size) + return -1; + vdx += indexes[d] * accum; + accum *= size; + } + return vdx; +} + +/* + * Derive the histogram grid budget requested by PostgreSQL's ANALYZE machinery. + * The planner caps the cell count via three heuristics that take the requested + * attstattarget, the histogram dimensionality, and the underlying row count + * into account. Double precision arithmetic keeps the intermediate products in + * range so the cap behaves consistently across build architectures. + */ +static inline int +histogram_cell_budget(double total_rows, int ndims, int attstattarget) +{ + double budget; + double dims_cap; + double rows_cap; + double attstat; + double dims; + + if (ndims <= 0) + return 0; + + if (attstattarget <= 0) + attstattarget = 1; + + /* Requested resolution coming from PostgreSQL's ANALYZE knob. */ + attstat = (double)attstattarget; + dims = (double)ndims; + budget = pow(attstat, dims); + + /* Hard ceiling that keeps the statistics collector responsive. */ + dims_cap = (double)ndims * 100000.0; + if (budget > dims_cap) + budget = dims_cap; + + /* Small relations do not need a histogram that dwarfs the sample. */ + if (total_rows <= 0.0) + return 0; + + rows_cap = 10.0 * (double)ndims * total_rows; + if (rows_cap < 0.0) + rows_cap = 0.0; + + /* Keep intermediate computations in double precision before clamping. */ + if (rows_cap > (double)INT_MAX) + rows_cap = (double)INT_MAX; + + if (budget > rows_cap) + budget = rows_cap; + + if (budget >= (double)INT_MAX) + return INT_MAX; + if (budget <= 0.0) + return 0; + + return (int)budget; +} + +/* + * Compute the portion of 'target' covered by 'cover'. The caller supplies the + * dimensionality because ND_BOX always carries four slots. Degenerate volumes + * fold to zero, allowing the callers to detect slabs that ANALYZE sometimes + * emits for skewed datasets. + */ +static inline double +nd_box_ratio(const ND_BOX *cover, const ND_BOX *target, int ndims) +{ + int d; + bool fully_covered = true; + double ivol = 1.0; + double refvol = 1.0; + + for (d = 0; d < ndims; d++) + { + if (cover->max[d] <= target->min[d] || cover->min[d] >= target->max[d]) + return 0.0; /* Disjoint */ + + if (cover->min[d] > target->min[d] || cover->max[d] < target->max[d]) + fully_covered = false; + } + + if (fully_covered) + return 1.0; + + for (d = 0; d < ndims; d++) + { + double width = target->max[d] - target->min[d]; + double imin = Max(cover->min[d], target->min[d]); + double imax = Min(cover->max[d], target->max[d]); + double iwidth = Max(0.0, imax - imin); + + refvol *= width; + ivol *= iwidth; + } + + if (refvol == 0.0) + return refvol; + + return ivol / refvol; +} + +#endif /* POSTGIS_GSERIALIZED_ESTIMATE_SUPPORT_H */ ----------------------------------------------------------------------- Summary of changes: NEWS | 1 + configure.ac | 1 + postgis/cunit/Makefile.in | 43 ++++++ postgis/cunit/cu_tester.c | 173 ++++++++++++++++++++++++ postgis/gserialized_estimate.c | 192 +++----------------------- postgis/gserialized_estimate_support.h | 237 +++++++++++++++++++++++++++++++++ 6 files changed, 474 insertions(+), 173 deletions(-) create mode 100644 postgis/cunit/Makefile.in create mode 100644 postgis/cunit/cu_tester.c create mode 100644 postgis/gserialized_estimate_support.h hooks/post-receive -- PostGIS From trac at osgeo.org Fri Nov 7 10:35:20 2025 From: trac at osgeo.org (PostGIS) Date: Fri, 07 Nov 2025 18:35:20 -0000 Subject: [PostGIS] #5959: Query planner vastly underestimates number of rows matched by GIST index In-Reply-To: <049.94b8cd0de5de54d9df9629186637db25@osgeo.org> References: <049.94b8cd0de5de54d9df9629186637db25@osgeo.org> Message-ID: <064.2043e0c31d31776329d331e14b5be9bc@osgeo.org> #5959: Query planner vastly underestimates number of rows matched by GIST index ----------------------+--------------------------- Reporter: alexobs | Owner: pramsey Type: defect | Status: closed Priority: medium | Milestone: PostGIS 3.5.5 Component: postgis | Version: 3.5.x Resolution: fixed | Keywords: ----------------------+--------------------------- Comment (by Darafei Praliaskouski ): In [changeset:"b950728171f5d26fc3bbdd2f1cdf023cbffb0169/git" b9507281/git]: {{{#!CommitTicketReference repository="git" revision="b950728171f5d26fc3bbdd2f1cdf023cbffb0169" Prevent histogram target overflow when analysing massive tables Add CUnit tests for overflow scenarios Closes #5959 }}} -- Ticket URL: PostGIS The PostGIS Trac is used for bug, enhancement & task tracking, a user and developer wiki, and a view into the subversion code repository of PostGIS project. From trac at osgeo.org Fri Nov 7 10:35:20 2025 From: trac at osgeo.org (PostGIS) Date: Fri, 07 Nov 2025 18:35:20 -0000 Subject: [PostGIS] #5959: Query planner vastly underestimates number of rows matched by GIST index In-Reply-To: <049.94b8cd0de5de54d9df9629186637db25@osgeo.org> References: <049.94b8cd0de5de54d9df9629186637db25@osgeo.org> Message-ID: <064.6140dfd55976fea23b597deb3811b473@osgeo.org> #5959: Query planner vastly underestimates number of rows matched by GIST index ----------------------+--------------------------- Reporter: alexobs | Owner: pramsey Type: defect | Status: closed Priority: medium | Milestone: PostGIS 3.5.5 Component: postgis | Version: 3.5.x Resolution: fixed | Keywords: ----------------------+--------------------------- Comment (by Darafei Praliaskouski ): In [changeset:"a518412e516d4c7b216fd2f2797b20495d095e44/git" a518412/git]: {{{#!CommitTicketReference repository="git" revision="a518412e516d4c7b216fd2f2797b20495d095e44" Guard against histogram axis dimension underflow References #5959 References #5984 }}} -- Ticket URL: PostGIS The PostGIS Trac is used for bug, enhancement & task tracking, a user and developer wiki, and a view into the subversion code repository of PostGIS project. From trac at osgeo.org Fri Nov 7 10:35:21 2025 From: trac at osgeo.org (PostGIS) Date: Fri, 07 Nov 2025 18:35:21 -0000 Subject: [PostGIS] #5984: PostGIS selectivity is screwing up queries and forcing it to choose a spatial index when it's inappropriate In-Reply-To: <046.75677ad63d74709c6ed20baed0392698@osgeo.org> References: <046.75677ad63d74709c6ed20baed0392698@osgeo.org> Message-ID: <061.a6aa26143184c331a51f583be55d7451@osgeo.org> #5984: PostGIS selectivity is screwing up queries and forcing it to choose a spatial index when it's inappropriate -----------------------+--------------------------- Reporter: robe | Owner: komzpa Type: defect | Status: new Priority: critical | Milestone: PostGIS 3.4.5 Component: postgis | Version: 3.5.x Resolution: | Keywords: -----------------------+--------------------------- Comment (by Darafei Praliaskouski ): In [changeset:"a518412e516d4c7b216fd2f2797b20495d095e44/git" a518412/git]: {{{#!CommitTicketReference repository="git" revision="a518412e516d4c7b216fd2f2797b20495d095e44" Guard against histogram axis dimension underflow References #5959 References #5984 }}} -- Ticket URL: PostGIS The PostGIS Trac is used for bug, enhancement & task tracking, a user and developer wiki, and a view into the subversion code repository of PostGIS project. From trac at osgeo.org Fri Nov 7 10:35:21 2025 From: trac at osgeo.org (PostGIS) Date: Fri, 07 Nov 2025 18:35:21 -0000 Subject: [PostGIS] #5984: PostGIS selectivity is screwing up queries and forcing it to choose a spatial index when it's inappropriate In-Reply-To: <046.75677ad63d74709c6ed20baed0392698@osgeo.org> References: <046.75677ad63d74709c6ed20baed0392698@osgeo.org> Message-ID: <061.76c1d7aa7304dea9873a20129721cd06@osgeo.org> #5984: PostGIS selectivity is screwing up queries and forcing it to choose a spatial index when it's inappropriate -----------------------+--------------------------- Reporter: robe | Owner: komzpa Type: defect | Status: closed Priority: critical | Milestone: PostGIS 3.4.5 Component: postgis | Version: 3.5.x Resolution: fixed | Keywords: -----------------------+--------------------------- Changes (by Darafei Praliaskouski ): * resolution: => fixed * status: new => closed Comment: In [changeset:"21517f662368c90bbee0dfdebff3cb82059467fd/git" 21517f6/git]: {{{#!CommitTicketReference repository="git" revision="21517f662368c90bbee0dfdebff3cb82059467fd" Touch up NEWS Closes #5984 }}} -- Ticket URL: PostGIS The PostGIS Trac is used for bug, enhancement & task tracking, a user and developer wiki, and a view into the subversion code repository of PostGIS project. From git at osgeo.org Fri Nov 7 10:47:39 2025 From: git at osgeo.org (git at osgeo.org) Date: Fri, 7 Nov 2025 10:47:39 -0800 (PST) Subject: [SCM] PostGIS branch stable-3.5 updated. 3.5.4-5-g06fbe94ff Message-ID: <20251107184740.6179519FBB6@trac.osgeo.org> This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "PostGIS". The branch, stable-3.5 has been updated via 06fbe94ffe37c1e9d26328b7554c2050feaae1b6 (commit) via 6b8bae5486f12dc53634f1d7982186c6413d0ef5 (commit) from 949a623cd0afc9802ebb228d8dc7b477272b9b18 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 06fbe94ffe37c1e9d26328b7554c2050feaae1b6 Author: Darafei Praliaskouski Date: Thu Oct 30 03:46:26 2025 +0400 Guard against histogram axis dimension underflow References #5959 References #5984 diff --git a/NEWS b/NEWS index df71a295a..075318f84 100644 --- a/NEWS +++ b/NEWS @@ -6,11 +6,7 @@ PostgreSQL 12-18 required. GEOS 3.8+ required. Proj 6.1+ required. * Bug fixes * - - -* Bug Fixes * - - - #5959, Prevent histogram target overflow when analysing massive tables (Darafei Praliaskouski) + - #5959, #5984, Prevent histogram target overflow when analysing massive tables (Darafei Praliaskouski) PostGIS 3.5.4 @@ -3083,4 +3079,3 @@ PostGIS 0.1 - truely_inside() - rtree index support functions - gist index support functions - diff --git a/postgis/cunit/cu_tester.c b/postgis/cunit/cu_tester.c index b4dd46aa7..eb6ba5b0d 100644 --- a/postgis/cunit/cu_tester.c +++ b/postgis/cunit/cu_tester.c @@ -28,6 +28,7 @@ #include #include +#include #include #include "../gserialized_estimate_support.h" @@ -82,6 +83,23 @@ histogram_budget_clamps(void) CU_ASSERT_EQUAL(histogram_cell_budget((double)INT_MAX, 50000, INT_MAX), INT_MAX); } +static void +histogram_axis_allocation_guards(void) +{ + /* Baseline: evenly split a 10k target over two varying dimensions. */ + CU_ASSERT_EQUAL(histogram_axis_cells(10000, 2, 0.5), 100); + + /* Skewed axis ratios that collapse to tiny powers still return one cell. */ + CU_ASSERT_EQUAL(histogram_axis_cells(10000, 2, 1e-9), 1); + + /* Denormals, NaNs and negative ratios should not leak to the histogram. */ + CU_ASSERT_EQUAL(histogram_axis_cells(10000, 2, NAN), 1); + CU_ASSERT_EQUAL(histogram_axis_cells(10000, 2, -0.5), 1); + + /* Extremely aggressive ratios remain bounded by the square root of the budget. */ + CU_ASSERT_EQUAL(histogram_axis_cells(INT_MAX, 2, 1.0), (int)sqrt((double)INT_MAX * 2.0)); +} + static void nd_stats_indexing_behaviour(void) { @@ -138,6 +156,7 @@ main(void) goto cleanup; if (!CU_add_test(suite, "histogram budget clamps", histogram_budget_clamps) || + !CU_add_test(suite, "histogram axis guards", histogram_axis_allocation_guards) || !CU_add_test(suite, "nd_stats value index guards", nd_stats_indexing_behaviour) || !CU_add_test(suite, "nd_box ratio edge cases", nd_box_ratio_cases)) { diff --git a/postgis/gserialized_estimate.c b/postgis/gserialized_estimate.c index 072d60a1c..e561fd5dd 100644 --- a/postgis/gserialized_estimate.c +++ b/postgis/gserialized_estimate.c @@ -1516,11 +1516,10 @@ compute_gserialized_stats_mode(VacAttrStats *stats, AnalyzeAttrFetchFunc fetchfu * Scale the target cells number by the # of dims and ratio, * then take the appropriate root to get the estimated number of cells * on this axis (eg, pow(0.5) for 2d, pow(0.333) for 3d, pow(0.25) for 4d) - */ - histo_size[d] = (int)pow((double)histo_cells_target * histo_ndims * edge_ratio, 1/(double)histo_ndims); - /* If something goes awry, just give this dim one slot */ - if ( ! histo_size[d] ) - histo_size[d] = 1; + * The dedicated helper clamps pathological floating point inputs so we + * do not resurrect the NaN propagation reported in #5959 on amd64. + */ + histo_size[d] = histogram_axis_cells(histo_cells_target, histo_ndims, edge_ratio); } histo_cells_new *= histo_size[d]; } diff --git a/postgis/gserialized_estimate_support.h b/postgis/gserialized_estimate_support.h index 0d3a23d75..6b372a43e 100644 --- a/postgis/gserialized_estimate_support.h +++ b/postgis/gserialized_estimate_support.h @@ -151,6 +151,46 @@ histogram_cell_budget(double total_rows, int ndims, int attstattarget) return (int)budget; } +/* + * Allocate histogram buckets along a single axis in proportion to the observed + * density variation. The caller passes in the global histogram target along + * with the number of axes that exhibited variation in the sampled data and the + * relative contribution of the current axis (edge_ratio). Earlier versions + * evaluated the pow() call directly in the caller, which exposed the planner to + * NaN propagation on some amd64 builds when the ratio was denormal or negative + * (see #5959). Keeping the calculation in one place allows us to clamp the + * inputs and provide a predictable fallback for problematic floating point + * combinations. + */ +static inline int +histogram_axis_cells(int histo_cells_target, int histo_ndims, double edge_ratio) +{ + double scaled; + double axis_cells; + + if (histo_cells_target <= 0 || histo_ndims <= 0) + return 1; + + if (!(edge_ratio > 0.0) || !isfinite(edge_ratio)) + return 1; + + scaled = (double)histo_cells_target * (double)histo_ndims * edge_ratio; + if (!(scaled > 0.0) || !isfinite(scaled)) + return 1; + + axis_cells = pow(scaled, 1.0 / (double)histo_ndims); + if (!(axis_cells > 0.0) || !isfinite(axis_cells)) + return 1; + + if (axis_cells >= (double)INT_MAX) + return INT_MAX; + + if (axis_cells <= 1.0) + return 1; + + return (int)axis_cells; +} + /* * Compute the portion of 'target' covered by 'cover'. The caller supplies the * dimensionality because ND_BOX always carries four slots. Degenerate volumes commit 6b8bae5486f12dc53634f1d7982186c6413d0ef5 Author: Darafei Praliaskouski Date: Thu Oct 30 02:55:59 2025 +0400 Prevent histogram target overflow when analysing massive tables Add CUnit tests for overflow scenarios Closes #5959 diff --git a/NEWS b/NEWS index 72aaf5e0d..df71a295a 100644 --- a/NEWS +++ b/NEWS @@ -8,6 +8,10 @@ PostgreSQL 12-18 required. GEOS 3.8+ required. Proj 6.1+ required. +* Bug Fixes * + + - #5959, Prevent histogram target overflow when analysing massive tables (Darafei Praliaskouski) + PostGIS 3.5.4 2025/10/16 diff --git a/configure.ac b/configure.ac index a60828c9d..0411756d7 100644 --- a/configure.ac +++ b/configure.ac @@ -1905,6 +1905,7 @@ AC_CONFIG_FILES([GNUmakefile libpgcommon/Makefile libpgcommon/cunit/Makefile postgis/Makefile + postgis/cunit/Makefile postgis/sqldefines.h sfcgal/Makefile $SFCGAL_MAKEFILE_LIST diff --git a/postgis/cunit/Makefile.in b/postgis/cunit/Makefile.in new file mode 100644 index 000000000..483e4ca10 --- /dev/null +++ b/postgis/cunit/Makefile.in @@ -0,0 +1,43 @@ +# ********************************************************************** +# * +# * PostGIS - Spatial Types for PostgreSQL +# * http://postgis.net +# * +# * Copyright 2025 Darafei Praliaskouski +# * +# * This is free software; you can redistribute and/or modify it under +# * the terms of the GNU General Public Licence. See the COPYING file. +# * +# ********************************************************************** + +srcdir = @srcdir@ +top_builddir = @top_builddir@ + +CC=@CC@ +LIBTOOL=@LIBTOOL@ +CFLAGS = @CFLAGS@ @CPPFLAGS@ @PGSQL_BE_CPPFLAGS@ @CUNIT_CPPFLAGS@ -I.. -I$(top_builddir) -I at top_srcdir@/liblwgeom -I at top_builddir@/liblwgeom -I at top_srcdir@/libpgcommon -I at top_builddir@/libpgcommon +LDFLAGS = @CUNIT_LDFLAGS@ -lm + +VPATH = $(srcdir) + +OBJS = cu_tester.o + +# Build the standalone histogram helper tester. +all: cu_tester + +# Execute the suite directly; no installation step is required. +check: all + $(LIBTOOL) --mode=execute ./cu_tester + +# Link the tester with libtool; all helper code is header-only. +cu_tester: $(OBJS) + $(LIBTOOL) --mode=link $(CC) $(CFLAGS) -o $@ $(OBJS) $(LDFLAGS) + +%.o: %.c + $(CC) $(CFLAGS) -c -o $@ $< + +clean: + rm -f $(OBJS) cu_tester + +clobber distclean: clean + rm -f Makefile diff --git a/postgis/cunit/cu_tester.c b/postgis/cunit/cu_tester.c new file mode 100644 index 000000000..b4dd46aa7 --- /dev/null +++ b/postgis/cunit/cu_tester.c @@ -0,0 +1,154 @@ +/********************************************************************** + * + * PostGIS - Spatial Types for PostgreSQL + * http://postgis.net + * + * This file is part of PostGIS + * + * PostGIS is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 2 of the License, or + * (at your option) any later version. + * + * PostGIS is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with PostGIS. If not, see . + * + ********************************************************************** + * + * Copyright 2025 (C) Darafei Praliaskouski + * + **********************************************************************/ + +#include "postgres.h" + +#include +#include +#include + +#include "../gserialized_estimate_support.h" + +static ND_BOX +make_box(float minx, float miny, float minz, float minm, float maxx, float maxy, float maxz, float maxm) +{ + ND_BOX box; + + memset(&box, 0, sizeof(box)); + box.min[0] = minx; + box.min[1] = miny; + box.min[2] = minz; + box.min[3] = minm; + box.max[0] = maxx; + box.max[1] = maxy; + box.max[2] = maxz; + box.max[3] = maxm; + return box; +} + +static void +histogram_budget_clamps(void) +{ + /* Zero or negative row counts disable histogram construction. */ + CU_ASSERT_EQUAL(histogram_cell_budget(0.0, 2, 100), 0); + CU_ASSERT_EQUAL(histogram_cell_budget(-1.0, 4, 100), 0); + + /* Degenerate dimensionality cannot allocate histogram space. */ + CU_ASSERT_EQUAL(histogram_cell_budget(1000.0, 0, 100), 0); + + /* Matches the classic pow(attstattarget, ndims) path. */ + CU_ASSERT_EQUAL(histogram_cell_budget(1e6, 2, 100), 10000); + CU_ASSERT_EQUAL(histogram_cell_budget(1e6, 3, 50), 125000); + + /* attstattarget^ndims exceeds ndims * 100000 and must be clamped. */ + CU_ASSERT_EQUAL(histogram_cell_budget(1e6, 4, 50), 400000); + + /* attstattarget<=0 is normalised to the smallest viable target. */ + CU_ASSERT_EQUAL(histogram_cell_budget(1e6, 2, 0), 1); + + /* Row clamp shrinks the grid for small relations. */ + CU_ASSERT_EQUAL(histogram_cell_budget(1.0, 2, 100), 20); + + /* Large tables now preserve the dimensional cap instead of overflowing. */ + CU_ASSERT_EQUAL(histogram_cell_budget(1.5e8, 2, 100), 10000); + + /* Regression for #5984: huge attstat targets stabilise instead of wrapping. */ + CU_ASSERT_EQUAL(histogram_cell_budget(5e6, 2, 10000), 200000); + + /* Trigger the INT_MAX guard once both other caps exceed it. */ + CU_ASSERT_EQUAL(histogram_cell_budget((double)INT_MAX, 50000, INT_MAX), INT_MAX); +} + +static void +nd_stats_indexing_behaviour(void) +{ + ND_STATS stats; + const int good_index[ND_DIMS] = {1, 2, 0, 0}; + const int bad_index[ND_DIMS] = {1, 5, 0, 0}; + + memset(&stats, 0, sizeof(stats)); + stats.ndims = 3; + stats.size[0] = 4.0f; + stats.size[1] = 5.0f; + stats.size[2] = 3.0f; + + /* Three-dimensional index (x=1, y=2, z=0) collapses into 1 + 2 * 4. */ + CU_ASSERT_EQUAL(nd_stats_value_index(&stats, good_index), 1 + 2 * 4); + /* Any request outside the histogram bounds triggers a guard. */ + CU_ASSERT_EQUAL(nd_stats_value_index(&stats, bad_index), -1); + + /* Regression for #5959: ndims higher than populated sizes still honours guards. */ + stats.ndims = 4; + CU_ASSERT_EQUAL(nd_stats_value_index(&stats, good_index), -1); +} + +static void +nd_box_ratio_cases(void) +{ + ND_BOX covering = make_box(0.0f, 0.0f, 0.0f, 0.0f, 2.0f, 2.0f, 2.0f, 0.0f); + ND_BOX interior = make_box(0.5f, 0.5f, 0.5f, 0.0f, 1.5f, 1.5f, 1.5f, 0.0f); + ND_BOX partial = make_box(0.0f, 0.0f, 0.0f, 0.0f, 0.5f, 0.5f, 0.5f, 0.0f); + ND_BOX target = make_box(0.0f, 0.0f, 0.0f, 0.0f, 1.0f, 1.0f, 1.0f, 0.0f); + ND_BOX flat = make_box(0.0f, 0.0f, 0.0f, 0.0f, 1.0f, 1.0f, 0.0f, 0.0f); + ND_BOX touch = make_box(2.0f, 0.0f, 0.0f, 0.0f, 3.0f, 1.0f, 1.0f, 0.0f); + + /* Full coverage should evaluate to one regardless of the extra extent. */ + CU_ASSERT_DOUBLE_EQUAL(nd_box_ratio(&covering, &interior, 3), 1.0, 1e-12); + /* A shared octant carries one eighth of the reference volume. */ + CU_ASSERT_DOUBLE_EQUAL(nd_box_ratio(&partial, &target, 3), 0.125, 1e-12); + /* Degenerate slabs have zero volume in three dimensions. */ + CU_ASSERT_DOUBLE_EQUAL(nd_box_ratio(&covering, &flat, 3), 0.0, 1e-12); + /* Boxes that only touch along a face should not count as overlap. */ + CU_ASSERT_DOUBLE_EQUAL(nd_box_ratio(&covering, &touch, 3), 0.0, 1e-12); +} + +int +main(void) +{ + CU_pSuite suite; + unsigned int failures = 0; + if (CU_initialize_registry() != CUE_SUCCESS) + return CU_get_error(); + + suite = CU_add_suite("gserialized_histogram_helpers", NULL, NULL); + if (!suite) + goto cleanup; + + if (!CU_add_test(suite, "histogram budget clamps", histogram_budget_clamps) || + !CU_add_test(suite, "nd_stats value index guards", nd_stats_indexing_behaviour) || + !CU_add_test(suite, "nd_box ratio edge cases", nd_box_ratio_cases)) + { + goto cleanup; + } + + CU_basic_set_mode(CU_BRM_VERBOSE); + CU_basic_run_tests(); + +cleanup: + failures = CU_get_number_of_tests_failed(); + CU_cleanup_registry(); + return failures == 0 ? CUE_SUCCESS : 1; +} diff --git a/postgis/gserialized_estimate.c b/postgis/gserialized_estimate.c index 24984e9d7..072d60a1c 100644 --- a/postgis/gserialized_estimate.c +++ b/postgis/gserialized_estimate.c @@ -19,11 +19,10 @@ ********************************************************************** * * Copyright 2012 (C) Paul Ramsey + * Copyright 2025 (C) Darafei Praliaskouski * **********************************************************************/ - - /********************************************************************** THEORY OF OPERATION @@ -112,10 +111,12 @@ dimensionality cases. (2D geometry) &&& (3D column), etc. #include "stringbuffer.h" #include "liblwgeom.h" #include "lwgeodetic.h" -#include "lwgeom_pg.h" /* For debugging macros. */ +#include "lwgeom_pg.h" /* For debugging macros. */ #include "gserialized_gist.h" /* For index common functions */ +#include "gserialized_estimate_support.h" #include +#include #if HAVE_IEEEFP_H #include #endif @@ -144,8 +145,7 @@ Datum _postgis_gserialized_stats(PG_FUNCTION_ARGS); /* Local prototypes */ static Oid table_get_spatial_index(Oid tbl_oid, int16 attnum, int *key_type, int16 *idx_attnum); -static GBOX * spatial_index_read_extent(Oid idx_oid, int idx_att_num, int key_type); - +static GBOX *spatial_index_read_extent(Oid idx_oid, int idx_att_num, int key_type); /* Other prototypes */ float8 gserialized_joinsel_internal(PlannerInfo *root, List *args, JoinType jointype, int mode); @@ -186,13 +186,6 @@ Datum geometry_estimated_extent(PG_FUNCTION_ARGS); */ #define SDFACTOR 3.25 -/** -* The maximum number of dimensions our code can handle. -* We'll use this to statically allocate a bunch of -* arrays below. -*/ -#define ND_DIMS 4 - /** * Minimum width of a dimension that we'll bother trying to * compute statistics on. Bearing in mind we have no control @@ -219,68 +212,6 @@ Datum geometry_estimated_extent(PG_FUNCTION_ARGS); #define FALLBACK_ND_SEL 0.2 #define FALLBACK_ND_JOINSEL 0.3 -/** -* N-dimensional box type for calculations, to avoid doing -* explicit axis conversions from GBOX in all calculations -* at every step. -*/ -typedef struct ND_BOX_T -{ - float4 min[ND_DIMS]; - float4 max[ND_DIMS]; -} ND_BOX; - -/** -* N-dimensional box index type -*/ -typedef struct ND_IBOX_T -{ - int min[ND_DIMS]; - int max[ND_DIMS]; -} ND_IBOX; - - -/** -* N-dimensional statistics structure. Well, actually -* four-dimensional, but set up to handle arbitrary dimensions -* if necessary (really, we just want to get the 2,3,4-d cases -* into one shared piece of code). -*/ -typedef struct ND_STATS_T -{ - /* Dimensionality of the histogram. */ - float4 ndims; - - /* Size of n-d histogram in each dimension. */ - float4 size[ND_DIMS]; - - /* Lower-left (min) and upper-right (max) spatial bounds of histogram. */ - ND_BOX extent; - - /* How many rows in the table itself? */ - float4 table_features; - - /* How many rows were in the sample that built this histogram? */ - float4 sample_features; - - /* How many not-Null/Empty features were in the sample? */ - float4 not_null_features; - - /* How many features actually got sampled in the histogram? */ - float4 histogram_features; - - /* How many cells in histogram? (sizex*sizey*sizez*sizem) */ - float4 histogram_cells; - - /* How many cells did those histogram features cover? */ - /* Since we are pro-rating coverage, this number should */ - /* now always equal histogram_features */ - float4 cells_covered; - - /* Variable length # of floats for histogram */ - float4 value[1]; -} ND_STATS; - typedef struct { /* Saved state from std_typanalyze() */ AnalyzeAttrComputeStatsFunc std_compute_stats; @@ -318,13 +249,12 @@ text_p_get_mode(const text *txt) char *modestr; if (VARSIZE_ANY_EXHDR(txt) <= 0) return mode; - modestr = (char*)VARDATA(txt); - if ( modestr[0] == 'N' ) + modestr = (char *)VARDATA(txt); + if (modestr[0] == 'N') mode = 0; return mode; } - /** * Integer comparison function for qsort */ @@ -372,7 +302,7 @@ total_double(const double *vals, int nvals) int i; float total = 0; /* Calculate total */ - for ( i = 0; i < nvals; i++ ) + for (i = 0; i < nvals; i++) total += vals[i]; return total; @@ -425,33 +355,6 @@ stddev(const int *vals, int nvals) } #endif /* POSTGIS_DEBUG_LEVEL >= 3 */ -/** -* Given a position in the n-d histogram (i,j,k) return the -* position in the 1-d values array. -*/ -static int -nd_stats_value_index(const ND_STATS *stats, int *indexes) -{ - int d; - int accum = 1, vdx = 0; - - /* Calculate the index into the 1-d values array that the (i,j,k,l) */ - /* n-d histogram coordinate implies. */ - /* index = x + y * sizex + z * sizex * sizey + m * sizex * sizey * sizez */ - for ( d = 0; d < (int)(stats->ndims); d++ ) - { - int size = (int)(stats->size[d]); - if ( indexes[d] < 0 || indexes[d] >= size ) - { - POSTGIS_DEBUGF(3, " bad index at (%d, %d)", indexes[0], indexes[1]); - return -1; - } - vdx += indexes[d] * accum; - accum *= size; - } - return vdx; -} - /** * Convert an #ND_BOX to a JSON string for printing */ @@ -722,50 +625,6 @@ nd_box_overlap(const ND_STATS *nd_stats, const ND_BOX *nd_box, ND_IBOX *nd_ibox) return true; } -/** -* Returns the proportion of b2 that is covered by b1. -*/ -static inline double -nd_box_ratio(const ND_BOX *b1, const ND_BOX *b2, int ndims) -{ - int d; - bool covered = true; - double ivol = 1.0; - double vol2 = 1.0; - - for ( d = 0 ; d < ndims; d++ ) - { - if ( b1->max[d] <= b2->min[d] || b1->min[d] >= b2->max[d] ) - return 0.0; /* Disjoint */ - - if ( b1->min[d] > b2->min[d] || b1->max[d] < b2->max[d] ) - covered = false; - } - - if ( covered ) - return 1.0; - - for ( d = 0; d < ndims; d++ ) - { - double width2 = b2->max[d] - b2->min[d]; - double imin, imax, iwidth; - - vol2 *= width2; - - imin = Max(b1->min[d], b2->min[d]); - imax = Min(b1->max[d], b2->max[d]); - iwidth = imax - imin; - iwidth = Max(0.0, iwidth); - - ivol *= iwidth; - } - - if ( vol2 == 0.0 ) - return vol2; - - return ivol / vol2; -} - /* How many bins shall we use in figuring out the distribution? */ #define MAX_NUM_BINS 50 #define BIN_MIN_SIZE 10 @@ -894,9 +753,9 @@ nd_increment(ND_IBOX *ibox, int ndims, int *counter) { int d = 0; - while ( d < ndims ) + while (d < ndims) { - if ( counter[d] < ibox->max[d] ) + if (counter[d] < ibox->max[d]) { counter[d] += 1; break; @@ -905,7 +764,7 @@ nd_increment(ND_IBOX *ibox, int ndims, int *counter) d++; } /* That's it, cannot increment any more! */ - if ( d == ndims ) + if (d == ndims) return false; /* Increment complete! */ @@ -1321,9 +1180,9 @@ gserialized_joinsel_internal(PlannerInfo *root, List *args, JoinType jointype, i PG_FUNCTION_INFO_V1(gserialized_gist_joinsel); Datum gserialized_gist_joinsel(PG_FUNCTION_ARGS) { - PlannerInfo *root = (PlannerInfo *) PG_GETARG_POINTER(0); + PlannerInfo *root = (PlannerInfo *)PG_GETARG_POINTER(0); /* Oid operator = PG_GETARG_OID(1); */ - List *args = (List *) PG_GETARG_POINTER(2); + List *args = (List *)PG_GETARG_POINTER(2); JoinType jointype = (JoinType) PG_GETARG_INT16(3); int mode = PG_GETARG_INT32(4); @@ -1512,22 +1371,13 @@ compute_gserialized_stats_mode(VacAttrStats *stats, AnalyzeAttrFetchFunc fetchfu #endif } - /* - * We'll build a histogram having stats->attr->attstattarget - * (default 100) cells on each side, within reason... - * we'll use ndims*100000 as the maximum number of cells. - * Also, if we're sampling a relatively small table, we'll try to ensure that - * we have a smaller grid. - */ #if POSTGIS_PGSQL_VERSION >= 170 - histo_cells_target = (int)pow((double)(stats->attstattarget), (double)ndims); POSTGIS_DEBUGF(3, " stats->attstattarget: %d", stats->attstattarget); + histo_cells_target = histogram_cell_budget(total_rows, ndims, stats->attstattarget); #else - histo_cells_target = (int)pow((double)(stats->attr->attstattarget), (double)ndims); POSTGIS_DEBUGF(3, " stats->attr->attstattarget: %d", stats->attr->attstattarget); + histo_cells_target = histogram_cell_budget(total_rows, ndims, stats->attr->attstattarget); #endif - histo_cells_target = Min(histo_cells_target, ndims * 100000); - histo_cells_target = Min(histo_cells_target, (int)(10 * ndims * total_rows)); POSTGIS_DEBUGF(3, " target # of histogram cells: %d", histo_cells_target); /* If there's no useful features, we can't work out stats */ @@ -1836,8 +1686,6 @@ compute_gserialized_stats_mode(VacAttrStats *stats, AnalyzeAttrFetchFunc fetchfu return; } - - /** * In order to do useful selectivity calculations in both 2-D and N-D * modes, we actually have to generate two stats objects, one for 2-D @@ -1875,7 +1723,6 @@ compute_gserialized_stats(VacAttrStats *stats, AnalyzeAttrFetchFunc fetchfunc, } } - /** * This function will be called when the ANALYZE command is run * on a column of the "geometry" or "geography" type. diff --git a/postgis/gserialized_estimate_support.h b/postgis/gserialized_estimate_support.h new file mode 100644 index 000000000..0d3a23d75 --- /dev/null +++ b/postgis/gserialized_estimate_support.h @@ -0,0 +1,197 @@ +/********************************************************************** + * + * PostGIS - Spatial Types for PostgreSQL + * http://postgis.net + * + * This file is part of PostGIS + * + * PostGIS is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 2 of the License, or + * (at your option) any later version. + * + * PostGIS is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with PostGIS. If not, see . + * + ********************************************************************** + * + * Internal helpers shared between the gserialized selectivity + * implementation and the unit tests. + * + * Keeping the routines header-only ensures the planner code and the + * harness evaluate the exact same floating-point flows without the + * cross-object plumbing that previously complicated maintenance. + * Nothing here is installed; the header is meant for + * gserialized_estimate.c and for the dedicated CUnit suite only. + * + ********************************************************************** + * + * Copyright 2012 (C) Paul Ramsey + * Copyright 2025 (C) Darafei Praliaskouski + * + **********************************************************************/ + +#ifndef POSTGIS_GSERIALIZED_ESTIMATE_SUPPORT_H +#define POSTGIS_GSERIALIZED_ESTIMATE_SUPPORT_H + +#include "postgres.h" + +#include +#include + +/* The maximum number of dimensions our statistics code supports. */ +#define ND_DIMS 4 + +/* Lightweight n-dimensional box representation for selectivity math. */ +typedef struct ND_BOX_T { + float4 min[ND_DIMS]; + float4 max[ND_DIMS]; +} ND_BOX; + +/* Integer counterpart used for histogram cell iteration. */ +typedef struct ND_IBOX_T { + int min[ND_DIMS]; + int max[ND_DIMS]; +} ND_IBOX; + +/* On-disk representation of the histogram emitted by ANALYZE. */ +typedef struct ND_STATS_T { + float4 ndims; + float4 size[ND_DIMS]; + ND_BOX extent; + float4 table_features; + float4 sample_features; + float4 not_null_features; + float4 histogram_features; + float4 histogram_cells; + float4 cells_covered; + float4 value[1]; +} ND_STATS; + +/* + * Return the flattened index for the histogram coordinate expressed by + * 'indexes'. A negative result signals that one of the axes fell outside + * the histogram definition. + */ +static inline int +nd_stats_value_index(const ND_STATS *stats, const int *indexes) +{ + int d; + int accum = 1; + int vdx = 0; + + for (d = 0; d < (int)(stats->ndims); d++) + { + int size = (int)(stats->size[d]); + if (indexes[d] < 0 || indexes[d] >= size) + return -1; + vdx += indexes[d] * accum; + accum *= size; + } + return vdx; +} + +/* + * Derive the histogram grid budget requested by PostgreSQL's ANALYZE machinery. + * The planner caps the cell count via three heuristics that take the requested + * attstattarget, the histogram dimensionality, and the underlying row count + * into account. Double precision arithmetic keeps the intermediate products in + * range so the cap behaves consistently across build architectures. + */ +static inline int +histogram_cell_budget(double total_rows, int ndims, int attstattarget) +{ + double budget; + double dims_cap; + double rows_cap; + double attstat; + double dims; + + if (ndims <= 0) + return 0; + + if (attstattarget <= 0) + attstattarget = 1; + + /* Requested resolution coming from PostgreSQL's ANALYZE knob. */ + attstat = (double)attstattarget; + dims = (double)ndims; + budget = pow(attstat, dims); + + /* Hard ceiling that keeps the statistics collector responsive. */ + dims_cap = (double)ndims * 100000.0; + if (budget > dims_cap) + budget = dims_cap; + + /* Small relations do not need a histogram that dwarfs the sample. */ + if (total_rows <= 0.0) + return 0; + + rows_cap = 10.0 * (double)ndims * total_rows; + if (rows_cap < 0.0) + rows_cap = 0.0; + + /* Keep intermediate computations in double precision before clamping. */ + if (rows_cap > (double)INT_MAX) + rows_cap = (double)INT_MAX; + + if (budget > rows_cap) + budget = rows_cap; + + if (budget >= (double)INT_MAX) + return INT_MAX; + if (budget <= 0.0) + return 0; + + return (int)budget; +} + +/* + * Compute the portion of 'target' covered by 'cover'. The caller supplies the + * dimensionality because ND_BOX always carries four slots. Degenerate volumes + * fold to zero, allowing the callers to detect slabs that ANALYZE sometimes + * emits for skewed datasets. + */ +static inline double +nd_box_ratio(const ND_BOX *cover, const ND_BOX *target, int ndims) +{ + int d; + bool fully_covered = true; + double ivol = 1.0; + double refvol = 1.0; + + for (d = 0; d < ndims; d++) + { + if (cover->max[d] <= target->min[d] || cover->min[d] >= target->max[d]) + return 0.0; /* Disjoint */ + + if (cover->min[d] > target->min[d] || cover->max[d] < target->max[d]) + fully_covered = false; + } + + if (fully_covered) + return 1.0; + + for (d = 0; d < ndims; d++) + { + double width = target->max[d] - target->min[d]; + double imin = Max(cover->min[d], target->min[d]); + double imax = Min(cover->max[d], target->max[d]); + double iwidth = Max(0.0, imax - imin); + + refvol *= width; + ivol *= iwidth; + } + + if (refvol == 0.0) + return refvol; + + return ivol / refvol; +} + +#endif /* POSTGIS_GSERIALIZED_ESTIMATE_SUPPORT_H */ ----------------------------------------------------------------------- Summary of changes: NEWS | 3 +- configure.ac | 1 + postgis/cunit/Makefile.in | 43 ++++++ postgis/cunit/cu_tester.c | 173 ++++++++++++++++++++++++ postgis/gserialized_estimate.c | 192 +++----------------------- postgis/gserialized_estimate_support.h | 237 +++++++++++++++++++++++++++++++++ 6 files changed, 474 insertions(+), 175 deletions(-) create mode 100644 postgis/cunit/Makefile.in create mode 100644 postgis/cunit/cu_tester.c create mode 100644 postgis/gserialized_estimate_support.h hooks/post-receive -- PostGIS From trac at osgeo.org Fri Nov 7 10:47:41 2025 From: trac at osgeo.org (PostGIS) Date: Fri, 07 Nov 2025 18:47:41 -0000 Subject: [PostGIS] #5959: Query planner vastly underestimates number of rows matched by GIST index In-Reply-To: <049.94b8cd0de5de54d9df9629186637db25@osgeo.org> References: <049.94b8cd0de5de54d9df9629186637db25@osgeo.org> Message-ID: <064.d2eee37640cff0715821166508d4f34a@osgeo.org> #5959: Query planner vastly underestimates number of rows matched by GIST index ----------------------+--------------------------- Reporter: alexobs | Owner: pramsey Type: defect | Status: closed Priority: medium | Milestone: PostGIS 3.5.5 Component: postgis | Version: 3.5.x Resolution: fixed | Keywords: ----------------------+--------------------------- Comment (by Darafei Praliaskouski ): In [changeset:"6b8bae5486f12dc53634f1d7982186c6413d0ef5/git" 6b8bae5/git]: {{{#!CommitTicketReference repository="git" revision="6b8bae5486f12dc53634f1d7982186c6413d0ef5" Prevent histogram target overflow when analysing massive tables Add CUnit tests for overflow scenarios Closes #5959 }}} -- Ticket URL: PostGIS The PostGIS Trac is used for bug, enhancement & task tracking, a user and developer wiki, and a view into the subversion code repository of PostGIS project. From trac at osgeo.org Fri Nov 7 10:47:42 2025 From: trac at osgeo.org (PostGIS) Date: Fri, 07 Nov 2025 18:47:42 -0000 Subject: [PostGIS] #5959: Query planner vastly underestimates number of rows matched by GIST index In-Reply-To: <049.94b8cd0de5de54d9df9629186637db25@osgeo.org> References: <049.94b8cd0de5de54d9df9629186637db25@osgeo.org> Message-ID: <064.660538655b3babee6d9e3fcf85e50934@osgeo.org> #5959: Query planner vastly underestimates number of rows matched by GIST index ----------------------+--------------------------- Reporter: alexobs | Owner: pramsey Type: defect | Status: closed Priority: medium | Milestone: PostGIS 3.5.5 Component: postgis | Version: 3.5.x Resolution: fixed | Keywords: ----------------------+--------------------------- Comment (by Darafei Praliaskouski ): In [changeset:"06fbe94ffe37c1e9d26328b7554c2050feaae1b6/git" 06fbe94/git]: {{{#!CommitTicketReference repository="git" revision="06fbe94ffe37c1e9d26328b7554c2050feaae1b6" Guard against histogram axis dimension underflow References #5959 References #5984 }}} -- Ticket URL: PostGIS The PostGIS Trac is used for bug, enhancement & task tracking, a user and developer wiki, and a view into the subversion code repository of PostGIS project. From trac at osgeo.org Fri Nov 7 10:47:42 2025 From: trac at osgeo.org (PostGIS) Date: Fri, 07 Nov 2025 18:47:42 -0000 Subject: [PostGIS] #5984: PostGIS selectivity is screwing up queries and forcing it to choose a spatial index when it's inappropriate In-Reply-To: <046.75677ad63d74709c6ed20baed0392698@osgeo.org> References: <046.75677ad63d74709c6ed20baed0392698@osgeo.org> Message-ID: <061.8c874fa5f26f469abf7be1dd187c7b8b@osgeo.org> #5984: PostGIS selectivity is screwing up queries and forcing it to choose a spatial index when it's inappropriate -----------------------+--------------------------- Reporter: robe | Owner: komzpa Type: defect | Status: closed Priority: critical | Milestone: PostGIS 3.4.5 Component: postgis | Version: 3.5.x Resolution: fixed | Keywords: -----------------------+--------------------------- Comment (by Darafei Praliaskouski ): In [changeset:"06fbe94ffe37c1e9d26328b7554c2050feaae1b6/git" 06fbe94/git]: {{{#!CommitTicketReference repository="git" revision="06fbe94ffe37c1e9d26328b7554c2050feaae1b6" Guard against histogram axis dimension underflow References #5959 References #5984 }}} -- Ticket URL: PostGIS The PostGIS Trac is used for bug, enhancement & task tracking, a user and developer wiki, and a view into the subversion code repository of PostGIS project. From git at osgeo.org Fri Nov 7 13:15:09 2025 From: git at osgeo.org (git at osgeo.org) Date: Fri, 7 Nov 2025 13:15:09 -0800 (PST) Subject: [SCM] PostGIS branch master updated. 3.6.0rc2-176-gc7cd09187 Message-ID: <20251107211509.C81993DC@trac.osgeo.org> This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "PostGIS". The branch, master has been updated via c7cd09187e922254625a89f75cd724fa5f3612f5 (commit) from b185e01e95149f17cbfe486f3906dc6f921108e9 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit c7cd09187e922254625a89f75cd724fa5f3612f5 Author: Paul Ramsey Date: Fri Nov 7 13:14:52 2025 -0800 Ignore generated makefile diff --git a/.gitignore b/.gitignore index f9d05e3ca..9b374cdea 100644 --- a/.gitignore +++ b/.gitignore @@ -144,6 +144,7 @@ postgis/uninstall_legacy.sql postgis/uninstall_postgis.sql postgis/vector_tile.pb-c.c postgis/vector_tile.pb-c.h +postgis/cunit/Makefile raster/loader/Makefile raster/loader/raster2pgsql raster/Makefile ----------------------------------------------------------------------- Summary of changes: .gitignore | 1 + 1 file changed, 1 insertion(+) hooks/post-receive -- PostGIS From git at osgeo.org Fri Nov 7 13:30:35 2025 From: git at osgeo.org (git at osgeo.org) Date: Fri, 7 Nov 2025 13:30:35 -0800 (PST) Subject: [SCM] PostGIS branch master updated. 3.6.0rc2-178-g8e7b03874 Message-ID: <20251107213035.5F71916A8@trac.osgeo.org> This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "PostGIS". The branch, master has been updated via 8e7b0387401764518cf28b072bb114b19514fa58 (commit) via ac07b86e9d8a2161a1a01544389d33c76051d8be (commit) from c7cd09187e922254625a89f75cd724fa5f3612f5 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 8e7b0387401764518cf28b072bb114b19514fa58 Merge: c7cd09187 ac07b86e9 Author: Paul Ramsey Date: Fri Nov 7 13:30:24 2025 -0800 Merge branch 'master-6012' commit ac07b86e9d8a2161a1a01544389d33c76051d8be Author: Paul Ramsey Date: Fri Nov 7 13:23:05 2025 -0800 fix memory leak, references #6012 diff --git a/liblwgeom/lwcircstring.c b/liblwgeom/lwcircstring.c index 71a1556bf..ec0d05b27 100644 --- a/liblwgeom/lwcircstring.c +++ b/liblwgeom/lwcircstring.c @@ -34,14 +34,13 @@ void printLWCIRCSTRING(LWCIRCSTRING *curve); void lwcircstring_release(LWCIRCSTRING *lwcirc); char lwcircstring_same(const LWCIRCSTRING *me, const LWCIRCSTRING *you); -LWCIRCSTRING *lwcircstring_from_lwpointarray(int32_t srid, uint32_t npoints, LWPOINT **points); +LWCIRCSTRING *lwcircstring_from_lwpointarray(int32_t srid, uint32_t npoints, const LWPOINT **points); LWCIRCSTRING *lwcircstring_from_lwmpoint(int32_t srid, LWMPOINT *mpoint); LWCIRCSTRING *lwcircstring_addpoint(LWCIRCSTRING *curve, LWPOINT *point, uint32_t where); LWCIRCSTRING *lwcircstring_removepoint(LWCIRCSTRING *curve, uint32_t index); void lwcircstring_setPoint4d(LWCIRCSTRING *curve, uint32_t index, POINT4D *newpoint); - /* * Construct a new LWCIRCSTRING. points will *NOT* be copied * use SRID=SRID_UNKNOWN for unknown SRID (will have 8bit type's S = 0) @@ -138,49 +137,44 @@ lwcircstring_same(const LWCIRCSTRING *me, const LWCIRCSTRING *you) * LWCIRCSTRING dimensions are large enough to host all input dimensions. */ LWCIRCSTRING * -lwcircstring_from_lwpointarray(int32_t srid, uint32_t npoints, LWPOINT **points) +lwcircstring_from_lwpointarray(int32_t srid, uint32_t npoints, const LWPOINT **points) { - int zmflag=0; uint32_t i; POINTARRAY *pa; - uint8_t *newpoints, *ptr; - size_t ptsize, size; + int has_z = 0, has_m = 0; + POINT4D pt; /* * Find output dimensions, check integrity */ for (i = 0; i < npoints; i++) { - if (points[i]->type != POINTTYPE) + const LWGEOM *lwg = (LWGEOM*)points[i]; + if (lwg->type != POINTTYPE) { - lwerror("lwcurve_from_lwpointarray: invalid input type: %s", - lwtype_name(points[i]->type)); + lwerror("%s: invalid input type: %s", __func__, lwtype_name(lwg->type)); return NULL; } - if (FLAGS_GET_Z(points[i]->flags)) zmflag |= 2; - if (FLAGS_GET_M(points[i]->flags)) zmflag |= 1; - if (zmflag == 3) break; + has_z |= lwgeom_has_z(lwg); + has_m |= lwgeom_has_m(lwg); + if (has_z && has_m) break; } - if (zmflag == 0) ptsize = 2 * sizeof(double); - else if (zmflag == 3) ptsize = 4 * sizeof(double); - else ptsize = 3 * sizeof(double); - /* * Allocate output points array */ - size = ptsize * npoints; - newpoints = lwalloc(size); - memset(newpoints, 0, size); + pa = ptarray_construct(has_z, has_m, npoints); - ptr = newpoints; for (i = 0; i < npoints; i++) { - size = ptarray_point_size(points[i]->point); - memcpy(ptr, getPoint_internal(points[i]->point, 0), size); - ptr += ptsize; + const LWPOINT *lwpt = points[i]; + if (!getPoint4d_p(lwpt->point, 0, &pt)) + { + lwerror("%s: failed getPoint4d_p", __func__); + return NULL; + } + ptarray_set_point4d(pa, i, &pt); } - pa = ptarray_construct_reference_data(zmflag&2, zmflag&1, npoints, newpoints); return lwcircstring_construct(srid, NULL, pa); } ----------------------------------------------------------------------- Summary of changes: liblwgeom/lwcircstring.c | 42 ++++++++++++++++++------------------------ 1 file changed, 18 insertions(+), 24 deletions(-) hooks/post-receive -- PostGIS From trac at osgeo.org Fri Nov 7 13:30:48 2025 From: trac at osgeo.org (PostGIS) Date: Fri, 07 Nov 2025 21:30:48 -0000 Subject: [PostGIS] #6012: Memory leak in function lwcircstring_from_lwpointarray In-Reply-To: <050.c313b633ffae4edc5682f59a3f451b19@osgeo.org> References: <050.c313b633ffae4edc5682f59a3f451b19@osgeo.org> Message-ID: <065.00406b034bcd4f9a8a57ff7596831518@osgeo.org> #6012: Memory leak in function lwcircstring_from_lwpointarray -----------------------+--------------------------- Reporter: ezimanyi | Owner: pramsey Type: defect | Status: new Priority: medium | Milestone: PostGIS 3.6.1 Component: postgis | Version: 3.5.x Resolution: | Keywords: -----------------------+--------------------------- Comment (by Paul Ramsey ): In [changeset:"ac07b86e9d8a2161a1a01544389d33c76051d8be/git" ac07b86e/git]: {{{#!CommitTicketReference repository="git" revision="ac07b86e9d8a2161a1a01544389d33c76051d8be" fix memory leak, references #6012 }}} -- Ticket URL: PostGIS The PostGIS Trac is used for bug, enhancement & task tracking, a user and developer wiki, and a view into the subversion code repository of PostGIS project. From git at osgeo.org Fri Nov 7 13:33:19 2025 From: git at osgeo.org (git at osgeo.org) Date: Fri, 7 Nov 2025 13:33:19 -0800 (PST) Subject: [SCM] PostGIS branch stable-3.6 updated. 3.6.0-29-g9b1071fcd Message-ID: <20251107213320.1E424253D@trac.osgeo.org> This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "PostGIS". The branch, stable-3.6 has been updated via 9b1071fcdb51c53af15942a6bd31cb51e57d602b (commit) via b85bce7cbeb15595e9e4f59cb7098f58c0274706 (commit) from 21517f662368c90bbee0dfdebff3cb82059467fd (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 9b1071fcdb51c53af15942a6bd31cb51e57d602b Author: Paul Ramsey Date: Fri Nov 7 13:33:10 2025 -0800 News entry for #6012 diff --git a/NEWS b/NEWS index 7a06e124f..7e316b9ee 100644 --- a/NEWS +++ b/NEWS @@ -15,6 +15,7 @@ PostGIS 3.6.1 authored: Andrey Borodin (Yandex), reported by Sergey Bobrov (Kaspersky) - #5754, ST_ForcePolygonCCW reverses lines (Paul Ramsey) - #5959, #5984, Prevent histogram target overflow when analysing massive tables (Darafei Praliaskouski) + - #6012, Remove memory leak from lwcircstring_from_lwpointarray (Paul Ramsey) PostGIS 3.6.0 commit b85bce7cbeb15595e9e4f59cb7098f58c0274706 Author: Paul Ramsey Date: Fri Nov 7 13:23:05 2025 -0800 fix memory leak, references #6012 diff --git a/liblwgeom/lwcircstring.c b/liblwgeom/lwcircstring.c index 71a1556bf..ec0d05b27 100644 --- a/liblwgeom/lwcircstring.c +++ b/liblwgeom/lwcircstring.c @@ -34,14 +34,13 @@ void printLWCIRCSTRING(LWCIRCSTRING *curve); void lwcircstring_release(LWCIRCSTRING *lwcirc); char lwcircstring_same(const LWCIRCSTRING *me, const LWCIRCSTRING *you); -LWCIRCSTRING *lwcircstring_from_lwpointarray(int32_t srid, uint32_t npoints, LWPOINT **points); +LWCIRCSTRING *lwcircstring_from_lwpointarray(int32_t srid, uint32_t npoints, const LWPOINT **points); LWCIRCSTRING *lwcircstring_from_lwmpoint(int32_t srid, LWMPOINT *mpoint); LWCIRCSTRING *lwcircstring_addpoint(LWCIRCSTRING *curve, LWPOINT *point, uint32_t where); LWCIRCSTRING *lwcircstring_removepoint(LWCIRCSTRING *curve, uint32_t index); void lwcircstring_setPoint4d(LWCIRCSTRING *curve, uint32_t index, POINT4D *newpoint); - /* * Construct a new LWCIRCSTRING. points will *NOT* be copied * use SRID=SRID_UNKNOWN for unknown SRID (will have 8bit type's S = 0) @@ -138,49 +137,44 @@ lwcircstring_same(const LWCIRCSTRING *me, const LWCIRCSTRING *you) * LWCIRCSTRING dimensions are large enough to host all input dimensions. */ LWCIRCSTRING * -lwcircstring_from_lwpointarray(int32_t srid, uint32_t npoints, LWPOINT **points) +lwcircstring_from_lwpointarray(int32_t srid, uint32_t npoints, const LWPOINT **points) { - int zmflag=0; uint32_t i; POINTARRAY *pa; - uint8_t *newpoints, *ptr; - size_t ptsize, size; + int has_z = 0, has_m = 0; + POINT4D pt; /* * Find output dimensions, check integrity */ for (i = 0; i < npoints; i++) { - if (points[i]->type != POINTTYPE) + const LWGEOM *lwg = (LWGEOM*)points[i]; + if (lwg->type != POINTTYPE) { - lwerror("lwcurve_from_lwpointarray: invalid input type: %s", - lwtype_name(points[i]->type)); + lwerror("%s: invalid input type: %s", __func__, lwtype_name(lwg->type)); return NULL; } - if (FLAGS_GET_Z(points[i]->flags)) zmflag |= 2; - if (FLAGS_GET_M(points[i]->flags)) zmflag |= 1; - if (zmflag == 3) break; + has_z |= lwgeom_has_z(lwg); + has_m |= lwgeom_has_m(lwg); + if (has_z && has_m) break; } - if (zmflag == 0) ptsize = 2 * sizeof(double); - else if (zmflag == 3) ptsize = 4 * sizeof(double); - else ptsize = 3 * sizeof(double); - /* * Allocate output points array */ - size = ptsize * npoints; - newpoints = lwalloc(size); - memset(newpoints, 0, size); + pa = ptarray_construct(has_z, has_m, npoints); - ptr = newpoints; for (i = 0; i < npoints; i++) { - size = ptarray_point_size(points[i]->point); - memcpy(ptr, getPoint_internal(points[i]->point, 0), size); - ptr += ptsize; + const LWPOINT *lwpt = points[i]; + if (!getPoint4d_p(lwpt->point, 0, &pt)) + { + lwerror("%s: failed getPoint4d_p", __func__); + return NULL; + } + ptarray_set_point4d(pa, i, &pt); } - pa = ptarray_construct_reference_data(zmflag&2, zmflag&1, npoints, newpoints); return lwcircstring_construct(srid, NULL, pa); } ----------------------------------------------------------------------- Summary of changes: NEWS | 1 + liblwgeom/lwcircstring.c | 42 ++++++++++++++++++------------------------ 2 files changed, 19 insertions(+), 24 deletions(-) hooks/post-receive -- PostGIS From trac at osgeo.org Fri Nov 7 13:33:21 2025 From: trac at osgeo.org (PostGIS) Date: Fri, 07 Nov 2025 21:33:21 -0000 Subject: [PostGIS] #6012: Memory leak in function lwcircstring_from_lwpointarray In-Reply-To: <050.c313b633ffae4edc5682f59a3f451b19@osgeo.org> References: <050.c313b633ffae4edc5682f59a3f451b19@osgeo.org> Message-ID: <065.14f88141fce01bb514944e81dc5c6da6@osgeo.org> #6012: Memory leak in function lwcircstring_from_lwpointarray -----------------------+--------------------------- Reporter: ezimanyi | Owner: pramsey Type: defect | Status: new Priority: medium | Milestone: PostGIS 3.6.1 Component: postgis | Version: 3.5.x Resolution: | Keywords: -----------------------+--------------------------- Comment (by Paul Ramsey ): In [changeset:"b85bce7cbeb15595e9e4f59cb7098f58c0274706/git" b85bce7/git]: {{{#!CommitTicketReference repository="git" revision="b85bce7cbeb15595e9e4f59cb7098f58c0274706" fix memory leak, references #6012 }}} -- Ticket URL: PostGIS The PostGIS Trac is used for bug, enhancement & task tracking, a user and developer wiki, and a view into the subversion code repository of PostGIS project. From trac at osgeo.org Fri Nov 7 13:33:46 2025 From: trac at osgeo.org (PostGIS) Date: Fri, 07 Nov 2025 21:33:46 -0000 Subject: [PostGIS] #6012: Memory leak in function lwcircstring_from_lwpointarray In-Reply-To: <050.c313b633ffae4edc5682f59a3f451b19@osgeo.org> References: <050.c313b633ffae4edc5682f59a3f451b19@osgeo.org> Message-ID: <065.8ca3679a312792b29c13502396527fb1@osgeo.org> #6012: Memory leak in function lwcircstring_from_lwpointarray -----------------------+--------------------------- Reporter: ezimanyi | Owner: pramsey Type: defect | Status: closed Priority: medium | Milestone: PostGIS 3.6.1 Component: postgis | Version: 3.5.x Resolution: fixed | Keywords: -----------------------+--------------------------- Changes (by pramsey): * resolution: => fixed * status: new => closed -- Ticket URL: PostGIS The PostGIS Trac is used for bug, enhancement & task tracking, a user and developer wiki, and a view into the subversion code repository of PostGIS project. From git at osgeo.org Sat Nov 8 00:03:46 2025 From: git at osgeo.org (git at osgeo.org) Date: Sat, 8 Nov 2025 00:03:46 -0800 (PST) Subject: [SCM] PostGIS Buildbots; Jenkins jobs and instructions for setting up winnie and debbie bots. branch master updated. fd680d0534a8f33863c577f6e6a8f1d90084bbd4 Message-ID: <20251108080346.8A00642D6@trac.osgeo.org> This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "PostGIS Buildbots; Jenkins jobs and instructions for setting up winnie and debbie bots.". The branch, master has been updated via fd680d0534a8f33863c577f6e6a8f1d90084bbd4 (commit) from 79e81aca3cf11cbb77ef13ce371392e57a6830ae (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit fd680d0534a8f33863c577f6e6a8f1d90084bbd4 Author: Regina Obe Date: Sat Nov 8 03:03:36 2025 -0500 build script for pgsphere diff --git a/windows/mingw64/scripts/build_pgsphere.sh b/windows/mingw64/scripts/build_pgsphere.sh new file mode 100644 index 0000000..7ebaed8 --- /dev/null +++ b/windows/mingw64/scripts/build_pgsphere.sh @@ -0,0 +1,89 @@ +#export OS_BUILD=64 +#export GCC_TYPE= +#export PG_VER=18 +export PROJECTS=/projects +export PGPATH=${PROJECTS}/postgresql/rel/pg${PG_VER}w${OS_BUILD}${GCC_TYPE} +export PGPATHEDB=${PGPATH}edb +export PATH="/mingw/bin:/bin:/c/Windows/system32" +export PATH="${PGPATH}/bin:${PGPATH}/lib:${PATH}" +#export PGPORT=5455 +export PGUSER=postgres +export SOURCES=/sources +export HEALPIX_VER=3.82.1 +export REL_HEALPIX=/projects/pgsphere/rel_healpix_${HEALPIX_VER}w${OS_BUILD}${GCC_TYPE} + +export CFITSIO_VER=4.6.3 +export REL_CFITSIO=/projects/pgsphere/rel_cfitsio_${CFITSIO_VER}w${OS_BUILD}${GCC_TYPE} +cd /projects +cd pgsphere/pgsphere + + +# build cfitsio +if false; then + wget -N https://heasarc.gsfc.nasa.gov/FTP/software/fitsio/c/cfitsio-${CFITSIO_VER}.tar.gz + export PATH="${PGPATH}/bin:${PGPATH}/lib:${PATH}" + rm -rf cfitsio-${CFITSIO_VER} + tar xvf cfitsio-${CFITSIO_VER}.tar.gz + cd cfitsio-${CFITSIO_VER} + mkdir build + #./configure --prefix=${REL_CFITSIO} + cmake -Bbuild -G"Ninja" -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX:PATH=REL_CFITSIO + #cmake --build . + cmake --install . + exit +fi +# build healpix +if false; then + export PATH="${PGPATH}/bin:${PGPATH}/lib:${REL_CFITSIO}/bin:${PATH}" + wget -N https://sourceforge.net/projects/healpix/files/Healpix_3.82/healpix_cxx-${HEALPIX_VER}.tar.gz/download -O healpix_cxx-${HEALPIX_VER}.tar.gz + tar xvf healpix_cxx-${HEALPIX_VER}.tar.gz + cd healpix_cxx-${HEALPIX_VER} + ./configure --prefix=${REL_HEALPIX} + exit +fi +#export PGSPHERE_VER=master +#export PGSPHERE_VER=1.5.1 + +#git clean -fdx +#git reset --hard +#git checkout ${PGSPHERE_VER} +#git pull +#clear +export USE_PGXS=1 +#git checkout 1.4.0 +#git checkout ${PGSPHERE_VER} +git pull +git clean -fdx +make clean +#export USE_PGXS=1 +make USE_HEALPIX=0 +rm ${PGPATH}/lib/pg_sphere.dll +make USE_HEALPIX=0 install +cp pg_sphere.dll ${PGPATHEDB}/lib/ + +make installcheck USE_HEALPIX=0 + +export RELDIR=${PROJECTS}/pgsphere/binaries/ +export RELVERDIR=pgsphere-pg${PG_VER}-binaries-${PGSPHERE_VER}w${OS_BUILD}${GCC_TYPE} +outdir="${RELDIR}/${RELVERDIR}" +package="${outdir}.zip" +verfile="${outdir}/pg_sphere_version.txt" +rm -rf $outdir +rm -f $package +strip *.dll +mkdir -p ${outdir} +mkdir -p ${outdir}/lib +mkdir -p ${outdir}/share/extension +echo "PGSPHERE VERSION: ${PGSPHERE_VER} https://github.com/postgrespro/pgsphere.git $(git rev-parse HEAD)" >> $verfile +echo "USE_HEALPIX=0" >> $verfile +cp COPYRIGHT.pg_sphere ${outdir} +cp *.dll ${outdir}/lib/ +cp *.control ${outdir}/share/extension/ +cp *.sql ${outdir}/share/extension/ +cd ${RELDIR} +cp README ${outdir} +zip -r $package ${RELVERDIR} +md5sum ${RELVERDIR}.zip > ${package}.md5 + +cp $package ${PROJECTS}/postgis/win_web/download/windows/pg${PG_VER}/buildbot/extras +cp ${package}.md5 ${PROJECTS}/postgis/win_web/download/windows/pg${PG_VER}/buildbot/extras \ No newline at end of file ----------------------------------------------------------------------- Summary of changes: windows/mingw64/scripts/build_pgsphere.sh | 89 +++++++++++++++++++++++++++++++ 1 file changed, 89 insertions(+) create mode 100644 windows/mingw64/scripts/build_pgsphere.sh hooks/post-receive -- PostGIS Buildbots; Jenkins jobs and instructions for setting up winnie and debbie bots. From git at osgeo.org Sat Nov 8 00:43:53 2025 From: git at osgeo.org (git at osgeo.org) Date: Sat, 8 Nov 2025 00:43:53 -0800 (PST) Subject: [SCM] PostGIS Buildbots; Jenkins jobs and instructions for setting up winnie and debbie bots. branch master updated. c960912a7a628aafacec039863d0b0377d6edd05 Message-ID: <20251108084353.6A54D488A@trac.osgeo.org> This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "PostGIS Buildbots; Jenkins jobs and instructions for setting up winnie and debbie bots.". The branch, master has been updated via c960912a7a628aafacec039863d0b0377d6edd05 (commit) from fd680d0534a8f33863c577f6e6a8f1d90084bbd4 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit c960912a7a628aafacec039863d0b0377d6edd05 Author: Regina Obe Date: Sat Nov 8 03:43:48 2025 -0500 Fix install in edb diff --git a/windows/mingw64/scripts/build_pgsphere.sh b/windows/mingw64/scripts/build_pgsphere.sh index 7ebaed8..77a3381 100644 --- a/windows/mingw64/scripts/build_pgsphere.sh +++ b/windows/mingw64/scripts/build_pgsphere.sh @@ -60,6 +60,8 @@ make USE_HEALPIX=0 rm ${PGPATH}/lib/pg_sphere.dll make USE_HEALPIX=0 install cp pg_sphere.dll ${PGPATHEDB}/lib/ +cp pg_sphere*.sql ${PGPATHEDB}/share/extension/ +cp pg_sphere.control ${PGPATHEDB}/share/extension/ make installcheck USE_HEALPIX=0 ----------------------------------------------------------------------- Summary of changes: windows/mingw64/scripts/build_pgsphere.sh | 2 ++ 1 file changed, 2 insertions(+) hooks/post-receive -- PostGIS Buildbots; Jenkins jobs and instructions for setting up winnie and debbie bots.