[postgis-devel] performance test suite

[Daniel Baston]

I agree that this would be very useful, not only for catching regressions
but also to help us promote the performance improvements make it in to each
release. We itemize performance improvements in the changelog, but we don't
generally quantify what they mean for typical use cases. It would be nice
to say by upgrading to release 2.5, typical point-in-polygon queries are
improved by 20%, K-means is improved by X%, etc.

To keep the perl/python to a minimum, could we rely on pg_stat_statements
to do the bulk of the work for us? So it be something as simple as:

1) a script that loads or generates test data
2) a SQL file that runs a bunch of queries capturing typical usages of
PostGIS
3) something that parses the output of pg_stat_statements

Dan

On Tue, May 1, 2018 at 4:42 PM, Regina Obe <lr at pcorp.us> wrote:

> Bjorn,
>
>
>
> Oh you are a man after my own heart.  Yes definitely.  Performance testing
> is a very weak spot in our testing.  I hate finding out about this when
> users complain J
>
>
>
> I think starting it off as a separate project is a good idea but I'd
> really love to see it eventually as part of PostGIS core that say we can
> flip on and have enabled for some bots or when we are about to release.
> How we keep record of timings etc, seems to me a bot end thing the testing
> bot reporting to some mothership database.
>
>
>
> As to whether it should be done in perl or something else – to be honest
> Perl scares the shit out of me.  Python sadly I haven't warmed up to
> either.  I always feel like I'm fumbling thru a mine field with both.  Okay
> that's an exaggeration.
>
>
>
> But then again Perl is a dependency we are used to having, so whatever you
> do ideally shouldn't add any crazy dependencies and if additional
> dependencies – a dependency that can run on all platforms.  It's okay to
> have extra dependencies as long as they are not required for regular
> testing.  I think Komzpa already put in some logic for code coverage
> testing via lcov for example, which is fine since it's not a requirement.
>
>
>
>
>
> Thanks,
>
> Regina
>
>
>
>
>
> *From:* postgis-devel [mailto:postgis-devel-bounces at lists.osgeo.org] *On
> Behalf Of *Paul Ramsey
> *Sent:* Tuesday, May 01, 2018 1:08 PM
> *To:* Björn Harrtell <bjorn at wololo.org>; PostGIS Development Discussion <
> postgis-devel at lists.osgeo.org>
> *Subject:* Re: [postgis-devel] performance test suite
>
>
>
> I think perhaps “do it as a separate project”. It’s going to be complex,
> it’s going to be brittle, it’s going to eventually break and I’d rather not
> have it sitting around broken inside the main source tree. The only way to
> find the regressions is going to be longitudinally testing and keeping
> track of numbers over time, so it’ll be quite a complex piece of work.
>
>
>
> P
>
>
>
> On May 1, 2018, at 10:05 AM, Björn Harrtell <bjorn.harrtell at gmail.com>
> wrote:
>
>
>
> Hi devs,
>
>
>
> In recent times I've been pondering on about how to make a sensible test
> suite specifically for performance. Hacking/extending run_test.pl to
> accommodate for this has been the only suggested path forward but to me
> it's a dead end mostly because of perl (sorry)-
>
>
>
> The reason why this has become a to me apparent missing thing is due to:
>
>
>
> 1. My own work on https://trac.osgeo.org/postgis/ticket/4076.
>
>
>
> 2. The recently discovered large performance regression of ST_Union
> tracked by https://trac.osgeo.org/postgis/ticket/4075. Even if it's in
> GEOS and could perhaps be performance tested there, I think it would not be
> wrong to also performance test ST_Union without consideration of underlying
> implementation.
>
>
>
> Any additional thoughts on the subject? do it in perl or don't do it? :)
>
>
>
> Regards,
>
>
>
> /Björn
>
> _______________________________________________
> postgis-devel mailing list
> postgis-devel at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/postgis-devel
>
>
>
> _______________________________________________
> postgis-devel mailing list
> postgis-devel at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/postgis-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/postgis-devel/attachments/20180501/6f2cd477/attachment.html>