[postgis-devel] performance test suite

[Even Rouault]

On mardi 1 mai 2018 19:05:12 CEST Björn Harrtell wrote:
> Hi devs,
> 
> In recent times I've been pondering on about how to make a sensible test
> suite specifically for performance. Hacking/extending run_test.pl to
> accommodate for this has been the only suggested path forward but to me
> it's a dead end mostly because of perl (sorry)-
> 
> The reason why this has become a to me apparent missing thing is due to:
> 
> 1. My own work on https://trac.osgeo.org/postgis/ticket/4076.
> 
> 2. The recently discovered large performance regression of ST_Union tracked
> by https://trac.osgeo.org/postgis/ticket/4075. Even if it's in GEOS and
> could perhaps be performance tested there, I think it would not be wrong to
> also performance test ST_Union without consideration of underlying
> implementation.
> 
> Any additional thoughts on the subject? do it in perl or don't do it? :)

When last year I worked on openjpeg improvements, one task I tackled was to try to put in 
place performance tests that would run when continuous integration triggers so as to be 
able to detect precisely when performance loss or improvement would happen. The only 
semi reliable way I found with Travis-CI (hope nobody is following some thread on 
board at list.osgeo.org !) is to do a build and run on the same machine before (previous 
commit or last stable version) and after the change, with elimination of outliers etc. In the 
openjpeg context, it turned out to be not reliable enough to be used to flag a build as KO 
performance-wise, since I was chasing for changes in performance of  1% to 2% and so that 
was within the noise of the measurement (noise due to the non deterministic nature of 
modern OS + probably the effect of cloud based machines). If you just need to detech 
regressions of speed > 10% that could probably be reliable enough.

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/postgis-devel/attachments/20180501/a69d2f4a/attachment.html>