[Wps-discuss] WPS Benchmarking session at FOSS4G 2014 in PDX

Sun Apr 6 13:59:58 PDT 2014

Gerald, 

I welcome the idea, even it's much more demanding, the original shootout idea
always was. Comparism is good.

I would suggest to use OSGeo-live, for reference distribution, since most of you
are already in, environment is known to everybody and to add new project  (like
PyWPS) should be relatively easy.

Jachym

On Fri, Apr 04, 2014 at 06:36:49PM +0200, Fenoy Gerald wrote:
> Dear OGC WPS friends,
> 
> I would like to invite all of you to discuss the possibility of getting a WPS
> Benchmark Session during the FOSS4G which will occur in PDX this year. I would
> like to try in this first email to present my personal vision of this session
> and then will wait for your confirmations for submitting the abstract, for
> joining the effort, any comments and feedbacks. I am personally expecting this
> presentation to happen during the FOSS4G this year and as a new Sherif Star
> owner, I have the responsibility that it happen but I will need you. This
> effort requires your help and can only happen thanks to your involvement.
> 
> The main goal of this session would be to promote the WPS use and the
> implementations available in the OpenSource world. We should be able to
> provide tangible comparisons of the implementations, such as dependencies
> required, ressources usage, response time, compliancy and so on. 
> 
> The expectations for an attendee of this talk should be to have a good
> overview of good / bad points, possible limitations of available
> implementation, have an overview of what can distinguish an implementation
> from another and what are the ressources / dependencies required by any
> implementation.
> 
> I would like to mention that we should be fully open to any commercial
> implementation until they are willing to join the effort and contribute
> effectively as other participants will be asked to. 
> 
> Anyway, the first question I would like to ask is, what do you think of making
> a presentation of the WPS Benchmark Results at FOSS4G this year ?
> 
> Supposing now that the answer is "we should have this presentation!", then I
> have cooked some kind of plan that I would like to discuss with you [1]. In
> short, the idea is to have a central server to access every implementation and
> to run tests against on demand, each participant to the effort will have a
> user account to fix / update his setup and can run the tests again. Now, let
> start the long version ...
> 
> So, supposing we decide to submit the presentation and we go for the benchmark
> work, I would like to ask for all the WPS implementation developer teams to
> decide if they want to participate to this effort and I invite them to join
> and send an email for confirmation to wps-discuss [2] mailing list for their
> participation and for providing a contact mail address for the team.
> 
> As I introduced to you already, I hope to be able to provide tangible results
> out of this work for being able to compare implementations. I think that this
> imply having a dedicated infrastructure to handle all the benchmarking work
> and run the tests locally, removing any network issue which may occur during
> the testing procedure, fixing issue with xml validation by using local cache,
> things like that... So, I would like to propose to host a dedicated GNU /
> Linux server (available, initially, from april 15th 2014 to 2015) and setup
> any GNU / Linux environment which may be required for testing purpose. I
> didn’t planned to use anything else than GNU / Linux as I think that all the
> implementations can run on this Operating System, this can be changed if it is
> required by any team.
> 
> This infrastructure would be used to run tests / scenarios that we have to
> define together. I planned to offer one environment per project involved in
> the effort and another global environment to gather all the solutions
> together. This will help us to first easily determine requirements /
> ressources usage for each project and see how they can be setup on the same
> host. Unfortunately, it will also imply more work for installer as they will
> have to setup twice the instance, in their environment then in the global one. 
> 
> The contact person would be used to share ip server addresses, ssh port, user
> and password to access the dedicated environment and run setup before  running
> tests. I think that it can help to use a fresh host to setup solutions from
> each participating team, but it will imply involvement of the contact person
> here. Indeed, once he have access to his test environment, he will be asked to
> make the setup on the server and make sure that everything is in place, then
> he will ask for the tests to be run against the new setup or run them on his
> own.
> 
> As I think that this effort should run for long time to give a chance for each
> team to potentially fix things after testing procedure / scenarios have been
> run, I propose to host if for 12 months. This way, even after the FOSS4G each
> project can continue to take advantages of the tests procedures / scenarios we
> defined together. Anyway, this make the effort too much global by now as we
> are only discussing to start this effort and submit an abstract to FOSS4G to
> confirm that this effort will occur. I would also like to make the tests
> available and accessible as soon as possible, that is why I planned to give
> access to the infrastructure so early. We may probably start by setting up an
> updated version of the tests we run against implementations for the FOSS4G
> 2011 in Denver.
> 
> In parallel of the described process, I think that a web site should be setup
> to gather informations about the ongoing effort and probably in future offer
> some eyes candies to access the informations. At the beginning, the OSGeo Wiki
> should be enough [1].
> 
> Obviously, all that I said in this email is open to discussion and engage only
> me. I simply preferred to prepare something that I can propose to you (as
> clear as possible at this stage) before asking about the FOSS4G presentation,
> I hope you don’t mind that I have started preparing something already and that
> you don’t have anything totally opposed to the proposal made here. One more
> time, I would like to make this an open effort where all can be decided
> through discussions.
> 
> I hope to hear back from you and we can make good work together for the
> benefit of all the participating teams (if any) and to have the presentation
> done in FOSS4G 2014, Best regards, 
> 
> (sorry for any cross posting)
> 
> [1] http://wiki.osgeo.org/wiki/WPS_benchmark_2014 [2]
> http://lists.osgeo.org/mailman/listinfo/wps-discuss
> 
> 
> Gérald Fenoy gerald.fenoy at geolabs.fr
> 

-- 
Jachym Cepicky
URL: http://geosense.cz
e-mail: jachym.cepicky at geosense cz
PGP: http://les-ejk.cz/pgp/JachymCepicky.pgp
@jachymc