[Wps-discuss] WPS Benchmarking session at FOSS4G 2014 in PDX

Mon Apr 7 00:25:14 PDT 2014

Hello Gerald, all

As the shootout gathered teams but also quiet a few FOSS4G attendees in
2011 and 2013, i think it is a very good idea to organize it from now,
submit a WPS community abstract and get a dedicated time slot at FOSS4G PDX
(so Gerald can give the Sherif Star to someone else too !). It may be more
work if we tackle it using a slightly different approach this year but yet
possible if we start early.

I like Gerald's idea of setting up a open infrastructure for the project
teams, and i agree with Jachym that it should host OSGeoLive or a ready to
use environment that every team knows.
Once every teams will be onboard (perhaps starting from here
http://wiki.osgeo.org/wiki/WPS_benchmark_2014#Projects_and_people ?), i
agree we should work on scenarios, may be installations and services tuning
too, and finally on simple solutions to render and compare the test results
as well.

I would be glad to help for the shootout organization, first by hacking the
wiki and then working on visualization of tests results.

What do other projects/people think ?
Get your hats and colts gentlemen :)

Best regards,

Nicolas

2014-04-06 22:59 GMT+02:00 Jachym Cepicky <jachym.cepicky at geosense.cz>:

> Gerald,
>
> I welcome the idea, even it's much more demanding, the original shootout
> idea
> always was. Comparism is good.
>
> I would suggest to use OSGeo-live, for reference distribution, since most
> of you
> are already in, environment is known to everybody and to add new project
>  (like
> PyWPS) should be relatively easy.
>
> Jachym
>
> On Fri, Apr 04, 2014 at 06:36:49PM +0200, Fenoy Gerald wrote:
> > Dear OGC WPS friends,
> >
> > I would like to invite all of you to discuss the possibility of getting
> a WPS
> > Benchmark Session during the FOSS4G which will occur in PDX this year. I
> would
> > like to try in this first email to present my personal vision of this
> session
> > and then will wait for your confirmations for submitting the abstract,
> for
> > joining the effort, any comments and feedbacks. I am personally
> expecting this
> > presentation to happen during the FOSS4G this year and as a new Sherif
> Star
> > owner, I have the responsibility that it happen but I will need you. This
> > effort requires your help and can only happen thanks to your involvement.
> >
> > The main goal of this session would be to promote the WPS use and the
> > implementations available in the OpenSource world. We should be able to
> > provide tangible comparisons of the implementations, such as dependencies
> > required, ressources usage, response time, compliancy and so on.
> >
> > The expectations for an attendee of this talk should be to have a good
> > overview of good / bad points, possible limitations of available
> > implementation, have an overview of what can distinguish an
> implementation
> > from another and what are the ressources / dependencies required by any
> > implementation.
> >
> > I would like to mention that we should be fully open to any commercial
> > implementation until they are willing to join the effort and contribute
> > effectively as other participants will be asked to.
> >
> > Anyway, the first question I would like to ask is, what do you think of
> making
> > a presentation of the WPS Benchmark Results at FOSS4G this year ?
> >
> > Supposing now that the answer is "we should have this presentation!",
> then I
> > have cooked some kind of plan that I would like to discuss with you [1].
> In
> > short, the idea is to have a central server to access every
> implementation and
> > to run tests against on demand, each participant to the effort will have
> a
> > user account to fix / update his setup and can run the tests again. Now,
> let
> > start the long version ...
> >
> > So, supposing we decide to submit the presentation and we go for the
> benchmark
> > work, I would like to ask for all the WPS implementation developer teams
> to
> > decide if they want to participate to this effort and I invite them to
> join
> > and send an email for confirmation to wps-discuss [2] mailing list for
> their
> > participation and for providing a contact mail address for the team.
> >
> > As I introduced to you already, I hope to be able to provide tangible
> results
> > out of this work for being able to compare implementations. I think that
> this
> > imply having a dedicated infrastructure to handle all the benchmarking
> work
> > and run the tests locally, removing any network issue which may occur
> during
> > the testing procedure, fixing issue with xml validation by using local
> cache,
> > things like that... So, I would like to propose to host a dedicated GNU /
> > Linux server (available, initially, from april 15th 2014 to 2015) and
> setup
> > any GNU / Linux environment which may be required for testing purpose. I
> > didn't planned to use anything else than GNU / Linux as I think that all
> the
> > implementations can run on this Operating System, this can be changed if
> it is
> > required by any team.
> >
> > This infrastructure would be used to run tests / scenarios that we have
> to
> > define together. I planned to offer one environment per project involved
> in
> > the effort and another global environment to gather all the solutions
> > together. This will help us to first easily determine requirements /
> > ressources usage for each project and see how they can be setup on the
> same
> > host. Unfortunately, it will also imply more work for installer as they
> will
> > have to setup twice the instance, in their environment then in the
> global one.
> >
> > The contact person would be used to share ip server addresses, ssh port,
> user
> > and password to access the dedicated environment and run setup before
>  running
> > tests. I think that it can help to use a fresh host to setup solutions
> from
> > each participating team, but it will imply involvement of the contact
> person
> > here. Indeed, once he have access to his test environment, he will be
> asked to
> > make the setup on the server and make sure that everything is in place,
> then
> > he will ask for the tests to be run against the new setup or run them on
> his
> > own.
> >
> > As I think that this effort should run for long time to give a chance
> for each
> > team to potentially fix things after testing procedure / scenarios have
> been
> > run, I propose to host if for 12 months. This way, even after the FOSS4G
> each
> > project can continue to take advantages of the tests procedures /
> scenarios we
> > defined together. Anyway, this make the effort too much global by now as
> we
> > are only discussing to start this effort and submit an abstract to
> FOSS4G to
> > confirm that this effort will occur. I would also like to make the tests
> > available and accessible as soon as possible, that is why I planned to
> give
> > access to the infrastructure so early. We may probably start by setting
> up an
> > updated version of the tests we run against implementations for the
> FOSS4G
> > 2011 in Denver.
> >
> > In parallel of the described process, I think that a web site should be
> setup
> > to gather informations about the ongoing effort and probably in future
> offer
> > some eyes candies to access the informations. At the beginning, the
> OSGeo Wiki
> > should be enough [1].
> >
> > Obviously, all that I said in this email is open to discussion and
> engage only
> > me. I simply preferred to prepare something that I can propose to you (as
> > clear as possible at this stage) before asking about the FOSS4G
> presentation,
> > I hope you don't mind that I have started preparing something already
> and that
> > you don't have anything totally opposed to the proposal made here. One
> more
> > time, I would like to make this an open effort where all can be decided
> > through discussions.
> >
> > I hope to hear back from you and we can make good work together for the
> > benefit of all the participating teams (if any) and to have the
> presentation
> > done in FOSS4G 2014, Best regards,
> >
> > (sorry for any cross posting)
> >
> > [1] http://wiki.osgeo.org/wiki/WPS_benchmark_2014 [2]
> > http://lists.osgeo.org/mailman/listinfo/wps-discuss
> >
> >
> > Gérald Fenoy gerald.fenoy at geolabs.fr
> >
>
>
>
> --
> Jachym Cepicky
> URL: http://geosense.cz
> e-mail: jachym.cepicky at geosense cz
> PGP: http://les-ejk.cz/pgp/JachymCepicky.pgp
> @jachymc
> _______________________________________________
> Wps-discuss mailing list
> Wps-discuss at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/wps-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/wps-discuss/attachments/20140407/d68f9b28/attachment-0001.html>