[Wps-discuss] WPS Benchmarking at FOSS4G 2015
Daniel Nüst
d.nuest at 52north.org
Tue Jul 7 02:13:09 PDT 2015
Am 07/07/2015 um 01:21 schrieb Jody Garnett:
> Thinking about this more, it changes one problem (implementations are
> dependent on their geospatial library for speed) with another
> (implementations are dependent on their xml parsing / format encoding
> technology for speed).
>
> So not sure of the value here, other than a discussion point :)
I think the value is, that the crucial question is what we really want
to "benchmark". Is plain performance outside of the algorithm really a
relevant criterion for a WPS, or is it just the easiest (and
quantifiable) thing we can measure?
From my perspective, any process that is worth publishing as a
standardized WPS will probably take an order of magnitude longer than
the parsing/encoding tasks. The interesting part of the studies/papers I
know is the behavior with multiple requests requests at the same time.
Regarding things that are hard to measure: In my opinion these are also
the strengths of the different open source implementations and goals of
the different communities, which makes them even harder to compare in a
shootout-fashion. Geoserver surely has a strength in processing the
integrated data! What are the strengths of other projects? Maybe we can
"benchmark" open source projects at a whole: For any given problem, is
there an open source WPS that solves it?
Looking forward to hear more views!
/Daniel
> --
> Jody Garnett
>
> On 30 June 2015 at 15:50, Jody Garnett <jody.garnett at gmail.com
> <mailto:jody.garnett at gmail.com>> wrote:
>
> I am happy with any approach that gives us a chance to both talk
> about our implementation(s) and promote WPS. We could also use the
> opportunity to discuss what we are looking forward to in WPS 2.0 etc...
>
> GeoServer may be a bit hampered with the echo process idea (as its
> strength is on processing data served up by the services) but that
> is fine ... always room for improvement.
>
> --
> Jody Garnett
>
> On 23 June 2015 at 06:40, Benjamin Proß <b.pross at 52north.org
> <mailto:b.pross at 52north.org>> wrote:
>
> Dear WPS developers,
>
> As I mentioned in a previous mail, we originally planned a
> interoperability testing for this years WPS benchmark.
> I had a discussion with my colleagues lately and we came to the
> conclusion that we will not have results ready for the FOSS4G in
> September.
> As a plan b, I think we could re-use the test concept from last
> year, as it was quite sophisticated.
> However, we had some discussion during the preparation for the
> last benchmark about the execute tests.
> I brought up that we could use a simple echo process, which
> would have some benefits:
>
> * we could easily test execute with all three data types
> * we are independent from the libraries used by the
> different WPS implementations for geospatial computations
> (i.e. we are testing only the performance of the WPS)
> * it could be re-used for WPS 1.0/2.0 compliance tests at
> the OGC
>
> Some points regarding the echo process from earlier mails:
>
> Am 19.08.2014 um 16:03 schrieb Benjamin Proß:
>> We do not yet have any results to compare but we think that
>> benchmarking with "real" processes that compute something
>> could falsify the results, as the computation in most cases is
>> done by a underlying library (GeoTools in our case, but this
>> could also be exchanged).
>> So the results would be influenced by how fast the
>> geo-computation lib would do its job. Not sure that we want
>> that?!
>> We also do not test whether the process really did its job
>> (e.g. buffer the input geometry). So do we need to use "real"
>> processes?
>> We would propose to use an echo process that could have three
>> inputs, Literal-, Complex- and BBoxData. The process then
>> simply returns what it gets (input/output types would have to
>> match).
>> We would probably also need a asynchronous version that has a
>> delay or something.
>> That way we could test the input-/output handling capabilities
>> (of all three data types) of the services and we would still
>> test everything regarding execute that is tested right now
>> (i.e. we would not loose functionality).
>
> Am 21.08.2014 um 11:16 schrieb Fenoy Gerald:
>> I think it is a great idea to handle testings of the Execute request the way you described, by using a simple echo service.
>>
>> By now for synchronous requests, the tests look like the following:
>>
>> • Test 1: value as JSON
>> • Test 2: value as GML
>> • Test 3: value as reference as GML
>> • Test 4: same as test 3, result as reference
>> • Test 5: value as reference as GML using POST request
>> • Test 6: same as test 5, result as reference
>>
>> And for asynchronous request, we have the following:
>>
>> • Test 1': value as reference as GML
>> • Test 2': same as test 1, result as reference
>> • Test 3': value as reference as GML using POST request
>> • Test 4': same as test 3, result as reference
>>
>> Personally, I think that the Test 1 and Test 2 should become unique now as there is no need for parsing any value from the service itself as it will simply return the value without any treatment. So we should use only the Test 2 for our next run.
>>
>> I also suppose that we should multiply the number of tests as we will need to ask for each available outputs individually and all the possible combinaisons of them to also test this cases.
>>
>> So, I would like to propose the following updates for the tests run for both synchronous and asynchronous requests. We run each individual test 7 times: first to have only once output at a time (so 3 tests), then to test all possible combinaisons of 2 outputs names (3 tests) and finally to have all the output returned (1 test).
>>
>> I hope that you can confirm that I’m right thinking of this updates to the current tests.
>>
>> I’ve added one column to the table [1] listing all WPS team participating to write the name of the echo service when it will be available.
>>
>> Best regards,
>>
>> [1]http://wiki.osgeo.org/wiki/WPS_benchmark_2014#Available_infrastructure
>
> What do you think about this approach?
>
> Cheers,
>
> Benjamin
>
> --
> Benjamin Proß
> Software Engineer
> 52°North Geoprocessing Community
>
> 52°North Initiative for Geospatial Open Source Software GmbH
> Martin-Luther-King-Weg 24
> Fon:+49-(0)-251–396371-42 <tel:%2B49-%280%29-251%E2%80%93396371-42>
> Fax:+49-(0)-251–396371-11 <tel:%2B49-%280%29-251%E2%80%93396371-11>
> b.pross at 52north.org <mailto:b.pross at 52north.org>
> http://52north.org/
>
> General Managers: Dr. Albert Remke, Dr. Andreas Wytzisk
> Local Court Muenster HRB 10849
>
>
> _______________________________________________
> Wps-discuss mailing list
> Wps-discuss at lists.osgeo.org <mailto:Wps-discuss at lists.osgeo.org>
> http://lists.osgeo.org/mailman/listinfo/wps-discuss
>
>
>
>
>
> _______________________________________________
> Wps-discuss mailing list
> Wps-discuss at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/wps-discuss
>
--
Daniel Nüst
52°North Initiative for Geospatial Open Source Software GmbH
Martin-Luther-King-Weg 24
48155 Münster, Germany
E-Mail: d.nuest at 52north.org
Fon: +49-(0)-251–396371-36
Fax: +49-(0)-251–396371-11
http://52north.org/
Twitter: @FiveTwoN
General Managers: Dr. Albert Remke, Dr. Andreas Wytzisk
Local Court Muenster HRB 10849
More information about the Wps-discuss
mailing list