[FOSS4G2016] [Program] Abstract review process draft

[Marc Vloemans]

Hi
Although I have only voted as a community member, I really applaud the care you guys have applied in approaching this delicate matter!!!

We present a very well balanced program.

Some typos aside, send it out as-is. It is the thought(fullness) that counts. Kudos to Volker et al.

Vriendelijke groet,
Marc Vloemans


> Op 28 apr. 2016 om 21:37 heeft Volker Mische <volker.mische at gmail.com> het volgende geschreven:
> 
> Hi LOC and Program-Committee,
> 
> here's the draft of a blog post I'd like to publish tomorrow morning, so
> there isn't much time for a review :) It's about our review process.
> This blog post will be mentioned in the acceptance/decline emails I'll
> also send out tomorrow.
> 
> Cheers,
>  Volker
> 
> 
> Abstract review process for FOSS4G 2016
> =======================================
> 
> The final selection of the presentations for the FOSS4G used to be a
> controversy. Reviewing so many abstracts takes a lot of effort, so it's
> not possible to tell the rejected presenters in detail why their
> abstract wasn't accepted. Though it's possible to be transparent about
> the process that was applied. That's what this blog post is about.
> 
> Let's start with some numbers. There were 281 abstracts submitted, 181
> were accepted. There was a community review and a separate review by the
> program committee. Both reviews were blind, without knowing the author's
> name.
> 
> Next the program committee met several times using a Google Docs
> spreadsheet which had the community rank, reviewer's rank and the
> submitted details in it. This time it was including the author's name.
> 
> 
> The selection process
> ---------------------
> 
> ### Step 1: Get rid of the lowest ranked ones
> 
> We were picking a number to eliminate the abstracts that had *both* a
> low rank (less thank 180) by the community and by the reviewers. This
> way we got the numbers down by 47 to 233.
> 
> 
> ### Step 2: Go through highest ranked presentations and limit those by
> author and company
> 
> Only the abstracts where both the community as well as the program
> committee were ranking them above 180 were taken into account, that were
> 142 talks. We went trough them to limit the number of presentations to a
> maximum of two per person (with exceptions: some people are so active in
> the FOSS4G community, that they represent so many different projects, so
> that it make sense to have them give more than two talks. We also
> limited the number of talks by company. This wasn't a hard-number, but
> depending on how many different topics were covered. This reduced the
> total talks by 11 to around 222.
> 
> 
> ### Step 3: Go through the "undecided" ones
> 
> Those presentations where the community and the program committee didn't
> agree on got special care. Those were about 90 talks. We sorted them by
> the community voting. Then everyone from the committee was able to argue
> for talks being in our out. This also included talks being excluded
> because of the limitations outlined in step 2. This was the most time
> consuming step which lead to a lot of good discussions. In the end it
> brought the total count down by 41 to the 181 we accepted.
> 
> 
> ### Step 4: Group talks by topic
> 
> As we were meeting already twice, we decided to take the grouping
> offline. All members were working on the spreadsheet of the final
> selection and adding tags of what the talks are about. The program
> committee chair then took the time to do the final grouping for the website.
> 
> 
> ### Impact of the community vote
> 
> If you compare the top 181 talks from the community voting with the
> final program the overlap is 77% (when taking the talks removed in step
> 2 into account it's even 80%).
> 
> When the committee made decisions that were contrary to the community
> voting, it had numerous reasons. For example all OSGeo project should
> get a stage to present, whether they were voted highly by the community
> or not. Other considerations were to include non-developer centric talks
> like case studies, about business models or talks matching one of our
> four key topics: remote sensing for earth observation, disaster
> management, open data and land information.
> _______________________________________________
> FOSS4G2016 mailing list
> FOSS4G2016 at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/foss4g2016