[FOSS4G2016] [Program] Abstract review process draft

[Arnulf Christl]

Hi Volker,
thanks for all the work so far. The blog post is excellent and should
probably go into the FOSS4G Handbook as a kind of template how to do the
selection.

Have fun,
Arnulf

On 29.04.2016 09:36, Christian Mayer wrote:
> Hi Volker,
> 
> great blog post! Thank you very much!
> 
> Cheers
> Chris
> 
> Am 28.04.2016 um 21:37 schrieb Volker Mische:
>> Hi LOC and Program-Committee,
>>
>> here's the draft of a blog post I'd like to publish tomorrow morning, so
>> there isn't much time for a review :) It's about our review process.
>> This blog post will be mentioned in the acceptance/decline emails I'll
>> also send out tomorrow.
>>
>> Cheers,
>>   Volker
>>
>>
>> Abstract review process for FOSS4G 2016
>> =======================================
>>
>> The final selection of the presentations for the FOSS4G used to be a
>> controversy. Reviewing so many abstracts takes a lot of effort, so it's
>> not possible to tell the rejected presenters in detail why their
>> abstract wasn't accepted. Though it's possible to be transparent about
>> the process that was applied. That's what this blog post is about.
>>
>> Let's start with some numbers. There were 281 abstracts submitted, 181
>> were accepted. There was a community review and a separate review by the
>> program committee. Both reviews were blind, without knowing the author's
>> name.
>>
>> Next the program committee met several times using a Google Docs
>> spreadsheet which had the community rank, reviewer's rank and the
>> submitted details in it. This time it was including the author's name.
>>
>>
>> The selection process
>> ---------------------
>>
>> ### Step 1: Get rid of the lowest ranked ones
>>
>> We were picking a number to eliminate the abstracts that had *both* a
>> low rank (less thank 180) by the community and by the reviewers. This
>> way we got the numbers down by 47 to 233.
>>
>>
>> ### Step 2: Go through highest ranked presentations and limit those by
>> author and company
>>
>> Only the abstracts where both the community as well as the program
>> committee were ranking them above 180 were taken into account, that were
>> 142 talks. We went trough them to limit the number of presentations to a
>> maximum of two per person (with exceptions: some people are so active in
>> the FOSS4G community, that they represent so many different projects, so
>> that it make sense to have them give more than two talks. We also
>> limited the number of talks by company. This wasn't a hard-number, but
>> depending on how many different topics were covered. This reduced the
>> total talks by 11 to around 222.
>>
>>
>> ### Step 3: Go through the "undecided" ones
>>
>> Those presentations where the community and the program committee didn't
>> agree on got special care. Those were about 90 talks. We sorted them by
>> the community voting. Then everyone from the committee was able to argue
>> for talks being in our out. This also included talks being excluded
>> because of the limitations outlined in step 2. This was the most time
>> consuming step which lead to a lot of good discussions. In the end it
>> brought the total count down by 41 to the 181 we accepted.
>>
>>
>> ### Step 4: Group talks by topic
>>
>> As we were meeting already twice, we decided to take the grouping
>> offline. All members were working on the spreadsheet of the final
>> selection and adding tags of what the talks are about. The program
>> committee chair then took the time to do the final grouping for the website.
>>
>>
>> ### Impact of the community vote
>>
>> If you compare the top 181 talks from the community voting with the
>> final program the overlap is 77% (when taking the talks removed in step
>> 2 into account it's even 80%).
>>
>> When the committee made decisions that were contrary to the community
>> voting, it had numerous reasons. For example all OSGeo project should
>> get a stage to present, whether they were voted highly by the community
>> or not. Other considerations were to include non-developer centric talks
>> like case studies, about business models or talks matching one of our
>> four key topics: remote sensing for earth observation, disaster
>> management, open data and land information.
>> _______________________________________________
>> FOSS4G2016 mailing list
>> FOSS4G2016 at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/foss4g2016
> 
> _______________________________________________
> FOSS4G2016 mailing list
> FOSS4G2016 at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/foss4g2016
> 


-- 
Arnulf Christl

We help you organize your spatial data
Open Source - Open Data - Open Standards
Metaspatial: http://metaspatial.net/