Talk:Critical issues presentations

Notification of acceptance

the result will be announced today ? --Touzrimounir (talk) 11:21, 27 January 2016 (UTC)[reply]

@MLWatts, Marcok: ping. --Yi yi 11:40, 27 January 2016 (UTC)[reply]

I edited the page reflecting the recent postponing of the date of the notification of acceptance. Here is the message I sent to the Wikimania mailing list today:

Dear all,

This is to inform you that the assessment phase for the submissions we received for our "Critical Issues" talks at Wikimania 2016 is in progress. The reviews are, however, taking more time than initially expected; the members of the Programme Committee needed some effort to get familiar with the EasyChair conference platform we are using, which, as I can testify having configured it and managed it so far, is not entirely intuitive to interact with. Needing more time to complete the reviews, we have to postpone the date of the notification of acceptance to February 1st. If you need further information, please do not hesitate to ask.

Sorry for the inconvenience to all who are affected.

Yours, --MLWatts (talk) 19:43, 27 January 2016 (UTC)[reply]

@MLWatts: I think the result should be announced yesterday --Touzrimounir (talk) 08:46, 2 February 2016 (UTC)[reply]

I'm sorry, we're doing our best. --MLWatts (talk) 11:09, 2 February 2016 (UTC)[reply]

We are sorry for the delay. but we decided to only allow ourselves 2 weeks for the reviews (which is quite complicated with a new system and we did not know how long it takes): we set such a short time to make sure we could provide the reviews to the committee evaluating the scholarships; we want people who are selected for the programme to have good chances to actually get a scholarship to come. Bridging together the two processes is not obvious but we trust it makes sense. The short delay did not compromise the objective. Thank you very much to all the reviewers for the great work and the fast and effective response! Thanks to the people who got involved last minute to fill gaps and thanks to MLWatts (WAW!) and Marcok for the management of this experiment. --iopensa (talk) 09:56, 4 February 2016 (UTC)[reply]

Feedback and Evaluation

WHY

Background. Please note that we have been working on the Wikimania program since August 2013 with discussions among former participants, an analysis of the comments and feedback of previous editions, a comparative analysis of the previous programs and formats, and a meeting at Wikimania Mexico City. You find the program described already when we submitted the bid for 2015 and then updated for 2016 and our objective for the program in 2016 is to increase the number of informal gatherings, to create a program suitable for different attendees and to experiment new systems.

The specific reasons why we introduced a blind peer-review process for a selection of presentations (not all of them) is specifically to respond to a series of issues. a. We had the impression that our current selection process on wiki does not allow new people to get in (maybe it is also due to the fact that when you decide between a 9 and 10 if you know the person it is easier to give a higher score: please note that the selection is really between the highest scores). b. For many years the reviewers where more or less always the same and - even if they are great people - we wanted to boost some change. We made a call for new reviewers and you are more than welcome to get involved in the future. More reviewers more reviews. c. We had the impression that there are too many presentations from wmf. I think the staff of wmf has been presenting great proposals, but maybe we need less great proposals and more involvement of other people, simply because we want to hear what they have to say and we need to make space for them (for this reason we set a limit for the presentations of wmf staff members).

B PLAN. At Wikimania Esino Lario there are different processes to submit proposals: critical issues are reviewed (42 presentations of this kind are included in the conference), but you can also suggest in different ways and with different coordinators talks, topics, posters, trainings, discussions, special guests and meetups. Submissions presented as “critical issues” which do not fit the format, can be moved to other sessions (you can resubmit them and we will do it too by looking at the “non selected”).

Thank you. Thanks to everyone who submitted his/her ideas and contributions and thanks to the reviewers who participating in highlighting their favorite topics/approaches. A review is never a neutral process but we trust there is always respect for the work of everyone involved. Thanks! --iopensa (talk) 09:31, 4 February 2016 (UTC)[reply]

Weakness of the peer-review process (please sign your comment)

People can not look at other proposals to check if a topic is already submitted (Lodewijk). --iopensa (talk) 09:31, 4 February 2016 (UTC)[reply]
The abstract and the information provided could be easily improved if someone could make a first review to the proposals and provide some feedback. (Lodewijk) --iopensa (talk) 09:31, 4 February 2016 (UTC)[reply]
The feedback of the reviewers is not always satisfactory. (pigsonthewing, Maarten, Amir). --iopensa (talk) 09:31, 4 February 2016 (UTC)[reply]
Sharing the content on wiki requires a lot of copy and paste. (Lodewijk) --iopensa (talk) 09:31, 4 February 2016 (UTC)[reply]
Time. As a first experience for us we had difficulties to estimate the time required for the reviews. We also allowed ourselves very little time because we wanted the results to be quickly available for the evaluation of the scholarships (to make sure a person who is submitting a proposal considered valuable has more chances to get a scholarship - since we really want that person to come and present). --iopensa (talk) 09:31, 4 February 2016 (UTC)[reply]
Short reviews are definitely not helping in addressing the frustration of rejection, yet are quite common in academic peer reviewing, especially for conferences. (dj) --iopensa (talk) 09:31, 4 February 2016 (UTC)[reply]
Double blind peer review (not knowing who is reviewed, and not knowing who reviews) is a standard in Academia, although some perceive it as contributing to lack of responsibility (especially true in competitive journal submissions). (dj) --iopensa (talk) 09:31, 4 February 2016 (UTC)[reply]
It could be useful to sensitize the reviewers that the main purpose of the review is to help the author to do better next time. (dj) --iopensa (talk) 09:31, 4 February 2016 (UTC)[reply]
I would have thought knowing who was proposing the presentation was a significant factor in its evaluation as that provides the critical point of whether the person has the knowledge to speak about the issue. Such games make the process appear to be unprofessional offering very little in the way of addressing critical issues and potentially opening up the community to abuse. (Gnangarra) --iopensa (talk) 09:31, 4 February 2016 (UTC)[reply]
ensure the reviews dont do harm to the person receiving them. (Gnangarra) --iopensa (talk) 09:31, 4 February 2016 (UTC)[reply]
ensure the reviewed dont do harm to (and are polite with) the reviewers. --iopensa (talk) 09:31, 4 February 2016 (UTC)[reply]
Concerns about the skills-expertise of the reviewers. (Amir) --iopensa (talk) 09:31, 4 February 2016 (UTC)[reply]
It should be made clear to submitters what is happening and if Wikimania is an academic conference or not. (Phoebe). --iopensa (talk) 09:31, 4 February 2016 (UTC)[reply]
If I ever run Wikimania again, I think I'd try having *no* formal talks: only discussions and lightning talks. (Phoebe). --iopensa (talk) 09:31, 4 February 2016 (UTC)[reply]
Two reviews per submission might work if we had a clear set of criteria that the reviewers were following and sufficient training of the reviewers that they were broadly consistent in their marking. (WhereSpielChequers) --iopensa (talk) 09:31, 4 February 2016 (UTC)[reply]
While anonymisation is intended to improve fairness, it appears to prevent the reviewers from being able to evaluate the submitter's expertise (especially since there is no paper submitted, only a brief abstract). Coren (talk) 18:34, 4 February 2016 (UTC)[reply]

Strengths of the peer-review process (please sign your comment)

The feedback can be shared with the people who applied. (Lodewijk) --iopensa (talk) 09:31, 4 February 2016 (UTC)[reply]
Two reviewers per submission is absolutely on par with the conference standards I'm used to. Sometimes there are three, but two is absolutely acceptable (although a third opinion should be used if the two disagree too much). (dj) --iopensa (talk) 09:31, 4 February 2016 (UTC)[reply]
In case of two very different evaluation we asked for another review. For limited time (and a limited number of available reviewers), we only did it when one of the score was already very high. --iopensa (talk) 09:31, 4 February 2016 (UTC)[reply]
Proposals which have not been selected can be moved to other sessions or presented as posters, topics of discussions, workshops... Among these there will be other reviews made also by the community. --iopensa (talk) 09:31, 4 February 2016 (UTC)[reply]
All this is volunteer work. We should be, generally, grateful to reviewers (but in the same time grateful to the contributors, too). (dj) --iopensa (talk) 09:31, 4 February 2016 (UTC)[reply]

Does the reviewers followed the criteria ?

Well I think that double blind peer review hardly make sense here, at least from the reviewer POV, as we are in fact small community and it is easy to guess who was a submiter in most cases.

Also judging from the several reviewers comments which I saw already - they did not follow the very vague criteria which was posted here:

https://wikimania2016.wikimedia.org/wiki/Critical_issues_presentations#Evaluation

Normally - at least in Academia - reviewers are forced directly (by the review form) to address their opinion in relation to the criteria. The criteria were:

   problems and possible solutions in a specific field
   proposals for others to replicate
   issues (positive or negative) which have emerged from projects
   issues you want to raise which you feel have not been discussed yet
   issues which are at the centre of an online debate that you would like to address offline

1-4 are IMHO relatively easy to evaluate - I would expect from the reviewers to answer yes or no to them. 5 is a bit tricky as it depends strongly of what the reviewer think is "at the centre" - but I would expect that they at least explain in few words here what they think is "at the centre" or not :-).

The other issue is - that having such a vague criteria and having only 2 reviewers - the results very much depends on those 2 reviewer POV - and in fact actually just one - as one negative opinion was usually enough to reject submission even if the second one was positive. Polimerek (talk) 11:02, 4 February 2016 (UTC)[reply]

Wrong license for presentations?

Hello, I am afraid that, technically, the requirement to release presentation slides and have the presentation recorded and broadcasted under "Creative Commons Attribution Share-Alike Licence 3.0" prohibits the display of the Wikimania 2016 logo anywhere on the slides, around the speaker etc. That's because the logo is licensed under CC BY-SA 4.0 and, if I am not mistaken, if you reuse something that is under 4.0, you need to release it under 4.0 again, not 3.0. I appreciate the effort for presentations under free licenses, but 4.0 might have been a better choice. --Blahma (talk) 11:39, 24 June 2016 (UTC)[reply]