A PDF version of the meta-reviewer guidelines (download)
We are grateful for your willingness to serve as a meta-reviewer for WASPAA 2023. Your role is akin to that of an associate editor for a journal and thus pivotal in ensuring the quality and success of the workshop.
What Happens Next?
Update your information for the Toronto Paper Matching System (TPMS) and for conflict domains. Shortly after the first submission deadline you will be asked to bid for papers and identify conflicts of interest. Based on this information the TPC chairs will assign 4-6 paper submissions to you. As a meta-reviewer for those papers you will have 3 main responsibilities: (1) to assign 3 qualified reviewers for each of them; (2) to lead the discussion phase and steer it towards consensus; and (3) to provide a meta-review and submit a recommendation to the TPC chairs. Below we outline some guidelines to assist you in this process.
Paper Assignment:
- Do an initial scan of the papers as soon as you receive them. This scan will help inform your decisions on reviewer assignments, and to identify potential conflicts of interest, disclosure of authors’ identities, and other issues needing quick clarification from authors or the TPC chairs.
- Note that the review process is double-blind this year. We are asking authors not to identify themselves and their institutions in the paper, but acknowledge that they may decide to post their papers on arXiv, while we still discourage advertising the work on social media until accepted. Therefore, don’t go out of your way to identify the authors, and if by any chance you are able to do so, try your best not to let that knowledge affect the outcome of the review process. If inadvertently identifying the authors leads to a potential conflict of interest, let the TPC chairs know immediately.
- All your papers will be initially assigned reviewers automatically using a combination of COI, bidding information and the Toronto Paper Matching System (TPMS). Review these initial assignments and make changes as needed to ensure each paper has a set of relevant and qualified reviewers.
- Complete the assignment process within the specified time, so that the reviews can be completed without delays.
Review Process, Rebuttal and Discussion:
- Communicate with reviewers immediately before and after the deadline to make sure all reviews are received. If a reviewer is unresponsive, identify an alternative reviewer and ask for an emergency review.
- Ask the reviewers to expand or improve their reviews if they are of insufficient quality to inform a decision.
- All reviews, whether original, improved or emergency, need to be received no more than 1 week after the initial review deadline at the latest. Authors will be sent all reviews and given two weeks to complete their rebuttals so time is of the essence.
- Read each paper, all reviews and the rebuttal letters prior to the start of the discussion phase so that you can be effective in steering the conversation. Keep in mind that, if accepted, only minor revisions are typically allowed for the camera-ready version of the paper. Any suggestion of a major revision in the rebuttal should be backed by evidence and only considered in special circumstances.
- Message reviewers within a day or two of the start of the discussion period. We recommend summarizing the main points of (dis)agreement amongst the reviewers and any salient points from the rebuttal letter, as well as encouraging reviewers to read and acknowledge it.
- If the reviewers are unresponsive, we suggest sending repeated reminders and asking specific questions.
- While polarization is sometimes inevitable, we encourage you to strive for a consensus among reviewers. If minds are changed during the discussion, encourage reviewers to update their reviews accordingly.
Meta-Review:
- Use the review form and its comment section to provide a meta-review for each paper, keeping in mind that it is visible to the authors.
- We recommend the meta-review summarizes: the strengths and weaknesses of the paper as observed by you and the reviewers; the main conclusions of the rebuttal and discussion; and, if applicable, the main revisions expected in a final version of the paper (grouped into major and minor revisions).
- We request that you avoid including your recommendation to accept or not in the comment section, which is visible to the authors, as the TPC chairs may still decide differently. Private comments (not for the authors) are often necessary and helpful to the TPC chairs. They can be added to the private comments section of the form.
- That said, please provide definitive accept/reject recommendations in your review scores, and avoid marginal recommendations as much as possible. Your scores won’t be visible to authors.
- If you encounter an exceptional paper, based on the reviewers’ recommendations and your own judgment, we encourage you to nominate the paper for an award and provide a brief rationale. Your nomination won’t be visible to the authors.
Thank you once again for your invaluable service as a meta-reviewer. If you have any questions or concerns, please do not hesitate to contact us. We truly appreciate your contribution and look forward to working with you.
Timeline for the Review Process
| Timeline | Action Item |
| 4/21 | Abstract/title deadline (for authors) |
| 4/24-5/3 | Paper bidding (for reviewers / meta-reviewers) |
| 4/28 | Full paper deadline (for authors) |
| 5/5 | Meta-reviewer assignments complete (for TPC chairs) |
| 5/12 | Reviewer assignments complete (for meta-reviewers) |
| 6/2 | Review deadline (for reviewers) |
| 6/12 | Reviews released to the authors |
| 6/23 | Rebuttal deadline (for authors) |
| 7/4 | Discussion period ends (reviewers / meta-reviewers) |
| 7/7 | Meta-reviews deadline (meta-reviewers) |
| 7/12 | Final notifications (TPC chairs) |