A few tips for the Area Chairs of ML conferences

Emtiyaz Khan  •  December 16, 2022  •  4 minutes read

What makes a good meta-review? It clearly describes the whole review process and gives clear reasons behind the decisions. Below are a few tips for writing Area Chairs (or Editors) of Machine Learning Conferences and Journals. These grew out of my own experiences as AC and Senior AC (SAC) over the years. These are not perfect, and I am more than happy to get any feedback to improve the list.

  1. [Summarize the paper] including its strengths and weaknesses, and make sure to add comments regrading novelty, significance, and relevance/impact.
  2. [Summarize the post-rebuttal discussion] Did reviewers engage in discussion or not? Did their opinion change? Did they change their score, and by how much? If there was no discussion, why did the AC took that particular decision? In case of rejection, please also give the authors some feedback to improve the paper, and suggestions about the next appropriate venue to submit their work.
  3. [Include one sentence that clearly indicates your decision] For example, you can say "I recommend to accept/reject this paper". Or add "... accept this paper as a poster/oral". This way the SAC/Chairs can double check your decision and make sure that it matches the content of the meta-review.
  4. [Discuss borderline paper with the Reviewers/SAC/Chairs] No explanations required for this. I have also found that a quick meeting often helps to arrive a good decision. If people don't respond, sending emails sometimes helps. Be considerate because sometimes they are just busy with other things, and can't respond even if they want to. The best way to schedule a meeting would be to give a few open slots, and let them choose. You can meet with the reviewers/SAC/chairs separately, and consolidate their opinions in the meta-reviews.
  5. [Avoid overriding reviewers decision] If you disagree with the reviewers and you must override, please inform the SAC/Chairs and discuss the paper with them. This helps to avoid any confusion later, for example, if the authors raise an issue with the chairs, the chairs can look up your explanation and take appropriate actions.
    • [Avoid making decisions on your own reading the paper without informing the reviewers] I have seen this much more often than I would like. Some ACs would take a lot of time to read and judge the papers on their own, and then completely exclude reviewers' account in the meta-review and just write their own opinion. Even though this was a lot of work for the AC who feel now that they have done their job so well, this is not right! Meta-reviewers are not reviewers (obviously), so you should avoid using only your opinion. If the reviewers have not participated or made mistakes, first, write a message to them saying that you are going to override their opinions. Give them a choice to respond. Then in the meta-review clearly indicate that you are making the decision based on your own opinion, your own reading of the reviews, and then discounted them (explain reasons why).
  6. [Push the paper towards acceptance] Most papers are the results of many months of hard work of many individuals. Our hope is that the review process helps them to get their work eventually published at an appropriate venue. I believe we should not look for reasons to reject, rather find ways to get the work accepted, either at this conference or in some other future conferences. Therefore, it is my opinion that the decision and meta review should help the authors towards acceptance of their work.
  7. [Give feedback the chairs about good/bad reviewers] If you think some reviewers were not responsive, please consider making a list and write briefly what they did wrong and can improve for the future. Same for amazing reviewers -- write what they did great. You can send this to the SAC/chairs. They can then take appropriate actions, and give feedback to the reviewers, which may improve their work for the future. Currently, such a system is lacking in our review process, but I am hopeful that some day, we have a system that trains and improves the reviewer pool over the years.