The referee report

In this exercise, you will review a paper that has not been formally published yet.

The paper must be a preprint posted to arXiv within the last two weeks, not yet published at a venue, and not a revision of something older. Check the submission history on the abstract page: you want a v1 dated within your window, with no “accepted at” note on it.

That constraint is the whole exercise. A new preprint has no published reviews to fall back on, no citation count to tell you what the field has decided, and no accumulated opinion to draw on. Whatever you conclude about it, you have concluded.

Why not your reproduction paper? Refereeing a paper you already know was accepted is a different task: you are checking a verdict rather than reaching one. The reproduction already gives you the paper from the inside. 6 months after the course is finished, check whether the paper was published in a formal venue.

Finding one

The recent listing for a category is at arxiv.org/list/cs.CV/recent and new submissions are announced each weekday. Computer Vision and Pattern Recognition (cs.CV) is the obvious place for this course, but Machine Learning (cs.LG) and Artificial Intelligence (cs.AI) carry vision work too.

Choose something you can review:

A paper making an empirical claim, with experiments you could argue about. Not a survey, not a position paper, not a benchmark release with no method attached.
Close enough to the course that you can judge the baselines. A paper whose related work you have never heard of is one you will review on presentation alone.
Long enough to have something to say about. A four-page workshop note does not carry a full review.

If you are unsure whether a paper qualifies, say so early rather than a day before the session.

Peer review

Before a paper appears at a venue, other researchers in the same field read it and judge whether it should be accepted. That is peer review. The people doing it are unpaid, working on the paper alongside their own research, and in machine learning they are often reviewing several papers in the same fortnight.

Most machine-learning venues review double-blind: the reviewers are not told who wrote the paper, and the authors are not told who reviewed it. The reason is that a judgement should rest on what the paper shows rather than on the reputation of the group that produced it, and the evidence is that unblinded review favours well-known authors and well-known institutions [1]. In that study, one conference ran both kinds of review on the same submissions, and the reviewers who could see the authors were more likely to accept papers from famous authors and from top-ranked institutions than the reviewers who could not. The blinding is imperfect. A preprint already on arXiv, a self-citation phrased as “our earlier work”, or a system with a distinctive name will often identify a group to anyone who follows the area.

Your review is not blind at all — you chose the paper, and its authors’ names are on it. Noticing where that knowledge pulls your judgement is part of the exercise.

Role	What they do
Author	Submits the paper and answers the reviews: once, in a rebuttal, at a conference; over as many rounds of revision as the editor allows at a journal.
Reviewer, or referee	Reads the paper, writes the report, scores it, and recommends acceptance or rejection. This is the part you are doing.
Area chair (conferences)	Handles a batch of papers, recruits and chases the reviewers, moderates discussion between them when they disagree, and writes a meta-review summarising why the decision went as it did.
Senior area chair or programme chair (conferences)	Oversees the area chairs, settles the cases they cannot, and takes responsibility for the final list.
Associate editor, or handling editor (journals)	The journal equivalent of an area chair: chooses the reviewers and decides whether the paper is rejected, accepted, or returned for minor or major revision.
Editor-in-chief (journals)	Assigns each submission to an associate editor and holds the final decision.

The two paths differ in shape. A conference decides once: reviews, a rebuttal, then discussion among the reviewers and the area chair, in which a reviewer swayed by the rebuttal can usually raise or lower their score. After that, it is accept or reject, with no route back except submitting again elsewhere months later. A journal often iterates: a paper can go through two or three rounds of revision with the same reviewers, and major revision is a normal outcome rather than a soft rejection.

Outcomes

A decision comes back in one of a small number of named forms, and the names are used the same way across most venues.

Outcome	What it means
Desk rejection	Rejected by the editor or chair without going to reviewers at all: out of scope, over length, anonymity broken, or plainly below the venue’s bar. It arrives within days.
Reject	Rejected after review. At a conference that is the end of it for that venue, and the paper goes to the next one. At a journal it can sometimes come with an invitation to resubmit as a new paper.
Major revision	The claims may well stand, but not on the evidence as it is: experiments to add, analysis to redo, a baseline to strengthen. It goes back to the same reviewers, and it is a normal outcome rather than a soft rejection.
Minor revision	Accepted in substance. Clarifications, a missing detail, presentation. Usually checked by the editor alone rather than sent round again.
Accept	Published as it stands. Rare on a first journal submission. At a conference the decision also sets the format, which is usually a poster and occasionally a talk.

Journals use all five. Conferences in this field mostly use two, accept and reject, with the rebuttal and the reviewer discussion doing the work that a revision round does elsewhere — which is why a conference review has to say what would change your mind, since there is no second round in which to find out.

Reviewer recommendations

Those two are the paper’s outcome. Your own recommendation as a reviewer is finer-grained than that, because the area chair needs to know not just which side you are on but how hard you would argue for it. NeurIPS asks for one of six:

Score	Label
6	Strong accept
5	Accept
4	Borderline accept / Weak accept
3	Borderline reject / Weak reject
2	Reject
1	Strong reject

Regardless of the terminology, the middle two mean the same thing: the reasons on one side outweigh those on the other, and not by much. NeurIPS asks reviewers to use them sparingly, which is worth knowing, because the middle of a scale is where a reviewer who has not decided tends to sit.

The gap between a 5 and a 6 is not just about your enthusiasm. A strong accept says you would argue for this paper against the other reviewers, and a strong reject says you would argue against it after reading their reports. If you would not, the score below is the honest one. The number is also not the review: an area chair weighing a borderline reject with a specific, unanswerable objection against a confident accept with three vague sentences will go with the first.

Reviewer confidence

Most venues ask for a confidence score alongside the recommendation, on a separate scale of 1 to 5. It answers a different question: not how good the paper is, but how much weight your opinion should carry. At the top of NeurIPS’s scale, you are certain, know the related work, and have checked the details; at the bottom, your assessment is an educated guess on a paper outside your area.

The point of it is that an area chair can discount a review honestly rather than guess. A confident borderline reject outweighs an uncertain strong reject, and saying you are uncertain is how a reviewer who is out of their depth avoids doing damage. Inflating it is a common failure case — your report is worth more with a 2 you can justify than with a 5 you cannot.

What a review contains

The form below is the one NeurIPS gives its reviewers, with the same section names. CVPR’s differ in wording but not in substance.

Read both before you start writing. They are the actual instructions sent to the people who decide whether a paper appears at some of the biggest conferences in the world; they take about fifteen minutes each, and they say a good deal that this page does not — on confidentiality, on conflicts of interest, and on what reviewers are told not to do. Where the two disagree, the disagreement is worth noticing: it tells you which parts of reviewing are settled and which are a venue’s own house rules.

Write every section. A review that omits its questions or limitations is not a short review; it is an incomplete one.

Section	What it must contain
Summary	The paper’s claim and contribution, in your own words, in a paragraph. The test is that its authors would recognise it and not object.
Strengths	What the paper gets right, each point naming where in the paper it is.
Weaknesses	What threatens the claims, each point naming where in the paper it is.
Questions	What you would ask the authors, phrased so that an answer is possible. Not rhetorical questions with the criticism buried in them.
Limitations	Which limitations the authors state, and which they do not. A paper that states its own limitations well should be credited for it.
Quality and clarity	A rating for each, on NeurIPS’s four-point scale — 4 excellent, 3 good, 2 fair, 1 poor — with one sentence justifying each number. The sentence is the part that is marked.
Recommendation	Accept or reject, a confidence from 1 to 5 on the scale in the guidelines, and the experiment or evidence that would move you.

Around 800 to 1200 words.

Two constraints

Name the claim and where it is. “Table 3’s baseline is undertrained, see §4.2” and not “the experiments are weak”. A judgement that does not point at something is not reviewable by the authors and is not markable by us.

You may not recommend rejection without naming the experiment that would change your mind. If you cannot name one, the objection is either not about the evidence or not yet thought through.

Reviewing without a safety net

A two-week-old preprint is the one case where the usual shortcuts are unavailable, and it is worth being clear about which ones.

There are no published reviews. On an older paper, you could write your report, then read OpenReview and quietly discover you had missed the obvious objection. Here, there is nothing to check yourself against until the session.

There is no citation count. A paper that has been cited four hundred times has been judged by the field, and it is nearly impossible to referee such a paper as though that had not happened. A paper posted last Tuesday has been judged by nobody.

An LLM has probably not read it either. Models are trained up to a cutoff, so a preprint from the last fortnight is usually outside what any of them know, and asking one about the paper tends to produce a plausible review of a paper that does not exist. You may still use an LLM the way you would on any reading — checking a definition, working through a derivation you are stuck on, asking what a baseline usually does. The judgements have to be yours, and in the session it becomes obvious whose they are.

You will find this harder than reviewing a known paper, and that difficulty is the point. It is also the ordinary condition of reviewing: real referees work without knowing what anyone else thinks.

The defence

Reports are written in pairs. Two of you take the same preprint, read it separately, write your reports without conferring, and then argue them in the session — which is how refereeing actually works: two referees who have not spoken reaching their own verdicts and then meeting.

Do not show each other your report beforehand. The disagreement is the point, and a pair who have aligned in advance has nothing to argue about and little to be marked on.

Each of you submits your own report through Moodle before the session starts. Submitting first is what makes the independence real: once both reports are in, neither of you can quietly take on the other’s objections in the room and call them your own.

Once both are submitted, swap them and read your partner’s before the session. Knowing where you disagree is what turns the session into an argument rather than two reviews read aloud, and the disagreements you find are the ones worth taking into the room.

In the session, expect to be asked which of your weaknesses you would drop, and which single one you would keep if you could keep only one. If the two reports agree, expect to be pushed on the strongest objection neither of you made.

Changing your mind during the session does not result in a loss of marks. Changing it without being able to say what changed it is.

How it is marked

Criterion	Weight	What earns full marks
Comprehension	25%	The summary is one the authors would accept, and the review engages with what the paper claims rather than what it is about.
Evidence	30%	Every judgement points to a figure, a table cell or a section. The score justifications say what earned each number.
Honesty	20%	Credit given where the paper is strong, uncertainty stated as uncertainty, and a recommendation you are willing to be wrong about.
Delivery	25%	You hold the argument in the session: you defend what survives, concede what does not, and can say why.

These are the four criteria from the strand rubric weighted for this piece. Evidence carries the most because a review without locations is an opinion, and Delivery is a quarter because the defence is where it becomes clear whether the report is yours.

The two reports in a pair are marked independently. A partner who reviews badly cannot cost you, and agreeing with a partner who turns out to be right earns nothing on its own.

# The referee report {.unnumbered} ## The referee report In this exercise, you will review a paper that has not been formally published yet. The paper must be a preprint posted to arXiv within the last two weeks, not yet published at a venue, and not a revision of something older. Check the submission history on the abstract page: you want a v1 dated within your window, with no "accepted at" note on it. That constraint is the whole exercise. A new preprint has no published reviews to fall back on, no citation count to tell you what the field has decided, and no accumulated opinion to draw on. Whatever you conclude about it, you have concluded. ::: {.column-margin} **Why not your reproduction paper?** Refereeing a paper you already know was accepted is a different task: you are checking a verdict rather than reaching one. The reproduction already gives you the paper from the inside. 6 months after the course is finished, check whether the paper was published in a formal venue. ::: ## Finding one The recent listing for a category is at [arxiv.org/list/cs.CV/recent](https://arxiv.org/list/cs.CV/recent) and new submissions are announced each weekday. Computer Vision and Pattern Recognition ([cs.CV](https://arxiv.org/list/cs.CV/recent)) is the obvious place for this course, but Machine Learning ([cs.LG](https://arxiv.org/list/cs.LG/recent)) and Artificial Intelligence ([cs.AI](https://arxiv.org/list/cs.AI/recent)) carry vision work too. Choose something you can review: - **A paper making an empirical claim**, with experiments you could argue about. Not a survey, not a position paper, not a benchmark release with no method attached. - **Close enough to the course** that you can judge the baselines. A paper whose related work you have never heard of is one you will review on presentation alone. - **Long enough to have something to say about.** A four-page workshop note does not carry a full review. If you are unsure whether a paper qualifies, say so early rather than a day before the session. ## Peer review Before a paper appears at a venue, other researchers in the same field read it and judge whether it should be accepted. That is peer review. The people doing it are unpaid, working on the paper alongside their own research, and in machine learning they are often reviewing several papers in the same fortnight. Most machine-learning venues review **double-blind**: the reviewers are not told who wrote the paper, and the authors are not told who reviewed it. The reason is that a judgement should rest on what the paper shows rather than on the reputation of the group that produced it, and the evidence is that unblinded review favours well-known authors and well-known institutions [@tomkins2017]. In that study, one conference ran both kinds of review on the same submissions, and the reviewers who could see the authors were more likely to accept papers from famous authors and from top-ranked institutions than the reviewers who could not. The blinding is imperfect. A preprint already on arXiv, a self-citation phrased as "our earlier work", or a system with a distinctive name will often identify a group to anyone who follows the area. Your review is not blind at all — you chose the paper, and its authors' names are on it. Noticing where that knowledge pulls your judgement is part of the exercise. | Role | What they do | | ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Author** | Submits the paper and answers the reviews: once, in a rebuttal, at a conference; over as many rounds of revision as the editor allows at a journal. | | **Reviewer**, or referee | Reads the paper, writes the report, scores it, and recommends acceptance or rejection. This is the part you are doing. | | **Area chair** (conferences) | Handles a batch of papers, recruits and chases the reviewers, moderates discussion between them when they disagree, and writes a *meta-review* summarising why the decision went as it did. | | **Senior area chair** or **programme chair** (conferences) | Oversees the area chairs, settles the cases they cannot, and takes responsibility for the final list. | | **Associate editor**, or handling editor (journals) | The journal equivalent of an area chair: chooses the reviewers and decides whether the paper is rejected, accepted, or returned for minor or major revision. | | **Editor-in-chief** (journals) | Assigns each submission to an associate editor and holds the final decision. | : {tbl-colwidths="[28,72]"} The two paths differ in shape. A conference decides once: reviews, a rebuttal, then discussion among the reviewers and the area chair, in which a reviewer swayed by the rebuttal can usually raise or lower their score. After that, it is accept or reject, with no route back except submitting again elsewhere months later. A journal often iterates: a paper can go through two or three rounds of revision with the same reviewers, and *major revision* is a normal outcome rather than a soft rejection. ### Outcomes A decision comes back in one of a small number of named forms, and the names are used the same way across most venues. | Outcome | What it means | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Desk rejection** | Rejected by the editor or chair without going to reviewers at all: out of scope, over length, anonymity broken, or plainly below the venue's bar. It arrives within days. | | **Reject** | Rejected after review. At a conference that is the end of it for that venue, and the paper goes to the next one. At a journal it can sometimes come with an invitation to resubmit as a new paper. | | **Major revision** | The claims may well stand, but not on the evidence as it is: experiments to add, analysis to redo, a baseline to strengthen. It goes back to the same reviewers, and it is a normal outcome rather than a soft rejection. | | **Minor revision** | Accepted in substance. Clarifications, a missing detail, presentation. Usually checked by the editor alone rather than sent round again. | | **Accept** | Published as it stands. Rare on a first journal submission. At a conference the decision also sets the format, which is usually a poster and occasionally a talk. | : {tbl-colwidths="[22,78]"} Journals use all five. Conferences in this field mostly use two, accept and reject, with the rebuttal and the reviewer discussion doing the work that a revision round does elsewhere — which is why a conference review has to say what would change your mind, since there is no second round in which to find out. #### Reviewer recommendations Those two are the paper's outcome. Your own recommendation as a reviewer is finer-grained than that, because the area chair needs to know not just which side you are on but how hard you would argue for it. NeurIPS asks for one of six: | Score | Label | | ----- | ------------------------------- | | 6 | Strong accept | | 5 | Accept | | 4 | Borderline accept / Weak accept | | 3 | Borderline reject / Weak reject | | 2 | Reject | | 1 | Strong reject | : {tbl-colwidths="[14,86]"} Regardless of the terminology, the middle two mean the same thing: the reasons on one side outweigh those on the other, and not by much. NeurIPS asks reviewers to use them sparingly, which is worth knowing, because the middle of a scale is where a reviewer who has not decided tends to sit. The gap between a 5 and a 6 is not just about your enthusiasm. A strong accept says you would argue for this paper against the other reviewers, and a strong reject says you would argue against it after reading their reports. If you would not, the score below is the honest one. The number is also not the review: an area chair weighing a borderline reject with a specific, unanswerable objection against a confident accept with three vague sentences will go with the first. #### Reviewer confidence Most venues ask for a **confidence score** alongside the recommendation, on a separate scale of 1 to 5. It answers a different question: not how good the paper is, but how much weight your opinion should carry. At the top of NeurIPS's scale, you are certain, know the related work, and have checked the details; at the bottom, your assessment is an educated guess on a paper outside your area. The point of it is that an area chair can discount a review honestly rather than guess. A confident borderline reject outweighs an uncertain strong reject, and saying you are uncertain is how a reviewer who is out of their depth avoids doing damage. Inflating it is a common failure case — your report is worth more with a 2 you can justify than with a 5 you cannot. ## What a review contains The form below is the one [NeurIPS gives its reviewers](https://neurips.cc/Conferences/2025/ReviewerGuidelines), with the same section names. [CVPR's](https://cvpr.thecvf.com/Conferences/2025/ReviewerGuidelines) differ in wording but not in substance. Read both before you start writing. They are the actual instructions sent to the people who decide whether a paper appears at some of the biggest conferences in the world; they take about fifteen minutes each, and they say a good deal that this page does not — on confidentiality, on conflicts of interest, and on what reviewers are told not to do. Where the two disagree, the disagreement is worth noticing: it tells you which parts of reviewing are settled and which are a venue's own house rules. Write every section. A review that omits its questions or limitations is not a short review; it is an incomplete one. | Section | What it must contain | | ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Summary** | The paper's claim and contribution, in your own words, in a paragraph. The test is that its authors would recognise it and not object. | | **Strengths** | What the paper gets right, each point naming where in the paper it is. | | **Weaknesses** | What threatens the claims, each point naming where in the paper it is. | | **Questions** | What you would ask the authors, phrased so that an answer is possible. Not rhetorical questions with the criticism buried in them. | | **Limitations** | Which limitations the authors state, and which they do not. A paper that states its own limitations well should be credited for it. | | **Quality and clarity** | A rating for each, on NeurIPS's four-point scale — 4 excellent, 3 good, 2 fair, 1 poor — with one sentence justifying each number. The sentence is the part that is marked. | | **Recommendation** | Accept or reject, a confidence from 1 to 5 on the scale in the guidelines, and the experiment or evidence that would move you. | : {tbl-colwidths="[18,82]"} Around 800 to 1200 words. ## Two constraints **Name the claim and where it is.** "Table 3's baseline is undertrained, see §4.2" and not "the experiments are weak". A judgement that does not point at something is not reviewable by the authors and is not markable by us. **You may not recommend rejection without naming the experiment that would change your mind.** If you cannot name one, the objection is either not about the evidence or not yet thought through. ## Reviewing without a safety net A two-week-old preprint is the one case where the usual shortcuts are unavailable, and it is worth being clear about which ones. There are no published reviews. On an older paper, you could write your report, then read OpenReview and quietly discover you had missed the obvious objection. Here, there is nothing to check yourself against until the session. There is no citation count. A paper that has been cited four hundred times has been judged by the field, and it is nearly impossible to referee such a paper as though that had not happened. A paper posted last Tuesday has been judged by nobody. An LLM has probably not read it either. Models are trained up to a cutoff, so a preprint from the last fortnight is usually outside what any of them know, and asking one about the paper tends to produce a plausible review of a paper that does not exist. You may still use an LLM the way you would on any reading — checking a definition, working through a derivation you are stuck on, asking what a baseline usually does. The judgements have to be yours, and in the session it becomes obvious whose they are. You will find this harder than reviewing a known paper, and that difficulty is the point. It is also the ordinary condition of reviewing: real referees work without knowing what anyone else thinks. ## The defence Reports are written in pairs. Two of you take the same preprint, read it separately, write your reports without conferring, and then argue them in the session — which is how refereeing actually works: two referees who have not spoken reaching their own verdicts and then meeting. Do not show each other your report beforehand. The disagreement is the point, and a pair who have aligned in advance has nothing to argue about and little to be marked on. Each of you submits your own report through Moodle before the session starts. Submitting first is what makes the independence real: once both reports are in, neither of you can quietly take on the other's objections in the room and call them your own. Once both are submitted, swap them and read your partner's before the session. Knowing where you disagree is what turns the session into an argument rather than two reviews read aloud, and the disagreements you find are the ones worth taking into the room. In the session, expect to be asked which of your weaknesses you would drop, and which single one you would keep if you could keep only one. If the two reports agree, expect to be pushed on the strongest objection neither of you made. Changing your mind during the session does not result in a loss of marks. Changing it without being able to say what changed it is. ## How it is marked | Criterion | Weight | What earns full marks | | ----------------- | ------ | ---------------------------------------------------------------------------------------------------------------------------------- | | **Comprehension** | 25% | The summary is one the authors would accept, and the review engages with what the paper claims rather than what it is about. | | **Evidence** | 30% | Every judgement points to a figure, a table cell or a section. The score justifications say what earned each number. | | **Honesty** | 20% | Credit given where the paper is strong, uncertainty stated as uncertainty, and a recommendation you are willing to be wrong about. | | **Delivery** | 25% | You hold the argument in the session: you defend what survives, concede what does not, and can say why. | : {tbl-colwidths="[22,10,68]"} These are the four criteria from the [strand rubric](index.qmd) weighted for this piece. Evidence carries the most because a review without locations is an opinion, and Delivery is a quarter because the defence is where it becomes clear whether the report is yours. The two reports in a pair are marked independently. A partner who reviews badly cannot cost you, and agreeing with a partner who turns out to be right earns nothing on its own.