Reproduction

Choosing. The curated list below is the starting point. Every paper on it has been picked so that a meaningful slice of it fits on a free Colab GPU, usually because the original was small, or because a scaled-down version of its central experiment still shows the effect. You may propose something else; bring it to one of the opening sessions and say which experiment you intend to reproduce and what hardware it needs. Papers are claimed first-come, one per student, and confirmed at the second session.

The referee report is a different paper. That exercise needs a preprint posted within the last two weeks, so the paper you reproduce cannot also be the one you review.

Reproducing. You are not reproducing the paper. You are reproducing one claim from it, a figure or a table row or an ablation, on your own code. Say at the outset which one. A reproduction that runs at a quarter of the original scale and reports honestly what that changes is worth more than one that quietly reports the original’s numbers.

The write-up. Around 1,500 to 2,500 words plus figures, and mostly about what did not reproduce. The interesting content of a reproduction is the gap: the hyperparameter the paper omits, the preprocessing step you had to guess, the result that came out two points low and stayed there. Record what you tried. A failed reproduction, documented well, marks better than a vague success.

The viva. Fifteen minutes on your own code. Expect to be asked why a particular line is there, what happens if it is removed, and which of your numbers you would defend if a reviewer challenged it.

The paper list

A starting point, not a menu. Each entry names the claim that is worth reproducing and what it costs to run. “Colab” means a free T4 session; “Colab ×N” means it needs roughly N sessions, so plan for checkpointing, which Lab 6 covers.

Reproductions that fit comfortably

Paper	The claim worth reproducing	Cost
Chiyuan Zhang et al., Understanding deep learning requires rethinking generalization — ICLR 2017	A network fits CIFAR-10 with the labels randomly shuffled, to zero training error. Generalisation is therefore not explained by capacity alone.	Colab
Frankle & Carbin, The Lottery Ticket Hypothesis — ICLR 2019	A pruned subnetwork, rewound to its initialisation, trains to the dense network’s accuracy — and to nothing like it from a fresh initialisation. One trial on Conv-2 fits; the paper’s five-trial averages do not.	Colab ×2
Hongyi Zhang et al., mixup: Beyond Empirical Risk Minimization — ICLR 2018	Training on convex combinations of pairs of images and labels improves test error, and blunts memorisation of corrupted labels (§3.4). One-line change, real effect.	Colab
Geirhos et al., ImageNet-trained CNNs are biased towards texture — ICLR 2019	A pretrained classifier follows texture over shape on cue-conflict images. Chapter 3 cites this; the stimuli are released.	Colab
Selvaraju et al., Grad-CAM — ICCV 2017	The localisation maps, and the biased-versus-unbiased doctor-and-nurse pair, where the map moves when the training set does.	Colab
Recht et al., Do CIFAR-10 Classifiers Generalize to CIFAR-10? — arXiv 2018, never formally published	Accuracy drops 4–10 points on a new test set built to follow CIFAR-10’s own collection process. The paper reads the gap as a small residual distribution shift, not as adaptive overfitting. Mostly careful evaluation.	Colab

Reproductions that need scaling down, and should say so

Paper	The claim worth reproducing	Cost
Ting Chen et al., A Simple Framework for Contrastive Learning of Visual Representations (SimCLR) — ICML 2020	The augmentation ablation (Figure 5), cut down to crop against four partners — the paper’s grid is 7×7 and asymmetric, so all 49 cells will not fit the term. Run on CIFAR-10 at small batch, and report what the batch size costs you.	Colab ×3
Tong He et al., Bag of Tricks for Image Classification — CVPR 2019	The paper stacks its tricks cumulatively (Tables 4 and 6) and never isolates one. Pick four, isolate each against a single baseline on a smaller backbone, and see whether the gains still add up.	Colab ×3
Dosovitskiy et al., An Image is Worth 16×16 Words — ICLR 2021	The data-hunger claim: ViT loses to a BiT ResNet of comparable compute when pre-training is ImageNet-1k rather than JFT (Figures 3 and 4). Downscale to CIFAR-10 from scratch, and say that you changed the axis.	Colab ×3
Lin et al., Focal Loss for Dense Object Detection — ICCV 2017	Table 1b’s γ sweep — α is retuned at every γ, so sweep the pair — on a subset of COCO or on your own annotated set from Lab 5.	Colab ×3
Nakkiran et al., Deep Double Descent — ICLR 2020	The test-error curve falling, peaking at the interpolation threshold, then falling again as model width grows (Figure 4b). The peak only appears on CIFAR-10 with label noise added, so budget for the noise.	Colab ×4
Kaiming He et al., Masked Autoencoders Are Scalable Vision Learners — CVPR 2022	That a high masking ratio still permits reconstruction, and what the probe accuracy does as the ratio moves.	Colab ×4

Reproductions that are mostly evaluation

These need little training and a great deal of care, which makes them harder than they look.

Paper	The claim worth reproducing	Cost
Radford et al., Learning Transferable Visual Models From Natural Language Supervision (CLIP) — ICML 2021	Zero-shot accuracy on a dataset CLIP was not tuned for, and how much of it moves with the prompt template alone.	Colab
Buolamwini & Gebru, Gender Shades — FAccT 2018, no arXiv version	The disparity analysis, on a modern released model and a dataset you may lawfully use. Read the data-ethics note below before starting this one.	Colab
Caron et al., Emerging Properties in Self-Supervised Vision Transformers (DINO) — ICCV 2021	That the attention maps of a self-supervised ViT segment objects without ever being told to.	Colab
Kirillov et al., Segment Anything (SAM) — ICCV 2023	Single-point ambiguity: the oracle-versus-ranked gap, and centre versus random click (Figures 9a and 9d). Then push past the paper — move the click within one object and measure how far the mask moves.	Colab

If you want something outside this list, the test is: can you name one claim, one figure or table it lives in, and a plan to test it that fits the term.

How it is marked

The write-up and the viva are marked separately, and the viva is on your own code rather than on the paper.

The write-up, 7%.

Criterion	Weight	What earns full marks
Comprehension	20%	The claim you chose is stated precisely, with the figure or table it lives in, and you can say what the original did to support it.
Evidence	30%	Your numbers are put next to theirs, compared like for like, with any change of scale stated rather than quietly absorbed.
Honesty	35%	The gap is the subject: what did not reproduce, what you tried, and what you still cannot explain. A documented failure marks well here; a vague success does not.
Reproducibility	15%	A clean checkout, `conda env create` from your `environment.yml`, and one documented command reproduce every number and figure in the write-up, without anyone needing to ask you a question.

The viva, 5%.

Criterion	Weight	What earns full marks
Comprehension	25%	You can explain the method you implemented, not only the interface you called.
Evidence	25%	Asked why a number came out as it did, you go to the line or the plot that shows it.
Honesty	20%	You separate what you measured from what you assume, and say plainly which of your results you would not defend.
Delivery	30%	You hold a technical conversation about your own code in real time, without notes you did not write.

These are the four criteria from the strand rubric. Honesty carries the most on the write-up because it is the thing a reproduction is for: anyone can report the original’s numbers, and the mark is for saying where yours differed. Reproducibility replaces Delivery there, since a written report cannot show whether you can hold a conversation — that is what the viva is for, which is why Delivery carries the most there.

Practicalities

Compute. Everything here is sized for a free Colab GPU. If your plan needs more, say so while you are still choosing rather than in the closing weeks.

Handing in. A PDF to Moodle, carrying a link to a public git repository. Public means readable without an account: check it in a logged-out browser before you submit, because a repository nobody can open is a write-up with no evidence behind it.

Code. One repository per reproduction, with a README that states which claim you are testing and how to re-run it. Two things are required: a conda environment.yml pinning what you actually used, and a single documented command that goes from a clean checkout to every number and figure in the write-up. Assume it will be run.

That command is allowed to take hours — a reproduction that trains has to train — but it may not need an undocumented manual step, a path that exists only on your machine, or a file you never committed. Where a stage is too long to repeat, commit the checkpoint and have the command skip that stage when it finds one. Commit as you go; a repository with one commit is its own kind of evidence.

Using the authors’ code. Allowed, and often the sensible route. Getting somebody else’s research code to run is where most reproductions actually fail, and learning that is part of the point. Say which route you took.

The floor is the same either way. Re-running the authors’ script and reporting the numbers it prints is not a reproduction: nothing has been tested that the authors had not already tested. Your write-up must contain at least one number the paper does not — the claim at a quarter of the scale, a second seed, an ablation they did not run, a hyperparameter moved to see whether the effect survives. That number is what the viva is about.

Reimplementing from the paper alone is harder and is marked no differently. It tests something else worth testing, which is whether the paper says enough to rebuild the method, and the places you had to guess are exactly what the write-up is for.

What is not acceptable is presenting the authors’ results as output you produced.

Data ethics. Several papers here study faces or people. If you work with one of those, check the dataset’s licence and its consent position before you download it, and say in your write-up what you found. Chapter 1’s provenance discussion is the standard the course holds.

# Reproduction {.unnumbered} ## Reproduction **Choosing.** The curated list below is the starting point. Every paper on it has been picked so that a meaningful slice of it fits on a free Colab GPU, usually because the original was small, or because a scaled-down version of its central experiment still shows the effect. You may propose something else; bring it to one of the opening sessions and say which experiment you intend to reproduce and what hardware it needs. Papers are claimed first-come, one per student, and confirmed at the second session. ::: {.column-margin} **The referee report is a different paper.** [That exercise](referee-report.qmd) needs a preprint posted within the last two weeks, so the paper you reproduce cannot also be the one you review. ::: **Reproducing.** You are not reproducing the paper. You are reproducing *one claim* from it, a figure or a table row or an ablation, on your own code. Say at the outset which one. A reproduction that runs at a quarter of the original scale and reports honestly what that changes is worth more than one that quietly reports the original's numbers. **The write-up.** Around 1,500 to 2,500 words plus figures, and mostly about what did *not* reproduce. The interesting content of a reproduction is the gap: the hyperparameter the paper omits, the preprocessing step you had to guess, the result that came out two points low and stayed there. Record what you tried. A failed reproduction, documented well, marks better than a vague success. **The viva.** Fifteen minutes on your own code. Expect to be asked why a particular line is there, what happens if it is removed, and which of your numbers you would defend if a reviewer challenged it. ## The paper list A starting point, not a menu. Each entry names the claim that is worth reproducing and what it costs to run. "Colab" means a free T4 session; "Colab ×N" means it needs roughly N sessions, so plan for checkpointing, which Lab 6 covers. ### Reproductions that fit comfortably | Paper | The claim worth reproducing | Cost | |---|---|---| | Chiyuan Zhang et al., [*Understanding deep learning requires rethinking generalization*](https://arxiv.org/abs/1611.03530) — [ICLR 2017](https://openreview.net/forum?id=Sy8gdB9xx) | A network fits CIFAR-10 with the labels randomly shuffled, to zero training error. Generalisation is therefore not explained by capacity alone. | Colab | | Frankle & Carbin, [*The Lottery Ticket Hypothesis*](https://arxiv.org/abs/1803.03635) — [ICLR 2019](https://openreview.net/forum?id=rJl-b3RcF7) | A pruned subnetwork, rewound to its initialisation, trains to the dense network's accuracy — and to nothing like it from a fresh initialisation. One trial on Conv-2 fits; the paper's five-trial averages do not. | Colab ×2 | | Hongyi Zhang et al., [*mixup: Beyond Empirical Risk Minimization*](https://arxiv.org/abs/1710.09412) — [ICLR 2018](https://openreview.net/forum?id=r1Ddp1-Rb) | Training on convex combinations of pairs of images and labels improves test error, and blunts memorisation of corrupted labels (§3.4). One-line change, real effect. | Colab | | Geirhos et al., [*ImageNet-trained CNNs are biased towards texture*](https://arxiv.org/abs/1811.12231) — [ICLR 2019](https://openreview.net/forum?id=Bygh9j09KX) | A pretrained classifier follows texture over shape on cue-conflict images. Chapter 3 cites this; the stimuli are released. | Colab | | Selvaraju et al., [*Grad-CAM*](https://arxiv.org/abs/1610.02391) — [ICCV 2017](https://openaccess.thecvf.com/content_iccv_2017/html/Selvaraju_Grad-CAM_Visual_Explanations_ICCV_2017_paper.html) | The localisation maps, and the biased-versus-unbiased doctor-and-nurse pair, where the map moves when the training set does. | Colab | | Recht et al., [*Do CIFAR-10 Classifiers Generalize to CIFAR-10?*](https://arxiv.org/abs/1806.00451) — arXiv 2018, never formally published | Accuracy drops 4–10 points on a new test set built to follow CIFAR-10's own collection process. The paper reads the gap as a small residual distribution shift, not as adaptive overfitting. Mostly careful evaluation. | Colab | ### Reproductions that need scaling down, and should say so | Paper | The claim worth reproducing | Cost | |---|---|---| | Ting Chen et al., [*A Simple Framework for Contrastive Learning of Visual Representations*](https://arxiv.org/abs/2002.05709) (SimCLR) — [ICML 2020](https://proceedings.mlr.press/v119/chen20j.html) | The augmentation ablation (Figure 5), cut down to crop against four partners — the paper's grid is 7×7 and asymmetric, so all 49 cells will not fit the term. Run on CIFAR-10 at small batch, and report what the batch size costs you. | Colab ×3 | | Tong He et al., [*Bag of Tricks for Image Classification*](https://arxiv.org/abs/1812.01187) — [CVPR 2019](https://openaccess.thecvf.com/content_CVPR_2019/html/He_Bag_of_Tricks_for_Image_Classification_with_Convolutional_Neural_Networks_CVPR_2019_paper.html) | The paper stacks its tricks cumulatively (Tables 4 and 6) and never isolates one. Pick four, isolate each against a single baseline on a smaller backbone, and see whether the gains still add up. | Colab ×3 | | Dosovitskiy et al., [*An Image is Worth 16×16 Words*](https://arxiv.org/abs/2010.11929) — [ICLR 2021](https://openreview.net/forum?id=YicbFdNTTy) | The data-hunger claim: ViT loses to a BiT ResNet of comparable compute when pre-training is ImageNet-1k rather than JFT (Figures 3 and 4). Downscale to CIFAR-10 from scratch, and say that you changed the axis. | Colab ×3 | | Lin et al., [*Focal Loss for Dense Object Detection*](https://arxiv.org/abs/1708.02002) — [ICCV 2017](https://openaccess.thecvf.com/content_iccv_2017/html/Lin_Focal_Loss_for_ICCV_2017_paper.html) | Table 1b's γ sweep — α is retuned at every γ, so sweep the pair — on a subset of COCO or on your own annotated set from Lab 5. | Colab ×3 | | Nakkiran et al., [*Deep Double Descent*](https://arxiv.org/abs/1912.02292) — [ICLR 2020](https://openreview.net/forum?id=B1g5sA4twr) | The test-error curve falling, peaking at the interpolation threshold, then falling again as model width grows (Figure 4b). The peak only appears on CIFAR-10 with label noise added, so budget for the noise. | Colab ×4 | | Kaiming He et al., [*Masked Autoencoders Are Scalable Vision Learners*](https://arxiv.org/abs/2111.06377) — [CVPR 2022](https://openaccess.thecvf.com/content/CVPR2022/html/He_Masked_Autoencoders_Are_Scalable_Vision_Learners_CVPR_2022_paper.html) | That a high masking ratio still permits reconstruction, and what the probe accuracy does as the ratio moves. | Colab ×4 | ### Reproductions that are mostly evaluation These need little training and a great deal of care, which makes them harder than they look. | Paper | The claim worth reproducing | Cost | |---|---|---| | Radford et al., [*Learning Transferable Visual Models From Natural Language Supervision*](https://arxiv.org/abs/2103.00020) (CLIP) — [ICML 2021](https://proceedings.mlr.press/v139/radford21a.html) | Zero-shot accuracy on a dataset CLIP was not tuned for, and how much of it moves with the prompt template alone. | Colab | | Buolamwini & Gebru, [*Gender Shades*](https://proceedings.mlr.press/v81/buolamwini18a.html) — FAccT 2018, no arXiv version | The disparity analysis, on a modern released model and a dataset you may lawfully use. Read the data-ethics note below before starting this one. | Colab | | Caron et al., [*Emerging Properties in Self-Supervised Vision Transformers*](https://arxiv.org/abs/2104.14294) (DINO) — [ICCV 2021](https://openaccess.thecvf.com/content/ICCV2021/html/Caron_Emerging_Properties_in_Self-Supervised_Vision_Transformers_ICCV_2021_paper.html) | That the attention maps of a self-supervised ViT segment objects without ever being told to. | Colab | | Kirillov et al., [*Segment Anything*](https://arxiv.org/abs/2304.02643) (SAM) — [ICCV 2023](https://openaccess.thecvf.com/content/ICCV2023/html/Kirillov_Segment_Anything_ICCV_2023_paper.html) | Single-point ambiguity: the oracle-versus-ranked gap, and centre versus random click (Figures 9a and 9d). Then push past the paper — move the click within one object and measure how far the mask moves. | Colab | If you want something outside this list, the test is: can you name one claim, one figure or table it lives in, and a plan to test it that fits the term. ## How it is marked The write-up and the viva are marked separately, and the viva is on your own code rather than on the paper. **The write-up**, 7%. | Criterion | Weight | What earns full marks | | ----------------- | ------ | ----------------------------------------------------------------------------------------------------------------- | | **Comprehension** | 20% | The claim you chose is stated precisely, with the figure or table it lives in, and you can say what the original did to support it. | | **Evidence** | 30% | Your numbers are put next to theirs, compared like for like, with any change of scale stated rather than quietly absorbed. | | **Honesty** | 35% | The gap is the subject: what did not reproduce, what you tried, and what you still cannot explain. A documented failure marks well here; a vague success does not. | | **Reproducibility** | 15% | A clean checkout, `conda env create` from your `environment.yml`, and one documented command reproduce every number and figure in the write-up, without anyone needing to ask you a question. | : {tbl-colwidths="[22,10,68]"} **The viva**, 5%. | Criterion | Weight | What earns full marks | | ----------------- | ------ | ----------------------------------------------------------------------------------------------------------------- | | **Comprehension** | 25% | You can explain the method you implemented, not only the interface you called. | | **Evidence** | 25% | Asked why a number came out as it did, you go to the line or the plot that shows it. | | **Honesty** | 20% | You separate what you measured from what you assume, and say plainly which of your results you would not defend. | | **Delivery** | 30% | You hold a technical conversation about your own code in real time, without notes you did not write. | : {tbl-colwidths="[22,10,68]"} These are the four criteria from the [strand rubric](index.qmd). Honesty carries the most on the write-up because it is the thing a reproduction is for: anyone can report the original's numbers, and the mark is for saying where yours differed. *Reproducibility* replaces *Delivery* there, since a written report cannot show whether you can hold a conversation — that is what the viva is for, which is why Delivery carries the most there. ## Practicalities **Compute.** Everything here is sized for a free Colab GPU. If your plan needs more, say so while you are still choosing rather than in the closing weeks. **Handing in.** A PDF to Moodle, carrying a link to a public git repository. Public means readable without an account: check it in a logged-out browser before you submit, because a repository nobody can open is a write-up with no evidence behind it. **Code.** One repository per reproduction, with a README that states which claim you are testing and how to re-run it. Two things are required: a conda `environment.yml` pinning what you actually used, and a single documented command that goes from a clean checkout to every number and figure in the write-up. Assume it will be run. That command is allowed to take hours — a reproduction that trains has to train — but it may not need an undocumented manual step, a path that exists only on your machine, or a file you never committed. Where a stage is too long to repeat, commit the checkpoint and have the command skip that stage when it finds one. Commit as you go; a repository with one commit is its own kind of evidence. **Using the authors' code.** Allowed, and often the sensible route. Getting somebody else's research code to run is where most reproductions actually fail, and learning that is part of the point. Say which route you took. The floor is the same either way. Re-running the authors' script and reporting the numbers it prints is not a reproduction: nothing has been tested that the authors had not already tested. Your write-up must contain at least one number the paper does not — the claim at a quarter of the scale, a second seed, an ablation they did not run, a hyperparameter moved to see whether the effect survives. That number is what the viva is about. Reimplementing from the paper alone is harder and is marked no differently. It tests something else worth testing, which is whether the paper says enough to rebuild the method, and the places you had to guess are exactly what the write-up is for. What is not acceptable is presenting the authors' results as output you produced. **Data ethics.** Several papers here study faces or people. If you work with one of those, check the dataset's licence and its consent position before you download it, and say in your write-up what you found. [Chapter 1](../chapters/01-classical-vision.qmd)'s provenance discussion is the standard the course holds.