{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Lab 3 — Transfer learning on a dataset of your own\n", "\n", "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/kleinric/cv-labs/blob/main/lab-03.ipynb)\n", "\n", "**COMS4036A / COMS7050A Computer Vision**\n", "\n", "Groups of up to three; one member submits the group's notebook (`.ipynb`) on Moodle. Every member must be able to explain every cell.\n", "\n", "Companion reading is [Chapter 3 of the course book](https://courses.ms.wits.ac.za/~richard/cv/book/chapters/03-architectures-transfer.html). Lab 2 trained a small network from scratch on CIFAR-10, where the book's reference run reaches 77% validation accuracy after forty epochs. This lab classifies 37 cat and dog breeds, a harder problem with about a hundred training photographs per class where CIFAR-10 gave five thousand, and gets past 90% in six epochs. This week's network is the bigger one, but Section 6's control shows that is not where the difference lives. What changed is where the weights started.\n", "\n", "Two rungs of the skills ladder are marked this week: a **`Dataset` you write yourself**, from a directory of files and a text file of labels, and an **augmentation pipeline you look at before you train on it**.\n", "\n", "## 0. GPU, W&B, and the data\n", "\n", "Open this notebook in [Colab](https://colab.research.google.com) with the badge above, then **File ▸ Save a copy in Drive**. Set **Runtime ▸ Change runtime type** to a GPU as in Lab 2, and log in to W&B; this lab logs to a project called `cv-lab3`." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# Group members — fill in before submitting.\n", "MEMBERS = [\n", " # (\"Student name\", \"Student number\"),\n", "]\n", "for name, number in MEMBERS:\n", " print(f\"{number} {name}\")" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": {}, "source": [ "!nvidia-smi --query-gpu=name,memory.total --format=csv" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": {}, "source": [ "import wandb\n", "wandb.login()" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The dataset is the [Oxford-IIIT Pet dataset](https://www.robots.ox.ac.uk/~vgg/data/pets/) (Parkhi et al., 2012): 37 breeds of cat and dog, roughly 200 photographs each, with an official train/test split. It is licensed CC BY-SA 4.0. Two archives, about 800 MB together; the download takes a minute or two." ] }, { "cell_type": "code", "metadata": {}, "source": [ "!wget -q -nc https://thor.robots.ox.ac.uk/pets/images.tar.gz\n", "!wget -q -nc https://thor.robots.ox.ac.uk/pets/annotations.tar.gz\n", "!tar -xzf images.tar.gz && tar -xzf annotations.tar.gz\n", "!ls annotations | head && head -3 annotations/trainval.txt" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": {}, "source": [ "import random\n", "\n", "import matplotlib.pyplot as plt\n", "import numpy as np\n", "import torch\n", "import torch.nn as nn\n", "import torch.nn.functional as F\n", "from PIL import Image\n", "from torch.utils.data import DataLoader, Dataset, Subset\n", "from torchvision import transforms\n", "from torchvision.models import ResNet50_Weights, resnet50\n", "\n", "DEV = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n", "\n", "\n", "def seed_everything(seed=0):\n", " random.seed(seed)\n", " np.random.seed(seed)\n", " torch.manual_seed(seed)\n", " torch.cuda.manual_seed_all(seed)\n", "\n", "\n", "seed_everything(0)\n", "print(\"device:\", DEV)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. A `Dataset` of your own\n", "\n", "Lab 2's data arrived as `datasets.CIFAR10(...)`: one line, already split, already labelled. Real data does not. Here you have a directory of JPEGs and a text file, and the job of turning them into something a `DataLoader` can shuffle is yours.\n", "\n", "`annotations/trainval.txt` and `annotations/test.txt` have one line per image:\n", "\n", "```\n", "Abyssinian_100 1 1 1\n", "```\n", "\n", "The fields are the image's file stem, its class ID from 1 to 37, its species (1 cat, 2 dog), and a breed ID within the species. Only the first two matter here. The breed name is in the stem: strip the trailing `_100`.\n", "\n", "Three things about this directory will break a naive implementation, and all three are ordinary:\n", "\n", "- `images/` holds a few `.mat` files as well as `.jpg`, and it holds 41 more photographs than the two split files list. Read the split file; do not list the directory.\n", "- Three of the listed JPEGs decode to RGBA rather than RGB. `Image.open(...).convert(\"RGB\")` fixes them, and without it your batches will fail to stack part-way through a run rather than at the start of one.\n", "- Opening every image in `__init__` would exhaust Colab's memory. `__init__` reads the *list*; `__getitem__` reads the *image*.\n", "\n", "**Q1.1.** Implement `PetsDataset`. `__init__` takes the split name and a transform and builds two parallel lists, of file stems and of integer labels in $[0, 37)$, plus `self.classes`, the 37 breed names in class-ID order. `__len__` returns the number of images; `__getitem__(i)` opens image $i$, converts it to RGB, applies the transform, and returns `(image, label)`." ] }, { "cell_type": "code", "metadata": {}, "source": [ "class PetsDataset(Dataset):\n", " # YOUR CODE HERE\n", " ..." ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q1.2.** Build the two splits with `transform=None` for now and check them: how many images in each, how many classes, and the smallest and largest number of images any class has in the training split? Print `train_set.classes[:5]`. `wc -l annotations/trainval.txt annotations/test.txt` gives the counts your parser must reproduce; if yours disagree, your parser is wrong, not the dataset." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# YOUR CODE HERE" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Answer:*\n", "\n", "**Q1.3.** Display one photograph per class in a $5 \\times 8$ grid, titled with the breed name. Look at them. Which pairs of breeds do you expect the network to confuse, and why?" ] }, { "cell_type": "code", "metadata": {}, "source": [ "# YOUR CODE HERE" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Answer:*\n", "\n", "## 2. Augmentation, seen before it is trained on\n", "\n", "Two transform pipelines, doing different jobs. The evaluation pipeline is fixed: resize the short side to 256, take the centre $224 \\times 224$ crop, and normalise with **ImageNet's** channel statistics, the ones the pretrained weights were fitted against. The training pipeline adds the randomness: a random resized crop and a horizontal flip, so the network sees a slightly different photograph of the same animal at every epoch.\n", "\n", "**Q2.1.** Build `eval_tf` and `train_tf` as described (`transforms.RandomResizedCrop(224, scale=(0.6, 1.0))`, `RandomHorizontalFlip`, `ToTensor`, and the normalisation with mean `(0.485, 0.456, 0.406)` and standard deviation `(0.229, 0.224, 0.225)`). Rebuild both datasets with them, and build a training loader with `batch_size=32, shuffle=True, num_workers=2` and a test loader with `batch_size=64`." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# YOUR CODE HERE" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q2.2.** Now look at what you just built. Take one training batch and display its first twelve images, un-normalised for display (multiply by the standard deviation, add the mean, clip to $[0,1]$). Run the cell twice. This is the week's second rung: if a crop has cut the animal's head off, or the colours are wrong, or the image is upside down, you find out now rather than after an hour of training." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# YOUR CODE HERE" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q2.3.** Two augmentations you were *not* given: `RandomVerticalFlip` and `RandomRotation(180)`. For each, say whether you would include it here and why. Then name one augmentation that would be sensible for these photographs and is not in the list above.\n", "\n", "*Answer:*\n", "\n", "**Q2.4.** The normalisation numbers above are ImageNet's, not this dataset's. Compute the Pet training set's own channel means and standard deviations (a few hundred images is enough for two decimal places; say so if you subsample). Report both triples. Then answer: why does the pretrained backbone want ImageNet's numbers rather than the ones you just computed?" ] }, { "cell_type": "code", "metadata": {}, "source": [ "# YOUR CODE HERE" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Answer:*\n", "\n", "## 3. The backbone" ] }, { "cell_type": "code", "metadata": {}, "source": [ "model = resnet50(weights=ResNet50_Weights.IMAGENET1K_V1)\n", "print(model.fc)\n", "print(f\"{sum(p.numel() for p in model.parameters()):,} parameters\")" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Those weights are the output of a training run nobody in this room is going to repeat: 1.2 million labelled photographs, 1,000 classes, and the result is 76.1% top-1 accuracy on ImageNet's validation set. The last layer is 2048 → 1000, the only part whose shape is tied to those thousand nouns; how far back the specialisation reaches is what Section 6 measures.\n", "\n", "**Q3.1.** Write `make_model(recipe, n_classes=37)` returning a ResNet-50 with `fc` replaced by a fresh `nn.Linear(2048, n_classes)`, set up for one of two recipes:\n", "\n", "- `\"frozen\"`: every parameter outside `fc` has `requires_grad = False`;\n", "- `\"full\"`: everything trains.\n", "\n", "Print, for each recipe, the number of trainable parameters and their percentage of the total. Check the frozen head's count by hand from the layer's shape before you trust the code." ] }, { "cell_type": "code", "metadata": {}, "source": [ "def make_model(recipe, n_classes=37):\n", " # YOUR CODE HERE\n", " ..." ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Answer:*\n", "\n", "**Q3.2.** Chapter 3's third transfer practicality is batch norm: freeze its running statistics. Explain in two sentences what `model.eval()` does to a `BatchNorm2d` layer, and what would go wrong in the frozen recipe if you left the backbone in `train()` mode. (Your training loop in Section 4 has to act on this.)\n", "\n", "*Answer:*\n", "\n", "## 4. Train the frozen recipe\n", "\n", "**Q4.1.** Write `evaluate(model, loader)` and `train(model, epochs, opt, run_name, config, freeze_bn=False)`, as in Lab 2, with one addition. When `freeze_bn` is true, the model goes into `eval()` mode for the whole epoch and only the head is switched to `train()`, so the backbone's running statistics stay the ones ImageNet gave it. Log `train/loss`, `train/acc`, `val/loss`, `val/acc` per epoch to W&B and return the history." ] }, { "cell_type": "code", "metadata": {}, "source": [ "def evaluate(model, loader):\n", " # YOUR CODE HERE\n", " ...\n", "\n", "\n", "def train(model, epochs, opt, run_name, config, freeze_bn=False):\n", " # YOUR CODE HERE\n", " ..." ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q4.2.** Seed, build the frozen model, and train it for 6 epochs with SGD on the head at learning rate $10^{-2}$, momentum 0.9, weight decay $10^{-4}$. Report the final test accuracy. Compare it against 1/37, what uniform guessing over 37 classes gets, and against Lab 2's twenty-epoch CIFAR-10 result." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# YOUR CODE HERE" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Answer:*\n", "\n", "## 5. Train the full fine-tune\n", "\n", "**Q5.1.** Same six epochs, same optimiser, same head learning rate, but now every parameter moves and the *backbone* gets a learning rate ten times smaller than the head's. Build the optimiser with two parameter groups to do that. Report the final test accuracy beside the frozen recipe's, and plot both runs' validation accuracy per epoch on one set of axes." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# YOUR CODE HERE" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Answer:*\n", "\n", "**Q5.2.** Break it deliberately: fine-tune again with the backbone at the *same* learning rate as the head, with no tenfold reduction, and plot the first two epochs' validation accuracy against the previous run's. What happens in the first epoch, and what does that say about where the pretrained weights sit?" ] }, { "cell_type": "code", "metadata": {}, "source": [ "# YOUR CODE HERE" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Answer:*\n", "\n", "## 6. How deep to cut\n", "\n", "Chapter 3 quotes Yosinski et al., who spliced networks between two halves of ImageNet: early layers transferred with almost no loss, late layers were specific to the source task, and transferability fell off with depth in between. Their split was one dataset against itself, so your own data is a case their experiment did not cover. A **linear probe** freezes the backbone, takes the activations at some depth, averages them over space, and fits one linear layer to them, so whatever accuracy it reaches is credited to the representation alone.\n", "\n", "**Q6.1.** Write `features_at(model, loader, cut)` which runs the frozen backbone and returns the globally average-pooled activations at `cut` $\\in$ {`layer1`, `layer2`, `layer3`, `layer4`}, along with the labels. Use `torch.no_grad()` and the *evaluation* transform for both splits, with no augmentation, because you want the same feature for the same image every time. Report each cut's feature dimension; check them against the shapes in Chapter 3's architecture browser." ] }, { "cell_type": "code", "metadata": {}, "source": [ "def features_at(model, loader, cut):\n", " # YOUR CODE HERE\n", " ..." ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q6.2.** For each of the four cuts, fit a linear classifier on the cached training features and report test accuracy. (A single `nn.Linear` trained with Adam for a hundred passes over the cached features takes seconds, since no images are touched.) Plot accuracy against cut depth." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# YOUR CODE HERE" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q6.3.** Now the control. Repeat the whole of Q6.2 with `resnet50(weights=None)`: the same architecture, the same probe, random weights. Plot both curves together. What does the gap between them measure, and what does the *random* curve's shape tell you about how much of a convolutional network's usefulness is in its wiring rather than its weights?" ] }, { "cell_type": "code", "metadata": {}, "source": [ "# YOUR CODE HERE" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Answer:*\n", "\n", "## 7. How much data\n", "\n", "Chapter 3's recipe table turns on two questions: how much target data, and how far the domain sits from the pretraining. This section takes the first. At what training-set size does full fine-tuning overtake a frozen backbone? Section 8 takes the second.\n", "\n", "**Q7.1.** Write `subset_per_class(dataset, n, seed)` returning a `Subset` with `n` images from each of the 37 classes, drawn with a seeded permutation, or every image of a class that holds fewer than `n`." ] }, { "cell_type": "code", "metadata": {}, "source": [ "def subset_per_class(dataset, n, seed):\n", " # YOUR CODE HERE\n", " ..." ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q7.2.** For $n \\in \\{5, 25, 100\\}$ images per class, train both recipes: six runs, all logged, all seeded, all on the same epoch budget. Plot test accuracy against $n$ with one line per recipe. Which recipe wins at each size, and where do the lines cross? If they do not cross in your runs, say so: the honest answer is what your numbers show, not what the chapter's figure shows." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# YOUR CODE HERE" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Answer:*\n", "\n", "**Q7.3.** With 5 images per class you have 185 training images and 23.6 million parameters in the full recipe. Give the reason the frozen recipe is the safer choice at that size in terms Chapter 2 would recognise.\n", "\n", "*Answer:*\n", "\n", "## 8. A domain further away\n", "\n", "Pets are close to ImageNet: about 120 of ImageNet's 1,000 classes are dog breeds, so many of the features you have been reusing were fitted on photographs of these very animals. Textures are not. The [Describable Textures Dataset](https://www.robots.ox.ac.uk/~vgg/data/dtd/) (Cimpoi et al., 2014) holds 5,640 images across 47 texture classes, among them *banded*, *cracked* and *honeycombed*, and it is the problem Lab 1 attacked with a hand-built Gabor bank." ] }, { "cell_type": "code", "metadata": {}, "source": [ "!wget -q -nc https://thor.robots.ox.ac.uk/dtd/dtd-r1.0.1.tar.gz && tar -xzf dtd-r1.0.1.tar.gz\n", "!ls dtd/images | head -3 && head -2 dtd/labels/train1.txt" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q8.1.** Write a `Dataset` for DTD, where the class is the directory name and `dtd/labels/train1.txt` and `test1.txt` list one relative path per line, and repeat the depth probe of Q6.2 on it. Plot the Pets and DTD probe curves on one set of axes." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# YOUR CODE HERE" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q8.2.** The two curves have different shapes. Say what that difference means for a practitioner who has 600 images of something ImageNet has never seen, and which of Chapter 3's four recipe quadrants they are in.\n", "\n", "*Answer:*\n", "\n", "## 9. The record\n", "\n", "Paste links to your W&B runs below. At minimum: the frozen run, the full fine-tune, the broken same-learning-rate run, and the six runs of Section 7. Each needs a meaningful name and a complete config: recipe, learning rates, epochs, training-set size, seed, trainable parameter count.\n", "\n", "*W&B run links:*\n", "\n", "## 10. Before you submit\n", "\n", "- [ ] **Runtime ▸ Restart session and run all** on a GPU runtime, then read every output. The full re-run downloads both datasets, trains nine networks, and fits twelve linear probes; budget an hour.\n", "- [ ] Group members filled in; every member can explain every cell.\n", "- [ ] Every *Answer:* cell answered; W&B links pasted in Section 9.\n", "- [ ] **File ▸ Download ▸ Download .ipynb**; one member submits it on Moodle by the deadline shown there." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" } }, "nbformat": 4, "nbformat_minor": 4 }