{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Lab 2 — A CNN from scratch on CIFAR-10\n", "\n", "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/kleinric/cv-labs/blob/main/lab-02.ipynb)\n", "\n", "**COMS4036A / COMS7050A Computer Vision**\n", "\n", "Groups of up to three; one member submits the group's notebook (`.ipynb`) on Moodle. Every member must be able to explain every cell.\n", "\n", "Companion reading is [Chapter 2 of the course book](https://courses.ms.wits.ac.za/~richard/cv/book/chapters/02-learning-the-filters.html). The network you build here is *the* Lab 2 network, the one whose training curves, filter snapshots, and activation atlas appear in the chapter's figures, so you can check everything you produce against the book.\n", "\n", "Three pieces of practitioner craft are this week's skills-ladder rung, and all three are marked: **every run logged to Weights & Biases**, **everything seeded**, and the **overfit-one-batch ritual** before any real training.\n", "\n", "## 0. GPU and W&B\n", "\n", "Open this notebook in [Colab](https://colab.research.google.com), using the badge above or uploading it as in Lab 1, then **File ▸ Save a copy in Drive**. Two things are different this week.\n", "\n", "**The GPU.** Under **Runtime ▸ Change runtime type**, select a GPU (a T4 is fine). Training this lab's network on the CPU takes over an hour; on the T4 it is minutes. The cell below is Lab 1's runtime check; this time it should print a GPU.\n", "\n", "**Weights & Biases.** W&B records every run's configuration, curves, and outputs on a page you can share; from this lab to the end of the course, unlogged runs don't count. Before the login cell, each group member (not one per group, each of you):\n", "\n", "1. Create a free account at [wandb.ai](https://wandb.ai) **with your Wits email**, or, if you already have an account, add your Wits email to it under your account settings.\n", "2. Apply for the free **academic plan** from that account ([how and why](https://docs.wandb.ai/support/academic_plan_student/)). It upgrades your account at no cost, and the course's W&B team, which later labs log to, requires it.\n", "3. When the course's W&B team is ready, a Moodle form will collect your W&B username; submit it and accept the emailed invite. This lab still logs to your personal `cv-lab2` project.\n", "\n", "The login cell will ask for the API key from [wandb.ai/authorize](https://wandb.ai/authorize)." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# Group members — fill in before submitting.\n", "MEMBERS = [\n", " # (\"Student name\", \"Student number\"),\n", "]\n", "for name, number in MEMBERS:\n", " print(f\"{number} {name}\")" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": {}, "source": [ "!nvidia-smi" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": {}, "source": [ "import wandb\n", "wandb.login()" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Setup, and seeding everything" ] }, { "cell_type": "code", "metadata": {}, "source": [ "import random\n", "import time\n", "\n", "import matplotlib.pyplot as plt\n", "import numpy as np\n", "import torch\n", "import torch.nn as nn\n", "import torch.nn.functional as F\n", "from torchvision import datasets, transforms\n", "\n", "DEV = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n", "print(\"device:\", DEV)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A training run touches four random number generators: Python's, NumPy's, PyTorch's CPU one, and PyTorch's GPU one. Seed all four, every time, in one function. That is what makes a run *reproducible*: the same seed rebuilds the same setup, and the numbers come back close. (Bitwise-identical is more than GPU arithmetic promises; close is what you should expect.)" ] }, { "cell_type": "code", "metadata": {}, "source": [ "def seed_everything(seed=0):\n", " random.seed(seed)\n", " np.random.seed(seed)\n", " torch.manual_seed(seed)\n", " torch.cuda.manual_seed_all(seed)\n", "\n", "seed_everything(0)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "CIFAR-10: 50,000 training and 10,000 test images, $32 \\times 32$, ten classes. The test set serves as this lab's held-out validation split, which is what the `val/` metrics below are. The transforms normalise each channel with the dataset's own statistics; the training set additionally gets the chapter's light augmentation, random crops and horizontal flips." ] }, { "cell_type": "code", "metadata": {}, "source": [ "norm = transforms.Normalize((0.4914, 0.4822, 0.4465),\n", " (0.2470, 0.2435, 0.2616))\n", "train_tf = transforms.Compose([transforms.RandomCrop(32, padding=4),\n", " transforms.RandomHorizontalFlip(),\n", " transforms.ToTensor(), norm])\n", "test_tf = transforms.Compose([transforms.ToTensor(), norm])\n", "\n", "train_set = datasets.CIFAR10(\"data\", train=True, download=True, transform=train_tf)\n", "test_set = datasets.CIFAR10(\"data\", train=False, download=True, transform=test_tf)\n", "# num_workers=0: every random draw, shuffling and augmentation alike,\n", "# then flows from the seed, which the Q1.2 experiment depends on.\n", "train_loader = torch.utils.data.DataLoader(train_set, batch_size=128,\n", " shuffle=True, num_workers=0)\n", "test_loader = torch.utils.data.DataLoader(test_set, batch_size=512,\n", " num_workers=0)\n", "print(train_set.classes)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q1.1.** Display a grid of one training image per class, titled with its class name. (`train_set.data` holds the raw `uint8` images and `train_set.targets` the labels, so no un-normalising is needed if you read from there.)" ] }, { "cell_type": "code", "metadata": {}, "source": [ "# YOUR CODE HERE" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q1.2.** Now look at what the network actually sees. Take one batch from `train_loader` and display its first eight images (un-normalise for display: multiply by the std and add the mean, channel-wise, then clip). Run the cell twice: the crops and flips differ, because augmentation is random. Now **Runtime ▸ Restart session and run all**: the same augmented batch comes back. Why different within a session but identical across restarts, and which line guarantees it?" ] }, { "cell_type": "code", "metadata": {}, "source": [ "# YOUR CODE HERE" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Answer:*\n", "\n", "## 2. The MLP baseline\n", "\n", "The obvious network flattens the image and connects everything to everything. Build it, count it, train it: it is the baseline the CNN has to beat.\n", "\n", "**Q2.1.** Implement `MLP`: flatten, a hidden layer of 512 units with ReLU, then a linear layer to 10 classes. Implement `count_params(model)` returning the total number of trainable parameters, and report the MLP's count. Verify the first layer's share by hand: $3072 \\times 512$ weights plus 512 biases, the chapter's flatten-and-connect counting at this width." ] }, { "cell_type": "code", "metadata": {}, "source": [ "class MLP(nn.Module):\n", " # YOUR CODE HERE\n", " ...\n", "\n", "\n", "def count_params(model):\n", " # YOUR CODE HERE\n", " ..." ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q2.2.** Write the two functions every experiment in this course reuses, and log to W&B from the start. The loss, here and everywhere in this lab, is cross-entropy on the logits: `F.cross_entropy(model(x), y)`." ] }, { "cell_type": "code", "metadata": {}, "source": [ "def evaluate(model, loader):\n", " \"\"\"Return (mean loss, accuracy) over a loader, without touching\n", " gradients.\"\"\"\n", " # YOUR CODE HERE\n", " ...\n", "\n", "\n", "def train(model, epochs, opt, run_name, config):\n", " \"\"\"Train on train_loader, evaluate each epoch, log everything to W&B.\n", "\n", " Start a run with wandb.init(project=\"cv-lab2\", name=run_name,\n", " config=config); each epoch log train/loss, train/acc, val/loss,\n", " val/acc; wandb.finish() at the end. Return the history as a dict of\n", " lists under those same four keys.\"\"\"\n", " # YOUR CODE HERE\n", " ..." ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q2.3.** Train the MLP for 5 epochs with Adam (`torch.optim.Adam`, learning rate $10^{-3}$). Seed first, and give the run a name and a config recording at least the architecture, optimiser, learning rate, epochs, parameter count, and seed. Report the final test accuracy." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# YOUR CODE HERE" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3. The chapter's network\n", "\n", "The template, at lab scale: two blocks of two $3 \\times 3$ convolutions each (32 channels, then 64), a $2 \\times 2$ max-pool after each block, global average pooling, one linear layer to the classes. No batch norm; that arrives with Chapter 3.\n", "\n", "**Q3.1.** Implement it exactly. Every convolution has `padding=1` (so the $3 \\times 3$s keep the spatial size, which you can check with the output-size formula) and a ReLU after it. Name the layers `c1`, `c2`, `c3`, `c4`, and `fc`, as in the book's training code, a later question reads `model.c1.weight` by that name. Give `__init__` a `first_kernel=3` argument that sets `c1`'s kernel size (with padding `first_kernel // 2`); Section 6 uses it." ] }, { "cell_type": "code", "metadata": {}, "source": [ "class LabNet(nn.Module):\n", " \"\"\"[conv32-conv32-pool]-[conv64-conv64-pool]-GAP-linear.\"\"\"\n", " # YOUR CODE HERE\n", " ..." ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q3.2.** Before running `count_params`: walk the shapes on paper with the output-size formula, layer by layer, and compute the parameter count of each layer by hand ($C_{\\text{out}} \\times (C_{\\text{in}} \\times k \\times k + 1)$). Then check both against the code: `count_params(LabNet())` should agree with your sum exactly. How many MLP parameters buy one LabNet?" ] }, { "cell_type": "code", "metadata": {}, "source": [ "# YOUR CODE HERE" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Answer:*\n", "\n", "## 4. The ritual: overfit one batch\n", "\n", "Before any real training run, always: take a single batch and overfit it, so that every image in the batch is classified correctly and the loss is driven very small. A network that cannot memorise 128 images has a wiring bug — the loss is not connected to what you think, the labels are shuffled, the learning rate is absurd — and no amount of patience on the full set will fix it. Thirty seconds now saves an hour later.\n", "\n", "The ritual needs the batch held fixed and nothing working against the memorisation. Both hold here without extra work: you draw `xb, yb` once and reuse them, so the random crop and flip are applied a single time and then frozen; `LabNet` has no dropout; and Adam's default `weight_decay` is zero. Switch them off anywhere they are live: augmentation resampling each step, dropout on, weight decay set.\n", "\n", "**Q4.1.** Seed, then take one batch from `train_loader` and train a fresh `LabNet` on that batch alone, several hundred Adam steps (800 is plenty), printing the loss as it falls. Log this run to W&B too (name it so it is obviously the ritual, not a result). How many steps until all 128 images are classified correctly, and how many until the loss is below 0.01?" ] }, { "cell_type": "code", "metadata": {}, "source": [ "# YOUR CODE HERE" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q4.2.** Sabotage it: with a **fresh, seeded** `LabNet` and the same batch, shuffle the labels (`yb[torch.randperm(len(yb))]`) and run the ritual again. It still gets there, on labels that mean nothing. What does that tell you about what the ritual does and does not test? This is the random-label experiment of Zhang et al., *Understanding deep learning requires rethinking generalization*, in miniature; at full scale, on all 50,000 images, it is the result that made capacity alone an unsatisfying explanation of generalisation." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# YOUR CODE HERE" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Answer:*\n", "\n", "## 5. Train for real, and read the curves\n", "\n", "**Q5.1.** Seed, then train a fresh `LabNet`, kept in a variable named `model` because Section 6 reads its weights, for 20 epochs with Adam at $10^{-3}$. Adam is the lab's default optimiser: robust to a mediocre learning rate, at some cost in final accuracy. Config and curves to W&B as before. Plot training and validation accuracy per epoch from your returned history, and report the final test accuracy next to the MLP's." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# YOUR CODE HERE" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q5.2.** Read your curves against the chapter's four regimes (the training-curves widget). Which shape is this? The book's reference run, the same network trained with SGD with momentum at lr 0.01 for 40 epochs, reaches 77% validation accuracy. Where does yours land, and is the remaining train–validation gap overfitting or healthy?\n", "\n", "*Answer:*\n", "\n", "**Q5.3.** Break it on purpose. Train a fresh, seeded `LabNet` in a *separate* variable (say `bad`), so your Section 5 `model` survives for Section 6, for 8 epochs with SGD at learning rate 1.0 (momentum 0.9) and log it. Which of the chapter's pathologies do your curves show, and what is the tell?" ] }, { "cell_type": "code", "metadata": {}, "source": [ "# YOUR CODE HERE" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Answer:*\n", "\n", "## 6. The learned filter bank\n", "\n", "Chapter 1 ended with a designed filter bank. Nobody chose the numbers in yours, and some of them will still come out resembling the classical filters, because edges, colour contrasts and oriented patterns are useful for classifying CIFAR-10, not because the network was told to build a bank. Look.\n", "\n", "**Q6.1.** Pull the first-layer weights from your Section 5 network (`model.c1.weight`, shape $32 \\times 3 \\times 3 \\times 3$) and display all 32 filters as a grid of colour patches, normalising each filter to $[0,1]$ for display. At $3 \\times 3$ they are hard to read; note what you can." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# YOUR CODE HERE" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q6.2.** Make them legible, the same way the book's filter figure does: widen only the first layer to $7 \\times 7$ (`LabNet(first_kernel=7)`), retrain for 10 epochs (seeded, Adam, logged), and display the 32 filters again. Expect structure, not a pixel match to the book's filter figure, which trained longer with a different optimiser. Beside them, plot a Gabor bank — Lab 1's `gabor_kernel` code, three frequencies at four orientations. What has the network rediscovered, and what does it have that the Gabor bank does not?" ] }, { "cell_type": "code", "metadata": {}, "source": [ "# YOUR CODE HERE" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Answer:*\n", "\n", "**Q6.3.** Nobody designed your filters. State, in two sentences, what did the choosing instead, and where the structure in those $7 \\times 7$ patches came from.\n", "\n", "*Answer:*\n", "\n", "## 7. The record\n", "\n", "The runs are the deliverable as much as the notebook. Paste links to your W&B runs below. They should include, at minimum: the MLP baseline, the one-batch ritual, the 20-epoch training run, the broken run, and the $7 \\times 7$ retrain. Each must have a meaningful name and a complete config (architecture, optimiser, learning rate, epochs, parameter count, seed).\n", "\n", "*W&B run links:*\n", "\n", "## 8. Before you submit\n", "\n", "- [ ] **Runtime ▸ Restart session and run all** on a GPU runtime, then read every output. The full re-run trains everything and takes roughly half an hour; budget for it.\n", "- [ ] Group members filled in; every member can explain every cell and has their own W&B account.\n", "- [ ] Every *Answer:* cell answered; W&B links pasted in Section 7 and visible to a logged-out viewer or shared with the course staff.\n", "- [ ] **File ▸ Download ▸ Download .ipynb**; one member submits it on Moodle by the deadline shown there." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" } }, "nbformat": 4, "nbformat_minor": 4 }