{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Lab 5 — Annotate, then fine-tune\n", "\n", "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/kleinric/cv-labs/blob/main/lab-05.ipynb)\n", "\n", "**COMS4036A / COMS7050A Computer Vision**\n", "\n", "Groups of up to three; one member submits the group's notebook (`.ipynb`) on Moodle. Every member must be able to explain every cell.\n", "\n", "Companion reading is [Chapter 5 of the course book](https://courses.ms.wits.ac.za/~richard/cv/book/chapters/05-detection-one-stage.html). Every dataset you have used so far arrived labelled. This week you make the labels. About a hundred photographs, taken by your group, boxed by your group, and then fine-tuned on. The first number you measure is how much two of you disagree about what a box is.\n", "\n", "This week's rung of the skills ladder is **annotation**: a written protocol before the clicking, and a measured agreement between two annotators afterwards.\n", "\n", "## 0. GPU, W&B, and the tools\n", "\n", "Open this notebook in [Colab](https://colab.research.google.com) with the badge above, then **File ▸ Save a copy in Drive**. Set **Runtime ▸ Change runtime type** to a GPU, and log in to W&B; this lab logs to a project called `cv-lab5`." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# Group members — fill in before submitting.\n", "MEMBERS = [\n", " # (\"Student name\", \"Student number\"),\n", "]\n", "for name, number in MEMBERS:\n", " print(f\"{number} {name}\")" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": {}, "source": [ "!nvidia-smi --query-gpu=name,memory.total --format=csv" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`ultralytics` is the package that ships YOLO. It is pinned here so that the numbers in this lab and the arguments in its API mean the same thing for everyone." ] }, { "cell_type": "code", "metadata": {}, "source": [ "!pip install -q ultralytics==8.4.115" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": {}, "source": [ "import wandb\n", "wandb.login()" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": {}, "source": [ "import csv\n", "import glob\n", "import os\n", "import random\n", "import shutil\n", "import time\n", "from pathlib import Path\n", "\n", "import matplotlib.pyplot as plt\n", "import matplotlib.patches as patches\n", "import numpy as np\n", "import torch\n", "from PIL import Image\n", "from ultralytics import YOLO, settings\n", "\n", "settings.update({\"wandb\": True})\n", "DEV = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n", "\n", "\n", "def seed_everything(seed=0):\n", " random.seed(seed)\n", " np.random.seed(seed)\n", " torch.manual_seed(seed)\n", " torch.cuda.manual_seed_all(seed)\n", "\n", "\n", "seed_everything(0)\n", "print(\"device:\", DEV)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Annotate\n", "\n", "Take about a hundred photographs between you, of two to four kinds of thing that appear in your daily surroundings. Mugs and laptops. Bicycles and cars. Litter of two kinds on the campus lawn. The classes should be things you can point at and count, and the photographs should vary: different distances, different lighting, some with several objects and some with none.\n", "\n", "Before anybody clicks anything, agree the rules and write them down. Four questions decide almost every disagreement you are about to have:\n", "\n", "- **The class list.** Exactly which names, and what belongs to each. \"Bottle\" or \"plastic bottle\" and \"glass bottle\"?\n", "- **What counts as an instance.** A stack of six books: one box or six? A reflection in a window? A picture of a mug on a poster?\n", "- **Occlusion.** An object half behind another: box the visible part, or box where you believe the whole object is? Both are defensible; the dataset only breaks if you do not choose.\n", "- **Truncation and size.** Do you include an object running off the edge of the frame? An object twelve pixels across at the back of the room?\n", "\n", "**Q1.1.** Write your group's protocol here: the class list with a sentence defining each class, and your rules for instances, occlusion, truncation, and minimum size. Be specific enough that a fourth person could annotate to it without asking you anything.\n", "\n", "*Answer:*\n", "\n", "### The tool\n", "\n", "Use [Roboflow](https://roboflow.com), which is free for public projects and the lowest-friction option.\n", "\n", "1. Create an account and a new project. Project type **Object Detection**, annotation group whatever you called your class list.\n", "2. **Upload** your photographs. Assign them among the group so that each image is annotated by exactly one member, except the ten in Q1.2 below.\n", "3. Annotate: drag a box, choose the class, next image. Keyboard shortcuts are worth the two minutes it takes to learn them.\n", "4. When everyone is done, **Generate** a version. Set the train/validation/test split to **100% train**. Section 2 has you split it yourself, which is the point. Add no augmentation; YOLO applies its own during training.\n", "5. **Export** the version in format **YOLOv11** (or *YOLO Darknet*, which is the same text format), choose *download zip to computer*, and upload the zip to this Colab session, or use the `roboflow` download snippet Roboflow offers.\n", "\n", "[CVAT](https://www.cvat.ai) is the alternative if you would rather not use a hosted service; it exports the same format under *YOLO 1.1*. The rest of this notebook does not care which you used.\n", "\n", "::: {.callout-note}\n", "## If the export will not come\n", "\n", "Nobody should be stuck waiting on an annotation tool. The cell below builds a stand-in dataset in exactly the format a Roboflow export produces: 200 photographs from the Oxford-IIIT Pet dataset (Parkhi et al., 2012), Lab 3's download again, with the head of each animal boxed, in two classes, `cat` and `dog`. The Pet annotations ship as PASCAL VOC XML, so the cell also shows the VOC-to-YOLO conversion, which is a conversion you will meet again.\n", "\n", "It also supplies a stand-in *second annotator*, needed by Q1.2 and Q6.2: it jitters each box, drops one now and then, and invents one now and then. It is a simulation, not a person, and its numbers are not evidence about human annotators. Use your own two passes if you have them.\n", ":::" ] }, { "cell_type": "code", "metadata": {}, "source": [ "# THE FALLBACK. Skip this cell if your own export worked.\n", "import xml.etree.ElementTree as ET\n", "\n", "!wget -q -nc https://thor.robots.ox.ac.uk/pets/images.tar.gz\n", "!wget -q -nc https://thor.robots.ox.ac.uk/pets/annotations.tar.gz\n", "!tar -xzf images.tar.gz && tar -xzf annotations.tar.gz\n", "\n", "FALLBACK_CLASSES = [\"cat\", \"dog\"]\n", "\n", "\n", "def read_voc(stem):\n", " \"\"\"(width, height, [(class_id, x1, y1, x2, y2), ...]) from a Pet XML.\"\"\"\n", " root = ET.parse(f\"annotations/xmls/{stem}.xml\").getroot()\n", " size = root.find(\"size\")\n", " w, h = int(size.find(\"width\").text), int(size.find(\"height\").text)\n", " boxes = []\n", " for obj in root.findall(\"object\"):\n", " b = obj.find(\"bndbox\")\n", " boxes.append((FALLBACK_CLASSES.index(obj.find(\"name\").text),\n", " float(b.find(\"xmin\").text), float(b.find(\"ymin\").text),\n", " float(b.find(\"xmax\").text), float(b.find(\"ymax\").text)))\n", " return w, h, boxes\n", "\n", "\n", "stems = sorted(Path(p).stem for p in glob.glob(\"annotations/xmls/*.xml\"))\n", "by_class = {0: [], 1: []}\n", "for s in stems:\n", " _, _, b = read_voc(s)\n", " if len(b) == 1:\n", " by_class[b[0][0]].append(s)\n", "rng = np.random.RandomState(0)\n", "picked = sorted([*rng.permutation(by_class[0])[:100],\n", " *rng.permutation(by_class[1])[:100]])\n", "picked = [str(s) for s in rng.permutation(picked)]\n", "\n", "os.makedirs(\"export/images\", exist_ok=True)\n", "os.makedirs(\"export/labels\", exist_ok=True)\n", "for stem in picked:\n", " w, h, boxes = read_voc(stem)\n", " if not os.path.exists(f\"export/images/{stem}.jpg\"):\n", " os.symlink(os.path.abspath(f\"images/{stem}.jpg\"),\n", " f\"export/images/{stem}.jpg\")\n", " with open(f\"export/labels/{stem}.txt\", \"w\") as f:\n", " for c, x1, y1, x2, y2 in boxes: # VOC corners -> YOLO centres\n", " f.write(f\"{c} {(x1 + x2) / 2 / w:.6f} {(y1 + y2) / 2 / h:.6f} \"\n", " f\"{(x2 - x1) / w:.6f} {(y2 - y1) / h:.6f}\\n\")\n", "Path(\"export/classes.txt\").write_text(\"\\n\".join(FALLBACK_CLASSES) + \"\\n\")\n", "\n", "\n", "def second_annotator(boxes, w, h, rng, jitter=0.06, p_drop=0.12, p_add=0.10):\n", " \"\"\"A simulated second pass over the same image. Not a person.\"\"\"\n", " out = []\n", " for c, x1, y1, x2, y2 in boxes:\n", " if rng.rand() < p_drop:\n", " continue\n", " bw, bh = x2 - x1, y2 - y1\n", " j = rng.randn(4) * jitter * np.array([bw, bh, bw, bh])\n", " out.append((c, x1 + j[0], y1 + j[1], x2 + j[2], y2 + j[3]))\n", " if rng.rand() < p_add:\n", " bw, bh = w * 0.15, h * 0.15\n", " x1, y1 = rng.rand() * (w - bw), rng.rand() * (h - bh)\n", " out.append((rng.randint(len(FALLBACK_CLASSES)), x1, y1, x1 + bw, y1 + bh))\n", " return out\n", "\n", "\n", "print(len(picked), \"images in export/, classes\", FALLBACK_CLASSES)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Two people, ten images\n", "\n", "Ten of your images are annotated **twice**, by two different members, independently, without looking at each other's work. Do this before you read each other's boxes; it does not work afterwards.\n", "\n", "**Q1.2.** Implement `iou(a, b)` for two boxes in $(x_1, y_1, x_2, y_2)$ pixel corners, and `agreement(A, B, thr)` which greedily matches the two annotators' boxes in descending order of IoU, same class only and one partner each, and returns the mean IoU of the matched pairs, the number of boxes only A drew, and the number only B drew. Report those three numbers over your ten shared images at `thr=0.5`, and then report how many pairs still match at `thr=0.75` and at `thr=0.9`." ] }, { "cell_type": "code", "metadata": {}, "source": [ "def iou(a, b):\n", " # YOUR CODE HERE\n", " ...\n", "\n", "\n", "def agreement(A, B, thr=0.5):\n", " # YOUR CODE HERE\n", " ..." ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q1.3.** Look at what you just measured. Two boxes matched at IoU 0.5 can still be visibly different boxes. Draw one such pair if you have one. State what your mean IoU and your two unmatched counts imply about the best score any model trained on these labels could honestly reach, and name the one rule in your Q1.1 protocol you would rewrite now.\n", "\n", "*Answer:*\n", "\n", "## 2. The data as YOLO sees it\n", "\n", "The export is a directory of images and a directory of `.txt` files with matching stems. One line per object:\n", "\n", "```\n", "0 0.469880 0.243000 0.433735 0.358000\n", "```\n", "\n", "Five fields: the class index, then the box's centre-$x$, centre-$y$, width and height, all four **normalised by the image's own width and height**, so every number lies in $[0, 1]$ and the file says nothing about how big the image is. An image with no objects has an empty file, or no file at all.\n", "\n", "**Q2.1.** Read the class names into a list called `CLASSES`, in index order, from your export's `classes.txt` or from `FALLBACK_CLASSES` if you used the fallback. Then write `read_yolo(path)` returning a list of `(cls, cx, cy, w, h)` tuples, and `to_corners(img_w, img_h, lines)` converting them back to a list of `(cls, x1, y1, x2, y2)` in pixels. Print the raw text of one label file beside the pixel corners it decodes to. Then report, over your whole export: the number of images, the number of boxes, the boxes per class, the number of images with no boxes, and the median box area as a fraction of its image's area. Later cells use `CLASSES`, `read_yolo` and `to_corners` by those names." ] }, { "cell_type": "code", "metadata": {}, "source": [ "def read_yolo(path):\n", " # YOUR CODE HERE\n", " ...\n", "\n", "\n", "def to_corners(img_w, img_h, lines):\n", " # YOUR CODE HERE\n", " ..." ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q2.2.** Prove the conversion. Draw six images from the export with their decoded boxes on top, labelled with the class name. If a box is offset, or the width and height are swapped, or the boxes drift as the image aspect ratio changes, this is where you see it, not after training." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# YOUR CODE HERE" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q2.3.** Split the export into train, validation and test at 70/15/15, seeded, and lay it out as YOLO expects:\n", "\n", "```\n", "dataset/\n", " images/train/… images/val/… images/test/…\n", " labels/train/… labels/val/… labels/test/…\n", " data.yaml\n", "```\n", "\n", "`data.yaml` needs an absolute `path`, the three relative split directories, and `names:` mapping each index to its class name. Use the directory name `dataset/` exactly; Section 5's cells read from it. Split **by image**, not by object: every box from one photograph lands in the same split. Say why that matters, and what would go wrong if you shuffled the boxes instead." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# YOUR CODE HERE" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Answer:*\n", "\n", "## 3. Fine-tune\n", "\n", "`yolo11n.pt` is the smallest of the YOLO11 family: 2,624,080 parameters and 6.7 GFLOPs at $640 \\times 640$ by its own `model.info()`, trained on COCO's 80 classes. Fine-tuning it loads every weight whose shape does not depend on the number of classes and re-initialises the class-score layers of the detection head to fit your class list.\n", "\n", "**Q3.1.** The ritual first, in its detection form. Build a second `data.yaml` whose training split is eight images and whose validation split is those same eight images, and train for 100 epochs. A network that cannot memorise eight photographs has a wiring bug: a class index out of range, a label file that does not match its image, boxes in the wrong units. Every one of those is invisible until you look. Report mAP50 on the eight images and say what it would have to be for you to continue." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# YOUR CODE HERE" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q3.2.** Now the real run: `YOLO(\"yolo11n.pt\")`, `data` your dataset, 40 epochs, `imgsz=640`, `batch=16`, `seed=0`, `deterministic=True`, and `project=\"cv-lab5\"` with a `name` that says which label set and how many images it used. Ultralytics logs to W&B by itself once `settings.update({\"wandb\": True})` has run, and its config panel carries the model, data, epochs, image size, batch size and seed. Report mAP50 and mAP50-95 on your **validation** split, and the per-class mAP50." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# YOUR CODE HERE" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q3.3.** Before it was fine-tuned, that checkpoint was a working detector. Score the untouched `yolo11n.pt` on the same validation split with `YOLO(\"yolo11n.pt\").val(data=…)`, no training, and report its mAP50. Explain the number you get. Two things are going on, and one of them would still apply even if your class names happened to be COCO class names." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# YOUR CODE HERE" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Answer:*\n", "\n", "## 4. The thresholds\n", "\n", "Two numbers sit between the network's output and the boxes you see. **Confidence** discards detections scoring below it. **NMS IoU** decides how much two surviving boxes may overlap before the lower-scoring one is deleted. Both are arguments to `val()` and `predict()`, and neither is trained.\n", "\n", "**Q4.1.** Sweep them on the validation split: `conf` over $\\{0.001, 0.05, 0.25, 0.5, 0.7\\}$ and `iou` over $\\{0.3, 0.5, 0.7, 0.9\\}$, twenty `val()` calls in all, using your fine-tuned weights. Plot mAP50-95 against confidence with one line per NMS IoU." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# YOUR CODE HERE" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q4.2.** Which pair maximises mAP50-95? Now answer the harder half: would you deploy that pair, and if not, why does the metric prefer it? Your explanation has to say what mAP does with a detection that scores 0.02, and what the confidence threshold does to the precision–recall curve that mAP integrates.\n", "\n", "*Answer:*\n", "\n", "## 5. Speed\n", "\n", "Chapter 5's argument for a one-stage detector is not accuracy. It is cost: the whole thing is one pass over the image, with no second stage re-scoring a few hundred proposals, so it takes less time per image on the same hardware. Its like-for-like comparison, RetinaNet at 36.0 COCO AP against Faster R-CNN at 36.2 on one card, puts the difference at 122 ms against 172 ms per image: a factor of 1.4, not a factor of ten. Measure the gap on your own data and your own GPU.\n", "\n", "Timing a GPU is where this goes wrong. A CUDA call returns as soon as the work is *queued*, not when it is *done*, so a naive `time.time()` around an inference call measures how fast Python can enqueue work. Two rules fix it: run several inferences first so that cuDNN's autotuner and the memory allocator have settled, and call `torch.cuda.synchronize()` before you read the clock at each end." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# Lab 4's detector, with its box predictor resized to your classes, fine-tuned\n", "# on the same training split so that the comparison is like for like.\n", "import torchvision.transforms.functional as TF\n", "from torchvision.models.detection import (FasterRCNN_ResNet50_FPN_V2_Weights,\n", " fasterrcnn_resnet50_fpn_v2)\n", "from torchvision.models.detection.faster_rcnn import FastRCNNPredictor\n", "\n", "\n", "class BoxDataset(torch.utils.data.Dataset):\n", " \"\"\"(image tensor, {\"boxes\": Nx4 xyxy, \"labels\": N}) — torchvision's format,\n", " whose labels are 1-based because 0 is reserved for background.\"\"\"\n", "\n", " def __init__(self, split):\n", " self.items = sorted(glob.glob(f\"dataset/images/{split}/*\"))\n", "\n", " def __len__(self):\n", " return len(self.items)\n", "\n", " def __getitem__(self, i):\n", " p = self.items[i]\n", " img = Image.open(p).convert(\"RGB\")\n", " lab = f\"dataset/labels/{Path(p).parent.name}/{Path(p).stem}.txt\"\n", " rows = to_corners(*img.size, read_yolo(lab)) if os.path.exists(lab) else []\n", " return TF.to_tensor(img), {\n", " \"boxes\": torch.tensor([list(r[1:]) for r in rows],\n", " dtype=torch.float32).reshape(-1, 4),\n", " \"labels\": torch.tensor([r[0] + 1 for r in rows], dtype=torch.int64)}\n", "\n", "\n", "seed_everything(0)\n", "frcnn = fasterrcnn_resnet50_fpn_v2(\n", " weights=FasterRCNN_ResNet50_FPN_V2_Weights.COCO_V1, min_size=640, max_size=640)\n", "frcnn.roi_heads.box_predictor = FastRCNNPredictor(\n", " frcnn.roi_heads.box_predictor.cls_score.in_features, len(CLASSES) + 1)\n", "frcnn = frcnn.to(DEV)\n", "\n", "loader = torch.utils.data.DataLoader(BoxDataset(\"train\"), batch_size=4, shuffle=True,\n", " num_workers=2, collate_fn=lambda b: tuple(zip(*b)))\n", "opt = torch.optim.SGD([p for p in frcnn.parameters() if p.requires_grad],\n", " lr=5e-3, momentum=0.9, weight_decay=5e-4)\n", "sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, 10 * len(loader))\n", "for ep in range(10):\n", " frcnn.train()\n", " total = 0.0\n", " for imgs, tgts in loader:\n", " imgs = [i.to(DEV) for i in imgs]\n", " tgts = [{k: v.to(DEV) for k, v in t.items()} for t in tgts]\n", " loss = sum(frcnn(imgs, tgts).values())\n", " opt.zero_grad(set_to_none=True)\n", " loss.backward()\n", " opt.step()\n", " sched.step()\n", " total += loss.item()\n", " print(f\"epoch {ep + 1}/10 loss {total / len(loader):.4f}\")" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q5.1.** Write `images_per_second(predict_one, images, warmup=10)`: run `predict_one` on the first `warmup` images and throw the results away, synchronise, start the clock, run it on all of them one at a time, synchronise, stop the clock. Use it to time both detectors on the same images, your validation and test splits together and at most the first 50 of them, at batch size 1 on the same device, and report images per second for each and the ratio. State which of the two rules above you would have violated, and what number you would have reported instead, if you had left it out." ] }, { "cell_type": "code", "metadata": {}, "source": [ "def images_per_second(predict_one, images, warmup=10):\n", " # YOUR CODE HERE\n", " ..." ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q5.2.** Speed alone settles nothing. Score both detectors on those same images at `conf=0.25`, using the one evaluator below so that the two numbers are produced by identical code, and plot the pair as two points with images per second on one axis and mAP50-95 on the other." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# Lab 4's average precision, restated so that both detectors are scored by the\n", "# same code. dets: [(image_id, cls, score, (x1, y1, x2, y2))].\n", "# gts: {(image_id, cls): [(x1, y1, x2, y2), ...]}.\n", "def average_precision(dets, gts, thr):\n", " aps = []\n", " for c in range(len(CLASSES)):\n", " d = sorted([x for x in dets if x[1] == c], key=lambda x: -x[2])\n", " n_gt = sum(len(v) for (im, cc), v in gts.items() if cc == c)\n", " if n_gt == 0:\n", " continue\n", " used = {k: [False] * len(v) for k, v in gts.items()}\n", " tp, fp = np.zeros(len(d)), np.zeros(len(d))\n", " for i, (im, cc, sc, box) in enumerate(d):\n", " best_v, best_j = 0.0, -1\n", " for j, g in enumerate(gts.get((im, cc), [])):\n", " v = iou(box, g)\n", " if v > best_v:\n", " best_v, best_j = v, j\n", " if best_v >= thr and best_j >= 0 and not used[(im, cc)][best_j]:\n", " used[(im, cc)][best_j] = True\n", " tp[i] = 1\n", " else:\n", " fp[i] = 1\n", " ctp, cfp = np.cumsum(tp), np.cumsum(fp)\n", " mrec = np.concatenate(([0.0], ctp / n_gt, [1.0]))\n", " mpre = np.concatenate(([0.0], ctp / np.maximum(ctp + cfp, 1e-9), [0.0]))\n", " for i in range(len(mpre) - 2, -1, -1):\n", " mpre[i] = max(mpre[i], mpre[i + 1])\n", " k = np.where(mrec[1:] != mrec[:-1])[0]\n", " aps.append(float(np.sum((mrec[k + 1] - mrec[k]) * mpre[k + 1])))\n", " return float(np.mean(aps)) if aps else 0.0\n", "\n", "\n", "def map_5095(dets, gts):\n", " per = [average_precision(dets, gts, t) for t in np.arange(0.5, 0.96, 0.05)]\n", " return per[0], float(np.mean(per))" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": {}, "source": [ "# YOUR CODE HERE" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q5.3.** Two questions. First: your YOLO mAP50-95 from `map_5095` above will not equal the one `val()` printed in Q3.2, on overlapping data. Find out why rather than guessing. The ultralytics documentation and its `DetMetrics` source both say what it computes, and two of the differences are in this notebook's own arguments. Second: given your two points, which detector would you ship for a phone app that annotates a live camera feed, and which for an overnight batch job over a hundred thousand archive photographs?\n", "\n", "*Answer:*\n", "\n", "## 6. What the labels did\n", "\n", "Two things about your dataset could have been better: there could have been more of it, and the two annotators could have agreed more. This section costs one of them at a time and reads off which mattered more.\n", "\n", "**Q6.1.** Build a third dataset directory whose training split is a seeded half of your training images, validation and test unchanged, and fine-tune again with identical settings. Report mAP50 and mAP50-95 on the **test** split for both this run and Q3.2's, and note that this is the first time the test split has been touched." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# YOUR CODE HERE" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q6.2.** Now cost the agreement instead of the quantity. Build a fourth dataset with all the training images, but with the labels of one half replaced by the second annotator's boxes for those images, the same disagreement Q1.2 measured applied to half the training set. (On the fallback path, run `second_annotator` over that half with a fixed seed.) Report the disagreement you injected: mean IoU of matched pairs, boxes dropped, boxes invented. Then fine-tune and report test mAP50 and mAP50-95." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# YOUR CODE HERE" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q6.3.** Put the three test numbers in a table: full and clean, half and clean, full and disagreed. Which cost more, halving the data or corrupting half the labels? Then answer the question a client asks: they will pay for either another hundred photographs or a second annotation pass over the hundred you have. Which do you take, and what in your numbers supports it?\n", "\n", "*Answer:*\n", "\n", "## 7. The record\n", "\n", "Paste links to your W&B runs below. At minimum: the one-batch ritual, the full clean fine-tune, the half-data run, and the disagreed-labels run. Each needs a name that says which label set and how many training images it used, and a config carrying the model, data path, epochs, image size, batch size and seed.\n", "\n", "*W&B run links:*\n", "\n", "## 8. Before you submit\n", "\n", "- [ ] **Runtime ▸ Restart session and run all** on a GPU runtime, then read every output. The full re-run trains YOLO four times and Faster R-CNN once; budget about forty minutes on a T4, plus the download if you are on the fallback.\n", "- [ ] Your annotated images and label files are in the submission, or the notebook downloads them from somewhere the marker can reach.\n", "- [ ] Group members filled in; every member can explain every cell.\n", "- [ ] Every *Answer:* cell answered; W&B links pasted in Section 7.\n", "- [ ] **File ▸ Download ▸ Download .ipynb**; one member submits it on Moodle by the deadline shown there." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" } }, "nbformat": 4, "nbformat_minor": 4 }