{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Lab 4 — Measuring an object detector\n", "\n", "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/kleinric/cv-labs/blob/main/lab-04.ipynb)\n", "\n", "**COMS4036A / COMS7050A Computer Vision**\n", "\n", "Groups of up to three; one member submits the group's notebook (`.ipynb`) on Moodle. Every member must be able to explain every cell.\n", "\n", "Companion reading is [Chapter 4 of the course book](https://courses.ms.wits.ac.za/~richard/cv/book/chapters/04-detection-two-stage.html). Nothing is trained this week. A pretrained detector is handed to you and the work is measuring it. Every quantity in the detection literature is arithmetic small enough to do on paper: IoU, precision, recall, average precision, mAP. You do it on paper first, then in code, then against the library's version.\n", "\n", "The rung of the skills ladder is **implement IoU and mAP yourself before trusting a library's version**. The second theme is evaluation hygiene: what a number computed on 200 images is worth when the benchmark has 4,952.\n", "\n", "## 0. GPU, W&B, and VOC 2007\n", "\n", "Open this notebook in [Colab](https://colab.research.google.com) with the badge above, then **File ▸ Save a copy in Drive**. Set **Runtime ▸ Change runtime type** to a GPU, and log in to W&B as in the last two labs; this lab logs to a project called `cv-lab4`." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# Group members — fill in before submitting.\n", "MEMBERS = [\n", " # (\"Student name\", \"Student number\"),\n", "]\n", "for name, number in MEMBERS:\n", " print(f\"{number} {name}\")" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": {}, "source": [ "!nvidia-smi --query-gpu=name,memory.total --format=csv" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": {}, "source": [ "import wandb\n", "wandb.login()" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The data is the test half of [PASCAL VOC 2007](http://host.robots.ox.ac.uk/pascal/VOC/voc2007/) (Everingham et al., 2010): 4,952 photographs, 20 object classes, one box per object, about 450 MB. Every mAP number in Chapter 4's R-CNN lineage is quoted on this set, which is the reason for choosing it. The download takes a minute or two." ] }, { "cell_type": "code", "metadata": {}, "source": [ "import random\n", "import time\n", "\n", "import matplotlib.pyplot as plt\n", "import numpy as np\n", "import torch\n", "from matplotlib.patches import Rectangle\n", "from torchvision.datasets import VOCDetection\n", "from torchvision.models.detection import (\n", " FasterRCNN_ResNet50_FPN_V2_Weights,\n", " fasterrcnn_resnet50_fpn_v2,\n", ")\n", "from torchvision.models.detection.transform import resize_boxes\n", "from torchvision.ops import box_iou, nms\n", "\n", "DEV = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n", "\n", "\n", "def seed_everything(seed=0):\n", " random.seed(seed)\n", " np.random.seed(seed)\n", " torch.manual_seed(seed)\n", " torch.cuda.manual_seed_all(seed)\n", "\n", "\n", "seed_everything(0)\n", "voc = VOCDetection(\".\", year=\"2007\", image_set=\"test\", download=True)\n", "print(\"device:\", DEV, \"· images:\", len(voc))" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The 20 class names, in the order VOC uses, and the seeded subset everything below is measured on. Running the whole test set takes a quarter of an hour on a T4; 200 images takes under a minute. Section 3 asks what that shortcut costs." ] }, { "cell_type": "code", "metadata": {}, "source": [ "VOC_CLASSES = [\n", " \"aeroplane\", \"bicycle\", \"bird\", \"boat\", \"bottle\", \"bus\", \"car\", \"cat\",\n", " \"chair\", \"cow\", \"diningtable\", \"dog\", \"horse\", \"motorbike\", \"person\",\n", " \"pottedplant\", \"sheep\", \"sofa\", \"train\", \"tvmonitor\",\n", "]\n", "\n", "def subset(seed, n=200):\n", " return sorted(np.random.RandomState(seed).choice(len(voc), n,\n", " replace=False).tolist())\n", "\n", "SUBSET = subset(0)\n", "print(len(SUBSET), \"images ·\", SUBSET[:5])" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Inspect the detector output\n", "\n", "A classifier returns one vector per image, the same shape every time. A detector returns a list whose length depends on the photograph, and the length is part of what is being judged. That is what makes the metric awkward, and the first thing to do is look at the output." ] }, { "cell_type": "code", "metadata": {}, "source": [ "weights = FasterRCNN_ResNet50_FPN_V2_Weights.COCO_V1\n", "model = fasterrcnn_resnet50_fpn_v2(weights=weights).eval().to(DEV)\n", "prep = weights.transforms()\n", "print(f\"{sum(p.numel() for p in model.parameters()):,} parameters\")\n", "print(\"score_thresh\", model.roi_heads.score_thresh,\n", " \"· nms_thresh\", model.roi_heads.nms_thresh,\n", " \"· detections_per_img\", model.roi_heads.detections_per_img)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Those three numbers are the detector's output policy, and none of them is part of the network. Sections 4 and 6 change them.\n", "\n", "**Q1.1.** `voc[i]` returns a `(PIL image, dict)` pair, and the dict is the VOC XML file parsed into nested dictionaries: `target[\"annotation\"][\"object\"]` is a list, one entry per annotated object, each with a `name`, a `bndbox` of `xmin`/`ymin`/`xmax`/`ymax` strings, and a `difficult` flag. Write `ground_truth(target)` returning three arrays: boxes as float $(x_1, y_1, x_2, y_2)$, integer labels indexing `VOC_CLASSES`, and a boolean `difficult`. Then report, over `SUBSET`: the number of annotated objects, how many are flagged difficult, the mean and maximum number of objects per image, and the count of non-difficult objects in each class." ] }, { "cell_type": "code", "metadata": {}, "source": [ "def ground_truth(target):\n", " # YOUR CODE HERE\n", " ..." ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q1.2.** The detector was not trained on VOC. It was trained on COCO, whose 91-slot label space is `weights.meta[\"categories\"]`, and its label 1 is not VOC's class 1. All 20 VOC classes appear in COCO, but six of the names differ. Build `COCO_TO_VOC`, a dict from COCO label index to VOC class index, and print the six pairs whose names are not identical. Detections in the other 71 slots are dropped from here on: VOC does not annotate those categories, so a correct detection of a truck or a bench would score as a false positive." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# YOUR CODE HERE" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q1.3.** Take the busiest image in `SUBSET`, the one with the most annotated objects, and run the model on it: `model([prep(img).to(DEV)])[0]`, inside `torch.no_grad()`. Draw two panels side by side: the ground-truth boxes, and the detections scoring above 0.5 labelled with class and score. Print the number of detections above 0.05, above 0.5 and above 0.9, and the number of annotated objects." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# YOUR CODE HERE" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q1.4.** Explain the differences between the four counts from Q1.3. Why are there more low-scoring detections than annotated objects? What limits the number returned above 0.05? Identify any annotated object that is flagged difficult, occluded, or unusually small, and say why a detector is likely to miss it.\n", "\n", "*Answer:*\n", "\n", "## 2. Intersection over union\n", "\n", "Intersection over union of two axis-aligned boxes: the area they share divided by the area they cover between them. Matching, NMS and the threshold in `mAP@0.5` all rest on this one number.\n", "\n", "**Q2.1.** Before writing any code, compute IoU on paper for these three pairs, in $(x_1, y_1, x_2, y_2)$ form. Give each as an exact fraction and as a decimal.\n", "\n", "| | box $A$ | box $B$ |\n", "|---|---|---|\n", "| (a) | $(0, 0, 10, 10)$ | $(5, 5, 15, 15)$ |\n", "| (b) | $(0, 0, 10, 10)$ | $(2, 2, 6, 6)$ |\n", "| (c) | $(0, 0, 10, 10)$ | $(20, 20, 30, 30)$ |\n", "\n", "*Answer:*\n", "\n", "**Q2.2.** Now implement `iou(a, b)` for two boxes given as length-4 sequences, and check it against your three answers. Case (c) is the one that breaks naive implementations: subtracting the corners gives a negative width and a negative height, whose product is a positive \"intersection\". Your code must return exactly 0, and must not divide by zero when both boxes have zero area. Add a fourth case of your own that touches edge to edge without overlapping, and say what it should give." ] }, { "cell_type": "code", "metadata": {}, "source": [ "def iou(a, b):\n", " # YOUR CODE HERE\n", " ..." ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q2.3.** `iou` is called once per pair, and matching needs every pair. Write `iou_matrix(A, B)` taking arrays of shape $(N, 4)$ and $(M, 4)$ and returning the $(N, M)$ matrix of IoUs, with no Python loop over the pairs; broadcast instead. Verify it two ways: against `iou` on your four cases, and against `torchvision.ops.box_iou`. Report the largest absolute difference." ] }, { "cell_type": "code", "metadata": {}, "source": [ "def iou_matrix(A, B):\n", " # YOUR CODE HERE\n", " ..." ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q2.4.** Name the edge cases your own tests forced you to handle. Report the largest absolute difference between your implementation and `box_iou`, and explain why float32 and float64 need not produce identical results.\n", "\n", "*Answer:*\n", "\n", "## 3. Matching detections and computing AP\n", "\n", "A detection is correct if it overlaps a ground-truth object of the same class by at least the IoU threshold, and that object has not already been claimed. The rule is greedy: take detections in descending score order, and let each one claim the best still-unclaimed object it overlaps by at least the threshold. This is COCO's rule, and it is what you implement here. VOC's differs: it takes the object a detection overlaps most whether that object is claimed or not, so a detection whose best match is taken is a false positive there rather than falling through. On this dataset the choice is worth 0.0001 of `mAP@0.5`, but `pycocotools` and the VOC devkit are not interchangeable, and it is worth knowing which one produced a number you are comparing against. Everything unclaimed is a false negative; every detection that claims nothing is a false positive.\n", "\n", "One ground truth per detection is what makes a second box on the same object a false positive rather than a second success. Without that rule, a detector could report a hundred copies of every object and score perfectly.\n", "\n", "**Q3.1.** Do the metric on paper first. One image, one class, three ground-truth objects, five detections:\n", "\n", "| ground truth | box |\n", "|---|---|\n", "| $g_1$ | $(10, 10, 60, 60)$ |\n", "| $g_2$ | $(100, 100, 150, 150)$ |\n", "| $g_3$ | $(200, 30, 260, 90)$ |\n", "\n", "| detection | score | box |\n", "|---|---|---|\n", "| $d_1$ | 0.95 | $(12, 12, 58, 58)$ |\n", "| $d_2$ | 0.90 | $(15, 15, 65, 65)$ |\n", "| $d_3$ | 0.75 | $(98, 102, 152, 148)$ |\n", "| $d_4$ | 0.60 | $(300, 300, 350, 350)$ |\n", "| $d_5$ | 0.30 | $(205, 35, 225, 55)$ |\n", "\n", "At an IoU threshold of 0.5, and using the COCO rule above — each detection takes the best of the objects still unclaimed — mark each detection TP or FP and say which object it claimed. Then write out precision and recall after each detection in turn, and compute average precision by all-point interpolation: replace each precision by the highest precision at that recall or greater, and take the area under the resulting step function. Give AP as an exact fraction.\n", "\n", "*Answer:*\n", "\n", "**Q3.2.** Implement `match(det_boxes, det_scores, gt_boxes, iou_thr, gt_difficult)`, returning one flag per detection in the order the detections were given: 1 for a true positive, 0 for a false positive, and $-1$ for a detection matched to an object flagged difficult.\n", "\n", "Implement the **COCO protocol**, precisely:\n", "\n", "1. Take the detections in descending score order.\n", "2. For each detection, consider only the ground-truth objects of its class that are **not yet claimed**, and take the one it overlaps most.\n", "3. If that overlap is at least `iou_thr`, the detection claims that object; otherwise the detection is a false positive.\n", "4. A claimed object cannot be claimed again, so a second detection of the same object is a false positive.\n", "\n", "Two details are worth stating rather than inferring. Step 2 searches the *unclaimed* objects: a detection whose best overall match is already taken falls through to the next-best free object rather than being written off. This is where VOC differs, and the companion chapter measures what the difference is worth: 0.0001 of `mAP@0.5` over the whole VOC 2007 test set. The `difficult` flag is VOC's, not COCO's: VOC's protocol neither rewards nor punishes a detection that lands on a difficult object, so $-1$ means \"drop this detection from the count entirely\". Your `mean_ap` will need to honour that in Q3.4.\n", "\n", "Check your function reproduces your Q3.1 marking exactly." ] }, { "cell_type": "code", "metadata": {}, "source": [ "def match(det_boxes, det_scores, gt_boxes, iou_thr=0.5, gt_difficult=None):\n", " # YOUR CODE HERE\n", " ..." ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q3.3.** Write `pr_curve(flags, scores, n_positive)` returning the precision and recall after each detection, and `average_precision(prec, rec)` doing the all-point interpolation. Difficult matches leave the arrays before anything is accumulated, and `n_positive` counts only non-difficult objects. Run both on the Q3.1 table and check the AP against your fraction. It should agree to every printed digit." ] }, { "cell_type": "code", "metadata": {}, "source": [ "def pr_curve(flags, scores, n_positive):\n", " # YOUR CODE HERE\n", " ...\n", "\n", "\n", "def average_precision(prec, rec):\n", " # YOUR CODE HERE\n", " ..." ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q3.4.** Now the whole subset. Write `mean_ap(dets, gts, iou_thr=0.5, use_difficult=True)`, which pools every detection of a class across all images, sorts them by score, matches them image by image, and averages the per-class APs over the classes that occur. `use_difficult=True` is the VOC protocol of Q3.2 and Q3.3; `use_difficult=False` hands `match` an all-false difficult array instead, which Q3.5 needs. Run the model over `SUBSET` at its default settings, collect the detections into the form `mean_ap` wants, and report `mAP@0.5`, `mAP@0.75`, and the per-class AP table beside each class's object count from Q1.1. Log all of it to a W&B run named `defaults-subset0`, with a config carrying the weights enum, the subset seed and size, the three output-policy numbers, and the IoU threshold." ] }, { "cell_type": "code", "metadata": {}, "source": [ "def mean_ap(dets, gts, iou_thr=0.5, use_difficult=True):\n", " # YOUR CODE HERE\n", " ..." ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q3.5.** Rerun `mean_ap` with `use_difficult=False`, so that difficult objects become ordinary ground truth and detections matching them count as ordinary true positives. Report both numbers, say which is larger and by how much, and say which of the two you would quote in a paper. \"Difficult\" was decided by the people who drew the boxes, not by the detector." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# YOUR CODE HERE" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Answer:*\n", "\n", "**Q3.6.** 200 images is a twenty-fifth of the benchmark. Repeat Q3.4's evaluation on `subset(1)`, `subset(2)` and `subset(3)`, three more runs, all logged, differing only in the seed, and report the four `mAP@0.5` values and their spread. The book's run over the whole 4,952-image test set, same model, same settings, gives 0.897. Where does your seed-0 number sit relative to the other three and to 0.897, and what would you have reported if you had run only seed 0? Name the classes whose object count in Q1.1 is small enough to make their AP close to meaningless, and say what that does to a mean taken over 20 classes." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# YOUR CODE HERE" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Answer:*\n", "\n", "## 4. Non-maximum suppression\n", "\n", "The detector has been suppressing duplicates all along. Turn it off and the suppression becomes yours to write.\n", "\n", "**Q4.1.** Get the unsuppressed candidates. Set `model.roi_heads.nms_thresh = 1.0` (no IoU can exceed 1, so nothing is suppressed), `model.roi_heads.score_thresh = 0.01` and `model.roi_heads.detections_per_img = 300`, and run one pass over `SUBSET`, caching per image the boxes, scores and VOC labels of every candidate in a VOC class. Cache them properly: Sections 4 and 6 both sweep thresholds over these candidates, and neither needs the GPU again. Report the total number of candidates and the mean per image, and compare that mean against the number of objects per image from Q1.1." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# YOUR CODE HERE" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q4.2.** Implement `my_nms(boxes, scores, thr)`: sort by descending score, take the top box, discard every remaining box whose IoU with it exceeds `thr`, repeat. Return the kept indices. Then `per_class_nms(boxes, scores, labels, thr)`, which does this within each class. Two overlapping boxes of different classes are not duplicates. Check `my_nms` against `torchvision.ops.nms` at a threshold of 0.5, class by class, over every image in the subset, and report whether the kept sets are identical everywhere." ] }, { "cell_type": "code", "metadata": {}, "source": [ "def my_nms(boxes, scores, thr):\n", " # YOUR CODE HERE\n", " ...\n", "\n", "\n", "def per_class_nms(boxes, scores, labels, thr):\n", " # YOUR CODE HERE\n", " ..." ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q4.3.** Sweep the NMS IoU threshold from 0.1 to 1.0 in steps of 0.1. At each value, apply `per_class_nms` to the cached candidates at a score threshold of 0.05 and compute `mAP@0.5`; separately, apply it at a score threshold of 0.5 and compute the overall precision and recall at IoU 0.5. Plot all three against the threshold on one set of axes, mark the mAP maximum, and log the sweep to W&B." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# YOUR CODE HERE" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q4.4.** Report the NMS threshold that gives the highest mAP, and how much it beats torchvision's default of 0.5 by. Explain why mAP barely moves across most of the sweep while precision at a fixed score threshold falls by more than a factor of ten. Which of the two better describes what a user of the system actually sees?\n", "\n", "*Answer:*\n", "\n", "## 5. Proposal recall\n", "\n", "Chapter 4's two stages divide the work: the region proposal network says where to look, and the second stage says what is there and where exactly. The second stage only ever sees regions the first proposed, so what the proposals cover is a ceiling on what the detector can find.\n", "\n", "`model.rpn` returns its proposals as the first element of its output. A forward hook can copy them out, but they arrive in the resized frame the model works in: every image is scaled so its shorter side is 800 pixels before the backbone sees it. They have to be mapped back to the original image, which `model.transform` and `resize_boxes` do between them, the first reporting the resized size and the second doing the mapping.\n", "\n", "```python\n", "proposals, sizes = [], []\n", "model.rpn.register_forward_hook(lambda m, i, o: proposals.append(o[0][0].detach()))\n", "model.transform.register_forward_hook(lambda m, i, o: sizes.append(o[0].image_sizes[0]))\n", "# after each model([...]) call, for an image of size (W, H):\n", "boxes = resize_boxes(proposals.pop(), sizes.pop(), (H, W))\n", "```\n", "\n", "Q4.1 left the detector's output policy loosened, and this section runs the model itself rather than the cached candidates. Put the three numbers Section 1 printed back first: `model.roi_heads.score_thresh = 0.05`, `model.roi_heads.nms_thresh = 0.5`, `model.roi_heads.detections_per_img = 100`.\n", "\n", "**Q5.1.** With those restored, register the hooks, run the model over `SUBSET` once more, and cache the proposals per image alongside the ground truth. Report how many proposals arrive per image. Then draw the top 50 on the busiest image from Q1.3, with the ground-truth boxes over them in a different colour." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# YOUR CODE HERE" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q5.2.** Proposal recall: the fraction of non-difficult ground-truth objects that have at least one proposal overlapping them by at least a given IoU. Compute it for the top 50, 100, 300 and 1,000 proposals, at IoU 0.5, 0.75 and 0.9, a $4 \\times 3$ table." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# YOUR CODE HERE" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q5.3.** Using your table, answer the following:\n", "\n", "- At IoU 0.5, does raising the number of proposals substantially improve proposal recall?\n", "- What happens to proposal recall as the IoU threshold rises?\n", "- If the proposals do not themselves reach IoU 0.9, which of the two stages must be producing the final localisation accuracy?\n", "- Is that consistent with the gap you measured in Q3.4 between `mAP@0.5` and `mAP@0.75`?\n", "\n", "*Answer:*\n", "\n", "## 6. Score thresholds\n", "\n", "The score threshold is not trained, not tuned by the benchmark, and not fixed by the architecture. It is set by whoever deploys the thing, and it moves the system along a curve.\n", "\n", "**Q6.1.** Using the cached candidates from Q4.1 with NMS at 0.5, sweep the score threshold over $\\{0.01, 0.05, 0.1, 0.3, 0.5, 0.7, 0.9, 0.95\\}$. At each value report overall precision, overall recall and `mAP@0.5` at IoU 0.5, and the number of detections kept. Plot precision and recall against the threshold on one set of axes, and log the table to W&B." ] }, { "cell_type": "code", "metadata": {}, "source": [ "# YOUR CODE HERE" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Q6.2.** Choose a score threshold for each of two deployments. One is a safety alarm on a mine haul road that must not miss a person; a human reviews every alert. The other tags a photograph library automatically, and nobody checks. Pick a threshold from your sweep for each, quote the precision and recall it gives, and say what the cost of the errors is in each case. Then say what mAP fails to tell either operator, given that it integrates over every score threshold at once and so names none of them.\n", "\n", "*Answer:*\n", "\n", "## 7. W&B runs\n", "\n", "Paste links to your W&B runs below. At minimum: the default-settings evaluation of Q3.4, the three extra seeds of Q3.6, and the two sweeps of Q4.3 and Q6.1. Each needs a meaningful name and a complete config: weights enum, subset seed and size, score threshold, NMS threshold, detections per image, and the IoU threshold the metric used.\n", "\n", "*W&B run links:*\n", "\n", "## 8. Before you submit\n", "\n", "- [ ] **Runtime ▸ Restart session and run all** on a GPU runtime, then read every output. The full re-run downloads 450 MB and puts about 1,200 images through the detector; budget twenty minutes on a T4.\n", "- [ ] Group members filled in; every member can explain every cell.\n", "- [ ] Q2.1 and Q3.1 answered from paper, with the arithmetic shown, not copied from the code below them.\n", "- [ ] Every *Answer:* cell answered; W&B links pasted in Section 7.\n", "- [ ] **File ▸ Download ▸ Download .ipynb**; one member submits it on Moodle by the deadline shown there." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" } }, "nbformat": 4, "nbformat_minor": 4 }