3 Architectures and Transfer Learning

Learning objectives

Trace the AlexNet → VGG → GoogLeNet → ResNet lineage and what each step changed, with the parameter arithmetic to back it
Explain the two obstacles to depth, unstable gradients and then the degradation problem, and the two fixes: batch normalisation and residual connections
Compare architectures on accuracy against parameters and FLOPs, and say what depthwise separability saves MobileNet
Fine-tune a pretrained backbone, choosing between linear probe, frozen backbone, and full fine-tuning

Skills ladder. Custom Dataset/DataLoader classes; augmentation pipelines — and the habit of visualising augmented batches before training on them.

Chapter 2 closed on a question: a network’s filters are fitted to one dataset and one task, so how much of what it learned is about ImageNet, and how much is about images? This chapter reaches the answer by way of the architectures — the four years in which networks grew from eight layers to a hundred and fifty, top-5 error fell below the human estimate, and the fitted features turned out to be worth more than the task they were fitted to.

3.1 The ImageNet years

Figure 3.1 plots the challenge results from the hand-crafted era to ResNet. Four architectures carry the story.

Figure 3.1: The ImageNet ladder. Top-5 error by year, one point per system, bubble area proportional to parameter count. Every point is that year’s winner except VGG-16, which placed second in 2014 and is here because the section argues with it. Hand-crafted features end at 25.8% in 2011; AlexNet opens the convolutional era at 16.4%; VGG and GoogLeNet halve the error again in 2014 from opposite directions — 138M parameters against 7M; ResNet’s 3.6% in 2015 passes the human estimate of roughly 5%. Hover a point for its parameter count, its depth, and what the system was; the chips re-size the bubbles by depth instead, which reorders them completely. Error values from the challenge report [1] and the architecture papers; parameter counts computed from the published architectures.

ZFNet (2013, 11.7%) is AlexNet, debugged by looking. Zeiler and Fergus built the visualisation tools of the hierarchy section of Chapter 2 and pointed them at the winner: the first layer’s $11 \times 11$ stride-4 filters were aliasing, the failure Chapter 2 describes under downsampling, and some had learned nothing at all. Shrinking the first kernels to $7 \times 7$ stride 2 fixed both findings and took 1.6 points off the error in their own single-model comparison, 18.1% to 16.5% [2]. The 11.7% that won 2013 was the full competition entry, ensembling and all.

VGG (2014, 7.3%) is a single design rule applied without exception: every kernel $3 \times 3$, every stride 1, double the channels after every pool [3]. The rule rests on the parameter arithmetic of Chapter 2. Two stacked $3 \times 3$ layers see a $5 \times 5$ field through 73,856 parameters (at 64 channels) where one $5 \times 5$ layer needs 102,464; three of them see $7 \times 7$ for roughly half the parameters of a single $7 \times 7$. Each replacement also inserts an extra ReLU. At equal receptive field, a stack of $3 \times 3$ layers therefore costs fewer parameters and adds nonlinearity, which is the whole of the design rule. What VGG’s simplicity costs is elsewhere: 138 million parameters, of which 124 million are the dense head that Chapter 2 flagged as AlexNet’s worst habit, uncorrected.

GoogLeNet (2014, 6.7%) attacked exactly that. Its Inception block runs $1 \times 1$, $3 \times 3$, and $5 \times 5$ branches in parallel, giving several receptive fields per layer. The parameter saving comes from the $1 \times 1$ convolution of Chapter 2: before each expensive branch, a $1 \times 1$ layer compresses the channels, cutting the $5 \times 5$ branch of its first block from 153,632 parameters to 15,920 [4]. With the dense head replaced by global average pooling, the entire 22-layer network holds about 7 million parameters, a twentieth of VGG at similar accuracy. The expense had been the dense head and the channel product, not the depth.¹

Then the progression stopped. VGG’s own experiments say it plainly: 16 layers beat 11, 19 barely beat 16, and deeper configurations stopped helping. Both 2014 networks sat at depths around twenty for a reason. Something about depth itself was broken, and the next two ideas are what fixed it.

Interactive: the architectures, layer by layer

The two networks of Chapter 2, this section’s VGG-16 and GoogLeNet, and the ResNet-50 of the next section, as browsable stacks — every shape, parameter count, and receptive field computed live. Before opening VGG-16: from the design rule alone, how many of its 138M parameters do you expect in the first conv block, and how many in the head? Check, then find the single most expensive layer in each network.

3.2 Going deeper: two fixes

3.2.1 Why stacking fails: unstable gradients

Backpropagation, stated once more: the gradient at layer $\ell$ is the gradient at layer $\ell + 1$ pushed through one more local Jacobian. By the bottom of an $L$-layer network the signal has been multiplied $L$ times, and long products are unstable in only two directions. If the typical factor is slightly under one, the product vanishes: $0.9^{50} \approx 0.005$, and the early layers receive almost no gradient. Slightly over one and it explodes, $1.1^{50} \approx 117$. Sigmoid nonlinearities made the vanishing case the default, since their slope is at most ¼ and near zero once saturated. ReLU’s slope is exactly 1 wherever it is active, which removes that source of shrinkage and is most of why it displaced the sigmoid. He initialisation scales the initial weights to preserve variance, which keeps the product stable at the start of training. Neither fix controls it throughout.

3.2.2 Batch normalisation

The layer that does is batch normalisation [5]. Inserted after a convolution, it standardises each channel across the current minibatch, subtracting the batch mean and dividing by the batch standard deviation, then applies a learned scale $\gamma$ and shift $\beta$ per channel, so the network keeps its expressive range. The convention is conv → BN → ReLU, repeated everywhere. Figure 3.2 draws the operation.

A schematic showing four small stacks of feature maps labelled image 1 to image 4, each with its front map highlighted in gold, an arrow to the formulas for the channel mean and standard deviation, the standardisation, and the learned scale and shift, then an arrow to the words same shape, steadied scale. — Figure 3.2: Batch normalisation, one channel at a time. The statistics are taken across the whole minibatch, pooling every image’s copy of that channel. The channel is standardised against them, and the learned $\gamma_c$ and $\beta_c$ give the network back a scale and shift of its own choosing.

What it improves is conditioning. Each layer now trains against inputs of steady scale, whatever the layers below are doing, so gradients stay usable, learning rates can be set around ten times higher, and initialisation stops being delicate. The original paper explained this as removing “internal covariate shift”; later analysis attributes the benefit to a smoother optimisation landscape [6]. The explanation is still contested; the effect is not.

Two side effects matter in practice. Because each example’s normalisation depends on whichever examples share its batch, BN injects a little noise, which regularises, and networks using it mostly dropped dropout. And because at test time there is no batch, the layer switches to running averages of mean and variance collected during training. That switch is what model.eval() performs. Forgetting it normalises single test images against themselves, which degrades accuracy without raising an error.

BN’s dependence on the batch is also its weakness: at batch size 2 the statistics are noise. GroupNorm and LayerNorm normalise without the batch; LayerNorm’s real entrance is with the transformer (Chapter 8).

3.2.3 The degradation problem

With BN in place, very deep plain networks do train: the gradients flow and the losses descend. They are still worse. He et al. trained a 20-layer and a 56-layer plain network on CIFAR-10 and found the deeper one had higher error on the training set [7]. Read the curves the way Chapter 2 taught: not overfitting — an overfitted network wins on training data and loses on validation; this one loses on both. Figure 3.3 reproduces the experiment with this book’s own runs.

The result contradicts a simple argument. The 56-layer network contains the 20-layer one: set 36 layers to the identity and it could match its shallower sibling exactly. It does not, which says the failure is not capacity but optimisation — nothing steers three dozen layers of freshly initialised weights towards do nothing, because in a plain stack the identity is not a nearby function. It is a specific, delicate configuration of millions of parameters, and gradient descent never finds it.

Figure 3.3: The degradation problem, reproduced. Train and test error on CIFAR-10 for plain (no-skip) networks of 20 and 56 layers, batch norm included, trained identically for 60 epochs. The deeper network is worse on both — including on training error, which is what rules out overfitting. Hover any epoch for the two depths and the gap; the left panel is the one that matters. Setup follows He et al. [7], section 4.2, at a shorter schedule.

3.2.4 Residual connections

The repair is one addition sign. A residual block computes

\[ y = F(x) + x, \]

where $F$ is two or three conv-BN-ReLU layers and $x$ rides past them on an unmodified skip connection. The block no longer produces the output; it produces the change to the input, and the change defaults to zero. Where a plain stack must construct the identity, a residual stack starts there: push $F$’s weights towards zero, which is where weight decay pushes anyway, and the block reduces to its input. An unhelpful block can therefore be neutralised without being removed. It still holds parameters and still costs computation and activation memory; what it no longer costs is accuracy.

The backward pass explains the training curves. Differentiating $y = F(x) + x$ gives $\partial L/\partial x = \partial L/\partial y \,(1 + F'(x))$. That constant 1 is a term of the gradient reaching the block’s input untouched by any weight, at every block. The long product that opened this section now has an additive bypass running its whole length, and the vanishing argument no longer applies to the network, only to the optional detours.

ResNets are the template of Chapter 2 built from these blocks, VGG’s design rule kept ($3 \times 3$ kernels, double channels on downsample), GAP head, BN throughout [7]. Figure 3.4 draws the plain block, the residual block, and the bottleneck side by side. The deeper family members swap in that bottleneck block: $1 \times 1$ down to a quarter of the channels, $3 \times 3$ in the narrow space, $1 \times 1$ back up, at 70,016 parameters where two plain $3 \times 3$ layers at 256 channels would cost 1,180,160. A single 152-layer version reaches 4.49% top-5 on the validation set, and the six-model ensemble that won ImageNet 2015 reached 3.57% on the test set [7], under the ~5% human estimate. Rerun with skips, the degradation experiment inverts: within this family, deeper is now reliably better, and the follow-up trains past a thousand layers [8]. Depth had been a research obstacle for thirty years; with residual connections it became a design parameter.

Three block diagrams. The first shows input flowing through two stacked conv-BN-ReLU boxes to the output. The second shows the same two boxes with an additional arrow bypassing them from input to a circled plus sign that also receives the boxes' output. The third shows three boxes labelled one-by-one, three-by-three, one-by-one with a bypass arrow to a circled plus. — Figure 3.4: The residual block. Left: a plain block, whose output is whatever the weights construct, with the identity nowhere nearby. Middle: the residual block, where the skip carries $x$ past the weights, $F(x)$ is added as a correction, and the gradient’s $1 + F'$ splits the same way. Right: the bottleneck variant, $1 \times 1$ compress, $3 \times 3$, $1 \times 1$ expand, with the skip unchanged.

Skip connections outlived their birthplace. U-Net’s encoder–decoder skips (Chapter 6) and the transformer’s skips around every sublayer (Chapter 8) are there for the same gradient-path reason.

Interactive: watch the gradient die

A deep stack at random initialisation, one backward pass, the gradient’s size plotted per layer. Set fifty sigmoid layers with naive initialisation and predict the shape before running — then fix it three ways, one toggle at a time, and rank the fixes. Which survives depth 200 alone?

3.3 Efficiency: MobileNet and EfficientNet in brief

ResNet settled accuracy; the next constraint was everything else. A model that runs on a phone, a robot, or a free Colab GPU is measured on three costs at once, parameters, FLOPs and latency, so an architecture is a point on a frontier of accuracy against cost rather than a single accuracy number.

MobileNet [9] is the separable factorisation of Chapter 2 applied wall to wall: every $3 \times 3$ dense convolution becomes depthwise-plus-pointwise, the 8.7× saving compounding through the stack, and the whole ImageNet network lands near 4 million parameters. That is a thirtieth of VGG-16’s 138 million, for 70.6% top-1 against VGG’s 71.6% — a point of accuracy given up for a factor of thirty in size. Two multipliers scale it down further, one thinning every layer and one shrinking the input, tracing a family of models along the frontier rather than one point on it.

EfficientNet [10] made the frontier itself the object. Depth, width, and resolution all raise accuracy with diminishing returns, and the paper’s experiments show that scaling one alone saturates sooner than scaling all three together. Its compound rule grows them in fixed ratio, and the resulting family took over the frontier: B0 matches ResNet-50’s accuracy at roughly a tenth of the operations and a fifth of the parameters.

Figure 3.5 plots the chapter on one set of axes. The question an architecture answers stopped being how accurate and became how accurate at what cost, which is the form a deployment decision takes.

Figure 3.5: Accuracy against cost, one decade of architectures. ImageNet top-1 accuracy against multiply–accumulate operations per image, log scale. The frontier moves up and left. VGG-16 is dominated by ResNet-50 and by every EfficientNet, each of them both more accurate and cheaper; MobileNet holds the low-cost corner at an accuracy slightly below VGG’s; the EfficientNet family traces the 2019 frontier. Hover for each model’s three costs and where the number comes from, then switch the cost axis to parameters. The ordering changes, which is why a network is compared on more than one cost at a time.

3.4 Transfer learning and fine-tuning

3.4.1 Whose features are they?

Chapter 2’s closing question has an empirical answer, and it arrived almost immediately. In 2014 two groups took the penultimate activations of an ImageNet-trained network, the 4096 numbers before the classifier, computed by weights fitted only to ImageNet. One group used AlexNet’s, the other OverFeat’s. They froze those activations and fed them to simple classifiers on entirely different problems: birds, scenes, flowers, furniture [11], [12]. The frozen features beat the hand-crafted state of the art on nearly every benchmark tried. Razavian et al. said it in their title: an astounding baseline. So the features were not narrowly ImageNet’s. A thousand nouns and 1.2 million photographs had been varied enough that most of the network’s layers learned representations useful well beyond that label set, with only the last layers specific to the thousand nouns. Figure 3.6 draws the manoeuvre that observation licenses.

A row of shrinking blue blocks representing the convolutional layers, labelled the backbone, with an arrow to two small squares: a dashed gold one labelled its ImageNet head, discarded, and a solid one labelled your task's head, trained. — Figure 3.6: The transfer manoeuvre. The convolutional backbone, fitted to ImageNet and expensive to train, is kept. The classifier that made it an ImageNet model is discarded, and a head for the new task trains in its place.

That observation changed practice. If the features transfer, nobody needs to train them twice — and the expensive, data-hungry part of every new vision problem is already done, by someone else, once.

3.4.2 What transfers, by depth

Not all layers transfer equally, and the pattern follows the feature hierarchy of Chapter 2. Yosinski et al. spliced networks at each depth and measured the damage [13]: first-layer features transferred with almost no loss across their task split, and transferability fell off with depth until the last layers, which were specific to the source task. Their experiment used two halves of ImageNet, so the finding is about tasks of that kind rather than about any target domain whatever. The practical reading: early layers are the most reusable, late layers the least, and how much to keep depends on how far the new task sits from the pretraining.

Figure 3.7 runs that experiment on the two datasets Lab 3 uses. Cut a frozen ResNet-50 at five depths, average whatever comes out over space, and fit one linear layer to it: on cats and dogs the accuracy climbs all the way to the last block, while on textures it peaks a block earlier and then falls. The dashed line is the control that makes the rest of the figure mean anything — the same architecture with random weights, probed identically.

Two line charts sharing a vertical axis of linear-probe accuracy. In the left panel, the pretrained curve rises steadily from 0.12 at the stem to 0.918 at layer 4, while a dashed random-weights curve stays between 0.10 and 0.16. In the right panel, the pretrained curve rises to a circled peak of 0.682 at layer 3 then falls slightly at layer 4, with the same flat dashed control below it. — Figure 3.7: What transfers, measured at five depths. A frozen ResNet-50 is cut after the stem and after each of its four stages; the activations are global-average-pooled and a linear classifier is fitted to them. Left, Oxford-IIIT Pet, 37 breeds, a domain ImageNet already knows, where every extra block helps and the last one reaches 91.8%. Right, the Describable Textures Dataset, 47 classes, where accuracy peaks at 68.2% after stage 3 and drops at stage 4, because the last block’s features are specific to ImageNet’s object categories and a texture is not an object. The random-weight curves are the same architecture untrained, and they stay near chance at every depth: what transfers is the fitting, not the wiring. Our own runs (`figures/runs/train_transfer.py`).

3.4.3 The recipes

Using a pretrained backbone comes down to one decision, how much of it to let move, with three standard settings.

A linear probe freezes everything and trains one linear layer on the features. Cheapest, least adjustable, and diagnostically the most honest: whatever accuracy it reaches is credited to the representation alone, which is why probing returns as the standard measuring instrument for self-supervised methods (Chapter 11). A frozen backbone with a new multi-layer head adjusts a little more. Full fine-tuning unfreezes everything and continues training on the new task — almost always at a learning rate around ten times lower than training from scratch, because the pretrained weights are a good solution you are trying to nudge, not noise you are trying to replace, and Chapter 2 said what an oversized step does to a good solution.

Which setting wins is mostly two questions: how much target data, and how far from the pretraining domain. Small dataset, similar domain: freeze. The probe or frozen head uses the features as they are and cannot overfit much. Large dataset, any domain: fine-tune fully, since there is enough signal to improve even the general layers. Small dataset, distant domain is the uncomfortable square, where freezing keeps features that half-fit and tuning overfits, and the honest answer is try both and augment hard. The boundaries are soft, and they sit at a particular place for a particular problem: on the 37 breeds of Lab 3’s dataset, freezing leads by 6.1 points at one image per class and full fine-tuning has taken the lead by twenty-five, which is a few hundred images either side of the crossing. Lab 3 puts you in that comparison directly — frozen against full fine-tune on the same data, both runs logged. Figure 3.8 lays the choice out.

A two by two grid with axes more target data rightward and closer to the pretraining upward. The quadrants read: small data near domain, freeze; large data near domain, full fine-tune at a low learning rate; large data far domain, full fine-tune or train from scratch; and highlighted in gold, small data far domain, the uncomfortable square, try both and augment hard. — Figure 3.8: The recipe, by dataset. Two questions place a task: how much labelled data, and how close to the pretraining domain. Three quadrants have a standard answer. The fourth, small and far, is the one where nothing works cleanly and the practical advice is to try both recipes and lean on augmentation.

Three practicalities catch people. Preprocess with the pretraining statistics, because a backbone trained on ImageNet-normalised inputs expects the same at fine-tuning time. Replace the head to match your classes before anything else moves. And when fine-tuning with small batches, freeze BN’s running statistics, for the reason the section above gave: batch statistics from a batch of four are noise.

When transfer fails, it fails by domain gap. X-rays, aerial tiles, and spectrograms are far enough from photographs that the late layers contribute little, and for genuinely large target datasets even the early layers stop paying: given enough data and training time, a network trained from scratch catches up [14]. Pretraining gives faster convergence and better accuracy on small datasets, not a permanent advantage.

Interactive: freeze or fine-tune?

The recipe table’s data axis, run as an experiment. Both recipes start from the same pretrained ResNet-50 on the same 37 breeds, under the same training budget; only the number of training images changes. Commit to an answer before you look: at one image per class, 37 images in total, which recipe wins and by how much? Then find the size at which the answer changes.

Transfer learning inherits the source distribution. What did ImageNet pretraining bake into the features you just reused? The taxonomy, for one: WordNet’s, with person-categories so troubling that the maintainers later removed them. Measurable perceptual habits, for another. ImageNet-trained CNNs tend to classify by texture where humans classify by shape, so that a cat rendered with elephant skin is read as an elephant [15], and your fine-tuned model starts from that prior. And the collection itself was scraped without consent, the history Chapter 1’s provenance reading traced [16]. A pretrained backbone is a dependency like any other: audit what ships inside it.

3.5 The backbone era

This chapter built a noun the rest of the book relies on. A backbone is a pretrained feature extractor, canonically ResNet-50, kept frozen or gently tuned as the first component of a larger system. Every architecture in Part II begins “take a pretrained backbone”; detection, segmentation, and their descendants are structurally elaborate heads. That answers most of the ceiling Chapter 1 closed on: the features adapt to the task, and usually you download them rather than train them. It does not answer all of it, since the transfer holds only as far as the domain gap allows.

One assumption is still standing. Everything here rested on 1.2 million labels, produced by years of annotation. Whether the labels were ever necessary, and whether the images alone could have taught the same features, is the question of Chapter 11.

Exercises

The ladder’s arithmetic. Using Chapter 2’s parameter formula: verify that VGG-16’s conv1 block (two $3 \times 3$ layers, 3 → 64 → 64 channels) holds 38,720 parameters while its first dense layer holds 102,764,544; then explain, in two sentences and one computation, how GoogLeNet reaches ~7M parameters total while being deeper than VGG.
Receptive field per parameter. For 64-in, 64-out channels, compare one $7 \times 7$ layer against three stacked $3 \times 3$ layers: receptive field, parameter count, and number of ReLUs. State one thing the single $7 \times 7$ layer can compute that the stack cannot.
Batch norm by hand. A channel’s values across a batch of four are $(2, 4, 4, 6)$. Compute the normalised values — take the variance with divisor $4$, not $3$, and use no $\epsilon$ term — then the layer output for learned $\gamma = 2$, $\beta = 1$. Now the same input arrives at test time with running mean 3 and running variance 1 — compute the output, and name the bug that produces the training-time answer at test time.
The gradient highway. Differentiate $y = F(x) + x$ to show the input gradient contains a term independent of $F$’s weights. Explain in one sentence why stacking 100 such blocks does not reintroduce the vanishing product of Section 3.2.1.
Separability at network scale. A MobileNet-style layer, 512 in, 512 out, $3 \times 3$: compute dense and separable parameter counts and the ratio. As $C_{\text{out}}$ grows with $k$ fixed at 3, what does the saving approach, and why does the pointwise term dominate?
Choosing the recipe. Four situations: (a) 800 labelled photos of local birds; (b) 400,000 labelled product photos; (c) 600 chest X-rays; (d) 90,000 aerial tiles. For each: probe, frozen backbone, or full fine-tune; the learning rate you would start from relative to from-scratch; and your single biggest concern, one sentence each.
Audit the inheritance. Design a two-experiment probe of your own fine-tuned model for texture bias in the style of Geirhos et al. [15]: state the stimuli, the measurement, and the result that would convince you the bias transferred.

Lab 3

Lab 3: fine-tune a pretrained ResNet-50 on a small custom dataset; frozen backbone vs full fine-tune — log both runs and compare their curves; visualise your augmented batches before training on them.

Open in Colab

References

[1]

O. Russakovsky et al., “ImageNet large scale visual recognition challenge,” International Journal of Computer Vision, vol. 115, no. 3, pp. 211–252, 2015, doi: 10.1007/s11263-015-0816-y.

[2]

M. D. Zeiler and R. Fergus, “Visualizing and understanding convolutional networks,” in European conference on computer vision (ECCV), 2014, pp. 818–833. doi: 10.1007/978-3-319-10590-1_53.

[3]

K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in International conference on learning representations (ICLR), 2015. Available: https://arxiv.org/abs/1409.1556

[4]

C. Szegedy et al., “Going deeper with convolutions,” in IEEE conference on computer vision and pattern recognition (CVPR), 2015, pp. 1–9. doi: 10.1109/CVPR.2015.7298594.

[5]

S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in International conference on machine learning (ICML), 2015, pp. 448–456.

[6]

S. Santurkar, D. Tsipras, A. Ilyas, and A. Mądry, “How does batch normalization help optimization?” in Advances in neural information processing systems (NeurIPS), 2018, pp. 2483–2493.

[7]

K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in IEEE conference on computer vision and pattern recognition (CVPR), 2016, pp. 770–778. doi: 10.1109/CVPR.2016.90.

[8]

K. He, X. Zhang, S. Ren, and J. Sun, “Identity mappings in deep residual networks,” in European conference on computer vision (ECCV), 2016, pp. 630–645. doi: 10.1007/978-3-319-46493-0_38.

[9]

A. G. Howard et al., “MobileNets: Efficient convolutional neural networks for mobile vision applications,” arXiv preprint arXiv:1704.04861, 2017, Available: https://arxiv.org/abs/1704.04861

[10]

M. Tan and Q. V. Le, “EfficientNet: Rethinking model scaling for convolutional neural networks,” in International conference on machine learning (ICML), 2019, pp. 6105–6114.

[11]

J. Donahue et al., “DeCAF: A deep convolutional activation feature for generic visual recognition,” in International conference on machine learning (ICML), 2014, pp. 647–655.

[12]

A. S. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson, “CNN features off-the-shelf: An astounding baseline for recognition,” in IEEE conference on computer vision and pattern recognition workshops (CVPRW), 2014, pp. 806–813. doi: 10.1109/CVPRW.2014.131.

[13]

J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, “How transferable are features in deep neural networks?” in Advances in neural information processing systems (NIPS), 2014, pp. 3320–3328.

[14]

K. He, R. Girshick, and P. Dollár, “Rethinking ImageNet pre-training,” in IEEE international conference on computer vision (ICCV), 2019, pp. 4918–4927. doi: 10.1109/ICCV.2019.00502.

[15]

R. Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann, and W. Brendel, “ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness,” in International conference on learning representations (ICLR), 2019. Available: https://arxiv.org/abs/1811.12231

[16]

A. Birhane and V. U. Prabhu, “Large image datasets: A pyrrhic win for computer vision?” in IEEE winter conference on applications of computer vision (WACV), 2021, pp. 1536–1546. doi: 10.1109/WACV48630.2021.00158.

The paper also hangs two auxiliary classifiers off the middle of the network, extra losses meant to push gradient into the early layers — a scaffold that batch normalisation, below, made unnecessary.↩︎

# Architectures and Transfer Learning {#sec-architectures} ::: {.callout-note icon=false} ## Learning objectives - Trace the AlexNet → VGG → GoogLeNet → ResNet lineage and what each step changed, with the parameter arithmetic to back it - Explain the two obstacles to depth, unstable gradients and then the degradation problem, and the two fixes: batch normalisation and residual connections - Compare architectures on accuracy against parameters and FLOPs, and say what depthwise separability saves MobileNet - Fine-tune a pretrained backbone, choosing between linear probe, frozen backbone, and full fine-tuning ::: ::: {.thread .thread-skills} **Skills ladder.** Custom `Dataset`/`DataLoader` classes; augmentation pipelines — and the habit of visualising augmented batches *before* training on them. ::: Chapter 2 closed on a question: a network's filters are fitted to one dataset and one task, so how much of what it learned is about ImageNet, and how much is about images? This chapter reaches the answer by way of the architectures — the four years in which networks grew from eight layers to a hundred and fifty, top-5 error fell below the human estimate, and the fitted features turned out to be worth more than the task they were fitted to. ## The ImageNet years {#sec-imagenet-years} @fig-imagenet-ladder plots the challenge results from the hand-crafted era to ResNet. Four architectures carry the story. :::: {#fig-imagenet-ladder} ```{=html} <iframe src="../widgets/imagenet-ladder.html" class="widget-frame" height="600" loading="lazy" allow="fullscreen" allowfullscreen title="ImageNet top-5 error by year, one point per system, bubble area proportional to parameter count or depth, with a hover readout for each system's numbers."></iframe> ``` The ImageNet ladder. Top-5 error by year, one point per system, bubble area proportional to parameter count. Every point is that year's winner except VGG-16, which placed second in 2014 and is here because the section argues with it. Hand-crafted features end at 25.8% in 2011; AlexNet opens the convolutional era at 16.4%; VGG and GoogLeNet halve the error again in 2014 from opposite directions — 138M parameters against 7M; ResNet's 3.6% in 2015 passes the human estimate of roughly 5%. Hover a point for its parameter count, its depth, and what the system was; the chips re-size the bubbles by depth instead, which reorders them completely. Error values from the challenge report [@russakovsky2015ilsvrc] and the architecture papers; parameter counts computed from the published architectures. :::: **ZFNet** (2013, 11.7%) is AlexNet, debugged by looking. Zeiler and Fergus built the visualisation tools of the hierarchy section of @sec-cnns and pointed them at the winner: the first layer's $11 \times 11$ stride-4 filters were aliasing, the failure @sec-cnns describes under downsampling, and some had learned nothing at all. Shrinking the first kernels to $7 \times 7$ stride 2 fixed both findings and took 1.6 points off the error in their own single-model comparison, 18.1% to 16.5% [@zeiler2014visualizing]. The 11.7% that won 2013 was the full competition entry, ensembling and all. **VGG** (2014, 7.3%) is a single design rule applied without exception: every kernel $3 \times 3$, every stride 1, double the channels after every pool [@simonyan2015vgg]. The rule rests on the parameter arithmetic of @sec-cnns. Two stacked $3 \times 3$ layers see a $5 \times 5$ field through 73,856 parameters (at 64 channels) where one $5 \times 5$ layer needs 102,464; three of them see $7 \times 7$ for roughly half the parameters of a single $7 \times 7$. Each replacement also inserts an extra ReLU. At equal receptive field, a stack of $3 \times 3$ layers therefore costs fewer parameters and adds nonlinearity, which is the whole of the design rule. What VGG's simplicity costs is elsewhere: 138 million parameters, of which 124 million are the dense head that @sec-cnns flagged as AlexNet's worst habit, uncorrected. **GoogLeNet** (2014, 6.7%) attacked exactly that. Its Inception block runs $1 \times 1$, $3 \times 3$, and $5 \times 5$ branches in parallel, giving several receptive fields per layer. The parameter saving comes from the $1 \times 1$ convolution of @sec-cnns: before each expensive branch, a $1 \times 1$ layer compresses the channels, cutting the $5 \times 5$ branch of its first block from 153,632 parameters to 15,920 [@szegedy2015googlenet]. With the dense head replaced by global average pooling, the entire 22-layer network holds about 7 million parameters, a twentieth of VGG at similar accuracy. The expense had been the dense head and the channel product, not the depth.^[The paper also hangs two auxiliary classifiers off the middle of the network, extra losses meant to push gradient into the early layers — a scaffold that batch normalisation, below, made unnecessary.] Then the progression stopped. VGG's own experiments say it plainly: 16 layers beat 11, 19 barely beat 16, and deeper configurations stopped helping. Both 2014 networks sat at depths around twenty for a reason. Something about depth itself was broken, and the next two ideas are what fixed it. ### Interactive: the architectures, layer by layer {.unnumbered} The two networks of @sec-cnns, this section's VGG-16 and GoogLeNet, and the ResNet-50 of the next section, as browsable stacks — every shape, parameter count, and receptive field computed live. Before opening VGG-16: from the design rule alone, how many of its 138M parameters do you expect in the first conv block, and how many in the head? Check, then find the single most expensive layer in each network. ```{=html} <iframe src="../widgets/architecture-assembler.html?preset=vgg16" class="widget-frame" height="640" loading="lazy" allow="fullscreen" allowfullscreen title="Architecture browser: modernised LeNet-5, AlexNet, VGG-16, GoogLeNet and ResNet-50 as layer stacks with live shape, parameter, and receptive-field readouts, plus a free-build mode."></iframe> ``` ## Going deeper: two fixes {#sec-depth} ### Why stacking fails: unstable gradients {#sec-unstable-gradients} Backpropagation, stated once more: the gradient at layer $\ell$ is the gradient at layer $\ell + 1$ pushed through one more local Jacobian. By the bottom of an $L$-layer network the signal has been multiplied $L$ times, and long products are unstable in only two directions. If the typical factor is slightly under one, the product vanishes: $0.9^{50} \approx 0.005$, and the early layers receive almost no gradient. Slightly over one and it explodes, $1.1^{50} \approx 117$. Sigmoid nonlinearities made the vanishing case the default, since their slope is at most ¼ and near zero once saturated. ReLU's slope is exactly 1 wherever it is active, which removes that source of shrinkage and is most of why it displaced the sigmoid. He initialisation scales the initial weights to preserve variance, which keeps the product stable *at the start* of training. Neither fix controls it throughout. ### Batch normalisation The layer that does is **batch normalisation** [@ioffe2015batchnorm]. Inserted after a convolution, it standardises each channel across the current minibatch, subtracting the batch mean and dividing by the batch standard deviation, then applies a learned scale $\gamma$ and shift $\beta$ per channel, so the network keeps its expressive range. The convention is conv → BN → ReLU, repeated everywhere. @fig-batchnorm draws the operation. ![Batch normalisation, one channel at a time. The statistics are taken across the whole minibatch, pooling every image's copy of that channel. The channel is standardised against them, and the learned $\gamma_c$ and $\beta_c$ give the network back a scale and shift of its own choosing.](../figures/batchnorm.png){#fig-batchnorm fig-alt="A schematic showing four small stacks of feature maps labelled image 1 to image 4, each with its front map highlighted in gold, an arrow to the formulas for the channel mean and standard deviation, the standardisation, and the learned scale and shift, then an arrow to the words same shape, steadied scale."} What it improves is conditioning. Each layer now trains against inputs of steady scale, whatever the layers below are doing, so gradients stay usable, learning rates can be set around ten times higher, and initialisation stops being delicate. The original paper explained this as removing "internal covariate shift"; later analysis attributes the benefit to a smoother optimisation landscape [@santurkar2018bnhelp]. The explanation is still contested; the effect is not. Two side effects matter in practice. Because each example's normalisation depends on whichever examples share its batch, BN injects a little noise, which regularises, and networks using it mostly dropped dropout. And because at test time there is no batch, the layer switches to running averages of mean and variance collected during training. That switch is what `model.eval()` performs. Forgetting it normalises single test images against themselves, which degrades accuracy without raising an error. ::: {.column-margin} BN's dependence on the batch is also its weakness: at batch size 2 the statistics are noise. GroupNorm and LayerNorm normalise without the batch; LayerNorm's real entrance is with the transformer (@sec-attention). ::: ### The degradation problem With BN in place, very deep plain networks do train: the gradients flow and the losses descend. They are still worse. He et al. trained a 20-layer and a 56-layer plain network on CIFAR-10 and found the deeper one had higher error *on the training set* [@he2016resnet]. Read the curves the way Chapter 2 taught: not overfitting — an overfitted network wins on training data and loses on validation; this one loses on both. @fig-degradation reproduces the experiment with this book's own runs. The result contradicts a simple argument. The 56-layer network *contains* the 20-layer one: set 36 layers to the identity and it could match its shallower sibling exactly. It does not, which says the failure is not capacity but optimisation — nothing steers three dozen layers of freshly initialised weights towards *do nothing*, because in a plain stack the identity is not a nearby function. It is a specific, delicate configuration of millions of parameters, and gradient descent never finds it. :::: {#fig-degradation} ```{=html} <iframe src="../widgets/degradation.html" class="widget-frame" height="560" loading="lazy" allow="fullscreen" allowfullscreen title="Training and test error per epoch for plain 20-layer and 56-layer networks on CIFAR-10, with a hover readout giving both depths and the gap between them at any epoch."></iframe> ``` The degradation problem, reproduced. Train and test error on CIFAR-10 for plain (no-skip) networks of 20 and 56 layers, batch norm included, trained identically for 60 epochs. The deeper network is worse on both — including on training error, which is what rules out overfitting. Hover any epoch for the two depths and the gap; the left panel is the one that matters. Setup follows He et al. [-@he2016resnet], section 4.2, at a shorter schedule. :::: ### Residual connections The repair is one addition sign. A **residual block** computes $$ y = F(x) + x, $$ where $F$ is two or three conv-BN-ReLU layers and $x$ rides past them on an unmodified **skip connection**. The block no longer produces the output; it produces the *change* to the input, and the change defaults to zero. Where a plain stack must construct the identity, a residual stack starts there: push $F$'s weights towards zero, which is where weight decay pushes anyway, and the block reduces to its input. An unhelpful block can therefore be neutralised without being removed. It still holds parameters and still costs computation and activation memory; what it no longer costs is accuracy. The backward pass explains the training curves. Differentiating $y = F(x) + x$ gives $\partial L/\partial x = \partial L/\partial y \,(1 + F'(x))$. That constant 1 is a term of the gradient reaching the block's input untouched by any weight, at every block. The long product that opened this section now has an additive bypass running its whole length, and the vanishing argument no longer applies to the network, only to the optional detours. ResNets are the template of @sec-cnns built from these blocks, VGG's design rule kept ($3 \times 3$ kernels, double channels on downsample), GAP head, BN throughout [@he2016resnet]. @fig-residual-block draws the plain block, the residual block, and the bottleneck side by side. The deeper family members swap in that **bottleneck** block: $1 \times 1$ down to a quarter of the channels, $3 \times 3$ in the narrow space, $1 \times 1$ back up, at 70,016 parameters where two plain $3 \times 3$ layers at 256 channels would cost 1,180,160. A single 152-layer version reaches 4.49% top-5 on the validation set, and the six-model ensemble that won ImageNet 2015 reached 3.57% on the test set [@he2016resnet], under the ~5% human estimate. Rerun with skips, the degradation experiment inverts: within this family, deeper is now reliably better, and the follow-up trains past a thousand layers [@he2016identity]. Depth had been a research obstacle for thirty years; with residual connections it became a design parameter. ![The residual block. Left: a plain block, whose output is whatever the weights construct, with the identity nowhere nearby. Middle: the residual block, where the skip carries $x$ past the weights, $F(x)$ is added as a correction, and the gradient's $1 + F'$ splits the same way. Right: the bottleneck variant, $1 \times 1$ compress, $3 \times 3$, $1 \times 1$ expand, with the skip unchanged.](../figures/residual-block.png){#fig-residual-block fig-alt="Three block diagrams. The first shows input flowing through two stacked conv-BN-ReLU boxes to the output. The second shows the same two boxes with an additional arrow bypassing them from input to a circled plus sign that also receives the boxes' output. The third shows three boxes labelled one-by-one, three-by-three, one-by-one with a bypass arrow to a circled plus."} ::: {.column-margin} Skip connections outlived their birthplace. U-Net's encoder–decoder skips (@sec-segmentation) and the transformer's skips around every sublayer (@sec-attention) are there for the same gradient-path reason. ::: ### Interactive: watch the gradient die {.unnumbered} A deep stack at random initialisation, one backward pass, the gradient's size plotted per layer. Set fifty sigmoid layers with naive initialisation and predict the shape before running — then fix it three ways, one toggle at a time, and rank the fixes. Which survives depth 200 alone? ```{=html} <iframe src="../widgets/gradient-flow.html" class="widget-frame" height="600" loading="lazy" allow="fullscreen" allowfullscreen title="Interactive gradient flow: per-layer gradient magnitudes for one backward pass through a deep randomly initialised network, with depth slider and toggles for sigmoid or ReLU, naive or He initialisation, batch normalisation, and skip connections."></iframe> ``` ## Efficiency: MobileNet and EfficientNet in brief {#sec-efficiency} ResNet settled accuracy; the next constraint was everything else. A model that runs on a phone, a robot, or a free Colab GPU is measured on three costs at once, parameters, FLOPs and latency, so an architecture is a point on a frontier of accuracy against cost rather than a single accuracy number. **MobileNet** [@howard2017mobilenet] is the separable factorisation of @sec-cnns applied wall to wall: every $3 \times 3$ dense convolution becomes depthwise-plus-pointwise, the 8.7× saving compounding through the stack, and the whole ImageNet network lands near 4 million parameters. That is a thirtieth of VGG-16's 138 million, for 70.6% top-1 against VGG's 71.6% — a point of accuracy given up for a factor of thirty in size. Two multipliers scale it down further, one thinning every layer and one shrinking the input, tracing a family of models along the frontier rather than one point on it. **EfficientNet** [@tan2019efficientnet] made the frontier itself the object. Depth, width, and resolution all raise accuracy with diminishing returns, and the paper's experiments show that scaling one alone saturates sooner than scaling all three together. Its compound rule grows them in fixed ratio, and the resulting family took over the frontier: B0 matches ResNet-50's accuracy at roughly a tenth of the operations and a fifth of the parameters. @fig-efficiency-frontier plots the chapter on one set of axes. The question an architecture answers stopped being *how accurate* and became *how accurate at what cost*, which is the form a deployment decision takes. :::: {#fig-efficiency-frontier} ```{=html} <iframe src="../widgets/efficiency-frontier.html" class="widget-frame" height="600" loading="lazy" allow="fullscreen" allowfullscreen title="ImageNet top-1 accuracy against cost on a log axis, with a switch between operations and parameters as the cost axis and a hover readout giving each model's accuracy, operations, parameters, and source."></iframe> ``` Accuracy against cost, one decade of architectures. ImageNet top-1 accuracy against multiply–accumulate operations per image, log scale. The frontier moves up and left. VGG-16 is dominated by ResNet-50 and by every EfficientNet, each of them both more accurate and cheaper; MobileNet holds the low-cost corner at an accuracy slightly below VGG's; the EfficientNet family traces the 2019 frontier. Hover for each model's three costs and where the number comes from, then switch the cost axis to parameters. The ordering changes, which is why a network is compared on more than one cost at a time. :::: ## Transfer learning and fine-tuning {#sec-transfer} ### Whose features are they? Chapter 2's closing question has an empirical answer, and it arrived almost immediately. In 2014 two groups took the penultimate activations of an ImageNet-trained network, the 4096 numbers before the classifier, computed by weights fitted only to ImageNet. One group used AlexNet's, the other OverFeat's. They froze those activations and fed them to simple classifiers on entirely different problems: birds, scenes, flowers, furniture [@donahue2014decaf; @razavian2014offtheshelf]. The frozen features beat the hand-crafted state of the art on nearly every benchmark tried. Razavian et al. said it in their title: an astounding baseline. So the features were not narrowly ImageNet's. A thousand nouns and 1.2 million photographs had been varied enough that most of the network's layers learned representations useful well beyond that label set, with only the last layers specific to the thousand nouns. @fig-backbone-swap draws the manoeuvre that observation licenses. ![The transfer manoeuvre. The convolutional backbone, fitted to ImageNet and expensive to train, is kept. The classifier that made it an ImageNet model is discarded, and a head for the new task trains in its place.](../figures/backbone-swap.png){#fig-backbone-swap fig-alt="A row of shrinking blue blocks representing the convolutional layers, labelled the backbone, with an arrow to two small squares: a dashed gold one labelled its ImageNet head, discarded, and a solid one labelled your task's head, trained."} That observation changed practice. If the features transfer, nobody needs to train them twice — and the expensive, data-hungry part of every new vision problem is already done, by someone else, once. ### What transfers, by depth Not all layers transfer equally, and the pattern follows the feature hierarchy of @sec-cnns. Yosinski et al. spliced networks at each depth and measured the damage [@yosinski2014transferable]: first-layer features transferred with almost no loss across their task split, and transferability fell off with depth until the last layers, which were specific to the source task. Their experiment used two halves of ImageNet, so the finding is about tasks of that kind rather than about any target domain whatever. The practical reading: early layers are the most reusable, late layers the least, and how much to keep depends on how far the new task sits from the pretraining. @fig-probe-depth runs that experiment on the two datasets Lab 3 uses. Cut a frozen ResNet-50 at five depths, average whatever comes out over space, and fit one linear layer to it: on cats and dogs the accuracy climbs all the way to the last block, while on textures it peaks a block earlier and then *falls*. The dashed line is the control that makes the rest of the figure mean anything — the same architecture with random weights, probed identically. ![What transfers, measured at five depths. A frozen ResNet-50 is cut after the stem and after each of its four stages; the activations are global-average-pooled and a linear classifier is fitted to them. Left, Oxford-IIIT Pet, 37 breeds, a domain ImageNet already knows, where every extra block helps and the last one reaches 91.8%. Right, the Describable Textures Dataset, 47 classes, where accuracy peaks at 68.2% after stage 3 and drops at stage 4, because the last block's features are specific to ImageNet's object categories and a texture is not an object. The random-weight curves are the same architecture untrained, and they stay near chance at every depth: what transfers is the fitting, not the wiring. Our own runs (`figures/runs/train_transfer.py`).](../figures/probe-depth.png){#fig-probe-depth fig-alt="Two line charts sharing a vertical axis of linear-probe accuracy. In the left panel, the pretrained curve rises steadily from 0.12 at the stem to 0.918 at layer 4, while a dashed random-weights curve stays between 0.10 and 0.16. In the right panel, the pretrained curve rises to a circled peak of 0.682 at layer 3 then falls slightly at layer 4, with the same flat dashed control below it."} ### The recipes Using a pretrained **backbone** comes down to one decision, how much of it to let move, with three standard settings. A **linear probe** freezes everything and trains one linear layer on the features. Cheapest, least adjustable, and diagnostically the most honest: whatever accuracy it reaches is credited to the representation alone, which is why probing returns as the standard measuring instrument for self-supervised methods (@sec-ssl). A **frozen backbone** with a new multi-layer head adjusts a little more. **Full fine-tuning** unfreezes everything and continues training on the new task — almost always at a learning rate around ten times lower than training from scratch, because the pretrained weights are a good solution you are trying to nudge, not noise you are trying to replace, and Chapter 2 said what an oversized step does to a good solution. Which setting wins is mostly two questions: how much target data, and how far from the pretraining domain. Small dataset, similar domain: freeze. The probe or frozen head uses the features as they are and cannot overfit much. Large dataset, any domain: fine-tune fully, since there is enough signal to improve even the general layers. Small dataset, distant domain is the uncomfortable square, where freezing keeps features that half-fit and tuning overfits, and the honest answer is *try both and augment hard*. The boundaries are soft, and they sit at a particular place for a particular problem: on the 37 breeds of Lab 3's dataset, freezing leads by 6.1 points at one image per class and full fine-tuning has taken the lead by twenty-five, which is a few hundred images either side of the crossing. Lab 3 puts you in that comparison directly — frozen against full fine-tune on the same data, both runs logged. @fig-transfer-recipes lays the choice out. ![The recipe, by dataset. Two questions place a task: how much labelled data, and how close to the pretraining domain. Three quadrants have a standard answer. The fourth, small and far, is the one where nothing works cleanly and the practical advice is to try both recipes and lean on augmentation.](../figures/transfer-recipes.png){#fig-transfer-recipes fig-alt="A two by two grid with axes more target data rightward and closer to the pretraining upward. The quadrants read: small data near domain, freeze; large data near domain, full fine-tune at a low learning rate; large data far domain, full fine-tune or train from scratch; and highlighted in gold, small data far domain, the uncomfortable square, try both and augment hard."} Three practicalities catch people. Preprocess with the *pretraining* statistics, because a backbone trained on ImageNet-normalised inputs expects the same at fine-tuning time. Replace the head to match your classes before anything else moves. And when fine-tuning with small batches, freeze BN's running statistics, for the reason the section above gave: batch statistics from a batch of four are noise. When transfer fails, it fails by domain gap. X-rays, aerial tiles, and spectrograms are far enough from photographs that the late layers contribute little, and for genuinely large target datasets even the early layers stop paying: given enough data and training time, a network trained from scratch catches up [@he2019rethinking]. Pretraining gives faster convergence and better accuracy on small datasets, not a permanent advantage. ### Interactive: freeze or fine-tune? {.unnumbered} The recipe table's data axis, run as an experiment. Both recipes start from the same pretrained ResNet-50 on the same 37 breeds, under the same training budget; only the number of training images changes. Commit to an answer before you look: at one image per class, 37 images in total, which recipe wins and by how much? Then find the size at which the answer changes. ```{=html} <iframe src="../widgets/transfer-explorer.html" class="widget-frame" height="640" loading="lazy" allow="fullscreen" allowfullscreen title="Test accuracy against training-set size for a frozen backbone and a full fine-tune of the same pretrained ResNet-50, with a difference view and a table of all runs."></iframe> ``` ::: {.thread .thread-ethics} Transfer learning inherits the source distribution. What did ImageNet pretraining bake into the features you just reused? The taxonomy, for one: WordNet's, with person-categories so troubling that the maintainers later removed them. Measurable perceptual habits, for another. ImageNet-trained CNNs tend to classify by texture where humans classify by shape, so that a cat rendered with elephant skin is read as an elephant [@geirhos2019texture], and your fine-tuned model starts from that prior. And the collection itself was scraped without consent, the history Chapter 1's provenance reading traced [@birhane2021pyrrhic]. A pretrained backbone is a dependency like any other: audit what ships inside it. ::: ## The backbone era {#sec-backbone-era} This chapter built a noun the rest of the book relies on. A **backbone** is a pretrained feature extractor, canonically ResNet-50, kept frozen or gently tuned as the first component of a larger system. Every architecture in Part II begins "take a pretrained backbone"; detection, segmentation, and their descendants are structurally elaborate heads. That answers most of the ceiling Chapter 1 closed on: the features adapt to the task, and usually you download them rather than train them. It does not answer all of it, since the transfer holds only as far as the domain gap allows. One assumption is still standing. Everything here rested on 1.2 million *labels*, produced by years of annotation. Whether the labels were ever necessary, and whether the images alone could have taught the same features, is the question of @sec-ssl. ## Exercises {.unnumbered} 1. **The ladder's arithmetic.** Using Chapter 2's parameter formula: verify that VGG-16's conv1 block (two $3 \times 3$ layers, 3 → 64 → 64 channels) holds 38,720 parameters while its first dense layer holds 102,764,544; then explain, in two sentences and one computation, how GoogLeNet reaches ~7M parameters total while being deeper than VGG. 2. **Receptive field per parameter.** For 64-in, 64-out channels, compare one $7 \times 7$ layer against three stacked $3 \times 3$ layers: receptive field, parameter count, and number of ReLUs. State one thing the single $7 \times 7$ layer can compute that the stack cannot. 3. **Batch norm by hand.** A channel's values across a batch of four are $(2, 4, 4, 6)$. Compute the normalised values — take the variance with divisor $4$, not $3$, and use no $\epsilon$ term — then the layer output for learned $\gamma = 2$, $\beta = 1$. Now the same input arrives at test time with running mean 3 and running variance 1 — compute the output, and name the bug that produces the training-time answer at test time. 4. **The gradient highway.** Differentiate $y = F(x) + x$ to show the input gradient contains a term independent of $F$'s weights. Explain in one sentence why stacking 100 such blocks does not reintroduce the vanishing product of @sec-unstable-gradients. 5. **Separability at network scale.** A MobileNet-style layer, 512 in, 512 out, $3 \times 3$: compute dense and separable parameter counts and the ratio. As $C_{\text{out}}$ grows with $k$ fixed at 3, what does the saving approach, and why does the pointwise term dominate? 6. **Choosing the recipe.** Four situations: (a) 800 labelled photos of local birds; (b) 400,000 labelled product photos; (c) 600 chest X-rays; (d) 90,000 aerial tiles. For each: probe, frozen backbone, or full fine-tune; the learning rate you would start from relative to from-scratch; and your single biggest concern, one sentence each. 7. **Audit the inheritance.** Design a two-experiment probe of your own fine-tuned model for texture bias in the style of Geirhos et al. [-@geirhos2019texture]: state the stimuli, the measurement, and the result that would convince you the bias transferred. ## Lab 3 {.unnumbered} ::: {.lab-pointer} [Lab 3](../labs/lab-03.ipynb){download="lab-03.ipynb"}: fine-tune a pretrained ResNet-50 on a small custom dataset; frozen backbone vs full fine-tune — log both runs and compare their curves; visualise your augmented batches before training on them. [![Open in Colab](../figures/colab-badge.svg)](https://colab.research.google.com/github/kleinric/cv-labs/blob/main/lab-03.ipynb) ::: ## Further reading {.unnumbered} - Simonyan and Zisserman [-@simonyan2015vgg] — VGG; the ablation tables are the argument. - Szegedy et al. [-@szegedy2015googlenet] — GoogLeNet; read for how the parameter count is controlled. - Ioffe and Szegedy [-@ioffe2015batchnorm], and Santurkar et al. [-@santurkar2018bnhelp] — batch norm, and the later argument about why it works. - He et al. [-@he2016resnet] — ResNet; the degradation experiment in section 4 repays close reading. - Howard et al. [-@howard2017mobilenet] and Tan and Le [-@tan2019efficientnet] — the efficiency frontier, built and then formalised. - Yosinski et al. [-@yosinski2014transferable] — what transfers, measured layer by layer. - Geirhos et al. [-@geirhos2019texture] — texture bias; the stimuli alone are worth the visit. - He, Girshick, and Dollár [-@he2019rethinking] — the honest counterweight: when pretraining does not help. ## References {.unnumbered} ::: {#refs} :::