activation init extras
One backward pass through a randomly initialised stack (width 48, batch 32), no training: each bar is the size of the gradient reaching that layer, on a log axis. The product argument of the prose is the whole story — watch it with sigmoid + naive initialisation at depth 50, then repair it one toggle at a time. He initialisation sets the weight scale, batch norm re-standardises every layer's activations (modelled here by its scaling effect), and skip connections give the gradient a term no weight can shrink. Which fixes survive depth 200 alone, and which only together?