Every location inside the box can name the box. Subtract (l, t, r, b) from the point and you land back on the ground truth exactly — which is why the gold dashed rectangle and the blue rebuilt one coincide wherever you click. So the four regressed numbers cannot tell you which location to believe. Centre-ness can: √(min(l,r)/max(l,r) × min(t,b)/max(t,b)), one extra output channel, near 1 where the four distances balance and towards 0 in a corner. FCOS multiplies the classification score by it before the score threshold and NMS. The classification score here is held equal at every location — that is the slider, and it is an assumption, not a measurement: this book has no per-location classifier output for this photograph. Holding it equal is the point. With p the same everywhere, centre-ness is the only thing separating 440 locations that all claim the same box. Set a threshold you would be happy to send to NMS, then press score alone and count what comes back.