領 · 域 · 分 割
Segmentation Lab
Lab 01 asked what a convolution does to an image. This one asks a harder question:
how do you label every single pixel, how do you know whether you got it right, and
how do you get the resolution back after a network has thrown it away?
One
評 · 価
Accuracy lies, IoU does not
A segmentation prediction is just a mask. Compare it to the ground truth and every pixel
falls into one of four buckets. Drag the prediction below — or shrink the object with the
slider — and watch what happens to each score. One of them barely moves. That one is the
reason careless papers report numbers that mean nothing.
Ground truth vs prediction
True positive — both agree it is object
False positive — predicted, but isn't there
False negative — missed real object
True negative — background
Outline — the true object's edge
Drag the prediction anywhere, or use arrow keys once the panel has focus.
The dark outline never moves — that is where the object really is.
Two
損 · 失
What the network is actually punished for
IoU tells you how good a prediction is, but you cannot train on it directly — it is not
differentiable. So networks are trained on a stand-in. The three below make very different
bargains, and the one you pick decides what the network learns to ignore.
Below is a real prediction over a real mask, not a single abstract pixel — which matters,
because Dice is a property of two sets and says nothing about one pixel on its own.
The two heatmaps show where each loss is spending its attention. That is the whole
argument in one picture.
Truth & soft prediction
Cross-entropy, per pixel
Focal, per pixel
Drag the prediction · both heatmaps share one scale, so they are comparable
Totals on this mask
Share of the loss coming from background pixels
Explain which loss
Per-pixel behaviour
Cross-entropy and focal are per-pixel losses, so a per-pixel curve is an honest picture
of them. Dice is deliberately absent — it has no per-pixel value to plot.
This is not academic. In camouflaged object detection the animal may be two percent of the
pixels, so cross-entropy's easy background wins swamp the gradient from the handful of pixels
that matter. Dice normalises that away; focal loss starves the easy pixels of gradient on
purpose. Most working systems use cross-entropy and Dice added together.
Three
拡 · 大
Getting the resolution back
A classifier's job is to destroy spatial information — pool it away until only "cat" remains.
Segmentation needs it back at full resolution. There are three ways to enlarge a feature map,
and only one of them has anything to learn.
Feature map
Nearest neighbour
no parameters · blocky
Bilinear
no parameters · smooth
Transposed convolution
learnable
Nearest and bilinear are fixed rules — they invent no detail, they only choose between the
values they were given. The transposed convolution has actual weights, and in a real network
those weights are learned. They are set by hand here so you can compare two specific choices,
and the next section walks through how they land on the output.
Four
転 · 置
Transposed convolution, one cell at a time
A normal convolution gathers: many input cells collapse into one output. A transposed
convolution scatters: each input cell paints a whole copy of the kernel into the output,
and wherever those copies land on top of each other the values add up. Step through it and
watch the overlaps build — the uneven ones are the checkerboard artifact being born.
Input
Kernel
each input cell scatters this, scaled by its own value
Keyboard: ← → to step, space to play
Five
架 · 橋
What the skip connections were carrying
U-Net's shape gets drawn in every lecture, but the arrows across the middle are usually
waved at. Here is what they do. The encoder halves the image three times; the decoder puts
it back. Switch a skip off and the detail that level was carrying is gone for good —
there is nowhere else in the network it could have come from.
Read this as an analogy, not a simulation. There are no learned weights here —
the encoder is average pooling and the decoder is bilinear upsampling, so with every skip
connected the reconstruction comes back exactly. A trained U-Net never does that:
its convolutions transform features rather than preserving them. What transfers is the
structural point — a skip is the only route by which fine detail can reach the decoder —
and that point is real. Section six of lab 01, and the training section below, are where
actual weights get learned.
Input
64 × 64
Bottleneck
8 × 8 — everything the encoder kept
Decoder output
What was lost
Image
Reconstruction error
Six
学 · 習
Now train it to find the animal
Everything above was analysis. This is a real network — two convolutional layers, a ReLU
between them, a sigmoid at the end, weights in total — learning
by gradient descent, in your browser, right now. Its job is the one from my own research:
there is a shape hidden in that texture, and almost nothing separates it from the background
except statistics.
The interesting control is the loss. Section two argued that cross-entropy loses interest in
small objects. Here you can test that claim — train with each loss and compare the
IoU it reaches.
Input
the animal is in there
Target
Prediction
Loss & IoU
vermillion loss · sage IoU
Best IoU reached, by loss
One result worth keeping. Focal loss here uses α = 0.5. The value quoted
almost everywhere is α = 0.25, from the RetinaNet paper — and measured on this task it makes
focal markedly worse, because it down-weights exactly the rare class we are trying
to find. Detection and segmentation are not the same problem, and a hyperparameter that is
standard in one is not automatically right in the other. Inherit numbers from papers with
care, and measure them on your own data.
This is a two-layer network with a 5×5 effective receptive field and no pooling, so do not
read too much into the absolute IoU — a real segmenter sees far more context. What does
transfer is the comparison: same network, same data, same initialisation, only the objective
changed.
Seven
検 · 出
Boxes, and throwing most of them away
Detection answers a coarser question than segmentation — where is it, roughly — but the
same IoU appears again, this time with a closed form for rectangles. A detector proposes
hundreds of overlapping boxes, and non-maximum suppression is the almost embarrassingly
simple rule that cleans them up.
Box IoU — drag either box
Lower the threshold and NMS gets aggressive — two genuinely separate animals standing close
together get merged into one detection. Raise it and duplicates survive. There is no correct
value, only a value that suits your data, which is why it is a tuned hyperparameter rather
than a constant.