The standard advice

Transfer learning's usual pitch: take a model pretrained on a large, general dataset, reuse its learned features, and fine-tune a small new layer for your specific task. Less data needed, less training time, usually better results than starting from nothing. It's genuinely good advice, most of the time, and it's worth checking directly rather than assuming it holds for every task.

Two real models, the same small task

A small, from-scratch CNN, built specifically for handwritten digit recognition, with under ten thousand parameters:

class SmallCNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 8, 3, padding=1)
        self.conv2 = nn.Conv2d(8, 16, 3, padding=1)
        self.fc = nn.Linear(16 * 7 * 7, 10)

Against a real, pretrained MobileNetV2, trained on ImageNet's million-plus natural photographs, with its backbone frozen and only a new final layer trained on top:

backbone = models.mobilenet_v2(weights=models.MobileNet_V2_Weights.IMAGENET1K_V1)
for p in backbone.parameters():
    p.requires_grad = False
backbone.classifier[1] = nn.Linear(backbone.last_channel, 10)

Same task for both: recognize handwritten digits, trained on the same 2,000 real images, evaluated on the same 1,000 held-out images, three epochs each.

What actually won

from-scratch CNN:      accuracy = 0.8000, train_time = 0.9s, params = 9,098
transfer learning:      accuracy = 0.7670, train_time = 7.6s, trainable params = 12,810 (of 2,236,682 total)

The from-scratch model, with roughly a quarter as many trainable parameters and roughly a tenth the training time, scored higher. Not a close result, a 3.3 percentage point gap in accuracy, in the from-scratch model's favor, running faster the entire time.

Why the "obvious" shortcut lost

MobileNetV2's pretrained features were learned from natural photographs, cars, animals, furniture, faces, textures, real-world lighting and color. A handwritten digit is almost none of that: a small grayscale line drawing, no color, no texture, no real-world lighting to speak of. The features that make MobileNetV2 excellent at recognizing a golden retriever, edges organized into fur patterns, color gradients, object silhouettes against natural backgrounds, aren't the features that distinguish a handwritten 3 from a handwritten 8. Freezing the backbone means those specific, photograph-tuned features are exactly what the new final layer is stuck working with, no matter how well-trained that final layer becomes. The small from-scratch model, with no pretrained features at all, gets to learn representations specifically suited to line-drawing digit shapes from the very first layer, on a task simple enough that a few thousand parameters are genuinely enough to do it well.

What this actually means for when to reach for transfer learning

Not "transfer learning doesn't work." On tasks that stay closer to a pretrained model's original domain, other kinds of natural images, other photographs, transfer learning's usual advantage holds up well, and it's a real, well-established technique for good reason. The result above is specifically about domain mismatch: when the target task is far enough from what a model was actually pretrained on, borrowed features can be worse than no borrowed features at all, and a small model built specifically for the actual task can win outright, using a fraction of the compute.

The takeaway

"Use a pretrained model" is a real, generally good default, and it's still worth checking against a simple from-scratch baseline before committing to it, the same way it's checked here. The gap between a technique's usual reputation and its actual, measured performance on a specific task is exactly the kind of thing that only shows up by testing both directly, not by trusting which one sounds more sophisticated.