Disentangling Activation Scaling From Layer-Wise Learning-Rate Reweighting at the Final Representation Boundary

초록

Internal scale interventions entangle activation rescaling with compensation-induced learning-rate reweighting. We separate these effects by moving the global feature learning strength (FLS) output multiplier to internal representation boundaries and pairing each cut with a learning-rate-only control that preserves the upstream compensation without scaling activations. In a controlled CIFAR-100 ResNet18-CIFAR depth sweep, learning-rate reweighting reproduces most of the accuracy trend: the Spearman correlation with cut depth is 0.93 for boundary scaling and 0.89 for its LR-only counterpart. At the final representation boundary, however, activation scaling remains +1.643 pp above the exact same-cut control. Four final-interface diagnostics distinguish the two interventions: activation scaling increases feature norm, logit norm, and classifier-weight effective rank while decreasing classifier-input gradient norm. The accuracy ordering is not universal. Under a fixed practical ResNet18-CIFAR protocol, exact LR-only reweighting reaches 70.881%, compared with 70.690% for activation scaling. On practical VGG19-BN, the exact LR-only control becomes unstable for some runs under the 16× upstream learning rate with momentum, so the apparent activation-scaling improvement primarily reflects degraded LR-only performance. A cosine-normalized head yields identical paired outputs, while with pre-head LayerNorm, the two achieve nearly the same accuracy. Within the tested settings, learning-rate allocation explains much of the depth-dependent accuracy trend, while activation scaling can still lead to a distinct final-interface state whose accuracy effect depends on the optimization protocol and classifier design.

키워드