FP16 Quantization Quality Loss: We Blamed the Architecture, the Bug Was One Overflowed Tensor

Automated excerpt

Part 3 of this series ends with a case study called "When fp16 quantization quality loss is real: the LaMa failure". The measurements in it are real, and the failure is reproducible: rebuilding that conversion from the same weights today, the unfixed fp16 model averages 15. 8 dB against fp32's 29. 7 dB across our 14 acceptance cases, and its worst case scores 11. 1 dB — the watermark surviving, exactly as described. Not sixteen layers of masking logic — sixteen numbers.

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.