My Comment Section Designed My Next Experiment. Then It Made Me Freeze My Predictions.

Automated excerpt

Ten days ago I published an article about a failure mode: tell a language model "a scanner flagged this code" and some models agree with everything. Gemma's rejections split 51% on false alarms against 2% on real bugs. His fix: run the same 200 code slices again with the flag removed. Same 200 stratified slices from the OWASP Benchmark (100 ground-truth safe, 100 ground-truth real), same JSON schema, same three models. The frozen predictions, written before any run: gpt-4o-mini's confirms on the safe subset fall by ≥15 percentage points without the flag.

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.