Where Did My Data Go? Benchmarking LLMs as Data Pipeline Detectives

Automated excerpt

The model gets whatever logs are left and has to say what happened to the record or "can't tell" if the logs that are left don't prove any single answer. And these are three different things, a model that says "can't tell" to everything would never guess and still score only 44%. The cheapest model, GPT-6 Luna, never guessed once and got everything right on instinct for 18 cents per thousand investigations, while Sonnet guessed on almost every third cold case and Kaggle's default Gemini 3. 7 Flash cost almost 40 times more than Luna and still guessed.

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.