Beyond Leaderboard Illusions: Benchmarking Multi-Turn Agentic Feedback Loops in Autonomous Software Engineering with TFD-Bench

Automated excerpt

We benchmarked 5 frontier model configurations across Pass@1 Resolve Rate, AST Tool Calling Validity, Context Token Consumption, and Loop Stalling Frequency. Phase 4: Verification: Sandbox execution re-running the reproducer to assert resolution (exit_code == 0). Discovery #2: AST Tool Syntax Collapse Under Turn Drift In turns 1 through 4, models achieve ~92% valid JSON tool calls.

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.