Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

Automated excerpt

This raises the question: can local inference viably redistribute demand from centralized infrastructure? We evaluate 20+ state-of-the-art local LMs, 8 hardware accelerators (local and cloud), and 1M real-world single-turn chat and reasoning queries. Third, local accelerators achieve at least 1. 4x lower IPW than cloud accelerators running identical models, revealing significant headroom for local accelerator optimization.

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.