AI article
Counting LLM Tokens in Python: tiktoken vs a Dependency-Free Heuristic
Community description: A practical guide to exact and approximate token counting in Python, and how to hide both behind one interface with graceful fallback.
Dev.to | Oct 13, 2026 | Royal Simpson Pinto
Automated excerpt
Context windows are measured in tokens, cost is measured in tokens, and truncation happens in tokens. It gives you the real token count, the same number the model sees. Use it for relative comparison and offline profiling, and use tiktoken when the exact number carries consequences.
Selected automatically from source text; not independently written or fact-checked. Read the original for full context.