AI article
Code Search RAG: Chunk by Function With tree-sitter, Not by 512 Tokens
Community description: Fixed token windows cut functions in half. A runnable tree-sitter chunker that splits Python by function and class, scored against plain token windows.
Dev.to | Oct 14, 2026 | Gabriel Anhaia
Automated excerpt
Parse Python with tree-sitter, cut at function and class boundaries, carry the file path and the enclosing symbol as metadata, and split the definitions that are too big on statement boundaries. The next chunk gets the rest of the body with no name attached. Its chunks still get the file path in the header.
Selected automatically from source text; not independently written or fact-checked. Read the original for full context.