Better Vector Search for Long Documents: Chunking Inside Manticore Search

Automated excerpt

Manticore now handles this in the table definition: add chunk_strategy to the vector column in CREATE TABLE, and Manticore splits each document into chunks, embeds every chunk, and searches all of them:DROP TABLE IF EXISTS docs; That is the whole feature. Documentation pages, knowledge-base articles, contracts, transcripts, email threads, wiki pages, README files, incident postmortems. mean — one vector, but the whole documentv float_vector knn_type='hnsw' hnsw_similarity='cosine' Manticore splits the document, embeds every chunk, and averages the chunk vectors into a single normalized vector. A chunk never starts or ends mid-sentence.

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.