Ingestion
Also called Indexing Pipeline
The one-time-ish job of loading documents in: parse, chunk, embed, and store them for later search.
Think of it like
Stocking the pantry before you cook — everything gets sorted and shelved so it is ready when you need it.
Example
Nightly, new help articles are parsed, split into chunks, embedded, and written to the vector store so search stays current.
How it actually works
Ingestion is the write side of retrieval: extract text (from PDFs, HTML, wherever), clean it, chunk it, embed the chunks, and persist vectors plus metadata. It is unglamorous but decisive — parsing errors, bad chunking, or stale data here poison every downstream query. Keeping it incremental and idempotent is what keeps a knowledge base fresh without full rebuilds.
For product teams
Where data freshness and quality are won or lost; schedule and monitor it, don’t set-and-forget.
For engineers
Parse→clean→chunk→embed→upsert; should be incremental and idempotent to keep the index current cheaply.
Related
- Document Store — Feeds the document store and index.
- Chunking — Includes the chunking step.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome