Data pipeline
Also called Data Pipeline
The assembly line that turns raw data into clean, tokenized, training-ready batches.
Think of it like
A factory line: raw ingredients go in one end, cleaned and packaged product comes out the other.
Example
Raw web dumps flow through the pipeline — deduped, filtered, tokenized, shuffled, and sharded — before the trainer ever sees a batch.
How it actually works
The pipeline is the engineered sequence of steps between raw sources and the optimizer: ingestion, cleaning, filtering, deduplication, tokenization, shuffling, and sharding into an efficient on-disk format. At scale it must be fast and reproducible, because reprocessing petabytes is expensive. Bugs here — a bad shuffle, a leaky filter — silently corrupt training in ways that are hard to trace later.
For product teams
Invisible plumbing whose reliability quietly determines whether a training run is trustworthy.
For engineers
The reproducible ETL from raw sources to tokenized, shuffled, sharded training batches.
Related
- Data curation — The judgment-heavy part of the pipeline.
- Tokenization — A step near the end of it.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome