Decoder. plain-English AI glossary

Web crawl

● Core

Also called Web Crawl Data

Text scraped in bulk from across the public internet — the messy backbone of most pretraining data.

Think of it like

Emptying the whole internet into a bin and then sorting it — most of the volume, most of the mess.

Example

A team starts from a raw web crawl of billions of pages, then filters it down to the fraction worth training on.

How it actually works

Web crawls provide the sheer scale that pretraining needs, but they are enormously noisy — spam, boilerplate, duplicates, machine text, and worse. Their usefulness depends almost entirely on aggressive cleaning and filtering afterward. Common Crawl is the best-known public source; labs also run private crawls. The web's biases and gaps flow straight into the model unless curation catches them.

For product teams

Cheap scale, but its raw quality is low — the value is in what you do to it after.

For engineers

Bulk-scraped public web text; high-volume, low-signal until filtered and deduplicated.

Common Crawl — The best-known public crawl.
Pretraining corpus — The pool it feeds.

Read anything AI without the jargon

Look up any term in plain English, or save terms as you read with the free Chrome extension.

Open Decoder Add to Chrome