RWKV
An architecture that trains like a transformer but runs like an RNN, aiming for the best of both.
Think of it like
A hybrid car — plug-in convenience for the long haul, efficient engine for the daily drive.
Example
RWKV lets a long chat run with constant memory per token, unlike a transformer’s growing KV cache.
How it actually works
RWKV reformulates attention-like mixing into a recurrent form that can be trained in parallel like a transformer but executed step by step like an RNN, giving constant memory and linear time at inference. That makes very long or streaming generation cheap in memory. It’s part of the wave of architectures probing whether quadratic attention is truly necessary.
For product teams
Another contender for cheap long-context and streaming use, trading the transformer’s memory growth for a constant footprint.
For engineers
A linear-attention-style architecture with parallel training and RNN-like constant-memory inference.
Related
- Transformer — Blends the transformer and recurrent neural network.
- Mamba — A linear-scaling peer to Mamba.
- State Space Model — Related to state space models.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome