Audio Generation
Automatically creating sound or audio content from scratch or from a description.
Think of it like
A composer writing a melody - the model generates the notes and tones.
Example
Text prompt: "Dog barking at distance" -> generated audio of a dog bark. Or: "Electronic lo-fi hip-hop beat" - the model produces the music.
How it actually works
Raw waveform generation (difficult, high-dimensional) vs. spectrogram generation (compact, then vocoded back). Autoregressive (SampleRNN, WaveNet) vs. non-autoregressive (FastSpeech, diffusion). Text-to-speech, music generation, and sound effect synthesis use different architectures. Transformer-based models and diffusion models show promise. Challenge: long-range coherence, realistic timbre, and diversity.
For product teams
Enables text-to-speech, generative music, game sound design, accessibility, content creation.
For engineers
Architectures: Diffusion models (Riffusion, MusicGen), autoregressive transformers. Datasets: AudioSet, FSD50K. Metrics: mel-cepstral distortion (MCD), log-magnitude RMSE.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome