The Cascade Equivalence Hypothesis: When Do Speech LLMs Behave Like ASR$\rightarrow$LLM Pipelines?

The Cascade Equivalence Hypothesis: When Do Speech LLMs Behave Like ASR$\rightarrow$LLM Pipelines?

Overview

• Current speech LLMs largely perform implicit ASR: on tasks solvable from a transcript, they are behaviorally and mechanistically equivalent to simple Whisper$→$LLM cascades.
• The authors show this through matched-backbone testing across four speech LLMs and six tasks, controlling for the LLM backbone for the first time.
• Ultravox is statistically indistinguishable from its matched cascade ($\kappa=0.93$); logit lens reveals literal text emerging in hidden states; LEACE concept erasure confirms text representations are causally necessary in both architectures tested, collapsing accuracy to near-zero.
• Qwen2-Audio genuinely diverges, revealing cascade equivalence is architecture-dependent, not universal.
• For most deployed use cases, current speech LLMs are expensive cascades, and under noise, they are worse ones, with clean-condition advantages reversing by up to 7.6% at 0 dB.

The Cascade Equivalence Hypothesis

• Consider a spoken input $A$, its transcript $T = ASR(A)$, and a task label $Y$.
• The acoustic surplus for task $Y$ is $\Delta I_Y = I(A; Y) - I(T; Y)$.
• When $\Delta I_Y \approx 0$, the transcript preserves nearly all task-relevant information; we call such tasks text-sufficient (e.g., factual QA, topic classification, sentiment analysis).
• The Cascade Equivalence Hypothesis: on text-sufficient tasks ($I(A; Y | T) \approx 0$), a speech LLM should be behaviorally indistinguishable from a cascade using the same LLM backbone.

Experimental Setup

Systems Under Test

• Four end-to-end speech LLMs and five ASR$→$LLM cascades evaluated.
• Three "matched-backbone" cascades pair Whisper with the same LLM backbone used inside each open-weight speech LLM.
• Two additional cascades provide upper and lower reference points.

Evaluation Tasks

• Six tasks spanning text-sufficient (TriviaQA, AG News, SST-2, CSQA) and text-insufficient (MELD, MUStARD) domains.
• Noise perturbation conditions (multi-talker babble at 15, 10, 5, and 0 dB SNR) applied to two text-sufficient tasks.

Behavioral Metrics

• Cohen's $\kappa$: chance-corrected per-example agreement
• Conditional error overlap: $P(same\ wrong\ answer|both\ wrong)$
• McNemar's test: systematic directional bias

Probing and Intervention

• Acoustic probes: linear regression for energy and pitch $R^2$
• Text probes: CTC decodability, bag-of-characters $R^2$
• Logit lens: project hidden states through LLM's unembedding matrix
• LEACE concept erasure: surgically remove text information

Results

Agreement on Text-Sufficient Tasks

• Ultravox is near-cascade-equivalent, achieving $\kappa = 0.93$ on AG News, $0.78$ on CSQA, and $0.75$ on SST-2 against its matched cascade.
• Qwen2-Audio genuinely diverges, with $\kappa$ remaining moderate ($0.54$–$0.85$) even against its own Qwen2-7B backbone cascade.
• Cascade equivalence is a spectrum: Phi-4-MM and Gemini fall between these extremes.

Shared Error Signatures

• 84–96% of joint errors involve the identical wrong label, far above chance baselines of 17–33%.
• Mechanisms: LLM reasoning errors and ASR-induced errors.

Text-Insufficient Tasks

• $\kappa$ drops substantially for all pairs, as predicted.
• Backbone confound accounts for a substantial portion of this gap.

Noise Robustness

• Whisper-based cascades degrade substantially less than the E2E models tested.
• At 0 dB, Cascade-S loses 0.5–4.2% accuracy, while E2E models show 5.9–12.7% drops on SST-2 and CSQA.

Mechanistic Evidence

Layer-wise Probing

• Acoustic information is retained but restructured, with finer-grained features progressively overwritten as text representations emerge.
• Qwen2-Audio maintains stable text decodability through layers 0–20 before a sharp decline, while Ultravox shows progressive text emergence.

Logit Lens: Visualizing Text Emergence

• Ultravox reaches 0.34 bag precision at layer 31 with recognizable paraphrases, while Qwen2-Audio peaks at 0.23 with fragmented, multilingual output.

Is the Emergent Text Behaviorally Sufficient?

• On AG News, the implicit cascade (text-only) achieves $\kappa_{impl} = 0.943$ with Ultravox, higher than the explicit cascade.
• On SST-2, CSQA, and MELD, the decoded text is insufficient.

LEACE Causal Erasure

• Text erasure collapses both models to near-zero accuracy on every task.
• CTC and BoC erasure show an architecture-dependent pattern: CTC is devastating for Qwen2-Audio but less so for Ultravox, while BoC is more destructive for Ultravox.

Discussion

When to Use Cascades vs. End-to-End Models

• For text-sufficient tasks in clean conditions, cascades are informationally equivalent to current speech LLMs, offering lower latency, cost, and engineering complexity.
• For noisy environments, Whisper-based cascades prove more robust.
• The case for E2E architectures rests on tasks with genuine acoustic surplus (emotion, sarcasm, speaker intent).

Breaking Cascade Equivalence: A Path Forward

• The models have the acoustic information but do not use it.
• Potential directions: paralinguistic auxiliary losses, minimal-pair prosodic training, and architectural modality separation.

Implications for Benchmarking

• Compare E2E speech LLMs against matched-backbone cascades to isolate architectural effects.
• Include paralinguistic tasks, noisy conditions, and matched-backbone cascade baselines.

Conclusion

• Speech LLMs exhibit a spectrum of cascade equivalence, with Ultravox near-indistinguishable from its matched cascade and Qwen2-Audio genuinely diverging.
• LEACE erasure confirms text representations are causally necessary in both architectures, with architecture-dependent encoding patterns explaining the behavioral spectrum.
• Under noise, Whisper-based cascades prove substantially more robust than the E2E models tested.
• Three key takeaways for practitioners: evaluate with matched backbones, use cascades as the default for text-sufficient tasks, and invest in E2E only where acoustic surplus can be exploited.