Zero-training speech integration preserves vision-language ability

Automated excerpt

Adding discrete speech tokens to a frozen vision‑language backbone instantly yields functional audio understanding without any gradient updates. “Training‑Free Omni (TFO), a plug-and‑play framework that converts a frozen VLM into a speech‑centric omni model without architectural modification, or multimodal re‑alignment. ” [1] Before TFO, omnidirectional models required dedicated audio encoders and costly joint training on image‑audio‑text triples, tightly coupling the new modality to a specific backbone and often degrading the original visual reasoning abilities. On 56 benchmarks spanning 21 languages, TFO matches native omni models in audio‑visual understanding while improving average audio‑only performance across all five evaluated model settings.

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.