趋势解读：New open-source voice model listens nonstop and decides，讨论数据集与基础模型

The "Audio Interaction" AI model processes continuous audio streams and combines tasks such as dialog, translation, transcription and sound recognition in a single system. To do this, it breaks down the audio stream into 0.4-second segments and decides after each segment via a special token whether it should remain silent or generate a response. Trained with an artificial data set of 302,000 hours of audio, the model processes listening and speaking in parallel. This minimizes the waiting time for responses and allows the system to beat models such as Gemini 3 Flash in proactive noise detection tests. Researchers want to close the gap between today's audio speech models and real listeners. Their system handles dialog, translation, and sound recognition all at once. Today's audio voice models, like GPT-4o or Qwen 3.5-Omni , work like a dictation machine with a button: they only respond when the recording ends. Streaming systems like Moshi for dialog or Paraformer for live subtitles do listen in, but they can only handle one task at a time and treat sounds like coughing as background noise. Researchers from China, Hong Kong, and Singapore want to combine both approaches with "audio interaction." The model listens to an audio stream continuously, breaks it into 0.4-second chunks, and decides after each chunk whether to stay silent or speak. Translation, transcription, chatting, and reacting to everyday noises all run in a single three-billion-parameter model. Ad After each audio snippet, the model outputs either or . If it picks , it keeps listening. Only with does it start talking. Classic tasks like "Translate into English" become instructions within the same continuous stream. Ad DEC_D_Incontent-1 According to the paper , Audio-Interaction scored 58.15 points on the audio benchmark MMAU, narrowly beating its base model Qwen2.5-Omni-3B. It also comes close to much larger 7B models. On English-Chinese translation, the model improves a lot over the base. For the model to learn when to step in, the team needed the right training data. Existing audio datasets consist of short, isolated clips and lack long sequences with sparse response signals, the researchers say. Ad So they built their own scenes in three stages. First, a language model designed a plausible setting—say, a kitchen in the morning—with three to 15 sub-events. The system then searched a database for matching clips or had missing sounds like breaking glass created by audio models like AudioX or ElevenLabs. A preprocessing step then smoothed out the cut edges so the recordings sounded natural. The resulting StreamAudio-2M dataset contains 2.6 million units and about 302,000 hours of audio across seven skill areas and 28 subtasks. Ad DEC_D_Incontent-2 Two weaknesses kept showing up during training. First, the model forgot earlier content in long, noisy sequences. The fix: asking questions that point back to passages from much earlier in the audio, forcing the model to build up long-term memory. Ad