谷歌想让视频 AI 读懂人们在做事过程中的需求

A research team led by Google has introduced StreamVibe, a video agent designed to decide for itself when to speak up. The tests show above all how hard that remains for today's AI models. Researchers from Google, Stanford, Oxford, and several U.S. universities are working on an assistant that watches you cook or assemble furniture and speaks up only when needed. With StreamVibe, they've built a video agent designed to understand even vague requests such as "Please help me if I run into any difficulties." According to the paper , existing video assistants are built for clear-cut commands like "Let me know when the water starts boiling," so they rarely pick up on unspoken intentions. They also have a hard time tracking what's already been done, which causes them to repeat advice or respond too late, and when users change their requests midway through, the assistants often stick to the original instructions. To measure the problem, the team compiled a benchmark of about 4,100 examples from more than 183 hours of everyday life and software recordings. Gemini 2.5 Flash suggested when to offer help and what to say, while four human evaluators had to unanimously approve each example. The test covers three stages including clear instructions, vague goals, and requests that change over time. In one example, a user taking apart a toy excavator asks for progress updates and then switches to requesting a quality check after three minutes. From there, the researchers trained StreamVibe-VL, a model apparently based on Alibaba's Qwen3.5-9B that decides every second whether to stay quiet or speak. A second training stage uses reinforcement learning to make the model more reserved so it doesn't constantly interrupt. A separate memory system tracks which requests are pending and which steps are complete, though the model alone decides whether to speak and what to say. According to the researchers, StreamVibe-VL improves response timing and quality by 10.7 percent over other models, while the memory system is designed to boost proactive interaction by another 12.3 percent and cut down on unnecessary interruptions, among other things. StreamVibe-VL generally beats its base model and other open-source models by a wide margin, though Google's proprietary Gemini 3.8 Flash scores higher in several tests. According to the paper, the specialized training doesn't hurt general video comprehension. Models often make the right call when they only have to decide at specific moments whether a prompt is appropriate. But when they process an entire video and have to build on their own earlier responses, performance across all tested systems plummets as errors compound, and a key content metric for the responses drops by up to 61.5 percent based on the system's own history. In videos longer than five minutes, the models rarely pick the right moment and find it especially hard to track a changed instruction over a long stretch. In a GIMP tutorial on converting a photo to black-and-white documented in the paper and on the project page, the comparison model JoyAI says the finished image has already been saved even though the user hasn't opened it yet, while StreamVibe-VL gives the next step at the right moment, such as "Colors" and then "Desaturate." In a step-by-step guide for making rolled toast with jam, other models just describe what's happening on screen, such as the bread being flattened, while StreamVibe-VL advises cutting off the crust first. The researchers acknowledge that real-world situations are likely more complex than their test setup, and future work will examine how user preferences and the environment influence each other over time. Google has been working on AI systems that actively analyze videos for a while now. In September, Google introduced agent-based video analysis for Gemini , where the model itself decides which scenes to examine more closely. With Gemini 4 Argon , Google also recently showed how AI can track longer action s