AgentHands：在XR中为空间锚定的智能体对话生成交互式手势

Xun Qian, Research Scientist, and Ruofei Du, Interactive Perception & Graphics Lead, Google XR AgentHands is an LLM-powered XR prototype that augments conversational agents with synchronized, expressive hand gestures to provide spatially grounded guidance, bridging the mental mapping gap and enhancing user engagement in physical tasks. As AI assistants evolve from simple text interfaces to multimodal companions, we are seeing a shift toward more proactive, situated assistance. Recent innovations like Project Astra and Gemini 3.1 Flash Live already allow users to discuss their physical surroundings in real time, often utilizing visual bounding box overlays to identify objects in a camera feed. While these overlays are highly effective for 2D screens, the transition to immersive platforms like Android XR presents a unique challenge: how do we move beyond flat UI to create a truly embodied, spatially aware dialogue? To bridge this gap, we introduce AgentHands , published at CHI 2026 , a research prototype that brings the power of co-speech gestures to the 3D world. In human communication, our hands do more than just point; they describe shapes, mimic actions, and emphasize points, all synchronized with our voice. By leveraging the spatial understanding capabilities of Extended Reality (XR), AgentHands replicates this natural synergy. Following up our prior research in Human I/O and Sensible Agent , AgentHands further equips AI agents with expressive, synchronized hand gestures that transform abstract verbal instructions into intuitive, physical demonstrations, making conversations about your surroundings more natural and engaging. AgentHands demo: Empowering AI agents with expressive, synchronized hand gestures for spatially grounded conversations in XR. To start, we conducted a formative study with XR and human–computer interaction (HCI) experts at Google to determine what makes a virtual hand “legible” in a 3D environment. We distilled these insights into a multi-dimensional taxonomy that defines how an agent should use its hands to ground a conversation within a user's physical space. Handedness & gesture: Choosing between one or two hands and selecting from a library of forms, such as a “palm” for caution or a “cylindrical grip” to mimic holding a tool. Spatiality: Leveraging the depth of XR to determine where hands should live. They can be mid-air for general conversation, object-anchored for identifying specific parts, or user-relative for social cues. Temporal dynamics & visual effects (VFX): Gestures in XR aren't just static poses; they include animated motions like “pouring” or “tracing”. We also utilize XR’s unique visual layer by adding effects, such as a red glow to signify a heat warning. The AgentHands taxonomy diagram shows the six dimensions: Handedness, Gesture, Spatiality, Temporal Dynamics, Interactivity, and Visual Effects. The core innovation of AgentHands is its ability to map the high-level reasoning of LLMs into precise, real-time physical motions that match the agent's “voice” and the user's XR environment. We introduce the following key steps to compose the AgentHands workflow. The system begins with a lightweight object registration module. Using eye gaze and scene reconstruction, users can quickly “tag” items — like an orchid or a laptop — creating a spatial registry with 3D bounding boxes that the agent can reference.