Real-Time Voice Interaction with 3D AI Agents — Preview

Today we're sharing an early preview of real-time voice interaction with three.ws 3D AI avatars. Agents listen, reason with memory and emotions and tools, then respond in voice — natural conversation with a 3D character that has a face, a body, and a persistent identity.

How it works

Voice interaction in three.ws is built as a full duplex pipeline running entirely in the browser:

1. Microphone input. The browser's Web Audio API captures the user's voice. VAD (voice activity detection) handles turn-taking — no push-to-talk required.
2. Speech-to-text. The audio stream is transcribed in real time using Groq's Whisper endpoint for low-latency, high-accuracy transcription.
3. LLM reasoning. The transcription is passed to the agent's configured LLM brain (Claude, GPT, or others) along with the agent's memory context, emotion state, and active tools. The LLM responds with both the spoken reply and any tool calls or emotion state changes.
4. Text-to-speech. The reply is synthesized using Groq, Anthropic, or OpenAI TTS — selectable per agent. The audio stream is returned to the browser.
5. Lipsync. The TTS audio stream is analyzed in real time to extract viseme timing data, which drives the avatar's mouth movement frame-by-frame in WebGL.
6. Emotion and animation. Any emotion state changes from the LLM response are applied to the avatar simultaneously — the agent can be speaking, gesturing, and expressing an emotion all at once.

AR and VR support in development

Voice interaction is the prerequisite for AR/VR deployment. A 3D agent in a headset that you can only text with is a poor experience. With voice wired up, we're now integrating WebXR to bring three.ws agents into headset environments:

• WebXR passthrough (mixed reality) for compatible headsets
• Spatial audio — the agent's voice comes from where the avatar is in 3D space
• Gaze and gesture interaction as supplementary input alongside voice
• Persistent spatial anchoring so the agent appears in the same location across sessions

Voice model options

Developers can select the voice provider per agent from the studio:

• Groq — fastest end-to-end latency
• Anthropic — high quality, character-consistent voices
• OpenAI — widest voice selection

All three providers are configurable as environment variables — no code changes to switch.

Links

• Try it: three.ws/chat
• Docs: three.ws/docs
• Open source: github.com/nirholas/three.ws

← All posts