The idea
Most chat interfaces are a text box. OpenTomo is a companion you hold a conversation with: you speak or type, and it replies out loud through a character that actually reacts while it talks — mouth shapes driven by the audio, expression following the tone.
The goal was to make the exchange feel like talking to someone rather than querying something.
How a turn works
A turn is a short pipeline, and the interesting engineering is in keeping it from feeling like a queue:
- Speech in — the mic is transcribed to text
- Reply — the transcript plus recent conversation history goes to the model
- Speech out — the reply is spoken
- Lip-sync — the generated audio drives visemes on the avatar in real time, so the mouth matches what's actually being said rather than replaying a canned animation
Conversation history is stored server-side, so the companion carries context across a session instead of treating every message as a cold start.
Model-agnostic by design
The model layer talks to any OpenAI-compatible endpoint through the AI SDK rather than binding to one vendor, so the companion can run against a hosted provider or a local model without touching the conversation code. A safety layer sits in front of replies, and the interface is internationalised.
Stack
React 19 and TypeScript on Vite, with the avatar rendered in Three.js via
@react-three/fiber and VRM. The server is Hono on Bun with Drizzle over SQL for
conversation storage. State is Zustand; data fetching is TanStack Query.
