The project logs, technical runbooks, and problem-solving notes behind Ben’s engineering builds.
The dream of AI has always been natural, seamless interaction. We don’t want to click buttons or wait for “processing” bars; we want to talk to our computers like we talk to people.
This project uses Pipecat, a framework for conversational AI, to orchestrate a pipeline that captures a video frame the moment you stop speaking, sends it to a Vision-Language Model (VLM), and responds with synthesized speech in near real-time.
To achieve this, we combined several best-in-class local tools:
VLM: Qwen-3-VL (running via LM Studio) for visual and textual understanding. qwen-3-vl-2b-instruct was chosen as it has vision capability while being non-thinking. A non-thinking variant for a speech to speech AI is important to reduce the latency between when the user finishes his speech to the time the AI responds with speech. The 2b model was chosen to be lightweight so that inference would be snappy, while keeping VRAM usage reasonable.
STT: OpenAI Whisper (CUDA-accelerated) for converting speech to text.
TTS: Piper for blazing-fast, neural-quality text-to-speech.
Orchestration: Pipecat for managing the flow of data “frames” between these services.
Latency is the enemy of conversation. If a bot takes 3 seconds to respond, the “illusion of life” is broken. This implementation uses three specific strategies to stay fast:
The “Just-in-Time” Vision Capture Instead of constantly streaming video (which eats bandwidth and processing power), we use a custom VisionCaptureProcessor. It listens for a UserStoppedSpeakingFrame. The moment the Voice Activity Detector (VAD) detects you’ve finished your sentence, the camera snaps a single 432x240 frame and injects it into the LLM context. This ensures the AI sees exactly what you were talking about right when you finish talking.
Aggressive VAD Tuning Most voice AI feels slow because the system waits too long to ensure you’re actually done talking. In our LLMUserAggregatorParams, we set the user_turn_stop_timeout to 0.5 seconds. This is the “sweet spot”—it’s long enough to allow for a natural breath but short enough to trigger the AI response almost instantly.
Local-First Processing By using Piper TTS and a local Whisper instance, we eliminate the “round-trip” time to the cloud. Piper is particularly impressive because it is designed to run on low-power devices, generating audio frames faster than they are spoken.
This setup isn’t just a toy; it’s a blueprint for:
Accessibility Tools: AI that can describe what it sees to the visually impaired in real-time.
Education: A tutor that can look at a student’s handwritten math problem and talk them through the solution.
Privacy: Because everything runs on local computer, your camera feed and voice never leave your machine.
Github Repository: link
By combining efficient local models with a frame-based pipeline architecture, we’ve moved away from the “stop-and-start” feel of traditional AI. We are finally entering the era of fluid human-AI collaboration.