Build Log: A Local Voice-Agent Pipeline with Whisper and Piper
You can build a local voice agent that listens, transcribes, answers, and speaks without sending audio to a cloud service. The tradeoff is honest: privacy and offline operation come with slower responses and model quality that depends on your hardware, language, and accent. This guide is assembled from official documentation and community reports — we did not run this build ourselves. It maps the pipeline from microphone to speaker, then gives commands you can run and checks you can time on your own machine. If you have followed our local retrieval-augmented generation (RAG) pipeline build log, you already know the shape: text in, model out, text back. This build adds ears and a mouth. For broader agent patterns, see NiteAgent. If the goal is home control, pair it with the Smart Home Field Guide.
Materials and prerequisites
Start with Python 3.10 or later, a local Ollama server, and the Wyoming service packages for speech-to-text (STT) and text-to-speech (TTS). Add openWakeWord only if you want hands-free activation; otherwise use push-to-talk or VAD-only capture, where VAD means voice activity detection. Install each package with its documented setup script, then keep the model data directory writable.
# wyoming-piper
git clone https://github.com/OHF-Voice/wyoming-piper
cd wyoming-piper
script/setup
script/run --voice en_US-lessac-medium --uri 'tcp://0.0.0.0:10200' --data-dir /data --download-dir /data
# wyoming-faster-whisper
git clone https://github.com/OHF-Voice/wyoming-faster-whisper
cd wyoming-faster-whisper
script/setup
script/run --model tiny-int8 --language en --uri 'tcp://0.0.0.0:10300' --data-dir /data --download-dir /data
The STT service comes from wyoming-faster-whisper, the TTS service from wyoming-piper, and the protocol itself is documented in Wyoming. Ollama is a local model server, not part of the voice packages, so install it separately from Ollama API docs. If you prefer a C/C++ transcription engine instead of the Python stack, see whisper.cpp; if you want the original research model, see openai/whisper.
Use a virtual environment if you want isolation. Decide where audio will come from before installing anything: a browser microphone, a USB headset, or a Home Assistant satellite. Push-to-talk gives an explicit start and stop, while VAD-only capture relies on silence detection. Check that your audio device is visible to the operating system before running the services. A missing microphone often looks like a Whisper failure, but it is only an input routing problem.
| Stage | Tool | What it does | Can you skip it? |
|---|---|---|---|
| Wake word | openWakeWord | scores audio frames for a trigger phrase | Yes: use push-to-talk or VAD-only capture |
| Speech-to-text | wyoming-faster-whisper | transcribes audio through a local Whisper service | No |
| Intent/reply | Ollama or pattern matching | turns transcript into reply text | No |
| Text-to-speech | wyoming-piper | synthesizes reply text as PCM audio | No |
The table assumes a Wyoming-based setup. If you call a script directly, the same stages still apply, but your application owns the sockets and audio buffers.
Stage 1 (optional): wake word detection with openWakeWord
Wake word detection is optional because push-to-talk or VAD can feed audio to STT. openWakeWord processes 80 ms frames and returns a score from 0 to 1 per frame, with a default activation threshold of 0.5. It can run 15-20 models on one Raspberry Pi 3 core, but it is English only.
If you skip wake words, your client can start sending audio when a button is pressed and stop when released. This avoids false activations, but it makes the assistant feel less natural. Treat any wake score as a gate, not as a transcript.
The bundled models include alexa, hey jarvis, hey mycroft, hey rhasspy, weather, and timers. Its stated targets are a false-reject rate below 5 percent and a false-accept rate below 0.5 per hour. But those are README targets, not measured results from this guide. The training data is 100 percent synthetic TTS, and the included models used about 30,000 hours of negative data. Optional Silero VAD gating and Speex noise suppression are available on x86 and Arm64 Linux.
Stage 2: speech-to-text with wyoming-faster-whisper
Use wyoming-faster-whisper to run local Whisper STT as a TCP service. Start it with --model tiny-int8, --language en, and the documented URI and data directories. Add --vad-endpointing 0.7 to end a command after 0.7 seconds of Silero-detected silence, sending the transcript immediately. This is useful when you want hands-free segmentation without a wake word.
script/run --model tiny-int8 --language en --uri 'tcp://0.0.0.0:10300' --data-dir /data --download-dir /data --vad-endpointing 0.7
The endpointing behavior, meaning the rule for deciding when a spoken command has finished, has four documented parts: the silence timer starts only after speech is detected, brief pauses reset it when speech resumes, a command runs for at least one second, and audio after the endpoint is ignored until the client sends audio-stop. It is independent of --vad-filter, which filters during transcription, and --vad-clip, which trims the completed batch WAV, a common audio file format, before transcription.
The first --data-dir receives downloads, so keep it writable. --model can also accept a Hugging Face id such as Systran/faster-distil-whisper-small.en; Distil models are not compatible with name biasing. Weights auto-download from the Hugging Face Hub, a model repository, when you use a supported id.
Whisper was trained on 680,000 hours of multilingual and multitask supervision and generalizes zero-shot, meaning it works on new tasks without fine-tuning, according to the Whisper paper. faster-whisper reports up to 4 times faster than openai/whisper for the same accuracy while using less memory; see faster-whisper.
Model choice changes the feel. A tiny model may respond quickly but can mistranscribe names, numbers, and accented speech. A larger model can improve recognition, but it uses more disk and memory. Use the whisper.cpp footprint table as a planning sheet.
If you use --vad-endpointing, keep the microphone gain stable. Too much background noise can make VAD think speech is still present, while too little gain can delay the endpoint.
Stage 3: intent and replies with Ollama (or no LLM at all)
The transcript can go to Ollama for open-ended answers, or to a small pattern matcher for fixed home-control phrases. Ollama exposes a local API at http://localhost:11434/api, with an OpenAI-compatible endpoint at http://localhost:11434/v1. Local requests need no API key, but the server must be running. For predictable commands, regular expression (regex) matching can avoid large language model (LLM) latency and uncertainty.
curl http://localhost:11434/api/chat -d '{
"model": "<model-name>",
"messages": [{"role": "user", "content": "<transcript>"}],
"stream": false
}'
Send the transcript to POST /api/chat with {model, messages, stream:false}; the answer arrives in message.content. See Ollama API docs. If you want a hardware starting point, read our Ollama-on-a-mini-PC build log.
When you use an LLM, keep the prompt short and deterministic. Ask it to return only the action text, not explanations, because Piper will read everything. Pattern matching avoids that problem by converting a small set of transcripts into fixed replies.
Home Assistant recommends Whisper when you have powerful hardware and plan to extend beyond home control, for example by pairing your assistant with an LLM-based agent, according to the fully-local Assist docs.
For simple commands, pattern matching can call device actions directly, skipping the LLM. This can reduce variance, but it cannot answer arbitrary questions. For agent tutorials, see Hermes Tutorials.
Stage 4: text-to-speech with Piper
Piper turns the reply text into speech locally. Its voice ids follow <lang>-<speaker>-<quality>, such as en_US-lessac-medium, with low, medium, and high tiers. It includes espeak-ng phonemization, which converts text to pronunciation symbols, while Chinese, Japanese, and Thai need extra phonemizer packages. Wyoming-piper exposes a TCP service and an optional unauthenticated HTTP demo gateway.
script/setup
script/run --voice en_US-lessac-medium --uri 'tcp://0.0.0.0:10200' --data-dir /data --download-dir /data
# optional demo HTTP server
script/setup --http
script/run_http --uri 'tcp://localhost:10200'
The base project is Piper, a fast local neural TTS engine used by NVDA and Open Voice OS. On a Raspberry Pi, using medium quality models, it can generate 1.6 seconds of voice in one second, according to the fully-local Assist docs. Listen to voice samples at Piper samples.
Voices needing a missing phonemizer extra are not advertised until installed, so the voice list can change after you add zh, ja, or th. Python 3.10 or later is required.
For short replies, keep the synthesis text plain and speakable. Remove markdown, URLs, long numbers, and punctuation that does not belong in speech. If the LLM returns a table, summarize it before sending it to Piper.
Wire it together: trace one command through the pipeline
A command travels through Wyoming frames: audio enters STT, a transcript leaves it, your intent layer produces reply text, and Piper returns PCM audio, raw pulse-code modulation sound. Wyoming is line-delimited JSON, meaning one object per line, plus PCM over TCP, with no authentication or encryption by design. Run it on a trusted network and never port-forward it.
To measure latency, time two curl calls and subtract. Record timestamps at the VAD endpoint, transcript receipt, and first Piper audio byte. Spot-check beam_size=5 and beam_size=1 on the same clip. Compare synthesis throughput with the documented Raspberry Pi figure of 1.6 seconds of voice per second.
Home Assistant reports that on a Raspberry Pi 4 it takes around 8 seconds to process incoming voice commands, while on an Intel NUC it is done in under a second, according to the fully-local Assist docs.
The verbatim security note is useful: Wyoming has no authentication or encryption, by design. It is meant for a trusted network, and should not be exposed directly to the internet. Mitigation: keep services local, firewall the subnet, and avoid port-forwarding.
Trace example: audio arrives, STT emits a transcript, your script sends it to Ollama or a pattern matcher, the reply goes to Piper, and PCM audio returns to the speaker. Wake score or button gates the microphone, and audio-stop ends input.
The client code opens sockets, converts microphone frames into PCM bytes, respects Wyoming frames, and handles audio-stop. It also decides whether to interrupt a current reply when new speech arrives.
The Wyoming integration docs are useful for Home Assistant service discovery.
faster-whisper vs alternatives: which should you pick?
Choose an engine by your runtime, hardware, and deployment style. faster-whisper reports up to 4 times faster than openai/whisper for the same accuracy while using less memory. whisper.cpp is plain C/C++ with no dependencies and supports Metal/Core ML, AVX, Vulkan, CUDA, ROCm, OpenVINO, and CPU. For Apple Silicon, Core ML can be more than x3 faster than CPU-only execution.
The whisper.cpp documentation also says the ANEForge direct-ANE path is about 2x faster than the Core ML encoder from tiny to medium. Core ML is Apple’s machine learning framework, and ANE means Apple Neural Engine, the on-chip accelerator. It also supports integer quantization, a size reduction technique.
Recommendations: pick faster-whisper for Python pipelines and the CTranslate2 runtime, a model execution library; pick whisper.cpp for C/C++ builds, Apple backends, and embedded targets. Pick tiny-int8 or base for Raspberry Pi, small or medium for NUC-class machines, and large only when accuracy matters more than latency. Pick Wyoming when you want a reusable TCP service for Home Assistant or multiple clients; call a script directly when a local app owns the audio. Pick GPU or Apple Neural Engine (ANE) when available; use CPU when simplicity and memory matter.
Home Assistant’s local STT options include Whisper and Speech-to-Phrase. Speech-to-Phrase is close-ended and under one second on Home Assistant Green or a Raspberry Pi 4, but open-ended items such as shopping lists, naming a timer, and broadcasts are not usable out of the box.
faster-whisper’s CUDA benchmarks show large-v2 GPU int8 beam 5 taking 59 s and using 2926 MB, while small CPU int8 beam 5 takes 1 m 42 s and uses 1477 MB. Batching with batch_size=8 can reduce large-v2 GPU time to 16 s while raising memory to 4500 MB, and small CPU time to 51 s at 3608 MB.
The Python API can load models with WhisperModel("large-v3", device="cuda", compute_type="float16"); other documented compute types include int8_float16 for GPU and int8 for CPU.
The footprints below are from whisper.cpp.
| Model | Disk | Memory | Who should pick it |
|---|---|---|---|
| tiny | 75 MiB | ~273 MB | Raspberry Pi or quick tests |
| base | 142 MiB | ~388 MB | small local devices |
| small | 466 MiB | ~852 MB | NUC-class machines |
| medium | 1.5 GiB | ~2.1 GB | powerful hardware needing better accuracy |
| large | 2.9 GiB | ~3.9 GB | high accuracy, more latency |
| Criterion | faster-whisper | whisper.cpp |
|---|---|---|
| Runtime | Python with CTranslate2 | plain C/C++ with no dependencies |
| Speed claim | up to 4 times faster than openai/whisper | Core ML more than x3 faster than CPU-only on Apple Silicon; ANEForge about 2x faster than Core ML encoder from tiny to medium |
| Backends | CUDA GPU and CPU int8 | Metal/Core ML, AVX, Vulkan, CUDA, ROCm, OpenVINO, CPU |
| Deployment | Hugging Face Hub auto-download, Distil compatible | CLI whisper-cli, integer quantization, offline voice-assistant example, VAD support |
| Platform | GPU path needs NVIDIA CUDA libraries | macOS, iOS, Android, Linux, Windows, WebAssembly, Raspberry Pi, Docker |
The Wyoming layer is a transport. If you call a script directly, you may get lower overhead, but you lose the reusable service boundary. If you use Wyoming, another client can connect to the same STT and TTS services.
What breaks: five known failure modes and their fixes
Known failures usually come from model downloads, endpointing, unsupported languages, network exposure, or missing phonemizers. Keep the first --data-dir writable, because Whisper models download there. Verify that speech is detected before the VAD silence timer starts, or endpointing will not trigger. Install extra phonemizer packages for Chinese, Japanese, and Thai.
- Failure: the STT service cannot download models. Fix: point
--download-dirand the first--data-dirat a writable path, then restart the service. - Failure: endpointing sends nothing after silence. Fix: confirm that speech was detected before the silence timer began, and remember that a command runs for at least one second.
- Failure: the wake word misses non-English speech. Fix: use push-to-talk or VAD-only capture, because openWakeWord is English only.
- Failure: a Wyoming service is reachable from outside your local network. Fix: keep it on a trusted network, bind to local interfaces, firewall the subnet, and never port-forward it, because Wyoming has no authentication or encryption by design.
- Failure: a Piper voice is missing from the advertised list. Fix: install the required phonemizer extra for Chinese, Japanese, or Thai, since voices needing missing extras are not advertised until installed.
When debugging, change a layer at a time. First confirm the STT service responds to audio. Then confirm the intent layer returns text. Then confirm Piper can synthesize that text.
For code quality and tooling notes, see CodeIntel.
The bottom line
Build it if your audio must stay on your machine. Skip it if you need multilingual wake words, guaranteed transcript accuracy, or sub-second answers on weak hardware; the documented latency and language limits in this guide are the honest tradeoff.
| What you are scoring | Score | Why |
|---|---|---|
| Setup effort for a beginner | 6/10 | Four services, two Python environments and three ports: the commands are copy-paste, but process supervision is on you. |
| Privacy | 10/10 | Audio stays on your hardware; nothing in this chain sends it to a hosted API. |
| Cost shape | 9/10 | Audio processed locally incurs no cloud service fee; you pay in hardware, electricity and tuning time. |
| Latency on small hardware | 4/10 | Home Assistant documents around 8 seconds per command on a Raspberry Pi 4. |
| How swappable the parts are | 9/10 | Wyoming is an open standard, so speech-to-text, text-to-speech and intent can be replaced independently. |
Scores are editorial judgments about this documented stack, not measurements — we did not run the build.
For side-by-side views of these engines and their hosted alternatives, the ToolBrain comparisons hub is the place to start. If you already run a local model, our local RAG pipeline build log covers the retrieval half and our Ollama-on-a-mini-PC build log covers the hardware. To try request shapes without installing anything, use the free browser tools; for evaluation work, see our competitive AI writing arena.
If the finished assistant lives in a house rather than a terminal, the Smart Home Field Guide covers the voice-control side of that deployment, and Hermes Tutorials walks through the agent patterns that sit on top of a local model. If you are writing the glue code by hand, CodeIntel is where to look for code-quality tooling, and NiteAgent covers agent development in depth.
Write down the model, voice, prompt and endpointing settings you choose. A reproducible local voice stack is mostly a list of settings plus a small amount of glue code.
Back to all posts