JARVIS: Self-Hosted AI Assistant
Privacy-first voice assistant on a Raspberry Pi
Overview
Problem. Voice assistants require cloud APIs, sacrificing privacy and creating vendor lock-in. Your voice, your notes, and your habits end up on someone else's servers.
Solution. Built a self-hosted assistant running entirely on a Raspberry Pi 4, using local STT/TTS and a private RAG search over personal notes — reachable from a phone PWA or native assist screen.
Impact. It runs 24/7 as systemd services, answers voice queries in under a second, and every core feature works without a cloud dependency.
24/7uptime as systemd services
<1svoice reply latency (warm)
626indexed note chunks (RAG)
0cloud dependencies for core features
Key Features
- Voice I/O — Groq STT with local fallback, Piper TTS
- Local RAG search over a 626-chunk personal note index
- 24/7 systemd services with file-based health logging
- Web Push notifications (VAPID) to the phone PWA
Technical Architecture
┌─────────┐ ┌──────────────┐ ┌─────────┐
│ Phone │────▶│ Pi (Python) │────▶│ Hermes │
│ PWA/TWA │◀────│ + SQLite │◀────│ API │
└─────────┘ │ + RAG │ └─────────┘
│ + TTS/STT │
└──────────────┘
│
▼
┌──────────────┐
│ Signal │
│ Gateway │
└──────────────┘Backend
- Python 3.11
- SQLite
- systemd
- Piper TTS
- faster-whisper
Frontend
- Vanilla JS (ES6+)
- Service Worker (PWA)
- Web Push API
- MediaRecorder API
Infrastructure
- Raspberry Pi 4
- Tailscale HTTPS
- GitHub Pages
Integrations
- Groq (STT fallback)
- Signal (via signal-cli)
- Hermes Agent API
Performance Wins
Bars share a per-metric scale: each "before" bar is full width, "after" is proportional.
Lessons Learned
- The Pi's journal is unreliable — all monitoring moved to file logs after silent service failures.
- Piper claims stdout support but only writes files; the TTS worker uses temp .wav files with cleanup.
- Cache keys must use the spoken text, not display text, or "13:15" and "one fifteen p.m." synthesize twice.
Open Challenges
- STT accuracy degrades in noisy rooms — a local wake-word model could gate listening and cut false triggers.
- RAG recall drops as the note index grows; chunking strategy needs tuning beyond 626 chunks.
- Remote access depends on Tailscale; a self-hosted relay would remove the last external dependency.