Hermes: Your Offline AI Co-Pilot

Hermes is an ambitious project to build a personal AI assistant that never touches the cloud. It runs entirely on a Raspberry Pi 5 (8GB RAM) and uses a quantized LLM (Llama 3 8B Q4_K_M) for reasoning, a local embedding model (all-MiniLM-L6-v2) and ChromaDB for semantic indexing of personal documents, Whisper.cpp for speech-to-text, and piper-tts for text-to-speech. No internet connection is required after initial setup. The assistant listens via USB microphone, processes queries locally, and responds with synthesized speech. It indexes your local files (notes, emails, code, PDFs) and allows you to ask questions, summarize, draft replies, or set reminders. It also integrates with local MQTT for home automation control. The system is designed to be always available, low-latency, and completely private. The entire stack is open-source and fits on a single microSD card.

Existing AI assistants like Alexa, Siri, or Google Assistant rely on cloud servers, which raises privacy concerns, introduces latency, and requires constant internet connectivity. Users must trust tech giants with their personal data and conversations. In remote areas or during outages, these assistants become useless. Hermes solves this by running everything locally on a cheap edge device, ensuring data never leaves your home.

• Run entirely on a $100 Raspberry Pi 5 with 8GB RAM.
• Sub-2 second response time for common queries (e.g., 'What’s on my calendar today?').
• Index up to 10GB of personal documents (notes, emails, code) into a local vector store.
• Support full voice input and output with <500ms STT and TTS latency.
• Integrate with local email (IMAP) and calendar (CalDAV) for personal tasks.
• Consume under 5W average power.
• Work completely offline after initial model downloads.

• Not a cloud-connected service; no remote access or sync.
• Not a general-purpose chatbot; not designed to answer internet trivia or search the web.
• Not multi-user; single user focus with no authentication.
• Not a replacement for heavy cloud AI like GPT-4 for complex reasoning.
• No GUI; voice-only interface with optional SSH for debugging.

Hardware

• Raspberry Pi 5 (8GB RAM)
• USB microphone (e.g., Blue Snowball)
• Powered speaker (3.5mm or USB)
• 64GB microSD card

Software

• OS: Raspberry Pi OS Lite (64-bit)
• LLM: llama.cpp with Llama 3 8B Q4_K_M (or Phi-3 mini 3.8B for speed)
• STT: Whisper.cpp (tiny.en model)
• TTS: piper-tts (low quality)
• Vector DB: ChromaDB (in-memory)
• Embedding: all-MiniLM-L6-v2 via ONNX
• Audio: ALSA, PulseAudio
• Protocols: MQTT (Mosquitto) for home automation
• File watcher: inotify-tools

Hermes has four main pipelines:

1. Audio Input Pipeline: USB mic → ALSA → Whisper.cpp (runs on 4 CPU cores) → transcribed text via Unix socket.

2. Inference Pipeline: Text query → Context retrieval from ChromaDB (embed query, top-k=3 chunks) → Prompt assembly (system + context + query) → llama.cpp inference → response text → send to TTS.

3. Audio Output Pipeline: Response text → piper-tts (via pipe) → ALSA → speaker.

4. Knowledge Pipeline: inotify watches ~/Documents for changes → document loader (txt, md, pdf, email maildir) → chunker (256 tokens, overlap 32) → all-MiniLM-L6-v2 embedding → ChromaDB upsert. Runs on idle CPU.

All components communicate via local sockets or pipes. A Python orchestrator (FastAPI) manages wake word detection (porcupine) and event loop. No internet dependency after initial model downloads.

• Model size vs RAM: The 8B quantized model needs ~5GB VRAM; with OS and other processes, 8GB may be tight. Mitigation: use swap on USB 3.0 SSD or switch to Phi-3 (3.8B).
• Inference speed: Sub-2s may be difficult with 8B model. Mitigation: use 4-bit quant, fewer context tokens, or smaller model.
• STT accuracy: Whisper tiny.en may struggle in noisy environments. Mitigation: fine-tune on user’s voice or use medium model (slower).
• Knowledge indexing: Initial indexing of 10GB may take hours. Mitigation: background indexing with low priority.
• Heat: Continuous heavy load may throttle Pi. Mitigation: add heatsink and fan.

• Should we use a smaller model like Phi-3 3.8B (faster but less capable) or Qwen 2.5 7B (better reasoning)?
• Can we achieve real-time streaming TTS? Piper is fast but not streaming; research LPCNet or VITS-on-device.
• How to handle multi-modal inputs? Would require vision models – likely out of scope.
• Should we add a small OLED screen for status? Might be nice but adds complexity.
• How to handle model updates when new LLMs are released? Pull them manually via USB key?

The project lost momentum when the pace of small model releases accelerated in late 2024. Every month a newer, better quantized model appeared, making the original Llama 3 8B choice feel obsolete. I kept waiting for the 'final' model that never came, and life got busy with work. The hardware is still in a drawer, and the plan remains a good plan.

The name Hermes comes from the messenger god – fitting for a personal communicator. This project is a natural extension of the open-source edge ML movement (like Ollama, LocalAI). If I had more time, I’d build it on a custom ARM SBC with an NPU (e.g., RK3588) for better performance.