Loculus: A Private, On-Device AI Knowledge Base

Imagine an AI that knows everything you've written, read, or saved—but that knowledge never leaves your machine. Loculus aims to be that. It's a desktop app (and eventually mobile) that runs a small, fine-tuned LLM (e.g., Llama-3.2-3B or equivalent) entirely on-device, along with a vector embedding model (e.g., BGE-small) to index local documents. The core idea is to ingest your notes, journals, PDFs, emails (with IMAP access), and bookmarks into a fast, local vector database (like LanceDB). On top of this, Loculus builds a lightweight knowledge graph connecting entities, dates, and topics extracted from your texts. The user can then ask questions, get summaries, generate new ideas, or see connections the AI discovers—all while being absolutely certain that no data is collected, no analytics are sent, and no model weights are uploaded. The project uses open-source models (via llama.cpp with GPU acceleration) and a local-first architecture with end-to-end encryption for any optional sync between your own devices. It's a second brain that's truly yours.

Most AI assistants today require cloud connectivity, meaning your personal data—notes, journal entries, confidential work files—is processed on someone else's server. Even with privacy promises, the architecture inherently trusts a third party. For power users, knowledge workers, and privacy-sensitive individuals, there is no polished tool that offers a local-only AI experience with modern capabilities like semantic search, Q&A, and knowledge graph discovery. Existing solutions either compromise privacy (Notion AI, ChatGPT plug-ins) or require heavy technical setup (building from source). Loculus fills this gap by providing a turnkey, local-first AI assistant that respects user sovereignty.

• Ingest markdown files, PDFs, plaintext, and common note formats (Obsidian vaults, Notion exports) into a local vector store
• Run a 3B-parameter language model locally with real-time Q&A on indexed content
• Build a dynamic knowledge graph from extracted entities and relationships
• Provide a chat interface for exploratory dialogue and a search UI for fast retrieval
• Enable optional encrypted peer-to-peer sync between a user's own devices (no central server)
• Achieve <2 second response time for semantic search and <5 seconds for generation on consumer hardware
• Support offline-first: no internet needed after initial model download

• NOT an online service, SaaS, or API
• NOT collecting any usage data, telemetry, or error logs
• NOT supporting multi-user or sharing features (sync is only between user's devices)
• NOT indexing social media or real-time web content
• NOT verifying the accuracy of information in the user's files
• NOT providing a general-purpose chatbot (focused on user's own data)

• Model: Fine-tuned Llama-3.2-3B or Phi-3-mini via llama.cpp with 4-bit quantization
• Embeddings: BGE-small or all-MiniLM-L6-v2 via ONNX runtime for speed
• Vector Database: LanceDB (embedded, columnar) or HNSWLib for ANN search
• Graph: NetworkX or custom Rust module for in-memory knowledge graph, serialized as local JSON
• Frontend: Tauri (Rust) with a React + TypeScript UI, using Vite
• Backend: Python or Rust core—Python for rapid prototyping, Rust for final performance
• Sync: libp2p for encrypted peer-to-peer (optional, disabled by default)
• NLP Pipeline: spaCy for NER and relation extraction, custom rules for entity linking

Loculus is a monolithic desktop application with three main subsystems:

1. Ingestion & Indexing: Watches configured folders (or imports exports) and processes new files. Each file is chunked (by paragraph or semantic boundaries), embedded, and stored in LanceDB. Concurrently, a GPU-accelerated NER pipeline extracts entities, dates, and relationships, building a local knowledge graph stored as adjacency lists in JSON.
2. Query Engine: The UI sends a query. The engine first embeds the query and retrieves top-k relevant chunks via cosine similarity. These chunks, plus the current graph context (connected entities), are passed as context to the local LLM. The LLM generates a grounded response. The chat history is stored locally in a SQLite database.
3. UI & Sync: A Tauri window provides a search bar, chat panel, and graph viz (using D3.js). The user can edit the knowledge graph manually. Optional sync uses a direct peer-to-peer connection between authorized devices—no cloud intermediary. All data stays in user-specified directories.

• Performance: Running a local LLM and vector DB simultaneously may overwhelm lower-spec machines (8GB RAM). Mitigation: limit to 2B models and aggressive quantization.
• Model Quality: Small local models may hallucinate or miss connections that a cloud model would catch. Mitigation: allow user to plug in a larger model if hardware permits.
• Knowledge Graph Noise: NER and relation extraction are imperfect; the graph could be messy. Mitigation: provide manual editing tools and confidence filters.
• User Adoption: Many users are conditioned to cloud convenience. Mitigation: emphasize absolute privacy and offline capability as a differentiator.

• How to handle very large vaults (100k+ documents) without sacrificing performance? Should we implement tiered storage or summarization?
• What is the best way to fine-tune the LLM on personal data without leaking? Should we offer a local fine-tuning option via LoRA?
• Should we support images (OCR) and audio transcripts? That drastically increases scope.
• What export/import standards to support for interoperability (e.g., Obsidian vault, Roam research, Logseq)?

I spent six months building a prototype in Python that worked on my own vault, but then life got busy with a job change and a move. The Rust rewrite stalled when I realized the complexity of the sync layer. I still think it's a great idea, just didn't have the sustained energy to ship it.

The core insight is that with today's open-source models and efficient vector databases, local AI is viable for personal knowledge work. If someone picks this up, I'd recommend starting with a single-platform (macOS) and a single hardcoded folder watcher, then expanding. Also consider using MLX on Apple Silicon for better performance.