diff --git a/README.md b/README.md index 17f3a26..00d2c9d 100644 --- a/README.md +++ b/README.md @@ -1,42 +1,37 @@ # Knowledge Refinery -A local-first macOS Tahoe application that ingests heterogeneous document corpora, extracts structured knowledge via local LLMs (LM Studio), and provides semantic search with 3D concept visualization. +A local-first macOS application that ingests heterogeneous document corpora, extracts structured knowledge via local LLMs (LM Studio), and provides semantic search with 3D concept visualization. ## Installation ### Prerequisites -- **macOS Tahoe** (26.x) on Apple Silicon +- **macOS 15+** (Sequoia or later) on Apple Silicon - **Xcode** or Xcode Command Line Tools (for Swift 6.2+) -- **Python 3.12+** (system Python or from python.org) +- **Go 1.22+** from [go.dev](https://go.dev/dl/) or `brew install go` - **LM Studio** from [lmstudio.ai](https://lmstudio.ai) ### One-Line Install ```bash -git clone && cd LongLocalTimeHorizonInfoRetrieval && bash scripts/install.sh +git clone https://github.com/saymrwulf/KnowledgeRefinery.git && cd KnowledgeRefinery && bash scripts/install.sh ``` This will: -1. Check all prerequisites -2. Create a Python virtual environment and install dependencies -3. Build the SwiftUI application -4. Create a proper `.app` bundle -5. Install to `/Applications` +1. Check all prerequisites (Go, Swift, Xcode) +2. Run all Go daemon tests (89 tests) +3. Build the Go daemon + SwiftUI app into a `.app` bundle +4. Install to `/Applications` +5. Create the data directory at `~/.knowledge-refinery/` ### Manual Build ```bash -# Set up daemon -cd daemon -python3 -m venv .venv -.venv/bin/pip install -e ".[dev]" - -# Build app bundle -cd .. +# Build Go daemon + .app bundle make build -# Or just run in development mode -make app-run +# Or run in development mode +make daemon # Build Go daemon binary +make app-run # Run SwiftUI app (builds daemon first) ``` ### LM Studio Setup @@ -44,7 +39,7 @@ make app-run Before launching Knowledge Refinery: 1. Open LM Studio 2. Load models: - - **Chat**: `gemma-3-4b` (or any chat model) + - **Chat**: `qwen3-4b-2507` (or any small chat model) - **Embeddings**: `nomic-embed-text-v1.5` (768-dim) 3. Start the local server on port **1234** @@ -52,26 +47,25 @@ Before launching Knowledge Refinery: 1. Launch **Knowledge Refinery** from Applications or Spotlight 2. The dashboard shows LM Studio status (green = connected) -3. Click **New Workspace** — name it, add data lake folders -4. Click **Start All** to launch all workspace daemons and auto-start ingestion -5. Watch live pipeline progress: stage tracker, animated counters, activity log -6. Search, explore the concept universe, browse clusters +3. Click **New Workspace** — name it, add source folders +4. Open the workspace and click **Process Documents** to run the pipeline +5. Watch live progress: stage tracker, animated counters, activity log +6. Search, explore the concept universe, browse themes ## Architecture -- **SwiftUI Master Control App** — Multi-workspace dashboard, LM Studio monitoring, daemon lifecycle, live pipeline visibility -- **Python Daemon** (FastAPI) — Per-workspace instances with independent ports and data directories (`~/.knowledge-refinery/workspaces//`) -- **Live Pipeline Progress** — 1.5s fast polling during ingestion, enriched `/ingest/status` with per-stage progress, counters, and activity log -- **LanceDB** — Embedded vector store for semantic search -- **SQLite** — Metadata, graph store, pipeline state -- **LM Studio** — Local LLM inference (embeddings + chat) -- **WebGPU** — 3D concept universe visualization with auto-refresh during ingestion +- **SwiftUI App** — Multi-workspace dashboard, LM Studio monitoring, daemon lifecycle, live pipeline visibility +- **Go Daemon** (chi router) — 11MB single binary, zero dependencies, per-workspace instances on independent ports +- **SQLite** — All storage: metadata, vectors (as BLOBs with brute-force cosine search), graph, pipeline state +- **6-Stage Pipeline** — scan, extract, chunk, embed, annotate, conceptualize +- **LM Studio** — Local LLM inference at `127.0.0.1:1234` (embeddings + chat) +- **WebGPU / Canvas2D** — 3D concept universe visualization with interactive fallback ## Project Structure ``` apps/macos/KnowledgeRefinery/ SwiftUI macOS application -daemon/ Python backend daemon +daemon-go/ Go daemon (chi, SQLite, tiktoken) shared/ Prompt templates, schemas docs/ Architecture and operational docs scripts/ Build and install scripts @@ -83,19 +77,16 @@ dist/ Built .app bundle (after make build) ```bash make help # Show all commands -make test # Run daemon tests + Swift build check +make test # Run 89 Go tests + Swift build check make app-run # Run app via swift run (dev mode) -make daemon-run # Run daemon directly +make daemon-run # Run daemon standalone make clean # Remove build artifacts ``` ## Milestones -- **M1**: Core ingestion + search + evidence -- **M2**: LLM structured annotation -- **M3**: Concept clustering + labeling -- **M4**: WebGPU 3D Universe visualization -- **M5**: Semantic zoom + lenses -- **M6**: Extended format support (EPUB, archives, DICOM) +- **M1-M6**: Core pipeline, search, annotation, clustering, WebGPU, extended formats - **M7**: Master Control App (multi-workspace, LM Studio monitoring, daemon lifecycle) -- **M8**: Live Pipeline Visibility (real-time progress panel, activity log, universe auto-refresh) +- **M8**: Live Pipeline Visibility (real-time progress, activity log, universe auto-refresh) +- **M9-M10**: UX language overhaul, macOS sizing improvements +- **M11**: Go daemon rewrite (Python replaced with single Go binary, 89 tests) diff --git a/docs/architecture.md b/docs/architecture.md index 684ac67..cd38fbe 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -13,29 +13,29 @@ Knowledge Refinery is a local-first macOS application that ingests heterogeneous │ │ Search │ Evidence │ │ │ │ View │ Panel (QL) │ │ │ ├────────┼────────────┤ │ -│ │Pipeline│ Volume │ │ -│ │Progress│ Manager │ │ +│ │Pipeline│ Source │ │ +│ │Progress│ Folders │ │ │ │ Panel │ │ │ │ └────────┴────────────┘ │ -│ │ HTTP (localhost) │ -│ │ ┌──────────────────┐ │ -│ │ │ 1.5s poll loop │ │ -│ │ │ /ingest/status │◀─┐ │ -│ │ └──────────────────┘ │ │ -│ │ auto-stop on done │ │ -│ │ │ │ -│ │ ┌──────────────────┐ │ │ -│ │ │ 5s universe │ │ │ -│ │ │ auto-refresh │──┘ │ -│ │ └──────────────────┘ │ +│ │ HTTP (localhost) │ +│ │ ┌──────────────────┐ │ +│ │ │ 1.5s poll loop │ │ +│ │ │ /ingest/status │◀─┐ │ +│ │ └──────────────────┘ │ │ +│ │ auto-stop on done │ │ +│ │ │ │ +│ │ ┌──────────────────┐ │ │ +│ │ │ 5s universe │ │ │ +│ │ │ auto-refresh │──┘ │ +│ │ └──────────────────┘ │ └───────┼──────────────────────────┘ ▼ ┌──────────────────────────────────┐ -│ Refinery Daemon (Python) │ +│ Go Daemon (11MB binary) │ │ Per-workspace on independent │ │ port + data dir │ │ ┌──────────────────────┐ │ -│ │ FastAPI Server │ │ +│ │ chi Router + CORS │ │ │ └──────────┬───────────┘ │ │ ▼ │ │ ┌──────────────────────┐ │ @@ -54,10 +54,11 @@ Knowledge Refinery is a local-first macOS application that ingests heterogeneous │ │ (200-entry buf) │ │ │ └──────────────────────┘ │ │ │ -│ ┌─────────┬───────────┐ │ -│ │ SQLite │ LanceDB │ │ -│ │ (meta) │ (vectors) │ │ -│ └─────────┴───────────┘ │ +│ ┌──────────────────────┐ │ +│ │ SQLite (WAL mode) │ │ +│ │ metadata + vectors │ │ +│ │ + graph + state │ │ +│ └──────────────────────┘ │ └───────┼──────────────────────────┘ ▼ ┌──────────────────────────────────┐ @@ -78,16 +79,16 @@ Knowledge Refinery is a local-first macOS application that ingests heterogeneous ## Data Flow -Files → FileAsset → ContentAtom → Chunk → Embedding (LanceDB) + Annotation (SQLite) - ↓ - ConceptNode + GraphEdge +Files → FileAsset → ContentAtom → Chunk → Vector (SQLite BLOB) + Annotation + ↓ + ConceptNode + GraphEdge -## Live Progress Data Flow (M8) +## Live Progress Data Flow During pipeline execution, the daemon maintains in-memory state that the app polls: ``` -Pipeline Orchestrator +Pipeline Orchestrator (goroutine) │ ├──▶ live progress dict (per-stage status: pending/running/done) │ stage_name, progress_pct, item_count @@ -101,7 +102,7 @@ SwiftUI App polling loop (1.5s interval): GET /ingest/status ──▶ stages, counters, activity_log │ ├── Pipeline Progress Panel: checkmarks + progress bars per stage - ├── Animated counters: chunks, vectors, annotations, concepts, edges + ├── Animated counters: passages, indexed, insights, themes, links ├── Interaction indicators: App↔Daemon, Daemon↔LM Studio ├── Auto-scrolling activity log └── Auto-stop polling when pipeline status = idle/done @@ -112,10 +113,13 @@ Universe auto-refresh (5s timer during ingestion): ## Key Design Decisions -- **LanceDB** over Qdrant: Embedded, no separate server, local-first -- **SQLite** for metadata/graph: Simple, reliable, WAL mode for concurrency +- **Go single binary** over Python: Zero dependencies, instant startup, 11MB, no venv/pip issues +- **SQLite for everything**: Metadata, vectors (as BLOBs with brute-force cosine search), graph — one file, WAL mode +- **chi router**: Lightweight HTTP routing with path params, CORS middleware +- **modernc.org/sqlite**: Pure Go SQLite driver, no CGo, true single binary +- **tiktoken-go**: Accurate token counting matching OpenAI tokenizer - **Deterministic chunk IDs**: SHA-256(asset_id + anchor + normalized_text_hash) - **Versioned annotations**: Never overwrite, mark active by pipeline version - **Evidence-native**: Every derived insight links back to source file + location -- **Fast polling over WebSocket**: 1.5s HTTP polls are simpler and sufficient for pipeline status; avoids connection lifecycle complexity -- **Ring buffer for activity log**: Fixed 200-entry buffer prevents memory growth during long pipeline runs +- **Fast polling over WebSocket**: 1.5s HTTP polls are simpler and sufficient for pipeline status +- **Ring buffer for activity log**: Fixed 200-entry buffer prevents memory growth during long runs diff --git a/docs/data-model.md b/docs/data-model.md index b70bae0..3f789bd 100644 --- a/docs/data-model.md +++ b/docs/data-model.md @@ -14,7 +14,8 @@ ContentAtom │ ▼ (split into) │ Chunk ──────────────────────┤ │ │ - ├──▶ Vector (LanceDB) │ + ├──▶ Vector (SQLite) │ + │ 768-dim BLOB │ │ │ ├──▶ Annotation │ │ (versioned) │ @@ -26,7 +27,7 @@ Chunk ──────────────────────┤ (hierarchical) ``` -## Tables +## Tables (SQLite) ### file_assets Tracks every file in watched volumes. Status progresses through: @@ -38,7 +39,21 @@ Each atom has an evidence_anchor linking to exact source location. ### chunks Deterministic text segments (500-800 tokens). IDs are stable across re-processing. -Linked to LanceDB vectors via embedding_id. +Linked to vectors in `chunk_vectors` table via chunk ID. + +### chunk_vectors +Embedding vectors stored as binary BLOBs (768 x float32 = 3072 bytes per vector). +Loaded into memory at startup for brute-force cosine similarity search. + +| Field | Type | Description | +|-------|------|-------------| +| id | TEXT PRIMARY KEY | Matches chunks.id | +| vector | BLOB | 768-dim float32 embedding | +| text | TEXT | Chunk text | +| asset_id | TEXT | Source file | +| asset_path | TEXT | File path | +| evidence_anchor | TEXT | JSON anchor | +| pipeline_version | TEXT | Version tag | ### annotations LLM-generated structured metadata per chunk. **Never overwritten** - new annotations @@ -56,7 +71,7 @@ Each edge stores evidence references back to source chunks. ### pipeline_jobs Crash recovery: tracks job state so processing resumes after restart. -## Live Progress State (M8, In-Memory) +## Live Progress State (In-Memory) During pipeline execution, the daemon maintains ephemeral in-memory structures that are not persisted to SQLite: @@ -74,19 +89,10 @@ Per-stage status object returned in the `live` field of `/ingest/status`: } ``` -Each stage transitions through `pending` -> `running` -> `done`. - ### Activity Log Ring Buffer -A fixed-size circular buffer (200 entries) that records pipeline events. The API returns the most recent 50 entries. Each entry contains a timestamp and message string: - -```json -{"timestamp": "2026-02-12T10:30:03Z", "message": "Found 47 files, 12 new"} -``` - -The ring buffer prevents unbounded memory growth during long pipeline runs. It is reset at the start of each new pipeline execution. +A fixed-size circular buffer (200 entries) that records pipeline events. The API returns the most recent 50 entries. ### Enriched Status Counters -The `/ingest/status` response includes running totals updated as each stage completes: | Counter | Description | |---------|-------------| @@ -111,17 +117,3 @@ Every derived artifact links back to source via JSON evidence anchors: "line_end": 58 } ``` - -## Vector Schema (LanceDB) - -| Field | Type | Description | -|-------|------|-------------| -| id | string | Matches chunks.id | -| vector | float32[] | Embedding vector | -| text | string | Chunk text | -| asset_id | string | Source file | -| asset_path | string | File path | -| evidence_anchor | string | JSON anchor | -| topics | string | Comma-separated topics | -| atom_type | string | text/image/etc | -| pipeline_version | string | Version tag | diff --git a/docs/operational-notes.md b/docs/operational-notes.md index ae50140..12075e7 100644 --- a/docs/operational-notes.md +++ b/docs/operational-notes.md @@ -7,11 +7,9 @@ Each workspace has its own data directory under `~/.knowledge-refinery/workspace | Item | Path | |------|------| | Workspace root | `~/.knowledge-refinery/workspaces//` | -| SQLite DB | `~/.knowledge-refinery/workspaces//refinery.db` | -| Vector DB | `~/.knowledge-refinery/workspaces//vectors/` | -| Thumbnails | `~/.knowledge-refinery/workspaces//thumbnails/` | -| Temp files | `~/.knowledge-refinery/workspaces//tmp/` | +| SQLite DB (metadata + vectors + graph) | `~/.knowledge-refinery/workspaces//refinery.db` | | PID file | `~/.knowledge-refinery/workspaces//daemon.pid` | +| Workspace config | `~/.knowledge-refinery/workspaces.json` | ## Resetting @@ -22,12 +20,12 @@ rm -rf ~/.knowledge-refinery ## Monitoring -The daemon logs to stdout. Key log patterns: -- `Stage N: ...` - Pipeline stage progress -- `Embedded batch N: X chunks` - Embedding progress -- `ERROR` - Errors during processing +The Go daemon logs to stdout with structured messages. Key log patterns: +- `[pipeline] Stage: ...` - Pipeline stage progress +- `[embedder] Embedded batch: X chunks` - Embedding progress +- `[error]` - Errors during processing -### Live Pipeline Monitoring (M8) +### Live Pipeline Monitoring During pipeline execution, real-time progress is available via the enriched `/ingest/status` endpoint. The daemon maintains: @@ -35,7 +33,7 @@ During pipeline execution, real-time progress is available via the enriched `/in - **Counters**: chunk_count, annotation_count, concept_count, edge_count - **Activity log**: 200-entry ring buffer; the last 50 events are returned via the API -The SwiftUI app polls at 1.5-second intervals and renders a full Pipeline Progress Panel with stage checkmarks, animated counters, and an auto-scrolling activity log. Polling auto-stops when the pipeline reaches idle/done state. The 3D universe auto-refreshes every 5 seconds during ingestion using `mergeUniverse()` for incremental node injection. +The SwiftUI app polls at 1.5-second intervals and renders a Pipeline Progress Panel with stage checkmarks, animated counters, and an auto-scrolling activity log. Polling auto-stops when the pipeline reaches idle/done state. The universe visualization auto-refreshes every 5 seconds during processing using `mergeUniverse()` for incremental node injection. ## API Endpoints @@ -48,19 +46,21 @@ The SwiftUI app polls at 1.5-second intervals and renders a full Pipeline Progre | POST | /ingest/start | Start pipeline | | GET | /ingest/status | Pipeline status | | POST | /search | Vector search | -| GET | /search/quick?q=... | Quick search | | GET | /evidence/{asset_id} | Get asset info | | GET | /evidence/chunk/{chunk_id} | Get chunk details | | GET | /evidence/assets/all | List all assets | -| GET | /universe/snapshot | Universe snapshot | +| GET | /universe/snapshot?lod=macro | Universe snapshot | | POST | /universe/focus | Focus on node | | POST | /concepts/refine | Refine concept | | GET | /concepts/list | List concepts | +| GET | /concepts/{id} | Concept detail | ## Performance Considerations - Large files (>500MB) are skipped by default -- Embedding batch size defaults to 32 (adjustable) +- Embedding batch size defaults to 32 - SQLite uses WAL mode for concurrent reads -- Pipeline runs in a background thread +- Pipeline runs in a background goroutine - Incremental processing skips unchanged files (content hash comparison) +- Vector search: brute-force cosine similarity, all vectors loaded in memory (~150MB for 50K vectors) +- Go daemon starts in <100ms, uses ~30MB base memory diff --git a/docs/running.md b/docs/running.md index 24cd4fa..62a9196 100644 --- a/docs/running.md +++ b/docs/running.md @@ -2,36 +2,40 @@ ## Prerequisites -- macOS Tahoe (26.x) -- Python 3.12+ -- Xcode 26+ -- LM Studio running locally with at least one model loaded +- macOS 15+ (Sequoia or later) +- Go 1.22+ +- Xcode or Xcode Command Line Tools (Swift 6.2+) +- LM Studio running locally with models loaded ## 1. Start LM Studio 1. Open LM Studio -2. Load an embedding model (e.g., `nomic-embed-text-v1.5` or `text-embedding-3-small`) -3. Load a chat model (e.g., `llama-3.2-3b-instruct` or similar) +2. Load an embedding model: `nomic-embed-text-v1.5` (768-dim) +3. Load a chat model: `qwen3-4b-2507` (or similar small model) 4. Start the local server (default port 1234) 5. Verify: `curl http://127.0.0.1:1234/v1/models` ## 2. Start the Daemon +### Via the App (Recommended) + +The SwiftUI app auto-starts daemons for all workspaces on launch. Each workspace runs an independent Go daemon with its own port and data directory. + +### Manually + ```bash -cd daemon -source .venv/bin/activate -python -m knowledge_refinery.main +cd daemon-go +go build -o knowledge-refinery-daemon . +./knowledge-refinery-daemon ``` The daemon will: - Create data directory at `~/.knowledge-refinery/workspaces//` -- Initialize SQLite database +- Initialize SQLite database (metadata + vectors) - Connect to LM Studio - Write a PID file to `{data_dir}/daemon.pid` for process detection - Listen on its assigned port (default `http://127.0.0.1:8742`) -> **Tip**: Use **Start All** in the app toolbar to launch all workspace daemons at once. Each workspace runs an independent daemon with its own port and data directory. After connection, ingestion auto-starts. - ### Environment Variables | Variable | Default | Description | @@ -48,14 +52,15 @@ curl http://127.0.0.1:8742/health ## 3. Run the macOS App +### From .app Bundle ```bash -cd apps/macos/KnowledgeRefinery -swift run +make build +open "dist/Knowledge Refinery.app" ``` -Or open in Xcode: +### Development Mode ```bash -open Package.swift +make app-run ``` The app will: @@ -64,64 +69,35 @@ The app will: - Auto-restart crashed daemons (up to 3 times) - Show connection status in the toolbar -## 4. Ingest Documents +## 4. Process Documents -1. In the app, go to **Volumes** tab +1. In the app, go to **Source Folders** tab 2. Click **Add Folder** and select a directory -3. Go to **Ingest** tab and click **Start Ingestion**, or use **Start All** from the dashboard -4. Watch live pipeline progress in the **Pipeline Progress Panel**: - - **Stage tracker**: Each of the 6 stages (Scan, Extract, Chunk, Embed, Annotate, Conceptualize) shows a checkmark when complete or an animated progress bar when running - - **Animated counters**: Live tallies for chunks, vectors, annotations, concepts, and edges +3. Click **Process Documents** to run the pipeline +4. Watch live pipeline progress in the **Processing Steps** panel: + - **Stage tracker**: Each of the 6 stages shows a checkmark when complete or a progress bar when running + - **Animated counters**: Live tallies for passages, indexed, insights, themes, and links - **Interaction indicators**: Visual status of App-to-Daemon and Daemon-to-LM Studio connections - **Activity log**: Auto-scrolling log of the last 50 pipeline events -5. The dashboard card shows a compact spinner with the current stage name and chunk count -6. The 3D universe auto-refreshes every 5 seconds during ingestion, using incremental node injection - -The app polls `/ingest/status` every 1.5 seconds during pipeline execution and automatically stops polling when the pipeline completes. +5. The universe auto-refreshes every 5 seconds during processing ### Via API ```bash -# Add a volume +# Add a source folder curl -X POST http://127.0.0.1:8742/volumes/add \ -H "Content-Type: application/json" \ -d '{"path": "/path/to/documents"}' -# Start ingestion +# Start processing curl -X POST http://127.0.0.1:8742/ingest/start \ -H "Content-Type: application/json" \ -d '{}' -# Check status (enriched response with live progress) +# Check status curl http://127.0.0.1:8742/ingest/status ``` -The enriched `/ingest/status` response includes: - -```json -{ - "status": "running", - "stage": "embed", - "chunk_count": 142, - "annotation_count": 87, - "concept_count": 12, - "edge_count": 45, - "live": { - "scan": {"status": "done", "progress_pct": 100}, - "extract": {"status": "done", "progress_pct": 100}, - "chunk": {"status": "done", "progress_pct": 100}, - "embed": {"status": "running", "progress_pct": 64}, - "annotate": {"status": "pending", "progress_pct": 0}, - "conceptualize": {"status": "pending", "progress_pct": 0} - }, - "activity_log": [ - {"timestamp": "2026-02-12T10:30:01Z", "message": "Scanning 3 volumes..."}, - {"timestamp": "2026-02-12T10:30:03Z", "message": "Found 47 files, 12 new"}, - "..." - ] -} -``` - ## 5. Search Use the **Search** tab in the app, or: @@ -134,7 +110,8 @@ curl -X POST http://127.0.0.1:8742/search \ ## Troubleshooting -- **Daemon won't start**: Check that port 8742 is free +- **Daemon won't start**: Check that port 8742 is free (`lsof -i :8742`) - **LM Studio unavailable**: Ensure LM Studio server is running on port 1234 - **No embeddings**: Verify an embedding model is loaded in LM Studio - **App can't connect**: Check daemon is running on the expected port +- **Build fails**: Ensure Go 1.22+ and Swift 6.2+ are installed diff --git a/presentation.html b/presentation.html index f190cd7..5ede55f 100644 --- a/presentation.html +++ b/presentation.html @@ -687,7 +687,7 @@ h3 {
0
Cloud Calls
- macOS Tahoe · SwiftUI · Python 3.14 · FastAPI · LM Studio · LanceDB · WebGPU · Multi-Workspace + macOS · SwiftUI · Go · chi · LM Studio · SQLite · WebGPU · Multi-Workspace
@@ -787,13 +787,13 @@ h3 { 127.0.0.1:8742
-

⚙️ Python Daemon

+

⚙️ Go Daemon

- FastAPI server with CORS
+ chi router with CORS
6-Stage Pipeline orchestrator
- SQLite metadata + graph (WAL mode)
- LanceDB embedded vector store
- 7 Extractors + sandbox worker + SQLite metadata + vectors + graph (WAL)
+ 11MB single binary, zero deps
+ 7 Extractors + fallback chain
@@ -805,7 +805,7 @@ h3 {

🤖 LM Studio

- gemma-3-4b — Chat / Annotation
+ qwen3-4b-2507 — Chat / Annotation
nomic-embed-text-v1.5 — Embeddings
Running on 127.0.0.1:1234
@@ -844,7 +844,7 @@ h3 {
4
Embed - 768-dim vectors via nomic-embed-text stored in LanceDB + 768-dim vectors via nomic-embed-text stored in SQLite
→
@@ -877,7 +877,7 @@ h3 {
📑
PDF
-
.pdf · Per-page text · PyMuPDF
+
.pdf · Per-page text · pdftotext
Priority 20
@@ -895,7 +895,7 @@ h3 {
🩻
DICOM
-
.dcm · Medical metadata · Optional pydicom
+
.dcm · Medical metadata · Binary header parsing
Priority 15
@@ -1076,7 +1076,7 @@ ContentAtom -- text/image/table/metadata ↓ (split into) Chunk -- deterministic ID, 500-800 tokens ↓ - ├──> Vector (LanceDB, 768-dim) + ├──> Vector (SQLite BLOB, 768-dim) ├──> Annotation (versioned, immutable) └──> GraphEdge ↓ @@ -1096,8 +1096,7 @@ Chunk -- deterministic ID, 500-800 tokens }

Storage

- - +
SQLiteMetadata, graph, jobs (WAL mode)
LanceDBVectors + payload (embedded)
SQLiteMetadata, vectors (BLOB), graph, jobs (WAL mode)
Disk~/.knowledge-refinery/
@@ -1165,7 +1164,7 @@ Chunk -- deterministic ID, 500-800 tokens
📦

Multi-Workspace

-

Each workspace gets its own data directory, daemon port (8742, 8743, ...), SQLite database, and LanceDB vector store. Independent lifecycle management.

+

Each workspace gets its own data directory, daemon port (8742, 8743, ...), and SQLite database (metadata + vectors + graph). Independent lifecycle management.

🗺
@@ -1213,18 +1212,18 @@ Chunk -- deterministic ID, 500-800 tokens - +
macOSTahoe (26.x) on Apple Silicon
Xcode26.x or Command Line Tools
Python3.12+ (system or python.org)
Go1.22+ (from go.dev or Homebrew)
LM StudioFrom lmstudio.ai (free)

What the Installer Does

  1. Checks prerequisites -

    Validates macOS version, architecture, Xcode tools, Swift, and Python version.

    +

    Validates macOS version, architecture, Xcode tools, Swift, and Go version.

  2. - Creates Python venv -

    Isolated .venv with all dependencies: FastAPI, LanceDB, PyMuPDF, tiktoken, NumPy.

    + Builds Go daemon +

    Compiles the Go daemon into a single 17MB binary with zero runtime dependencies.

  3. Builds the .app bundle @@ -1260,10 +1259,7 @@ Knowledge Refinery.app/ KnowledgeRefinery # launcher KnowledgeRefinery-bin # Swift binary Resources/ - daemon/ # Python source - knowledge_refinery/ - pyproject.toml - shared/ + knowledge-refinery-daemon # Go binary (17MB) WebGPU/ # 3D renderer universe.html/js/wgsl
@@ -1350,7 +1346,7 @@ Knowledge Refinery.app/
↓
3 - LanceDB cosine similarity search + Brute-force cosine similarity search (in-memory vectors)
↓
@@ -1449,20 +1445,20 @@ Knowledge Refinery.app/
-

Backend (Python Daemon)

+

Backend (Go Daemon)

- - - - - - - - + + + + + + + +
RuntimePython 3.14.2 (venv isolated)
Web ServerFastAPI + Uvicorn
Vector StoreLanceDB (embedded, no server)
MetadataSQLite 3 (WAL mode)
LLM ClientOpenAI SDK → LM Studio
PDFPyMuPDF (fitz)
Tokenizertiktoken (cl100k_base)
MathNumPy (clustering, kNN)
LanguageGo 1.22+ (single 17MB binary)
HTTP Routerchi/v5 + CORS middleware
StorageSQLite (metadata + vectors + graph, WAL mode)
SQLite Drivermodernc.org/sqlite (pure Go, no CGo)
LLM Clientnet/http → LM Studio
PDFpdftotext (poppler-utils)
Tokenizertiktoken-go (cl100k_base)
MathPure Go (k-means++, cosine sim)

LM Studio Models

- +
Chatgoogle/gemma-3-4b
Chatqwen3-4b-2507
Embeddingsnomic-embed-text-v1.5 (768-dim)
@@ -1481,15 +1477,15 @@ Knowledge Refinery.app/ Clone, install, launch — from raw documents to searchable knowledge in minutes.

-
30
Tests Passing
+
97
Tests Passing
0
Compiler Warnings
-
~55
Source Files
+
~90
Source Files
100%
Local
-
📁 daemon/knowledge_refinery/
+
📁 daemon-go/
📱 apps/macos/KnowledgeRefinery/
-
🧪 daemon/tests/
+
🧪 daemon-go/internal/*/
🛠 scripts/install.sh
@@ -1497,7 +1493,7 @@ Knowledge Refinery.app/ git clone <repo> && cd LongLocalTimeHorizonInfoRetrieval && make install
- Built with SwiftUI, Python, FastAPI, LanceDB, WebGPU, and local LLMs + Built with SwiftUI, Go, chi, SQLite, WebGPU, and local LLMs