Knowledge Graph · Field notes

Running Local AI on 16 GB MacBook — the real stats

A 27B model, an embedding model, a transcription model, a Postgres database, a RAG server, an API, a worker, and a SvelteKit UI — all in 16 GB. Live stats from the machine, not benchmarks from the model card.

GitHub · paulvisciano/knowledge-graph

This is the technical companion to the Knowledge Graph project page. The project page tells you what it is. This tells you what it costs to run — in memory, in swap, in tokens per second, and in months of tuning. Every number here is live, measured from the actual running system, not from a benchmark script.

The machine: a 14-inch MacBook Pro with an Apple M2 Pro — 10-core CPU (6 performance, 4 efficiency), 16-core GPU, and 16 GB of unified memory. That means the CPU and GPU share one pool. There is no separate VRAM. Every model that runs on the GPU eats from the same 16 GB the OS, the browser, and the editor are using.

The model stack

Every model in the system is a file on disk. No API keys, no remote endpoints, no silent uploads. That constraint is not a limitation to work around; it is the design constraint that forced the architecture.

Bonsai-27B is a 27B-parameter model compressed with PrismML's 1-bit quantization (Q1_0_g128) — packed weights that are a fraction of the original size with enough reasoning quality for tool use and long context. The catch: the 1-bit Metal kernels do not exist in stock llama.cpp. The project builds the PrismML-Eng fork from source to get them. The Homebrew bottle will refuse to load the GGUF.

The same model serves both roles — chat and vision. Bonsai-27B is built on Qwen3.6-27B, which has a vision tower. In llama.cpp the vision encoder weights ship as a separate file: the base GGUF holds only the language model, and the mmproj file is the projection that maps image pixels into the model's embedding space. The server runs with -np 2 — two parallel processing slots — so the model can answer two tasks concurrently: slot 0 for interactive chat, slot 1 for the vision pipeline. You can send a message while an image is being processed, and neither blocks the other. The project's config caps VLM concurrency at 1 so the vision pipeline never eats both slots — one slot is always reserved for chat. Both slots share the same 16GB pool.

Embeddings are handled by BGE-M3, a 1024-dimensional multilingual model running on its own llama-server instance. Voice transcription is Whisper, running on whisper-server (from whisper.cpp) on the Metal GPU. Both are separate processes from the LLM so they can load and serve without competing for the same slot.

Architecture

The stack splits cleanly along a boundary: model servers run on the host (direct Metal access), and the service layer runs in Docker (isolated, composed, restartable). They talk to each other over host.docker.internal — the Docker bridge to the laptop's own loopback.

LightRAG is the memory engine. It does the entity-and-relation extraction that turns raw text into a queryable graph, and it runs the dual-level retrieval (local entity match + higher-level themes) that makes "what did I do last week" return something more than a keyword search. Its KV store, vector store, and document status all live in Postgres with pgvector and an HNSW index. The graph itself stays in NetworkX — in-process, no separate graph database — because the personal archive is small enough that an in-memory graph is faster than a network round trip.

The KG API is a FastAPI app that owns everything that is not LightRAG: image ingestion (EXIF extraction, optional face recognition, VLM image descriptions), chat (streaming SSE to the UI, job queue to the LLM worker), and the settings surface. It talks to LightRAG over HTTP inside the Docker network and to the host llama-servers through the Docker gateway. The API process serves HTTP only — a separate worker process owns all image processing so that a long VLM batch never blocks the website's event loop.

The worker is the quiet engine of the pipeline. It polls a Postgres jobs table with SELECT FOR UPDATE SKIP LOCKED, claims pending rows, and drives each one through a two-phase pipeline. No Redis, no Celery — Postgres is the queue, and pg_notify is the event bus that the API's SSE endpoints listen on. The architecture is deliberately boring: one database, one queue, one event stream.

The image pipeline

When a photo lands in the system, it does not go straight to the model. It goes through a two-phase pipeline that separates the cheap work from the expensive work, so a batch of 50 photos does not monopolize the GPU for an hour.

Phase 1 is CPU-bound: EXIF extraction with ExifRead and reverse geocoding with reverse_geocoder — an offline dataset, no network. The results — camera model, date, location — become entities and relations in the graph immediately via LightRAG's entity and relation creation API. This phase can run at high concurrency (up to 10 in parallel) because it never touches the LLM.

The codebase also includes face recognition — DeepFace with MTCNN detection and ArcFace 512-dimensional embeddings — but it is currently disabled. The setting face_detection_enabled is false in the database, and the API's _resolve_skip_faces function returns True when that setting is off, so every job skips the face phase. The known_faces/ directory has no reference photos in it. DeepFace and its TensorFlow dependencies are still in requirements.txt, the code is still in processors/face_recognizer.py, and the Docker volumes for face_crops and deepface_weights still exist — but none of it runs. It was turned off because DeepFace in a container competes for the same memory budget as the rest of the stack, and on a 16 GB machine that trade was not worth it. The code is there for when the machine changes.

Phase 2 is GPU-bound: the VLM. The photo is resized to max 768px (the mmproj caps image tokens at 280, so anything larger is wasted compute) and sent to Bonsai-27B with the vision projection. The model returns a natural-language description of the scene — what is in the photo, the setting, the mood. That description is ingested by LightRAG as text, which extracts its own entities and relations and adds them to the graph. A photo of a dinner in Lisbon becomes nodes for the food, the restaurant, the city — connected to the date, the camera, and each other.

The split exists because of the 16GB constraint. Phase 1 runs fast and cheap; phase 2 is the bottleneck. Running them separately means a batch upload can complete phase 1 in seconds and then trickle phase 2 through the single VLM slot overnight, while the other LLM slot stays free for chat.

The chat and query path

When you speak or type a question, the path is different from ingestion. The Nexus UI streams the message to the KG API, which saves it and creates an LLM job. The LLM worker picks up the job, calls the Bonsai-27B server with the chat context, and streams the response back via SSE. If the question needs memory — "what did I do in Lisbon?" — the worker calls LightRAG's query API first. LightRAG embeds the query with BGE-M3, retrieves the top-k entity subgraphs from the Postgres vector store, assembles a context block, and hands it to the LLM as part of the prompt. The model answers from your own data, not from its training set.

If the question has an audio component — a voice memo — Whisper transcribes it on the host GPU before the text reaches the LLM. The transcription becomes another node on the canvas, timestamped to the moment it was spoken, sitting in the same time layer as the photos and notes from that day.

The MCP server exposes the graph to external tools. Any agent that speaks the Model Context Protocol can query the graph, save to it, or search it — the same API the internal UI uses, but over a standard interface. The utility agent on your machine can ask the personal agent's memory without the personal agent's data ever leaving the laptop.

The graph right now

These are not benchmarks from a demo. They are the live numbers from the Postgres database running behind the Knowledge Graph, queried while the system was serving the conversation that produced this writing.

The LLM cache is the quiet win. 862 cached completions mean the model does not re-run extraction on text it has already seen — a big deal when each run costs GPU time on a shared 16GB pool. The entity-to-relation ratio (270 entities, 256 relations) is low by enterprise graph standards but dense for a personal archive: nearly every entity is connected to at least one other, because the source material is one person's life, not a web crawl. Database size: 243 MB — a personal archive, not a data lake.

Throughput

Measured live with a direct /v1/chat/completions call to the Bonsai-27B server on port 8080:

Fifteen tokens per second is not fast by cloud standards — a frontier API model will outpace it 10x. But fifteen tokens per second is fast enough for a conversation to feel responsive, and it is coming from a 27B model running entirely on a laptop with no network cable. The prompt eval at 10 tok/s is the real bottleneck: a long context (2,997 tokens in the last chat slot) takes ~5 seconds to process before the first token generates. That is the cost of Q1 quantization — the weights are small, but the computation per weight is higher because the dequantization happens in the kernel. The DRY sampler and XTC (both tuned aggressively in start-llama-servers.sh) exist because at 1-bit, the model paraphrase-loops without them.

The claim vs the machine

Model vendors publish performance numbers the way car companies publish MPG. The sticker says 42 mpg highway. That number came from a test track — a perfect road, 55 mph, no wind, no cargo, no passengers, no AC. You drive it off the lot and get 28. Nobody lied. The test was real. It was just not your life.

AI model cards work the same way. PrismML's HuggingFace model card opens with a headline: "~44 tok/s on an Apple M5 Pro laptop." The marketing page leads with "87 tok/s on M5 Max." An independent reviewer on a 48 GB M4 Pro measured 27 tok/s and called the gap "explainable rather than suspicious." Here is what those numbers look like next to the machine that actually runs the Knowledge Graph:

The gap between 44 tok/s and 15.1 tok/s is not a lie on PrismML's part — it is the difference between a benchmark and a workload. The vendor number comes from llama-bench: a single model, alone in memory, generating 128 tokens from a 512-token prompt. The 15.1 number comes from a system that is running two model slots (chat + vision), an embedding server, a Whisper transcription server, a Postgres database, a LightRAG server, a FastAPI app, a background worker, and a SvelteKit UI — all sharing 16 GB with a browser and an editor. The model is not alone in memory. It is fighting for every byte.

That is the number nobody puts on their model card: what happens when the model is not the only thing running. The HuggingFace page says "3.9 GB deployed footprint" and it is correct — the weights are 3.9 GB. But the live system at idle consumes ~3.5 GB of wired GPU just for the weights, plus 1.7 GB for Whisper, 0.4 GB for BGE-M3, and the OS's own 4.5 GB of wired memory — and the moment the model starts generating, wired memory spikes by another 5.8 GB for the KV cache. The deployed footprint is not the deployed cost. The deployed cost is the whole machine breathing between 4.9 GB and 10.7 GB of wired memory, 8.3 to 8.7 GB of swap, and 73 MB free — and the model still answers.

The benchmark tells you what the model can do in isolation. The machine tells you what it can do in your life. Fifteen tokens per second, under full load, with the fan running, is not a number PrismML will publish. It is the number that matters — the same way the MPG you actually get matters more than the one on the sticker. The sticker is tested on a perfect road. Your road has traffic, cargo, and weather. The model card is tested with one model alone in memory. Your machine has a browser, an editor, a database, and a fan that is working hard. Both numbers are real. Only one of them is yours.

16GB, accounted for

The memory budget is the hardest constraint and the reason the architecture looks the way it does. On Apple Silicon, the CPU and GPU share one unified memory pool — there is no separate VRAM. Model weights loaded with -ngl 99 (all layers on GPU) live in wired memory. But the system has two states, and they look very different.

Idle — no active conversation, no image processing

This is the machine at rest, with the stack running but nobody talking to the model:

The live container footprint at idle is small: the API at 80 MB, LightRAG at 45 MB, Postgres at 72 MB, Nexus at 7 MB. The LLM weights are resident in wired memory but the KV caches are empty — slot 0 has 25 prompt tokens, slot 1 has 2,997 from the last conversation, but neither is processing. The machine is at 15 GB used, 73 MB free, 8.3 GB of swap. The fan is quiet. This is the resting state.

Active — conversation or image pipeline running

The moment the model starts generating — whether from a chat message or an image going through the VLM — memory spikes. The KV cache fills, activations allocate, and the wired memory jumps by nearly 6 GB. Here is the same machine during an active conversation, measured while the model was generating tokens for this writing:

The 5.8 GB spike is the cost of thinking. Here is what actually happens inside that number.

What is the KV cache?

A transformer generates text one token at a time. To predict the next token, it needs to attend to every previous token in the conversation. Re-running the entire model on all prior tokens for every single new token would be impossibly slow. Instead, the model caches the intermediate results of its attention computation — the Keys and Values for every token it has already seen — so it can reuse them on the next step. That cache is called the KV cache.

Think of it as the model's short-term memory during a conversation. The weights are the long-term memory — they do not change. The KV cache is what the model builds as it reads your prompt and writes its response. Every token you send and every token it generates adds to the cache. When the conversation ends, the cache is discarded. The next conversation starts from scratch.

The size of the cache is determined by four things: how many layers the model has, how many tokens of context it holds, the width of each layer's internal representation, and how many slots are running at once. Bonsai-27B has 64 transformer layers, but its hybrid-attention architecture means only 16 of those layers grow a full KV cache — the other 48 use linear attention, which has a fixed state that does not scale with context length. That is the architectural trick that makes a 262K-token context possible on a laptop. A conventional 27B model with 64 full-attention layers would need ~20 GB of KV cache alone at 32K context — more than the entire machine. Bonsai needs ~5 GB at full 32K context per slot.

The server is started with -c 32768 -np 2 — 32K total context divided across two slots, so each slot gets 16K. That is a deliberate choice. The model supports 262K tokens natively, but at full 32K per slot the KV cache alone would consume ~5 GB of wired memory — on top of the 3.5 GB for weights, 1.7 GB for Whisper, and 4.5 GB for the OS, there would be nothing left. Running at 16K per slot cuts the KV cache in half, to ~2.5 GB, and leaves the machine breathing room for the rest of the stack. The trade is context depth for system stability — and on a personal archive, 16K tokens of conversation history is more than enough.

The server is started with -ctk q4_0 -ctv q4_0 — 4-bit quantized KV cache. That cuts the cache size by 4x compared to FP16. Without it, the full-attention cache alone would be ~10 GB at our running 16K per slot — already more than the machine can spare. The 4-bit quantization is near-lossless: the model card reports 0.0011 nats of output divergence from FP16 KV, which is invisible in practice. The trade is memory for precision you cannot see.

So the lifecycle is: idle, the weights sit in wired GPU memory at ~3.5 GB, the KV cache is empty, the fan is quiet. You send a message. The prompt is tokenized — 2,997 tokens for the last conversation — and each token's Keys and Values are computed and stored into the cache. That allocation is the spike. The model generates its response, adding more tokens to the cache with each step. When the conversation ends, the slot goes idle, the cache stays resident until evicted, and the next time the OS needs memory it reclaims those pages and the wired memory drops back down. The system breathes.

The OS responds by compressing 0.9 GB of inactive pages back into the compressor to make room. The swap grows slightly as the compressor itself needs space. The fan ramps up. And when the conversation ends and the slots go idle, the wired memory drops back down, the compressor relaxes, and the machine returns to its resting state.

That is the real shape of local AI on 16 GB. It is not a steady-state system. It breathes. Idle, the machine is tight but stable — the models are loaded, the containers are running, and there is headroom. Active, the machine is at its limit — wired memory nearly doubles, the compressor works overtime, and every byte is contested. The architecture exists to make sure that the spike is survivable: two-phase pipelines so the VLM does not compete with chat, a separate worker so image processing does not block the API, and container memory limits so Docker cannot eat the GPU's budget. The system was designed around the spike, not the baseline.

That budget is why the reranker was removed. bge-reranker-v2-m3 worked, but it added ~600MB of resident GPU memory for a quality bump that did not justify the cost on a machine already swapping. The architecture is willing to trade a retrieval refinement for a slot that stays free for chat. On a 32GB box the reranker comes back — it is a config change, not a rewrite.

And yet — the conversation works. Not in a demo, not in a benchmark, but right now, with the fan running hard to keep the chip cool, wired memory doubled from the spike, and VS Code and a browser open alongside the entire stack. You ask a question, the model retrieves from the graph, and the answer streams back in real time. No hiccups. No timeouts. No moment where the system freezes and you wonder if it crashed. It is slow by cloud standards and it is fast enough to use.

That did not happen on the first try, or the tenth. Getting a 27B model to run two slots — chat and vision — alongside an embedding model, a transcription model, a Postgres database, a LightRAG server, a FastAPI app, a background worker, and a SvelteKit UI, all inside 16 GB of shared memory, on a laptop that is also running a browser and an editor — that took months of tuning. Memory limits per container. Two-phase pipelines so the GPU is not monopolized. A separate worker process so the API's event loop never blocks. DRY and XTC samplers tuned to stop the 1-bit model from looping. A Whisper model that fits the budget instead of the biggest one. A reranker removed because 600 MB mattered. Each of those decisions was a failure that became a fix. The stack running right now is the survivor of a long list of things that did not work.

That is the honest version of local AI. It is not magic. It is a machine running at its limit, with a fan you can hear, doing something it was not designed to do — and doing it well enough that the person using it forgets the cost. The cost is real. So is the result.

What that result looks like, right now: a slick, modern UI on a Three.js infinite canvas. Photos from the whole month are visible — not as a grid, but as floating nodes on a time surface you scroll through. A conversation with the model holds up. You can see an image from two weeks ago, navigate to the conversation that produced it, and ask the model about it — all in the same view, all on the same machine, all offline.