Knowledge Graph · Field notes · Part 1 of 3
Knowledge Graph, Part 1: The canvas and the stack
Cut over to an infinite canvas, make time a depth axis, and run the whole stack on 16GB of shared memory — models that never leave the machine.
Knowledge Graph started as a bet that memory should be a place, not a thread. The first solid version of that bet was a 3D infinite canvas with time as a navigable dimension, local models for vision and speech, and a pipeline that kept originals on hardware you own.
The deeper motivation came from watching the first wave of agent projects. OpenClaw was the first publicly viral agent that could actually do things on your computer — a genuine leap in utility. Hermes and others chased the same frontier. But somewhere along the way the conversation shifted to multi-agent orchestration, swarms talking to each other, elaborate tool chains — and the most basic expectation quietly fell behind: the agent remembering your day, your people, the thing you mentioned last Tuesday. Memory, the thing a personal assistant is supposedly for, got treated as a solved problem while everyone worked on the complicated one. Knowledge Graph is the bet that memory is the complicated one — and that it is the one worth getting right first.
That bet pulls in two threads that the rest of this series keeps circling back to. There is the tension between an agent built for utility and one built to be personal — they look like the same product and they are not, and I wrote about that split separately in Utility vs Personal AI. And there is the fact that the moment you start bolting on API keys — one per service, one per model, one per integration — you have handed your data to yet another company, and the “AI on your machine” story stops being true. That one became its own piece: Who has your data?.
Why a canvas
Lists and chat windows flatten life into sequence. A spatial surface lets the same day hold photos, notes, and later conversations without forcing them into a single scroll order. Zoom out for months; zoom in for an afternoon. The camera is the primary navigation, not a sidebar of folders.
Local by default — on 16GB shared
Models run on the laptop’s own chip — no cloud involved. Speech recognition, vision, the memory index, the database, and the canvas all share one machine. There is no obligatory cloud ingest to make the product work — cloud is optional, not the path of least resistance.
The ultimate test for sovereignty is offline. Airplane mode. No Wi‑Fi. No fallback API. If the assistant still answers from weights on disk and data on disk, it is a local model in fact — not a thin client dressed up as “AI on your machine.” Anything that silently phones home when the cable is unplugged has already failed the test.
Why 16GB, specifically? Because that is the machine this is built on — not a hypothetical target spec. The dev box is a 14‑inch MacBook Pro with an Apple M2 Pro: a 10‑core CPU (6 performance, 4 efficiency), a 16‑core GPU, and 16GB of unified memory — meaning the processor and the graphics share one pool of memory, there is no separate graphics memory to spill into. That is the entire budget, and the operating system, the browser, the database, and the live canvas are already in it when the first model loads.
This is the part of the local‑AI story that most write‑ups quietly skip. The demos that get shared are almost always run on rented A100s or a workstation that cost more than a used car. Large orgs have the money to blow on GPUs — a frontier‑class rig is a line item, not a sacrifice. Most people do not. The Steam Hardware Survey is one of the few public windows into the machines real users actually own, and the RAM split tells the story plainly:
| System RAM | All platforms | macOS only |
|---|---|---|
| 8 GB | ~8% (and growing — RAM prices are forcing downgrades) | ~2% |
| 16 GB | ~41% — the single most common configuration | ~43.6% — the plurality on Mac |
| 32 GB | ~37% (slipping as kits stay expensive) | ~32% |
| 64 GB+ | under 5% | under 3% |
In other words, for nearly half the people reading this, 16GB is not a floor they are trying to climb above — it is the ceiling they are living under. If a local stack only runs on a $4,000 box most of them will never buy, then “local AI” is a hobby for the few, not a tool for the many.
That pool is the hard constraint — not “local in theory” but local in fact. The main model, the speech recognizer, the vision model, and the live canvas all compete for whatever is left once the operating system, the browser, and the database have taken theirs, which is why the stack leans on compressed models, reclaims memory the moment a tool is idle, and is careful about what loads first. Those are product decisions in this stack, not optimization trivia.
The pipeline
Ingest path
Part 2 picks up where phones, microphones, and concurrent clients hit that 16GB budget at once — and what had to change so multi-client local AI stayed usable without abandoning the constraint.