Sci-Fi Labs · Field notes
The Local Agent That Can Answer “What Have I Been Up To?”
Cloud agents feel useful because the vendor already holds your month. A local agent only starts when retrieval can see the whole set and fly the canvas there.
A local agent is only useful if it answers the questions you actually ask. For a personal memory system that question is usually some version of “what have I been up to this month?” or “show me last month.”
The model was never the missing piece. Retrieval was. Until a few days ago, Knowledge Graph still hit the classic ceiling. Vector search returned a top-k slice. Ask about August and you would get roughly seven photos out of the twenty-three that lived in the graph. The files were on disk. The agent never saw the full set. The answer was incomplete because the prompt was incomplete.
Closing the retrieval gap
This week’s commits closed that gap. The agent walks the graph for the date range, pulls every Photo entity linked by its real EXIF taken-on relationship, and injects the complete set into the response. Notes travel the same path. Relative language (“this month,” “last month,” a bare month name) is parsed into a proper range instead of being rewritten into a keyword salad by the language model.
At the same time a navigate tool landed. When you mention a time period the system posts a request. The UI polls, matches the target to a time bucket, and flies the Three.js canvas to that layer. You see the spatial layout of the month, not only a text reply.
Tool results are no longer dumped as raw JSON. Entities, relationships, and image paths render inline so the conversation stays readable. The truncation limit that previously cut month-scale context down to 8,000 characters was raised so the model actually receives the data the graph returns.
| Chat transcript | Canvas place | |
|---|---|---|
| What you get | A text summary of a top-k slice | Every photo and note in the month, on a time layer |
| Where you look | A scroll of bubbles | The spatial layout of that month |
| What shipped | Incomplete answer from incomplete retrieval | Date-range walk + fly-to-bucket |
Why this is the agentic difference
Cloud agents feel powerful because the vendor already holds the corpus. You hand the workflow over. The data has already left. A local agent has to earn the same usefulness on hardware you control. That means the retrieval layer, the date logic, the spatial navigation, and the prompt hygiene all have to be good enough that the model acts on a complete view of your own history.
Knowledge Graph is the test bed for that claim. It is a private spatial memory engine: conversations, photos, and voice notes floating as time layers on an infinite canvas. Inference runs through llama.cpp. Retrieval uses LightRAG. Voice is local Whisper. Kokoro JS speaks when the conversation surface needs it. The published site is a demo. The product is the copy that runs on your machine, answers questions about your own life, and never phones home.
The same principle applies to the rest of the Sci-Fi Labs stack. Where is Paul? is the public-facing spatial life map (globe, timeline, stories) that can feed moments into the private graph. Both are just websites, so they open in any browser, including Apple Vision Pro and other AR/VR headsets. The data layer stays local by design.
Ambient alternatives keep arriving
While the local retrieval work landed, the rest of the industry continued in the opposite direction. Comcast began offering Wi-Fi motion sensing on millions of home gateways. The feature detects movement through signal disruption, stores up to seven days of activity history in the company’s cloud, and can be disclosed under a valid subpoena or warrant. Convenient. Ambient. No longer yours the moment it leaves the house.
What “done” looks like
Unplug the network. Ask the agent what you did last month. Watch the canvas fly to the correct bucket and the answer arrive with every photo and note that belongs there. Offline is the proof. Everything else is marketing.
Memory you own, models you run. Published sites are demos. The product is the copy on your machine.