Sci-Fi Labs · Essay
The Local Agent That Can Answer “What Have I Been Up To?”
Cloud agents are convenient because they already have your data. Local agents become useful only when the retrieval layer can actually see the whole month.
A local agent is only useful if it can answer the questions you actually care about. For a personal memory system that question is usually some version of “what have I been up to this month?” or “show me last month.”
Until a few days ago, Knowledge Graph still hit the classic retrieval ceiling. Vector search returned a top-k slice. Ask about August and you would get roughly seven photos out of the twenty-three that lived in the graph. The model never saw the full set, so the answer was incomplete even though every file sat on disk.
Closing the retrieval gap
This week’s commits closed that gap. The agent now walks the graph for the date range, pulls every Photo entity linked by its real EXIF taken-on relationship, and injects the complete set into the response. Notes travel the same path. Relative language — “this month,” “last month,” a bare month name — is parsed into a proper range instead of being rewritten into a keyword salad by the language model.
At the same time a new navigate tool was added. When the user mentions a time period the system posts a request; the UI polls, matches the target to a time bucket, and flies the Three.js canvas to that layer. You see the spatial layout of the month, not only a text reply.
Tool results themselves are no longer dumped as raw JSON. Entities, relationships, and image paths are parsed and rendered inline so the conversation stays readable. The truncation limit that previously cut month-scale context down to 8 000 characters was raised so the model actually receives the data the graph returns.
Why this is the agentic difference
Cloud agents feel powerful because the vendor already holds the corpus. You hand the workflow over; the data has already left. A local agent has to earn the same usefulness on hardware you control. That means the retrieval layer, the date logic, the spatial navigation, and the prompt hygiene all have to be good enough that the model can act on a complete view of your own history.
Knowledge Graph is the test bed for that claim. It is a private spatial memory engine — conversations, photos, and voice notes floating as time layers on an infinite canvas. Inference runs through Llama.cpp. Retrieval uses LightRAG. Voice is local Whisper. The published site is a demo. The real product is the copy that runs on your machine, answers questions about your own life, and never phones home.
The same principle applies to the rest of the Sci-Fi Labs stack. Where is Paul? is the public-facing spatial life map — globe, timeline, stories — that can feed moments into the private graph. Both are just websites, so they open in any browser, including Apple Vision Pro and other AR/VR headsets. The data layer stays local by design.
Ambient alternatives keep arriving
While the local retrieval work landed, the rest of the industry continued in the opposite direction. Comcast began offering Wi-Fi motion sensing on millions of home gateways. The feature detects movement through signal disruption, stores up to seven days of activity history in the company’s cloud, and can be disclosed under a valid subpoena or warrant. Convenient. Ambient. No longer yours the moment it leaves the house.
What “done” looks like
Unplug the network. Ask the agent what you did last month. Watch the canvas fly to the correct bucket and the answer arrive with every photo and note that belongs there. That offline test is still the only proof that matters. Everything else is marketing.