Case Study: How We Built Our Own Agent

Where the problem started

The starting point was not technical ambition but a daily irritation. The project's own documentation puts it like this:

"I was running close to twenty AI tools at once to cover my own work. What one of them learned about me, the next one had never heard of. My memory was scattered, security was decided somewhere different in every tool, and the context was lost at every switch."

That is not a shortage of tools. It is an integration problem — and that distinction decides whether it is worth starting at all. If your trouble is that you have no good tool, buy a better one. If your trouble is that you have six good tools and none of them knows about the others, the missing piece is not a seventh tool.

What came out of it is a self-hosted framework, in public beta as of September 2026: close to sixty modules, more than two hundred skills, thirty-four MCP servers and twelve runnable engines, in six languages. Those numbers move every week, and they are not the interesting part. The decisions are.

What an agent does step by step, at the level of the loop itself, is set out in what an agent actually does, a piece of its own.

The first decision: a framework, not a chatbot

The documentation's opening line is blunt: "Not a chatbot. A real assistant." The practical consequence, though, is not the interface — it is the engine question.

The system can use any model provider, and inside that sits a decision that rarely comes up: subscription coding CLIs serve as engines too, with no separate API key. If you already pay for a coding assistant, the system calls it as an engine instead of buying the same capacity a second time over an API.

That is a cost decision, not a feature. And it has its counterpart on the trade-off side: with the big managed services the model is bundled into the subscription. Here it is not. The documentation says so plainly — the model is still your key or your subscription.

Whether building your own beats buying something finished is the question buy or build walks through, case by case.

Routing: the first step is not AI at all

This is the most transferable decision in the whole system. The path a call takes:

1. CHEAP SIGNALS       -> five signals, no paid call
2. MODEL CLASSIFIER    -> only if the cheap round is unsure
3. TIER SELECTION
4. BUDGET CHECK
5. FALLBACK CHAIN      -> if the provider is unreachable

The point is in the first line: the first round of classification costs nothing. A cascade of cheap signals decides — a function binding, keyword rules, message metadata, the conversation's own history, and a small classifier running locally — and only when all of those are still unsure does it ask a paid model. The cheapest optimisation in an agent is not calling a cheaper model; it is not calling a model at all.

The tiers a call gets sorted into:

The spending budget has three thresholds: it warns at 80 percent, drops to a cheaper tier by itself at 100 percent, and halts at 120 percent. That is interesting because the third one is the one that matters: everybody writes a warning, few write a hard stop.

Memory: five layers, and a bug that forced isolation

Memory splits into five layers, and what separates them is how long they live:

The key retrieval decision is that the model does not have to ask for memory. Two blocks go into every turn: a one-line index of every note, and the related earlier work. The index has a fixed budget, eight thousand characters by default — and that number is in the system because without it memory quietly eats the context window.

Capture has gates of its own: a message shorter than forty characters buys no note, one conversation is capped at twenty extraction calls, and a turn produces between zero and two notes. There is no keyword list, in any language. And capture runs after the reply has gone out: if it falls over, the cost is a missing note, not a missing answer.

And here comes the bug that justified an entire subsystem. One of the engines keeps its own memory on the host machine. Without isolation the extractor once read that host memory, reported back that the fact was "already recorded" — and EYAS's own store stayed empty. A silent failure: nothing crashed, nothing got written.

The fix has two parts: an isolated model context, and a non-model gate that denies every file-writing tool the foreign memory paths. The honest limit stayed where it was, though: two engines' protocol knows no isolation parameter, so there it cannot be done — and the interface says so rather than pretending it can.

What an agent should remember in general and what it should forget is detailed in agent memory, with the trade-offs named.

Security: three gates, and the first is not a model

Not every call goes through all three, and that is the soul of the system:

• a green action (reading, searching) → the first gate only
• yellow (writing, changing a setting) → the first and the second
• red (deletion, financial action, live deployment) → all three, plus human approval

Part of the deterministic gate is one simple rule: of "untrusted input", "sensitive data" and "external action", at most two may be true at once. If all three are needed, human approval is mandatory. That one sentence buys more protection than most policies manage in three pages.

Before every change a snapshot is taken — file, database, setting, git — with a restorable state. The phrasing we built it for: "'the AI got it wrong' is not an argument, it is a single command".

And the sentence without which this section would have no credibility: prompt injection is not solved. The documentation says exactly that, and adds that anyone claiming otherwise is worth a second look. The layers reduce the risk; they do not remove it.

Where to put the approval inside a process is unpacked by the human in the loop, which names three places worth guarding.

Narrowing permissions — giving the agent the least it can work with — is covered by least privilege as a design principle.

And the business consequences of prompt injection get a separate look in another post.

The dead end you could measure

There is exactly one decision in this project that a number stopped rather than an argument — and that is the most instructive one of the lot.

Search works from several sources: full-text search, vector search and a graph. The hits have to be merged, and there is a textbook method for that, reciprocal rank fusion applied plainly. Obvious, and what everybody does.

Measured: the naive fusion took recall in the top five from 85 percent down to 65. So one good hit in five vanished — from a step most systems never even put on the measuring bench. It has been forbidden in the code ever since.

What is worth taking away here is not the number itself. It is that you only notice this if you measure recall rather than your impression of it. After a failure like this the system is no slower and throws no error. It simply finds less, and hallucinates more because of it — which you then charge to the model's account.

In fairness: this is the only measurement of its kind published in the project. Every other decision is justified by reasoning, not by an A/B test. That is normal for a one-person project, but you have to know the difference.

What we deliberately did not build

This section matters more than the feature list. The documentation names twelve trade-offs out loud; here are six of them:

• An embedded database instead of a bigger one. The bigger one would have brought about fifteen megabytes of native dependencies; the simpler option covers roughly 95 percent of the need.
• No native desktop or mobile app. The documentation calls this "a real gap" in as many words, rather than explaining it away. The surface is a browser and a messenger.
• Full computer control works on one operating system only. On the other two it is not finished.
• There is no one-click restore onto an empty machine. The backup is complete; the restore is six manual steps.
• A single-owner system, not an enterprise one. No central sign-on, no certifications. The audit trail is complete, but it is single-machine and uncertified.
• The add-on catalogue is small, on purpose. Thirty-four MCP servers and nine extensions: a decision taken after the 2026 skill-registry incidents, because an open extension market is a demonstrated attack surface.

A list like that is information in itself. If a product's documentation has no "what we cannot do" section, that does not mean it can do everything.

The risks of self-hosting, incidentally, are gathered up by personal agents on your own machine one by one.

What to take away if you were to build one

1. NAME THE PROBLEM FIRST, NOT THE TOOL
   "I have no good tool" and "I have six tools and
   none of them knows about the others" are two projects.

2. THE CHEAPEST CALL IS THE ONE THAT NEVER STARTS
   Make the first round of classification deterministic.
   Ask a model only where you are genuinely unsure.

3. A SPENDING BUDGET NEEDS A HARD STOP
   A warning is not a limit. Neither is a downgrade.
   You need a level where it actually halts.

4. THE FIRST SECURITY GATE MUST NOT BE A MODEL
   Whatever a prompt can talk around is not a gate.
   Pattern matching is boring, and reliable for it.

5. MEASURE RECALL, NOT YOUR IMPRESSION
   Search degrades in silence. One bad merge
   can take a fifth of the hits away with it.

6. WRITE DOWN WHAT YOU CANNOT DO
   The trade-off list is for your own later self,
   and for whoever is going to use the thing.

Common misconceptions

• "Your own agent is cheaper." The model bill stays. Only the orchestration becomes yours — and so does the maintenance.
• "If it runs on my own machine, it is safe." The risk does not disappear, it moves: from the model to the permissions.
• "The agent will work out which model it needs." In a good system the first round of that decision is deterministic, before any model call.
• "Memory is the vector database." Storage is the easier half. The harder one is when and how much goes into the context without eating it.
• "With three security layers prompt injection is solved." It is not. The layers reduce the risk.

What is worth taking away

A home-built agent framework is not made good by how many modules it knows. It is made good by taking the expensive decisions cheaply: classification with no model call, blocking by pattern matching, restoring from a snapshot.

The three numbers worth remembering from this project: five milliseconds for the deterministic gate, 120 percent for the hard spending stop, and recall falling from 85 to 65 percent on a single badly chosen merge.

And the most useful thing you will find in the documentation of a system like this is not the feature list but the trade-off list. If you build, write your own while you still remember why you decided the way you did.

The standard for connecting tools, which is this system's backbone too, is explained in the post on MCP and how it works.