> Markdown version of https://archtenet.dev/blog/the-harness-not-the-brain — the same page without the site chrome.
> Index of everything published here: https://archtenet.dev/llms.txt

# The harness, not the brain: how a bluffing laptop model became an agent I trust

> Part 3 of a series on building a personal AI agent. How a small local model that bluffed about drafting emails became a daily assistant I trust: why lecturing a model backfires, what happened when one AI supervised another, and the discovery that my own workspace was the real culprit.

- **HTML version:** https://archtenet.dev/blog/the-harness-not-the-brain
- **Published:** 2026-11-17
- **Authors:** Ivan Baha
- **Series:** Building a Personal AI Agent
- **Tags:** ai, ai-agents, local-llm, privacy, self-hosted, architecture

Several months ago, I asked [Azek](https://archtenet.dev/blog/why-im-building-my-own-ai-agent.md) to draft and send an email. A few seconds later, it replied, politely and confidently, that the draft was ready. I opened my email client – nothing there. No draft had been created, and no email tool had even been touched. My milestone note from that evening is shorter than usual: *"No draft was created. No email was drafted. Azek lied."*

In the first article of this series, I set out the bet behind the project: can a small AI model running privately on my own hardware do real everyday work with my mail, messages, and calendar if I build enough scaffolding around it? In [the second](https://archtenet.dev/blog/two-flows-second-model-and-the-primary.md), I showed how a tiny helper model runs alongside the main one to handle small chores, and ended by promising to show what happens when that small model is asked to supervise the big one.

Between that phantom email back then and today lie months spent living with the bet. It wasn't a straight line. There were stretches when I doubted a laptop-sized model was capable enough for real life, and for a while I switched the main seat to cloud models just to get my daily work done while I kept building. This post is the story of how I got from an assistant that bluffed to one I actually trust, and why the turning point had much less to do with the AI's brain than with the workspace I built around it.

## Stop lecturing, take the choice away

When an AI assistant misbehaves, everyone reaches for the same lever first: write a stricter instruction. *"Always check the calendar first. Never make things up. Never stop halfway through a task."* It costs nothing and feels like managing.

Living with a small local model cures you of that habit fast. Take my morning briefing. Azek uses playbooks (skills) for recurring jobs, and the briefing playbook clearly instructed the model to review my context and unread messages before summarising the day. One morning, I watched the model read the playbook, pause for a second, and talk itself out of the work: *"standard context seems sufficient for a general briefing."* It skipped four of its seven lookups and improvised the rest. A few days later, during a voice briefing, it confidently reported unread emails from a bank, an insurer, and a newsletter, even though it had only checked the calendar – stitching together yesterday's conversation history and presenting it as live news.

My first instinct was to make the prompt firmer, but it didn't help. Worse, when I added stern rules elsewhere, telling the model never to end a turn by simply announcing what it planned to do next, the warnings backfired. The warnings tangled the model up so badly that, in my worst test run, it spent over three minutes thinking in circles and still ended with nothing but a promise to search next.

The fix wasn't a sterner lecture. It was taking away the choice.

If a morning briefing always needs my calendar, tasks, and unread mail, asking a small model to decide whether to fetch them invites laziness. So I built prefetch: the moment a playbook starts, Azek runs those mandatory lookups itself and lays the results on the model's desk before it speaks a word. No tool is left to skip because the data is already in front of it.

The same principle addressed the habit of stopping halfway. I asked Azek to find a payslip email from my employer. It ran a single narrow search, found nothing, replied, *"Let me broaden the search,"* and stopped there, waiting for me to pat its head. Rather than banning that phrasing in the prompt, I put a gate at the exit. When a turn is about to end, the gate checks whether the answer merely announces the next step. If so, the gate stays shut and nudges the model to take the step it just promised. And when retries run out and the turn truly has to wrap up, Azek removes every tool for one final pass – because without tools, *"I'll search X next"* is no longer an option. All the model can do is report what it found.

**Polite instructions are advice a small model can talk itself out of; a good harness prevents the wrong move rather than forbidding it.**

## The watcher on the shoulder

What about when a turn goes wrong in the middle, long before it reaches the exit door? That was the job of the experiment I teased at the end of [Part 2](https://archtenet.dev/blog/two-flows-second-model-and-the-primary.md): the Supervisor. Instead of waiting until a turn finished to notice the main model had drifted off course, I put the tiny helper model to work, watching the main model's live stream in real time – ready to interrupt a ramble, nudge it back on track, or tell it to stop overthinking and act.

Watching one AI supervise another immediately showed me where language models shine and where they are blind. One afternoon, the main model got stuck in a mental rut, repeating the phrase *"stop reasoning and act"* thirty-six times in a row. My shiny new AI judge watched the whole thing and kept offering polite hints about a minor tool error. It never noticed the loop – because a language model reads for *meaning*, and the meaning sounded perfectly sensible. A tiny, old-fashioned text check that simply counts repeated words caught it the next day without waking a model at all.

I learned a similar lesson whenever I asked the small watcher to weigh a rule with exceptions. Ask a small model a *"yes or no, except when..."* question, and it wavers in both directions. Instead, ask it to pick one label from four clear, non-overlapping buckets – *is this reply complete, announcing a next step, asking the user a question, or giving up?* – and let ordinary code decide what to do with that label. It becomes rock-solid. **Small models are poor judges of exceptions but are natural at sorting things into clear buckets.**

Still, even with playbooks, short, interruptible thinking steps, and the watcher looking over the model's shoulder, everyday life with the local model remained frustrating for a while. Too many turns felt fragile. Tasks that should have been simple took ages or stumbled in strange ways. After weeks of battling inconsistent results, disappointment set in, and I grew sceptical that a laptop-sized model could be dependable enough for daily life.

I needed an assistant I could actually use every day, not just an experiment to babysit. So, out of pure pragmatism, I switched the main conversation to larger open-weight models running in the cloud. They handled the rough edges noticeably better right out of the box, which let me rely on Azek for real work while I kept refining the harness underneath during my spare hour a day.

## The culprit wasn't the model

Eventually, I decided to settle the question properly. Instead of relying on a gut feeling that the local model wasn't good enough, I sat down to run live, side-by-side passes across dozens of real tasks – pitting my local models against cloud models, with the supervisor switched on and off, to see where the local brain actually broke down.

What came back was the biggest surprise of the whole journey. Time and again, when the local model did something that looked foolish, the fault wasn't in the model's brain at all. It was in what my own system was feeding it.

Here are three examples that quickly humbled me:

* **Fifty-four blank instruction labels.** Earlier, while investigating why the agent kept fumbling browser tasks, I noticed the model muttering in its reasoning that none of its tools listed any input parameters. It wasn't hallucinating. A quiet software update had renamed a single field in my code, leaving all fifty-four of Azek's tools with completely blank parameter descriptions. Every time I had blamed the local model for "clumsily guessing" how to call a tool, it had been reading a blank sheet and guessing from the tool's title alone.
* **The "no" nobody passed along.** During the test runs, I asked Azek to create a calendar event and clicked "Cancel" in the confirmation box. The local model stubbornly tried to create the declined event four times in a row. It looked like a rogue, disobedient AI – until I checked what Azek actually told the model when I clicked Cancel. The message said nothing about a person saying no; it looked like a technical failure, so the watcher nudged the model to repair the error and try again. Once Azek stated in plain text that *the user declined this action* and locked that tool for the rest of the turn, the local model obeyed on the first try: *"Per the rules, I must not call the tool again in this turn."*
* **Poisoned memory and shifted clocks.** Half of the morning briefings in my test run listed appointments an hour early – on both local and cloud models – because my calendar connector was handing them a confusing mix of time zones. In another test, a turn guessed which email address was connected to my Gmail, saved its own guess into memory, and four hours later even a fast frontier cloud model repeated that wrong address as a settled fact – because it trusted the memory notes my system had handed it.

That was the lightbulb moment. The cloud models hadn't succeeded earlier because local models were hopeless; the cloud models were just big enough to brute-force their way past some of my broken plumbing, while quietly falling into the rest of the traps whenever I wasn't looking closely. **Half of what looked like small-model stupidity was just my own workspace feeding the model bad information.**

And those tests delivered one more surprise: what was the live AI watcher actually contributing now that the workspace was clean?

When I ran the exact same tasks with the supervisor on and off, the supervised runs passed far more often – fourteen out of sixteen, compared with six. That looked like a triumph for the live AI judge until I opened the logs to see *which part* of the supervisor had made the difference. Almost every rescue came from that simple check at the exit door, which caught the local model when it announced *"let me search X"* and stopped too early. The mid-turn AI judge – the small model watching the big one stream and offering commentary – had barely changed a single outcome. Worse, on a single laptop GPU, having a second model wake up mid-turn to judge the first meant the two models were constantly fighting for memory and slowing each other down.

The live judge wasn't a mistake; it was scaffolding that kept the house standing while I learned where the walls needed to go. Every real failure pattern in my notes had been discovered by reading finished turns, not by one AI lecturing another mid-flight. Once those patterns were clear, the mid-turn judge had served its purpose, so I deleted it cleanly. What remained was faster, quieter, and more predictable: a small classifier that picks a state label when needed, and plain code that enforces the rules.

```mermaid
flowchart LR
    U(["Request"]) --> PRE["Prefetch & workspace<br/>lay needed data on the desk"]
    PRE --> MAIN["Main model works<br/>Local by default · Cloud for heavy jobs"]
    MAIN --> GATE{"Exit gate &<br/>code checks"}
    GATE -- "Stopped on a promise or<br/>unverified detail" --> MAIN
    GATE -- "Complete & grounded" --> ANS(["Answer"])
```

## Where the line actually sits

Once the plumbing defects were fixed and the noisy mid-turn judge was gone, I ran a proper testing campaign to determine the true boundary of the laptop model across five categories of my daily work. I set the trust bar in advance: no invented facts, no turns dying from time limits, and consistent completion across repeated runs.

The results finally put my earlier doubts to rest. For taking action and for chatting about my personal data – my emails, messages, calendar, and stored notes – the local model (`qwen3.8:27b-mlx`) met the trust bar cleanly. It was careful, grounded, and honest. Where it missed the bar on heavy morning briefings and open-ended web research, it didn't fail by hallucinating or doing the wrong thing – seven of its eight misses were simply due to the clock running out. The cloud model's real advantage wasn't greater care; it was raw speed.

So I changed how Azek chooses its brain. Instead of a single global switch for the whole agent, the tier now belongs to the job:

* **Private life runs locally by default.** Anything that reads or acts on my mail, Telegram messages, calendar, personal memory, or background summaries stays on my laptop.
* **Heavy open-ended work goes to the cloud by choice.** Long career-coaching sessions, news digests from public sources, and broad web research run on fast cloud models – intentionally, per job, when no private household data is at stake.

**You don't need a laptop model to beat a data centre at everything; you just need to know which jobs belong in your house and which can rent a faster engine down the road.**

## The email I don't edit anymore

After reverting the default back to the local model, I did the hardest thing for someone building their own tool: I left it alone for a full week and just used it.

It isn't magic, and it isn't instant. A local model on a laptop is still noticeably slower than a frontier cloud API – you can feel it taking its time when a turn is heavy, and using a real laptop brings physical realities you never think about in the cloud, such as managing memory when another project's containers spin up or waking up after the lid has been closed.

But the feeling of living with Azek has completely changed from those early weeks. It is calm and predictable, and it fully satisfies my everyday flow.

The simplest way I can measure how far it has come since that first phantom email – when it looked me in the eye and lied about creating a draft – is how I send emails through it today. Every action that reaches the outside world still brings up a confirmation card on my screen before anything leaves the laptop. I still click "Approve" every time. **But I don't edit the email drafts anymore – and most days, I don't even read them before hitting send.**

Getting a local model to use its tools honestly and finish its work solved the hardest problem on my list. That immediately exposed the next one: once an assistant lives with you for months, a single conversation is the easy part – the real mess is what it remembers about you tomorrow, and what happens when its memory starts playing tricks on it.