Why I'm building my own AI agent (and why it runs on my hardware)
Last month my AI agent spent eight minutes turning one email into a calendar event. Five of those minutes were pure waste – the model fumbling with tool parameters, failing validation, retrying, fumbling again. I sat there watching tool_input_invalid errors scroll past, produced by software I own completely, running on hardware I paid for, doing a task any cloud assistant finishes before you've put the phone down.
I'd do it all again tomorrow. This post is about why – and it's the opening move of a series about what it actually takes. (Part 2 looks at why a second, simpler model runs beside the primary.)
Because the obvious question deserves a straight answer. It's 2026. Capable AI agents are a commodity: sign up, connect Gmail, press "allow," done. Why would anyone build one from scratch and insist it runs on a laptop?
The non-negotiables
The first document I ever wrote in the repo answered that before a single line of code existed. It described "a self-hosted, privacy-first personal AI agent," "fully owned, fully controlled by its builder," where "every line of its core logic is transparent and modifiable" – unlike the commercial crowd: ChatGPT, Claude, OpenClaw, the lot.
Strip the manifesto tone and three non-negotiables remain: ownership, privacy, and learning. I wanted a system where nothing important happens in a component I can't open. I wanted my data to stay mine. And – this is the one that actually drives the project – I wanted the education. Azek isn't a race to build the best agent or plant a flag on some breakthrough. It's a deliberate journey through how modern AI systems really work, taken the only way that produces real understanding: design every layer yourself, hold the whole system in your own head, and live with the consequences. My day job runs the opposite way, and rightly so: enterprise work means proven components, as little custom code as possible, and delivery speed above everything. Which is exactly why a personal project is the only lab left where an engineer can still take the whole machine apart.
The privacy point deserves one more sentence, because it's the non-negotiable with teeth. This agent reads my email, my Telegram messages, and other communications. It sees my calendar, my notes, my half-formed questions at one in the morning – some of the most sensitive data there is about me and my family. A system with that level of access to your life should not be a black box operated by someone else's company. Not because those companies are evil, but because "trust us" is not an architecture. In my agent, personal data lives in a local database, and any request that does leave the machine carries only the minimum context that specific turn requires. That's enforced by context budgets and egress guards in code, not by a paragraph in a privacy policy.
Four constraints most people would call mistakes
The agent is called Azek, and it follows four unconventional rules.
No agent frameworks. The orchestrator, memory retrieval, supervision – all hand-rolled. LangGraph, LangChain, CrewAI, PydanticAI: evaluated, rejected, in writing. Framework fatigue is real, but the deeper reason is a mismatch of shape: frameworks solve orchestration-in-general, and a single-operator agent needs orchestration-in-particular. Every abstraction a framework adds is a layer I'd have to see through when the agent misbehaves at midnight. A turn loop I wrote is a turn loop I can audit end-to-end, and every bug in it teaches me something a framework would have quietly absorbed on my behalf.
Local-first inference. The main conversational loop runs on local models through Ollama, and there is deliberately no silent cloud fallback. If the local stack is down, the turn fails loudly instead of quietly shipping my conversation to a paid API. That decision was actually walked back and forth before it stuck, because a fallback is so comfortable, but that comfort is exactly the problem. A silent fallback erodes local-first one outage at a time. "Local-first" is a privacy stance before it is a cost stance; the money is a bonus.
One DB for everything. Conversations, the tool-invocation audit trail, background job state, rate limiting, and the entire semantic memory index. One database to back up, one to reason about, one connection to take care of.
No OpenAI. Not as an inference provider, not as an SDK dependency. Full disclosure: there's no technical argument here – I simply don't like the company's politics, and a personal project is the one place where that's reason enough. The interesting part is what happened to the feeling: an import-guard script runs with every typecheck and fails the build if a forbidden provider package appears anywhere in the tree. An opinion, automated, becomes architecture.
The rest of the stack follows the same minimalistic taste: Bun as the runtime, Hono – a 14 KB router – for HTTP and WebSockets, the Vercel AI SDK as the one thin provider-agnostic layer, Zod everywhere. Small enough to hold in one head. That's the point.
How this thing actually gets built
Here's the part that makes this a distinctly 2026 project, and the part where I owe you a definition of "building it yourself" – I don't write the code.
Nearly every line of Azek was implemented by AI coding agents (Claude, Copilot, Antigravity) working from milestone briefs I write and argue over. My loop is: research how people solve the problem, pick the approach that fits (or just the one I'm curious to try), argue the architecture out with Claude, turn it into a task, hand it to a coding agent to implement and live-test autonomously, then check the results and loop back to the drawing board when something smells wrong. No line-by-line reviews. Every decision that shaped the system is mine; almost none of the typing is. Two months in, the project rarely gets more than an hour of my attention a day. Most days it's whenever I find a few minutes to make a decision. And all of it fits inside the daily quotas of the cheapest subscription tiers that include coding tools – nine days out of ten that means Claude Code. No boosted plans, no API bills; when the quota runs out, that's the workday over.
That's also why building from scratch stopped being crazy. The traditional argument for frameworks was implementation cost, and coding agents just collapsed it. What's scarce now isn't code – it's architectural judgement and the willingness to own the consequences. And running agents at this level of autonomy is the other half of the curriculum: a project about mastering AI, built by learning to direct it.
This workflow has one consequence I own with zero regret: there are no unit tests. I didn't start there. The early weeks were by-the-book – proper coverage on everything. But when implementation is this cheap, and the destination is deliberately unfixed, the project pivots hard and often – and every pivot meant rewriting a test suite, often bigger than the code it guarded, built to protect decisions that no longer existed. At some point I made it policy: no unit tests at this stage; the gate is typechecking, a set of custom guard scripts, and actually running the app. That was the right call for my case, and I'm not sure I'll ever go back to test suites in a fast-evolving project.
What it costs
Honestly, this way of building has sharp edges, and I've bled on most of them.
The codebase optimises for the best end state, full stop – backward compatibility is redundant in a single-user system. Obsolete schemas are deleted, not aliased; database tables are dropped and recreated, not migrated; breakage is fixed forward. One milestone doc contains my favourite admission in the whole repo: "No data preservation is required." That's not recklessness – it's honest maths. If a code mistake takes the database down, restoring a half-hour-old automatic backup is a one-minute job. The full ceremony – data-preserving migrations, compatibility shims, separate test, stage and prod environments, version juggling – is the right workflow for a product with users. For a system with exactly one user who is the only developer at the same time, it's overkill that would eat the very hours the project runs on.
And the cloud providers aren't a fallback. They're the control group. Azek keeps frontier models wired in, one config switch away, for exactly one reason: to know where the boundary of a local model actually is. Test a new capability on a powerful frontier stack first; if it works there and fails locally, the model is the problem, not the idea. Skip that step, and you'll bury good ideas with a verdict of "doesn't work at all" when the honest verdict was "doesn't work on a laptop-sized model yet."
Then there's the cost that opened this article: local models fumble and, even worse, they bluff. Everyone knows LLMs will confidently invent things at the edge of their competence; living with small ones teaches you that the smaller the model, the bolder the bluff. Eight-minute calendar events happen. Which brings us to the actual bet.
The bet
Today, Azek's main loop runs mostly on qwen3.6 as the flagship model and gemma4:e4b for fast utility work. Two small local models, deliberately from different families; neither is anywhere near frontier. Both fit on a laptop.
The bet this whole series tests: can a small local model do real agent work if you build enough scaffolding around it?
Because the honest version of that question includes the ugly part: the scaffolding's real job is squeezing reliable behaviour out of a model that isn't reliable on its own – learning to out-trick a model that would happily trick you.
The answer is scattered along a long dive into deep AI waters: model tiers, token economics, a fast small model supervising a bigger one. Every day down there throws more light on the machinery most users never think about and teaches a bit more about handling raw LLMs across providers and tiers. It's going to be a fascinating trip. Fasten your seatbelt.