Edge-Native RAG — When the Small Stack Is the Right Stack

Every AI chatbot has the same problem: it forgets you.

You tell it your name in one conversation, and by the next one it has no idea who you're talking to. You tell it your preferences, and a week later it's making things up. The bigger you build, the worse this gets — and at some point the bill from your LLM provider stops being funny too.

The fix has a name: RAG, short for retrieval-augmented generation. You give the model a memory it can search before it answers. The catch is that the standard way to build it is overcomplicated — a vector database, an embedding service, a chat model API, and a server somewhere to glue them all together. Four services, four monthly bills, four different things that can wake you up at 3 AM. Three months in, you're paying close to $200 a month for what is, honestly, a chatbot that remembers things.

There's a smaller version of this — one file, one deploy command, and a bill that starts at zero and stays surprisingly low for surprisingly long. It's not the right answer for every product, and I'll get to where it breaks. But for a large slice of what people are actually building right now, it's the answer, and almost nobody is talking about it.

---

The Stack

The whole RAG loop — embed the question, search for context, generate the answer, save the new message — collapses into a single Cloudflare Worker:

• The Worker is the entire application. One handler. Receives the message, does the work, returns the answer.
• Workers AI runs an embedding model (bge-base-en-v1.5) and a generation model (Llama 3.1 8B or similar). Both live inside Cloudflare's network — no outbound HTTP call to OpenAI, no rate limit roulette.
• MongoDB Atlas Vector Search stores embeddings alongside their source text and whatever metadata you care about, and answers similarity queries.

That's it. No queue. No background worker. No separate API service. No Docker, no Kubernetes, no nginx. wrangler deploy and the thing is running in 300+ cities.

The first time I shipped something like this, the part that surprised me wasn't the cost. It was that I stopped thinking about the infrastructure entirely. Two weeks in, I realized I hadn't logged into a cloud console once. The thing was just running. That doesn't sound like a feature until you've spent a Sunday afternoon debugging why your staging database can't reach your inference service because someone rotated a VPC peering credential.

This is the actual pitch. Not "it's cheap." It's that the smallest possible version of a RAG stack is now genuinely small.

---

How the RAG Loop Works

Every request follows the same six steps, all inside a single Worker invocation:

1. Embed the incoming message. Workers AI returns a 384-dimensional vector.
2. Vector search. Query Atlas with $vectorSearch, cosine similarity > 0.7, top 3 results. Returns relevant past messages.
3. Build the prompt. System message contains retrieved context; user message is the current question.
4. Generate the answer. Workers AI runs the LLM call (Llama 3.1 8B Instruct or similar).
5. Embed the reply for future retrieval.
6. Persist. insertMany writes both messages with their embeddings, role, session_id, and timestamp.

Steps 1, 4, and 5 all hit Workers AI — one inference endpoint, no API keys to rotate. Steps 2 and 6 hit Atlas — one connection string. The Worker orchestrates the sequence with no external service dependencies at all.

---

What It Actually Costs

Here is where most posts about this stack lose me, because they quote the marketing numbers without doing the math. Let me do the math.

Cloudflare Workers

The free tier is 100,000 requests/day with 10ms of CPU per invocation. That's plenty for the Worker code itself — orchestration, JSON parsing, calling Workers AI and Atlas, formatting the response. The CPU limit doesn't include time spent waiting on those calls, which is most of the request.

The paid plan is $5/month with 10 million requests and 30 million CPU-ms included. Overage is $0.30 per million requests and $0.02 per million CPU-ms. Default CPU per invocation rises to 30 seconds. It's "$5/month base", not "$5/month flat" — but in practice, unless you're doing something exotic, the base covers it.

Workers AI

The unit Cloudflare charges in is the Neuron. You get 10,000 Neurons/day free. Beyond that, $0.011 per 1,000.

Here's the part nobody quantifies properly. A single embedding call costs about 1–3 Neurons. A single LLM generation on a 7–8B model is 50–150, depending on how much you ask it to write. A full RAG turn — embed the question, retrieve, generate, embed the answer for storage — runs roughly 60–160 Neurons.

So on the free tier you get 60–150 full RAG turns per day. That is not "thousands of conversations." Let's not pretend. It's enough to validate that the thing works, to run a demo, to support a tiny user base. On the paid plan, 1,000 turns a day costs about $33/month in inference. That's the real number.

MongoDB Atlas

The Atlas free cluster gives you 512 MB of storage, supports Atlas Vector Search, and never expires.

About that 512 MB. A 384-dimensional float32 embedding is 1.5 KB raw, and you'll see posts claiming this means 300,000 stored messages. That number is fiction. What you actually store per record is: the embedding, the original text, metadata (user_id, session_id, timestamps), MongoDB's BSON overhead, and the HNSW index that makes vector search fast — which is comparable in size to the raw vectors themselves. The realistic number is 50,000–150,000 records depending on how heavy your metadata is. Still a lot, for free.

When you outgrow the free cluster, Flex picks up at $8/month and scales with usage up to about $30. Dedicated clusters start at M10 around $57/month, which is where things get serious.

What you actually pay

Stage	Volume	Monthly cost
Prototype	< 60 RAG turns/day	$0
Side project	< 1,000 turns/day	~$5 (Workers Paid, Neurons trickle)
Small product	< 10,000 turns/day	~$40 (Workers + Flex Atlas + inference)
Past that	scale-dependent	Time to plan, not panic

Honest comparison to the tutorial stack: at prototype scale, OpenAI embeddings plus a Pinecone Starter index also costs $0. The edge stack doesn't win on price there.

It wins on how many things you have to keep track of.

The real cost gap shows up the moment the tutorial stack needs a server to run the orchestration code. The moment you're paying Railway or Fly $20 a month to keep a Node service alive around the clock, you're paying for something the edge version doesn't need at all.

---

Where This Actually Shines

The AI assistant that remembers you. The original problem. A user comes back tomorrow and expects the system to know they prefer brief answers, work in TypeScript, and asked about React Server Components last week. This is the canonical case for vector memory, and the edge stack handles it without you ever provisioning anything.

Internal knowledge bases. Embed the company wiki, the runbooks, the design docs. Every Slack thread that ends with "wait, where did we decide this?" gets a better answer. The 512 MB free tier is more than enough for any single team's documentation; you'll hit the writes/sec ceiling before you hit storage.

Personalization at consumer scale, in theory. A recommendation feed, a "more like this" widget, a search-by-meaning box. Embeddings are the right primitive and Atlas Vector Search is fast enough at this size. The edge story is real here — you can serve users in Lisbon from Lisbon, not from us-east-1.

Side projects you actually want to ship. A team of 10 generating 500 queries a day stays inside the free tier indefinitely. The deployment story means a backend engineer can own the entire system without it becoming an on-call liability. This is a category I keep underestimating: things that don't get built because the operational overhead exceeds the energy budget. The edge stack changes that math.

---

Where It Doesn't Shine

This is the section most posts skip. Here's where the stack breaks.

When you need a top-tier model. Workers AI runs smaller open models like Llama 3.1 8B. They handle most everyday tasks well — answering questions from your docs, summarizing, classifying. But they're not as strong as GPT-5 or Claude on hard, multi-step problems. If your product needs the smartest possible answer, you'll want to call a frontier API for that part — which adds back one external dependency.

When 512 MB stops being a lot. It happens faster than you think. Ten thousand support tickets and you're already eating into the budget. Flex gets you to ~5 GB. After that, the next step is M10 at ~$57/month, with nothing in between. If you're going to index a real corpus this year, plan for M10 now.

When your data can't leave the building. Healthcare, finance, government, anything with strict data residency rules. Atlas is third-party. Workers AI runs on Cloudflare's GPUs. If that doesn't fit your compliance story, you need a different stack — self-hosted models, your own database, your own ops.

When you need sub-300ms responses. A full RAG turn here is 400–800ms: embedding, retrieval, generation, all sequential. Fine for chat. Too slow for products where perceived latency is the experience.

When you write a lot. Atlas Free caps at 100 ops/sec. Flex scales, but not without limit. An agent that persists embeddings every few seconds, or a bulk document ingestion job, will hit the write ceiling before the storage one.

---

The Alternatives, Briefly

I don't want to pretend this stack is the only reasonable choice. It isn't.

Pure Cloudflare. Cloudflare Vectorize plus D1 plus Workers AI keeps the entire request path inside one network. No external service. Vectorize is younger and less feature-rich than Atlas Vector Search — the query API is more limited, the metadata filtering less expressive — but the simplification is real. Worth considering if you have no existing MongoDB footprint and aren't worried about migrating out of the Cloudflare ecosystem later.

The default tutorial stack. OpenAI or Anthropic + Pinecone + Railway. Best DX in the industry, best model quality, most mature ecosystem. Costs more, has more moving parts, more places to hit a rate limit at the wrong moment. The right call when answer quality genuinely is the product.

Self-hosted. Ollama or vLLM on a VPS, pgvector for storage, your own service for orchestration. The right call when data sovereignty or fine-tuned inference is a non-negotiable. The wrong call when you reach for it because it feels more "real" — managed services are less work, and less work is the whole point.

---

Four Things to Add Next

The demo at github.com/Ajjya/cf-worker-mdb covers the full RAG loop end to end — clone it, deploy it, watch it work. Once it's running, here's where to take it next. These four upgrades turn the basic loop into something genuinely useful in production.

Scope retrieval to the user

Atlas Vector Search lets you filter by indexed fields during the vector query — not after. This is the killer feature of vector search inside MongoDB, and the cleanest way to make the same stack work for many users at once:

{
  $vectorSearch: {
    index: "messages_vec",
    path: "embedding",
    queryVector,
    filter: { user_id: userId, session_id: sessionId },
    numCandidates: 200,
    limit: 5
  }
}

One line in the index definition, one line in the query. The filter can be as rich as any MongoDB query — ranges, arrays, compound conditions. Each user's memories stay scoped to that user, and the similarity search only runs over what's actually relevant. Most dedicated vector databases need namespace gymnastics for this. MongoDB just lets you write the filter.

Combine vector memory with a sliding window

Vector search is associative — it finds semantically similar past messages. It's less good at "what did the user just say two turns ago." For that, always inject the last N turns of the current session into the prompt directly, alongside whatever retrieval returns.

Two layers, each doing what it's good at. The window handles short-term coherence — the conversation you're in the middle of. The vector search handles long-term memory — things that came up days ago. Together they feel continuous. Either one alone feels like the model has amnesia.

Extract facts before you store them

Storing raw messages and searching them works, but it competes with itself. A user says "my name is Anna" once. A hundred turns later, that one message has to win a similarity race against everything else they've ever said. It usually loses.

The upgrade: before persisting a message, run a small LLM call that extracts structured facts into a separate collection:

{ "property": "name", "value": "Anna" }

Retrieve facts with an exact filter, not a similarity search. Precise facts shouldn't have to compete with semantic noise. It costs one extra LLM call per user message, and it's the difference between an assistant that remembers you and an assistant that hopes it gets lucky.

Index hygiene from day one

On 512 MB you can't keep every message forever, so build the cleanup in from the start. Set a TTL index on timestamp. Periodically deduplicate near-identical embeddings (cosine similarity > 0.95, keep the highest-signal one). This keeps retrieval quality steady as the collection grows, instead of degrading silently in the background.

Each of these is a small addition. Together they're what takes the stack from "neat demo" to "memory layer I can actually build on."

---

The Underlying Point

The interesting thing about this stack isn't that it's cheap. Cheap is a side effect.

The interesting thing is that it inverts what the tutorial ecosystem trained us to expect. The default assumption — for years now — has been that doing RAG properly means standing up infrastructure: a vector DB service, an inference service, an orchestration service, the connective tissue between them. That assumption made sense in 2023. It doesn't, anymore, for a large class of products.

Workers AI exists. Atlas Vector Search on the free tier exists. The embedding models are good enough. The open-weight generation models are good enough for everything except the cases where they aren't, and those cases are narrower than the marketing of frontier providers suggests. A lot of what people are building today is "the AI remembers things and answers questions about them" — and for that, the small version of the stack is not the compromised version. It's the appropriate version.

If you're about to start a new AI side project, or you're about to provision your seventh service to support a chatbot, this is the question worth sitting with: do you actually need the larger stack, or did you reach for it because that's what the tutorial showed you? Most of the time, in my experience, the answer is the second one — and most of the time, the smaller version would have worked, faster, with less to break.