The Request You Made 100 Times

A backend-for-frontend layer is one of the easier architectural decisions to get right. Each client gets a service that speaks its language, the domain services stay clean, and every team owns a box with a clear border drawn around it. An architect reviews the diagram and approves it — and the diagram is correct.

This is a field story about what the diagram doesn't show. Four correctly-written service calls, each behaving exactly as designed, turned one screen render into roughly a thousand queries against a sharded collection. The screen stopped loading, one collection's query time dwarfed every other collection in the cluster, and the ticket said "MongoDB is slow."

MongoDB was not slow. Nobody had ever counted what one user action actually costs.

Start with what the team actually saw

Before any explanation, here is what was on the dashboards:

• 1.5 days of database time per second on one collection — the time all its queries spent running, added up, for every second of wall clock. The next-worst collection in the same cluster: 54 seconds.
• 8.8 s average read latency on that same collection — how long a single read actually took. Its neighbours: 8 to 300 milliseconds.
• A list screen returning zero rows and spinning. It is the entry point to the whole product.
• 460 million accesses on _id_ against 1.5 million on the index the collection is sharded by. Three hundred to one.
• 53 indexes, 1.9 GB on one collection. The list query scans 405,000 keys to return one page.
• The webhook database scaled a tier up, roughly double a month — for events about changes that never happened.

Every collection on that chart lives in the same cluster and serves the same product. Four of them idle. One does not.

Not one of these is a cause. Each is a symptom, and — this is the part worth sitting with — each belongs to a different team's area of responsibility. That is why the incident survived several rounds of investigation: everybody looked inside their own box and found it healthy.

The same question, a hundred times

The first root cause is the one nobody suspects, because no single piece of code looks wrong: the same document was being read over and over inside one user request.

Not in a loop. Nothing as visible as a loop. A BFF composes a screen out of many domain calls, and the composition grew the way compositions grow — one feature at a time. The inbox list fetches an account per row to render a name. An authorization check fetches the same account to evaluate access. A mapper fetches it again to resolve a reference. A permission badge fetches it once more. Every one of those calls is cheap, correct, and written by someone who had no reason to think anyone else was asking.

A simple log analysis made it visible: count how many times the service reads an account, then count how many distinct accounts it actually needed. The first number climbs through the working day. The second stays flat along the floor. At peak, there are around a hundred reads for every distinct document anyone wanted. A hundred to one.

This failure mode is specific to a layer whose job is composition. Inside a single domain service, repeated reads of the same entity in one request are obvious during review — the code sits on one screen. Across a BFF, the repetition is spread over separate handlers, guards, interceptors and mappers, each with its own owner and its own unit tests. The duplication exists only in the sum, and the sum is nobody's file.

Every duplicate costs ten times what it looks like

The second cause multiplies the first. A hundred redundant reads would be unpleasant anywhere. On a sharded collection queried without its shard key, each of them is ten reads.

The collection is sharded by location + key. The read path is GET /accounts/:id, which runs findOne({ _id }). The _id is unique and the query is instant on any single shard — but the router has no idea which shard owns it, so it asks all of them and waits for the slowest.

// No shard key → mongos broadcasts to all 10 shards
db.accounts.findOne({ _id })

// Shard key present → mongos routes to exactly one
db.accounts.findOne({ location, key, _id })

The index statistics state the problem without ambiguity: 460 million accesses on _id_, against 1.5 million on location_1_key_1. On a collection that was deliberately distributed by location, the location index is used three hundred times less often than the one that defeats the distribution.

The writes do the same thing. Account edits, status changes, preference updates and the activity auto-update all filter by _id alone, so every one of them is a broadcast write. The irritating part: each of those methods already loads the document before writing it. location and key are sitting in memory, two fields away from the filter they are missing.

Multiply it out. One screen render asks for an account about a hundred times; each ask fans out across ten shards; that is on the order of a thousand shard-level queries for a single user action — before anything is written, and before the indexes make it worse.

Nobody wrote a loop. The hundred and the ten only exist once the four callers are added up — which is something no single file shows.

53 indexes nobody felt safe deleting

The third cause accumulated over years, one incident at a time. Adding an index at 2 a.m. feels free and fixes the page you are staring at. Deleting one feels like gambling with a query you have never read. Multiply by several years and several teams: 53 indexes on one collection, 1.9 GB of index data.

Indexes are not free storage. Every insert and every update maintains all of them, on every shard, inside the write lock. They compete for the same cache the documents need. And a cluttered index set makes the planner's job harder, not easier — with 53 candidates, the one it picks is often not the one you would have picked.

The list query behind the main screen was scanning 405,000 index keys to return a single page. Partly because no index matched its access pattern, and partly because of a collation mismatch: the query asked for a collation the index wasn't built with, which silently disqualifies the index and sends the query to a scan. Nothing in the application logs says so. You only see it in explain().

Rebuilding the set around the ESR rule — Equality fields first, then Sort, then Range — changed all three numbers at once:

	Before	After
Indexes on the collection	53	11
Index data	1.9 GB	~135 MB
Keys scanned by the list query	405,000	10

No new hardware, no new technology, no rewrite. The same data, asked for properly.

Events about nothing

The fourth cause leaves the database entirely and still gets paid for monthly.

Every activity written against an account triggered an auto-update on the account document itself. The handler bumped updatedAt, saved the document, and published an account.modified event carrying the full before and after documents — without ever checking whether a single field had changed. Most of the time, none had.

Three bills arrive for that one habit. A broadcast write on a sharded collection, because the auto-update filters by _id like everything else. A webhook payload containing two complete account documents instead of a diff. And a message retained in the webhook service's own database until every consumer has taken it.

Those events filled the webhook database fast enough that the instance had to be scaled a tier up — roughly double the monthly cost — to deliver notifications about changes that never happened. Downstream, every consumer wakes up, deserializes two documents, compares them, finds nothing, and goes back to sleep. That cost does not appear on anyone's dashboard at all.

updatedAt is the tell. A timestamp that changes on every touch rather than on every change turns "something happened near this document" into "this document changed", and every system downstream believes it.

The architecture was approved. The interactions weren't.

Here is the uncomfortable part: the architecture is good. Services are split by domain. A BFF per client keeps presentation logic out of the domain services. You can name the owner of any endpoint in a second. An architect reviewed this and approved it, and approving it was the right call.

A design review asks about boundaries. Does this service own the right data? Does it depend on the right neighbours? It does not ask about traffic: how many times, to draw one screen, does this system ask the same question?

Nothing answers that question, because a service boundary is built to hide it. Encapsulation means a caller does not know what a call costs. That is the point of it, and also the problem with it.

So four callers, each correct on its own and unaware of the other three, add up to a hundredfold read amplification. It exists in no file and in no service. It lives in the gaps between the boxes, and nobody's name is written in a gap.

Every per-service dashboard agreed. The BFF was fast, because its calls were fast. The domain service was fast per query, because each query was small. Only a view of the collection itself — total database time instead of per-service time — showed what was actually happening.

The failures underneath are organizational before they are technical:

• Components were reviewed alone. The composition was never reviewed at all, and no document in the project described it.
• The shard key was treated as a storage detail, not as part of the API. Once a collection is sharded, a query without the key is a design error.
• Indexes accumulated under incident pressure, and nobody owned removing them. Adding one was a fix; deleting one was a risk.
• Events were published on touch, not on change, which moves your write amplification onto other teams' budgets.
• Monitoring followed the org chart, service by service, instead of following one user action across all of them.

Correct boxes and correct boundaries do not add up to a correct system. The place where they fail is composition — and composition is the one surface no single team owns.

How to avoid the whole thing

• Measure reads per user action, not per service. One trace id through one screen render, then count how many times each document was fetched. If that number surprises you, it has been surprising you for months.
• De-duplicate inside the request before you cache across requests. A request-scoped cache or a DataLoader-style batch in the BFF removes the hundredfold amplification in an afternoon, with no invalidation strategy to get wrong. Distributed caching is the next conversation, not the first one.
• Make the shard key part of the repository API. If a collection is sharded, don't expose a method that takes only an id. Take location + key + id, and let the type system refuse the broadcast. Write paths already have both fields in hand.
• Confirm targeting with explain() on the hot paths — a healthy query touches one shard; a broken one fans out across all of them — and check that your collation matches the index's.
• Budget your indexes. Review $indexStats on a schedule, drop what nothing uses, order the survivors by ESR, and treat 50 indexes on one collection as the incident it is.
• Publish events on change, not on touch. Diff before you emit, send what changed rather than two full documents, and separate "this document was written" from "this document is different".
• Review the interaction, not only the boundary. Add one question to every design review: what does one user action cost in reads, writes and events? It takes ten minutes, and it is the only question that catches this class of bug before production does.

The diagram was never wrong. It just described responsibilities, and responsibilities don't add up to traffic. The most expensive line of code in this system was the cheapest one to write: a correct, well-named, perfectly reasonable call to fetch an account that somebody else had already fetched.