Skip to main content

ABAC in Production: The Migration — DaemonSet, a Node-Local Attribute Cache, and Policy Delivery from Git

· 8 min read
Vladyslava Prykhodko
Vladyslava Prykhodko
Engineering Technical Lead & Architect

Part 2 showed that OPA/OPAL sidecars hold down about a third of the cluster and, on top of that, force you to under-provision headroom. This part is about the migration itself: how to pull the PDP out of every pod without losing data freshness or hitting a latency wall.

The properties of externalized authorization don't change — single policy plane, no drift, tamper-resistant audit log. What changes is where the PDP lives and how policy and data reach it.

Everything below is illustrative and generalized — a reference model, not data or code from any specific system. Substitute your own.

Where we're moving from, and to​

In the sidecar topology every pod has its own OPA, its own OPAL client, and its own copy of the data. That's expensive (a multiplier on the pod count) and fragile (reservations get cut to the bone, because every extra milliCPU is multiplied by the number of pods). The goal of the migration is to keep exactly the same properties of externalized authorization, but to:

  • compute decisions in one OPA per node (DaemonSet) rather than in every pod;
  • distribute policy centrally from Git via a control plane;
  • distribute attributes through a node-local cache fed by an event stream, rather than pushed into every OPA's memory.

Three flows — decision traffic, policy, and attributes — are separated across different mechanisms. Here's what the runtime looks like on a single node after the migration:

DaemonSet Authorization — one shared PDP per node, policy and attributes arrive on separate paths

OPA as a DaemonSet​

Instead of a sidecar in every pod, one OPA per node deployed as a DaemonSet. Pods on a node send their ext-authz request to the node-local PDP. The instance count becomes the number of nodes — dozens instead of hundreds — and that's exactly where the savings and cheap headroom from Part 2 come from.

There's a trade-off worth stating explicitly. Pods now find OPA not over localhost but via the node's IP, through the Kubernetes Downward API (status.hostIP). This slightly changes the network boundary compared to a standard localhost sidecar: the ext-authz call now travels over the node's internal network rather than staying inside the pod.

Policy delivery: a control plane from Git​

Policy lives in Git — that's the PAP (Policy Administration Point) from Part 1's vocabulary. A separate component, the control plane, turns it into something OPA can consume:

  1. a change in the policy repo is detected (a push event from Git via mirror/webhook);
  2. the control plane builds a policy bundle and uploads it to object storage;
  3. each node-local OPA periodically polls the storage and pulls the fresh bundle — OPA's native Bundle API.

The whole chain is triggered by a push event from Git, but the policy reaches OPA itself via a timed pull. This replaces the sidecar setup's OPAL server with a simpler pull mechanism: no persistent WebSocket connections from hundreds of clients, just a bundle store and periodic polling from dozens of OPAs. The polling lag (seconds to tens of seconds) is fine for policy — it changes rarely, unlike attributes.

One thing to note: policy is per-node but identical on every node — it's a single centrally built version, they can't drift apart. The no-drift property from Part 1 is preserved and even strengthened: one source, one build.

Attribute delivery: the hard part​

Policy is static and changes rarely — distributing it is easy. Attributes (the PIP, Policy Information Point) change constantly and must be fresh at decision time. That's the non-trivial part.

The naive path is to sync the entire attribute dataset into every OPA's memory — what OPAL did in the sidecar setup. Per node, that means holding a full copy of the data in each instance. We don't want that: the data is smeared across instances, updating each is a separate operation, and memory grows as "dataset size × instance count."

Instead — a node-local cache (Valkey/Redis on every node), and OPA reads only the slice it needs at decision time. There's an important subtlety: OPA can't read RESP/Valkey natively during eval — Rego runs against data in memory. So the mechanics are specifically these:

  • a lightweight HTTP shim sits on the node, serving attributes over Valkey;
  • at decision time Rego makes an http.send to that shim for the subject's and resource's attributes;
  • since both the shim and Valkey are node-local, the call never leaves the node and costs a fraction of a millisecond, with no cross-AZ tail.

To avoid hitting the cache on every decision, OPA caches the http.send response itself with a short TTL (cache: true). It's a deliberate trade-off between freshness and latency: a TTL of seconds means an attribute change is visible almost immediately, while repeated decisions for the same subject within the TTL come from OPA's memory. In effect it's a hybrid of "pull at decision time" and "a small working set in memory" — with no full copy of the dataset anywhere.

Why a message broker for attributes​

That leaves one question: how does the cache on each node learn about attribute changes? The direct path — having the attribute-owning service write to the caches directly — is tight coupling between the data-owning team and the authorization platform. The attribute owner shouldn't even know that someone else's caches exist.

So between them sits a message broker. The owning service publishes the fact of a change to a topic, and its responsibility ends there. A separate syncer (owned by the platform team) reads the topic and updates the node-local caches. What this buys:

  • Team decoupling — the attribute owner doesn't call anyone else's systems, it only publishes an event.
  • Ordering — within a partition, changes are ordered.
  • Replay — a cache can be rebuilt by re-reading the log from a given offset, e.g. after a node restart.
  • Fan-out — many consumers (one per node) without the producer knowing.

This is exactly the boundary the sidecar setup blurred: there, data was simply pushed into OPA, and who put it there was fused with the delivery mechanism. Here the source, the broker, and the consumer are separated.

Being honest: on a small dataset this is about decoupling, not memory​

A node-local cache is often pitched as a way to save memory. On a small dataset (hundreds of thousands of records with a dozen attributes each — that's megabytes) it doesn't work that way: moving the data out of OPA's memory barely shows up in the bill. The bulk of the savings comes from the sidecar → DaemonSet move itself.

The real value of the cache + broker here is operational: attributes update without rebuilding bundles or redeploying OPA, and the data-owning team is decoupled from the authorization platform. Memory only becomes the primary argument on a large dataset — that's when "full copy × instance count" hits RAM, and the cache pays off on that front too.

The spectrum, end to end​

The three parts of this series are one spectrum — "where the decision is computed":

  • Part 1 — guards. PDP in-process. Cheap, simple, correct for a single language and domain-local rules. This is where most teams belong.
  • Part 2 — OPA/OPAL sidecars. PDP externalized, but multiplied across every pod. Delivers the properties you want — a single policy plane, no drift, audit — at the cost of a third of the cluster and forcibly thin headroom.
  • Part 3 — DaemonSet + node-local cache + broker. The same properties, but the PDP is multiplied per node rather than per pod; policy and attributes travel different paths. A third of the cluster comes back, headroom gets cheap, and the boundaries between teams become explicit.

Two things to take away. Externalizing authorization is worth it not because of scale itself, but because of fleet heterogeneity and governance requirements: a single language and autonomous teams are perfectly well served by guards, no matter how many services there are. But once the reasons to externalize are real, topology matters most: the default sidecar setup delivers the properties at the cost of a third of the cluster, whereas a DaemonSet with policy and attribute flows split apart delivers exactly the same properties cheaper and more resiliently.

Most teams end up with sidecars because that's what the quickstart ships. The decision to keep them — or change them — is worth making on measured data, not on defaults.

The spectrum exists so you can stop at the right point — not so you have to reach the end of it.