6. Compliance Infrastructure: Audit Trails, Policy-as-Code, and the Append-Only Principle

The Infrastructure That Looked Right

There’s a category of gap that’s almost impossible to catch by reading code. The class exists, the method gets called, the schema is well designed — and the actual compliance behaviour just never happens.

We found three of these in a single audit session, weeks after each control had been signed off as “implemented.” Our AuditLogger was wired into all five agent services and wrote to Loki only — the audit_events Postgres table had a clean schema and zero rows. Our OPA policy was a static .rego file mounted read-only, so changing a threshold meant rebuilding the image, with no record of what changed or when. And our configuration tables used plain SQL UPDATE — when the Sharia agent’s autonomy tier moved from SUPERVISED to FULL, the previous value simply disappeared.

None of this showed up in review — dashboards showed activity, the system looked governed. It took an auditor’s questions to show it wasn’t. What follows is how we fixed all three: the transactional outbox, a policy bundle server, and temporal table design.

Back to top . Next: Building a Real Audit Trail →

---

Building a Real Audit Trail

Confusing a log line with an audit event is exactly what created this gap, so it’s worth being precise about the difference.

A log line is operational — useful for debugging, ephemeral by design, and Loki keeps ours for 30 days. An audit event is a compliance record that has to answer a regulatory question years later: which agent made this credit decision, what fatwa supported that Sharia ruling. That needs records that are immutable, timestamped, attributed, and queryable long after the fact.

A well-formed audit event carries a controlled event type (SHARIA_CHECK_PASSED, not free text), an agent ID, an actor that’s never null for human actions, a task ID tying it to the decision trace, and an immutable timestamp.

Two Gaps, Stacked

We found two problems here, stacked on top of each other. AuditLogger.record() called structlog.info() and returned — the INSERT INTO audit_events that should have followed simply never got written, so for weeks the dashboards showed activity while the table sat at zero rows. Adding a fire-and-forget _persist_to_db() call fixed that, but it exposed the second problem: downstream systems — a SIEM, OpenMetadata, a reporting dashboard — also needed these events on a Redpanda topic, and writing to Postgres then publishing separately means anything between the two calls can fail silently.

The Transactional Outbox

The fix is a standard distributed-systems pattern: a second table, the outbox, in the same database as audit_events — so the write is one local Postgres transaction, no external coordinator needed.

• Atomic write: one transaction writes to audit_events and audit_outbox together — both succeed or both fail.
• Background polling: a poller runs every two seconds, publishes unpublished rows to Redpanda’s audit.events topic, marks them published.
• Guaranteed delivery: if the poller crashes mid-publish, the row stays unpublished and goes out next cycle. If it’s in Postgres, it eventually reaches the stream.

Back to top . Next: Policy-as-Code, Done Right →

---

Policy-as-Code, Done Right

When we first built the authorization layer, we did what most engineers do: wrote an OPA policy file, mounted it into the container, and enforced it at the one point we assumed mattered — the credit agent. Four problems in that setup only surfaced at audit.

• No hot reload. Changing a threshold meant editing the file, rebuilding the image, and restarting OPA — so thresholds mostly didn’t get updated.
• No audit trail of policy changes. Git shows when the file changed, not when the container restarted with it.
• Enforcement was partial. Only the credit agent called OPA — Sharia, Risk, Simulation, and the Orchestrator didn’t, so a request could reach the Sharia agent with no authorization check at all.
• Policy and code were coupled. Updating policy required a deployment, so changes shipped bundled with agent code even when the logic hadn’t changed.

Worth naming here: Rego policies default to allow := false, untouchable in our codebase, since default-allow can quietly turn a missing rule into a silent grant. Each agent gets back more than allow or deny — whether HITL is required, whether a Sharia evaluation must run first, whether the tier should be elevated — and Python never makes that call; OPA does.

The Bundle Server

The fix: package policies as a compressed bundle, upload to an object store, and have OPA pull it on a poll instead of mounting a file. We use MinIO, self-hosted and S3-compatible; OPA polls every 30 seconds and activates any bundle with a newer ETag — no restart, no redeploy.

Each bundle holds two files — authz.rego for the rules, data.json for the tier map — packaged for atomic activation by a shell script, bundle.sh, which also checks that OpenBao is reachable and the deploy token valid. An early version had a fail-open bug: an unreachable OpenBao made the token check return an empty string that was accidentally truthy, so the script deployed anyway. We now treat any validation failure as fatal.

Back to top . Next: Dual-Control for Policy Changes →

---

Dual-Control for Policy Changes

Hot reload solves the operational problem, but it opens a new one: if anyone with access to bundle.sh can deploy a policy instantly, the control can be bypassed faster than any human can intervene. Speed without accountability isn’t an improvement here.

The first control is GitOps: authz.rego lives in version control, changes need a pull request, and CODEOWNERS requires both the compliance officer and the security lead to review — neither can approve their own change. That doesn’t cover deployment, though. Merging a change and running bundle.sh are two different actions, and the same developer could do both unsupervised.

So the second control is a governed deploy endpoint, gated behind an admin key separate from the normal agent API key — stored in OpenBao, rotated quarterly, logged on every use. Merging a policy change doesn’t let you activate it. Every deployment also writes a row to opa_policy_versions — who deployed it, why, and an effective_from / effective_to pair — so “what policy was active at time T” is always answerable. For an Islamic finance platform, this gives the Sharia Board a real, blocking role in technical governance: rules about which products need Sharia pre-approval go through the same CODEOWNERS review as any other policy change.

Back to top . Next: Why UPDATE Fails Compliance →

---

Why UPDATE Fails Compliance

Every engineer who has built databases professionally has the same reflex: when data changes, you update the row. That reflex is correct for most systems and a liability in regulated AI.

Here’s the scenario. The Sharia agent starts at SUPERVISED. Three weeks later you promote it to FULL, and the instinct is UPDATE agent_registry SET autonomy_tier = 'FULL' WHERE agent_name = 'agent-sharia'. That’s correct, right up until a CBUAE examiner asks what tier the agent was operating under for a transaction approved on 14 March at 09:47. You can’t answer — the row says FULL because that’s what it is now, and what it was on 14 March no longer exists. The UPDATE erased it.

SR 11-7, the CBUAE Model Risk Guidelines, and AAOIFI Governance Standard 6 all require exactly that, each in its own language. If you ran UPDATE, you simply can’t.

The Fix: Soft-Close and Insert

Every row carries effective_from and effective_to (NULL means current). One transaction does two things: soft-close the current row (SET effective_to = now() WHERE effective_to IS NULL — the only UPDATE ever permitted, and it only stamps an end date), then insert a new row with the new tier, a fresh effective_from, and governance metadata — who changed it, who approved it, and why.

agent_name	autonomy_tier	effective_from	effective_to	tier_changed_by
agent-sharia	SUPERVISED	2026-02-01 08:00:00+00	2026-03-14 11:22:00+00	NULL (self-registered)
agent-sharia	FULL	2026-03-14 11:22:00+00	NULL (active)	alice@bank.ae

Querying at 09:47 with effective_from <= T AND (effective_to IS NULL OR effective_to > T) returns SUPERVISED — the promotion happened at 11:22, after the decision. Soft-close and insert, never delete or overwrite: the table only grows, and it’s queryable as its own audit trail.

The Tables That Live By This Rule

Not everything needs this — caches and session state can be overwritten freely. Four tables govern decisions, policies, models, and audit events, and all four are append-only:

• agent_model_registry — which model version ran for which agent, when. Survives a rollback.
• agent_registry — tier history with the four-eyes metadata above; an env-var restart can’t silently override a human-set tier.
• opa_policy_versions — the exact Rego rules active at any past timestamp, not reconstructed from Git blame.
• audit_outbox — a delivery queue, but the same rule applies: the only permitted change after insert is marking a row published.

Back to top . Next: Point-in-Time Reconstruction →

---

⏱️ Point-in-Time Reconstruction

A customer complains about a Murabaha offer from 14 March 2026, 09:47 UTC, claiming the profit rate was unfair. The regulator wants the full picture — which model ran, under which policy, at which agent tier, with what audit trail. With temporal tables and the outbox in place, that’s four queries against the same timestamp.

• Model version: agent_model_registry at 09:47 → v1.2, approved by Alice on 1 February.
• Autonomy tier: agent_registry, same timestamp → SUPERVISED, the tier before that morning’s promotion.
• Active policy: opa_policy_versions → bundle authz/bundles/2026-02-15.tar.gz, downloadable and inspectable.
• Audit trail: audit_events for that task ID → the full sequence, timestamped and attributed.

Without temporal design, some of those queries would return today’s state instead of March’s, and the reconstruction would just be wrong. The same tables pay off outside audits too — compliance officers can query them directly instead of opening a ticket, and root-cause work gets faster.

Put together, a single credit decision now produces a connected chain, linked by one task ID: a decision trace, the audit events in the outbox stream, the Sharia decision detail with fatwas cited, the full Langfuse trace of every LLM call, and the exact policy and model context.

Compliance Infrastructure Stack

Back to top . Next: Key Takeaways →

---

---