6 Governing with Intelligence: OPA Policies and Agent-Driven Compliance

---

⚠️ The Problem With Compliance-as-Code

A senior engineer writes validate_compliance(transaction). It works until a threshold changes, the number is updated inline, the PR merges, and nobody notices until audit time.

The real problem is that the rule lives in application code — hard to audit, easy to bypass, and slow to change. OPA fixes the data layer. Financial agents still need their own governance layer: rate limits, execution boundaries, and failure recovery.

Open Policy Agent (OPA) moves data rules out of application logic into tested, versioned policy. Agents still need governance: execution gates, rate limits, and failure recovery applied consistently across the platform — the layers described below.

↑ Back to top · Next: Layer 1: OPA Enforces Data Access →

---

⚖️ Layer 1: OPA Enforces Data Access

OPA runs as a sidecar service. The pipeline’s OPAPolicyEnforcementStage evaluates every write against a Rego policy before data reaches the database.

The Rego policy in infra/opa/pipeline_policy.rego uses role-based access control. A write is allowed only when the requester has the right role and passes four checks:

default allow = false

allow {
    role_permissions[input.identity][_] == input.action
    check_data_classification        # sensitive datasets → compliance/auditor only
    check_compliance                 # domain-specific transaction rules
    check_retention_policy           # 7 years for staged_data, 10 for audit_logs
    check_audit_requirements         # write operations must be audit_logged
}

The key rule is default allow = false. Policy errors, missing rules, or odd input shapes return deny, not allow. A bug stops the pipeline instead of opening data.

📅 Retention Baked Into Policy, Not Code

One of the more powerful aspects of this setup is that retention requirements live in the Rego file, not in the pipeline application:

check_retention_policy {
    input.resource.name == "staged_data"
    input.context.retention_days >= 2555   # 7 years — SOX Section 802
}

When the rule changes, the Rego file is updated, tested with opa test, and deployed. The application code does not change. That is policy-as-code in practice.

Next: apply consistent governance to agent behaviour across the platform.

↑ Back to top · Next: Layer 2: The Governance Middleware Decorator →

---

🔒 Layer 2: The Governance Middleware Decorator

With data access controlled, we must also control agent behaviour. How do we prevent runaway agents or resource exhaustion? Enter the @governed_execution decorator in libs/agents/governance_middleware.py:

class ComplianceAuditAgent:
    @governed_execution("compliance_audit")
    def process(self, state: AgentState) -> AgentState:
        # agent logic here
        ...

Before process() runs, the decorator checks permission and, afterwards, records the result. This keeps domain code simple and governance centralised.

↑ Back to top · Next: Layer 3: Circuit Breakers and Rate Limits →

---

🔁 Layer 3: Circuit Breakers and Rate Limits

The AgentGovernanceAgent maintains an AgentGovernancePolicy for each registered agent. The policy includes two protective mechanisms that operate automatically.

Rate limiting blocks an agent once it exceeds max_invocations_per_hour in a rolling window, and circuit breaking opens after repeated failures and disables the agent for a cooldown period.

Normal mode → N failures → Circuit OPEN (agent disabled) → cooldown → Circuit HALF-OPEN → probe → success → Circuit CLOSED

Without this, a failing agent can spam logs and alerts indefinitely. The breaker turns that sustained noise into one clear signal: this agent is disabled for now.

↑ Back to top · Next: Layer 4: Specialised Governance Agents →

---

🤖 Layer 4: Specialised Governance Agents

Three dedicated agents handle concerns that OPA and the middleware cannot address because they require reasoning, not just rule evaluation.

🔍 AuditAgent — Anomaly Detection in the Audit Trail

The AuditAgent in libs/agents/audit_agent.py watches for patterns individual records cannot reveal:

• Repeated failures — three or more failures within the last ten operations from the same actor suggests a misconfiguration or an attempted probe
• Unusual access patterns — sudden access to sensitive resources from an identity that does not normally touch them
• Rate limit violations — access frequency anomalies that policy alone did not catch

When suspicious activity is detected, the agent triggers the alert engine with severity based on the anomaly score. A score above 0.9 fires CRITICAL; above 0.75, SEVERE.

The AuditAgent also produces on-demand reports for operations, success rate, policy violations, and compliance rate.

📋 PolicyEnforcementAgent — Organisational Policies

Where OPA enforces data-level access rules, the PolicyEnforcementAgent enforces organisational policies across systems: data modification requires approval, expensive agents are scheduled, and API call counts stay within limits.

It manages four policy domains and records each violation with a timestamp and actor.

✅ ValidationAgent — Data Integrity Before It Enters the System

The ValidationAgent applies checks before data reaches OPA. It validates query inputs, checks financial data ranges, and detects structural anomalies such as oversized state or missing critical fields.

In strict mode, validation failures block execution entirely. In standard mode, they are recorded and the pipeline continues with degraded confidence.

↑ Back to top · Next: MCP Server: Governance as an API →

---

🌐 MCP Server: Governance as an API

All of the above is accessible through the MCP server’s governance routes (apps/mcp_server/routes/governance_routes.py). The endpoints expose governance state to the operational dashboard and to downstream tooling:

• POST /api/governance/policy-enforcement/check — check whether an operation is compliant before executing it
• GET /api/governance/audit/trail — filtered audit log by actor or event type
• GET /api/governance/audit/suspicious — current anomaly list with severity scores
• GET /api/governance/meta-governance/agent-status/{agent_name} — execution mode, success rate, circuit breaker state
• POST /api/governance/meta-governance/update-policy — dynamically update an agent’s governance policy

The last endpoint is worth calling out. An operator can update an agent’s rate limit, failure threshold, or execution mode at runtime — without a code change or deployment.

↑ Back to top · Next: Compliance Framework Mapping →

---

🗺️ Compliance Framework Mapping

Every Rego rule traces to a specific regulatory obligation. Maintaining this mapping is what turns the policy file into audit evidence rather than just configuration:

OPA Rule	Regulatory Obligation	Framework
default allow = false	Least-privilege access to personal data	GDPR Article 25
retention_days >= 2555	7-year minimum for financial records	SOX Section 802
audit_logged == true	Complete audit trail for all writes	SOX Section 404
check_data_classification	Sensitive data restricted to authorised roles	MiFID II, GDPR Article 9
EDD on high-value transactions	Enhanced due diligence above thresholds	FATF Recommendation 10, Basel III/IV

This table is maintained as inline comments in the Rego file and as a standalone compliance matrix. During audits, both are provided to show each requirement is implemented, machine-testable, and traceable to policy code.

↑ Back to top · Next: DAMA DMBOK Alignment →

---

📚 DAMA DMBOK Alignment

DAMA DMBOK defines eleven data management knowledge areas. This governance architecture directly addresses four of them:

Data Governance (KA 1): OPA policies are executable, version-controlled controls; CI/CD tests enforce governance changes.

Data Security (KA 7): OPA, execution gates, circuit breakers, and AuditAgent create layered defence.

Data Quality (KA 6): The ValidationAgent checks structure and domain rules before data enters the pipeline.

Metadata Management (KA 12): OpenMetadata keeps schema, lineage, quality, and ownership current after each run.

↑ Back to top · Next: The Decision Log Is the Audit Evidence →

---

🗂️ The Decision Log Is the Audit Evidence

Every OPA evaluation writes a structured log entry — decision ID, identity, action, result, timestamp. That log is the audit evidence for each batch run.

Not a narrative. Not a process document. A record of every allow and deny decision the policy engine made, queryable like any other structured log.

↑ Back to top · Next: Key Takeaways →

---