11. Zero-Trust Data Governance: Security Architecture for Agentic Platforms

---

⚠️ Why Traditional Security Falls Short

If you are a data engineer or security architect building a platform where AI agents make autonomous decisions against sensitive financial data — this section explains exactly why the perimeter security model fails for agentic systems. You will understand the two unique threat vectors agentic systems introduce, why zero-trust is the only viable architecture, and what that means concretely in terms of Kubernetes RBAC, OPA policy, and PostgreSQL role design.

Agentic platforms break the perimeter model in two ways:

Agents are inside the network by definition. An agent running in a Kubernetes pod has network access to PostgreSQL, Kafka, MinIO, and OPA. If the agent is compromised — through prompt injection, model poisoning, or a logic error in the workflow — it has full access to all these services unless explicit controls restrict it.

AI decision-making is opaque. Traditional software does exactly what it is programmed to do. An LLM-backed agent makes decisions based on patterns learned from training data, which are not fully interpretable. An agent that appears to behave correctly in testing may behave unexpectedly when presented with novel inputs or adversarial prompts.

Zero-trust security addresses both challenges: every service call is authenticated and authorised regardless of its origin, and agent capabilities are bounded by explicit policy controls that the agent itself cannot override.

↑ Back to top · Next: Zero-Trust Principles in Practice →

---

🛡️ Zero-Trust Principles in Practice

Zero-trust is not a product — it is an architecture principle with four core tenets:

• 👁️ Never trust, always verify. Every request — whether from a user, an application, or an agent — must be authenticated and authorised before access is granted. A pipeline pod running inside the cluster must present valid credentials to query the database; its network location alone is not sufficient.
• ⚙️ Least privilege access. Every identity (user, service account, agent) is granted only the permissions it needs for its specific function. The pipeline service account can read and write the staged_data table. It cannot read the audit_log table. It cannot drop tables.
• ⚡ Assume breach. Security controls are designed on the assumption that a component will eventually be compromised. The goal is to limit the blast radius: “if an agent is compromised, what is the maximum damage it can cause?”
• 🔍 Explicit verification. Access decisions are made based on identity, device health, data classification, and context — not just network location. OPA is the policy enforcement point for this explicit verification.

↑ Back to top · Next: Authentication and Authorisation →

---

🔑 Authentication and Authorisation

🏛️ Service-to-Service Authentication

Each platform component has a Kubernetes ServiceAccount with a corresponding RBAC role:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: pipeline-sa
  namespace: agentic-pfm
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pipeline-role
  namespace: agentic-pfm
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["pipeline-secrets"]
    verbs: ["get"]
  - apiGroups: [""]
    resources: ["configmaps"]
    resourceNames: ["pipeline-config"]
    verbs: ["get", "list"]

The pipeline ServiceAccount can read its own secrets and config maps. It cannot list all secrets in the namespace (which would expose other services’ credentials). This is least privilege at the Kubernetes RBAC layer.

For database access, each service has a dedicated PostgreSQL role with table-level permissions:

-- Pipeline service: can write staged_data, cannot read audit_log
CREATE ROLE pipeline_service;
GRANT INSERT, SELECT ON staged_data TO pipeline_service;

-- Compliance service: can read both tables, cannot write
CREATE ROLE compliance_service;
GRANT SELECT ON staged_data, audit_log TO compliance_service;

-- Analytics service: can read scrubbed data only, never original memos
CREATE ROLE analytics_service;
GRANT SELECT (
    transaction_date, amount, currency, transaction_type,
    scrubbed_memo, audit_metadata
) ON staged_data TO analytics_service;
-- original_memo column is NOT in this grant

The analytics_service column-level grant is how GDPR data minimisation is enforced at the database layer: the analytics team’s service account cannot access the original_memo column, even accidentally.

🤖 Agent Identity

Agents present a unique challenge: they are software components that must authenticate to services, but their execution environment is more dynamic than traditional service accounts. The MCP server (covered in Part 14) provides agent identity management:

• Each agent type has a registered identity in the MCP server
• When an agent invokes a tool, the MCP server validates the agent’s identity before routing the call
• OPA evaluates the agent’s policy — not just “is this agent authenticated?” but “is this agent authorised to invoke this specific tool with this specific parameter?”

class MCPServerAuthMiddleware:
    def validate_agent_request(self, agent_id: str, tool_name: str, params: dict) -> bool:
        policy_input = {
            "identity": agent_id,
            "action": "invoke_tool",
            "resource": {
                "type": "tool",
                "name": tool_name,
            },
            "params": params,
        }
        return self.opa_client.evaluate("agentic/tool_access", policy_input)

↑ Back to top · Next: Data Classification and Handling →

---

🗂️ Data Classification and Handling

Not all data in the platform deserves the same protection. The classification scheme maps data sensitivity to handling requirements:

Class	Description	Examples	Access	Encryption
Restricted	Personal data, regulated financial data	original_memo, customer IDs	Compliance team only	AES-256 at rest + in transit
Confidential	Business-sensitive, non-personal	amount, transaction_type	Internal teams	Encrypted in transit
Internal	Operational metadata	audit_metadata, timestamps	All internal services	Encrypted in transit
Public	Non-sensitive operational data	Aggregated metrics	No restriction	Optional

The original_memo column is Restricted: it contains PII before scrubbing. Access requires the compliance_service role or higher. The scrubbed_memo is Confidential: it has had PII removed, but it is still financial transaction data that must not leave the organisation without authorisation.

🇪🇺 GDPR Implementation

GDPR Article 25 (data protection by design) requires that privacy protections be built into the system architecture, not bolted on afterward. The platform implements GDPR through four mechanisms:

Data minimisation (Article 5(1)(c)): The PII scrubber ensures only the minimum necessary personal data is retained. original_memo is retained under the legal basis of regulatory obligation (SOX Section 802) with a defined retention period; it is not retained for any other purpose.

Retention limits (Article 5(1)(e)): The RetentionStage deletes records that have exceeded their configured retention period. The default retention is 2,555 days (7 years) to satisfy SOX requirements. Records outside this window are automatically purged.

Right of access (Article 15): A data subject can request all personal data held about them. The original_memo column provides the complete transaction history. The query to fulfil a data subject access request is:

SELECT transaction_date, amount, currency, transaction_type, original_memo
FROM staged_data
WHERE original_memo LIKE '%' || $1 || '%'  -- $1 = anonymised identifier
ORDER BY transaction_date;

Right to erasure (Article 17): When a data subject exercises the right to erasure, the original_memo is overwritten with a null value for that subject’s records. The transaction records themselves are retained (SOX obligation outweighs the erasure right for financial records), but the personal identifiers are removed.

📋 SOX Compliance Implementation

SOX Section 404 requires that financial reporting controls be documented, tested, and audited. Section 802 requires a minimum retention period for financial records. The platform satisfies both:

Audit trail: The audit_metadata JSONB column in every staged_data row constitutes an audit trail. Every record carries the batch ID, governance decision, processing timestamp, and pipeline version. An auditor can query the database to verify that every record was processed through the governance stage with an allowed decision.

Segregation of duties: Database roles prevent the pipeline service from modifying the audit_log table. The audit log is append-only (controlled by PostgreSQL row-level security). A developer who has write access to the pipeline code cannot modify audit log entries.

Retention: The 7-year default retention satisfies SOX. The RetentionStage logs every deletion to the audit log before executing, providing evidence that records were retained for the required period before deletion.

↑ Back to top · Next: Encryption →

---

🔒 Encryption

💾 At-Rest Encryption

All storage systems use encryption at rest:

• PostgreSQL: transparent data encryption via storage-layer encryption (AWS EBS/RDS encryption, Azure Disk Encryption)
• MinIO: server-side encryption for all stored objects
• Qdrant: vector data encrypted at the storage layer

Additionally, the original_memo field is encrypted at the application layer using AES-256 before it reaches the database. This provides defence-in-depth: even if the database storage encryption is compromised, the memo content remains protected:

from cryptography.fernet import Fernet

class MemoEncryptionService:
    def __init__(self, key: bytes):
        self.cipher = Fernet(key)

    def encrypt(self, memo: str) -> str:
        return self.cipher.encrypt(memo.encode()).decode()

    def decrypt(self, encrypted_memo: str) -> str:
        return self.cipher.decrypt(encrypted_memo.encode()).decode()

The encryption key is loaded from an environment variable (ENCRYPTION_KEY) that is injected from the secrets manager at deployment time. It is never stored alongside the data it protects.

🌐 In-Transit Encryption

All service-to-service communication uses TLS:

• PostgreSQL: sslmode=verify-full in connection strings
• Kafka: TLS listeners with certificate verification
• MinIO: HTTPS endpoint
• OPA: HTTPS
• OpenMetadata: HTTPS

In development (Docker Compose), TLS is optional to reduce setup complexity. In production (Kubernetes), it is mandatory, enforced by a network policy that rejects non-TLS connections.

↑ Back to top · Next: OWASP Top 10 Applied to Data Pipelines →

---

🛡️ OWASP Top 10 Applied to Data Pipelines

The OWASP Top 10 is typically discussed in the context of web applications, but the attack vectors apply equally to data pipelines:

• A01 Broken Access Control: Addressed by PostgreSQL role-based access control and OPA policy enforcement. Every data access decision is evaluated against policy.
• A02 Cryptographic Failures: Addressed by AES-256 for sensitive fields and TLS for all inter-service communication.
• A03 Injection: The pipeline uses parameterised SQL queries throughout. The execute_batch() call from psycopg2.extras uses parameterised bindings — user data never appears directly in SQL strings.
• A05 Security Misconfiguration: Addressed by infrastructure-as-code (Terraform) that enforces security settings declaratively, and by the pre-commit secret scanning hook that prevents credential exposure.
• A07 Identification and Authentication Failures: Addressed by Kubernetes ServiceAccount-based service identity and OPA agent authentication.
• A09 Security Logging and Monitoring Failures: Addressed by the audit log (structured JSONB in every row), OPA decision log, and Prometheus metrics that alert on anomalous access patterns.

↑ Back to top · Next: ISO/IEC 27001 Controls Mapping →

---

📋 ISO/IEC 27001 Controls Mapping

ISO/IEC 27001 defines an Information Security Management System (ISMS). The platform implements several key controls from Annex A:

Control	Implementation
A.9.4.1 Information access restriction	PostgreSQL role-based column grants, OPA policy enforcement
A.10.1.1 Policy on the use of cryptographic controls	AES-256 for sensitive fields, TLS for transit
A.12.4.1 Event logging	Audit log embedded in data rows, Prometheus metrics, OPA decision log
A.16.1.7 Collection of evidence	Immutable audit trail, retention controls, governance decision log
A.18.1.3 Protection of records	Retention policy, append-only audit log

↑ Back to top · Next: Secret Scanning and Prevention →

---

🔍 Secret Scanning and Prevention

Secrets accidentally committed to version control are one of the most common security incidents. The platform uses detect-secrets in a pre-commit hook to prevent this:

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/Yelp/detect-secrets
    rev: v1.4.0
    hooks:
      - id: detect-secrets
        args: ["--baseline", ".secrets.baseline"]

The baseline file (secrets.baseline) records known false positives (test fixture values, example configurations). The hook blocks any commit that contains a new string matching credential patterns: API keys, database connection strings, private keys.

In CI, a separate git-secrets or truffleHog scan covers the full commit history and pull request diffs, catching any secrets that bypassed the pre-commit hook.

↑ Back to top · Next: Frequently Asked Questions →

---

❓ Frequently Asked Questions

Common questions about zero-trust security and compliance for agentic data platforms, answered from real-world implementation experience.

What is zero-trust security and how does it apply to data pipelines?

Zero-trust means every request is authenticated and authorised regardless of where it originates. Traditional perimeter security trusts traffic inside the network — but an agentic platform has AI agents inside the network that can be compromised through prompt injection or model poisoning. Zero-trust removes implicit trust: a pipeline pod must present valid Kubernetes ServiceAccount credentials to reach the database; an agent must be evaluated against OPA policy before invoking any tool. Network location is not a credential. This is implemented through three layers: Kubernetes RBAC, OPA tool-access policy, and PostgreSQL role-based column grants.

How do you implement GDPR data minimisation at the database layer?

Use column-level PostgreSQL grants, not application-layer filtering. The analytics_service role is granted SELECT on specific columns of staged_data — explicitly excluding original_memo. This means the analytics team’s service account physically cannot retrieve the PII column, even if a developer writes a SELECT * query by mistake. Application-layer filtering can be bypassed; a database-layer column grant cannot. This satisfies GDPR Article 25 (data protection by design) because the minimisation is enforced at the storage layer, not as documentation.

How do you prevent secrets from being committed to a Git repository?

Pre-commit hook scanning with detect-secrets, backed by CI history scanning. The pre-commit hook checks every staged file against known credential patterns (API key formats, connection string patterns, private key headers) before the commit completes. A .secrets.baseline file records known false positives so the hook does not block test fixtures. In CI, truffleHog or git-secrets scans the full commit history and PR diffs, catching anything that bypassed the pre-commit hook. The two-layer approach catches both new commits and historical exposure.

How does OWASP Top 10 apply to data pipelines differently than to web applications?

The attack categories are the same; the implementation surface differs. A03 Injection in a web app means SQL injection via form fields; in a data pipeline it means injection via transaction memo fields processed by the PII scrubber — addressed with parameterised SQL in every execute_batch() call. A01 Broken Access Control in a web app means missing auth middleware; in a pipeline it means an agent accessing a data column it should not — addressed with PostgreSQL column grants. The mitigations map directly; the threat actors target the same vulnerability classes through different entry points.

↑ Back to top · Next: Key Takeaways →

---

---