17. Seventeen Parts Later: Lessons From Building an Agentic Data Platform

---

🎯 What This Series Set Out to Do

For data engineers building agentic systems in regulated environments, most tutorials stop where the real problems start. This series was built to go further: to show what agentic data engineering looks like when compliance is non-negotiable, audit trails are mandatory, and governance failures halt the batch rather than generate a log entry no one reads. By the end of this article, you will have a complete picture of the architectural decisions that made this platform work — and the three decisions that should have been made differently.

Financial data engineering is different. The data is regulated. The audit trail is mandatory. The security model has to be zero-trust from the first line of code, not bolted on when the compliance team asks. And increasingly, the intelligence layer — the agents that reason about data, flag anomalies, and recommend policy changes — is not optional. It is the differentiator.

This series was my attempt to build a tutorial-grade example of what enterprise-grade agentic data engineering actually looks like: not “here is how to use LangChain in a notebook” but “here is a production architecture with compliance, security, observability, and multi-agent coordination, explained decision by decision.”

↑ Back to top · Next: The 10 Lessons →

---

💡 The 10 Lessons

1. 🏛️ Architecture Is Policy

The single most important decision in the platform was making OPA the enforcement layer for data access control, not the advisory layer. The governance stage does not recommend that the pipeline follow policy — it enforces it. A governance denial halts the batch. The agent cannot route around it.

This design choice came from observing how “advisory” governance fails in practice: the advisory system flags violations, the flag goes into a log, the log is reviewed monthly, the violations persist. When governance is advisory, it is performative. When governance is architectural — when the pipeline cannot proceed without a policy approval — it becomes real.

This principle extends beyond OPA. The human approval gate in the LangGraph workflow covered in Part 12 is a topology constraint, not an optional node. The authentication middleware in the MCP server runs on every request, not on requests that come from untrusted sources. Zero-trust means the enforcement is in the path, not alongside it.

2. 🤖 Agents Are Interfaces, Not Magic

The most common mistake I see in AI engineering is treating agents as magic black boxes: you give them a prompt, they do the task, you declare success. The platform treats agents as software components with defined interfaces, observable behaviour, and testable outputs.

Every agent in the platform has:

• A defined input schema (what it receives in AgentState)
• A defined output schema (what it returns)
• Unit tests for the invariants its output must satisfy
• Prometheus metrics for its invocations, latency, and error rates
• A clear description — not a clever prompt — that tells the LLM planner exactly what the tool does

The description field in the MCP server’s tool registry is not boilerplate. It is the contract that the LLM planner uses to decide which tool to invoke. A tool with a vague description (“does data things”) will be misused or ignored. A tool with a precise description (“Validates a financial transaction against ISO 4217 currency codes and configured business rules. Returns per-record validation results with rule names.”) is reliably and correctly selected.

3. 🔗 The Pipeline Stage Contract Is Worth More Than It Looks

The two-method PipelineStage abstract base class — execute(data, pipeline_context) and name() — is the smallest useful abstraction in the codebase. Every stage implements it. Every stage can be tested in isolation by calling execute() with synthetic inputs. Every stage can be swapped for a different implementation without changing the PipelineRunner.

This contract enabled a development workflow that I did not anticipate when I first wrote it: stages could be developed and tested completely independently, then composed into pipelines. The PIIScrubberStage developer did not need to know about the DataIngestionStage. The OPAPolicyEnforcementStage developer did not need to know about the KafkaPublishStage. The contract was the only shared surface.

4. 🧪 Test Against Real Services

Property-based testing with Hypothesis found a bug the team had not anticipated: the PII scrubber’s regex for ID detection was matching five-digit sequences in transaction amounts formatted with thousands separators (1,000.50 → 1,000 matched as a national ID). No human-written test had constructed that specific input combination.

But the more important testing lesson was integration tests: run them against real PostgreSQL, real OPA, real Kafka — not mocks. Mocking the database connection for the DataIngestionStage test validates that the code calls the mock correctly. It does not validate that the SQL query is valid, that the column types match, or that the batch insert handles duplicate keys.

The observation that mock-based tests often pass while integration tests fail is not a testing philosophy argument. It is an empirical pattern. For data platform code, which is fundamentally about interactions with external systems, the integration test is the primary test.

5. 📉 Statistical Drift Is the Silent Failure Mode

Schema validation and business rules validation catch discrete failures — a specific record is invalid for a specific reason. Statistical validation catches something more insidious: the data is valid, but the distribution has shifted.

A batch where the mean transaction amount is 300% of the historical baseline is not invalid. Each record passes schema and business rules checks. But something has changed: a data source misconfiguration, a currency conversion error, a new transaction type that was not anticipated when the amount ranges were configured.

Statistical validation with a rolling baseline catches this. The first few pipeline runs establish the baseline. As the system accumulates history, the baseline becomes more accurate and the anomaly detection becomes more precise. This is one of the few areas in the platform where “it gets better over time” is not marketing — it is a property of the algorithm.

6. 🔒 Security Posture Is Set on Day One

The zero-trust security architecture was the hardest component to retrofit when I initially left it for “after we get the pipeline working.” The correct order is: define the security model first, then build everything else within it.

Concretely:

• Column-level PostgreSQL grants should be set when the schema is created, not after you discover the analytics team can read original_memo.
• The pre-commit secret scanning hook should be in .pre-commit-config.yaml before the first commit, not after a credential is accidentally pushed.
• OPA policies should be written before the pipeline stages they govern, not after the pipeline is already running in production.

The OWASP Top 10 applies to data pipelines exactly as much as to web applications. SQL injection through pipeline inputs is a real attack vector. Broken access control through missing database grants is a real vulnerability. Treating data pipeline security as an afterthought creates technical debt that is expensive to correct and dangerous in the meantime.

7. 🚀 The Agent Network Abstraction Pays Off in CI/CD

LocalAgentNetwork and KafkaAgentNetwork share the same AgentMessage protocol. In CI/CD, where running a Kafka cluster for every test would be prohibitively expensive, LocalAgentNetwork runs the entire multi-agent workflow as in-process function calls. The same orchestrator code, the same agents, the same state machine — just without the distributed transport.

This was not an accident of implementation. It was a deliberate architectural choice: the orchestrator should not know which transport is in use. Selecting the transport is an environment variable change, not a code change.

The pattern that made this possible — a clean interface with two implementations chosen by configuration — is one of the most reused patterns in software engineering. It is reliable precisely because it is boring. I mention it here because it is tempting to reach for something more sophisticated when a simple interface is all that is needed.

8. 🗺️ Lineage Is the Foundation of Trust

Data consumers trust data that they understand. They distrust data that arrives from an opaque source without explanation. For regulated financial data, this is not a preference — it is a compliance requirement.

OpenLineage provides the vocabulary for describing data provenance in a way that is both machine-readable (for automated auditing) and human-navigable (for compliance officers who need to trace a specific record). The columnLineage facet in the lineage event is what enables the GDPR right-to-explanation response: the data subject can be told not just that their data was processed, but exactly what transformations were applied, in what order, by what pipeline version.

The investment in lineage registration pays off not at ingestion time but months later, when an auditor asks “show me the complete processing history for these 500 transactions.” Without lineage, answering that question requires code archaeology. With lineage, it is a query.

9. 🔔 Alert Design Is Product Design

The alert system is not an engineering concern — it is a product concern. The question “which alerts should fire, at what severity, to which channels, with what cooldown?” is a question about what operators need to know, when they need to know it, and how much interruption they can absorb.

Alert fatigue — the condition where operators stop reading alerts because too many are low-value — is more dangerous than missing alerts. An operator who ignores the #data-alerts channel because it is flooded with INFO-level notifications will also miss the CRITICAL governance denial buried in the noise.

The severity discipline — INFO only to dashboard, WARNING to Slack, ERROR to email, CRITICAL to PagerDuty — was derived from a simple principle: each channel should carry only the signals that justify its interruption cost. A PagerDuty page at 2am justifies getting out of bed. A Slack message justifies a 30-second look. An INFO dashboard entry justifies nothing immediately. Apply the channel to the appropriate interruption threshold.

10. 🛡️ Red Team Findings Must Become Regression Tests

The red team exercise covered in Part 13 found two vulnerabilities that the test suite had missed: prompt injection through transaction memo fields could redirect the compliance agent’s reasoning, and amount-in-words formatting (“five thousand dollars”) could bypass the human approval gate’s regex-based amount extraction.

Both were fixed. Both became regression tests. And both regression tests are now in the CI pipeline, running on every merge.

↑ Back to top · Next: What I Would Do Differently →

---

🔄 What I Would Do Differently

Three decisions I would revisit if starting from scratch:

• 👁️ Start with the seed data model. The Transaction Pydantic model defines the shape of data that flows through the entire platform. Every stage, every agent, every schema definition, every lineage facet depends on it. In the first iteration, I treated it as a detail to be worked out later. The right approach is to define it first, make it explicit, and treat changes to it as breaking API changes.
• ⚙️ Deploy OPA before writing Rego. The first version of the governance stage tested Rego policies against unit tests only. The integration between the pipeline’s policy input format and the OPA server’s expectation was not validated until the full integration test ran — and it failed. The lesson: stand up the service, write the first test that calls it, and confirm the wire format before writing any logic. Infrastructure integration tests are the first tests, not the last.
• ⚡ Model the alert taxonomy before the first stage. The alert system was added incrementally as each stage was built. This meant the severity labels were inconsistent across stages (the validation stage used "error" for blocking issues; the Kafka stage used "critical" for connection failures), requiring a normalisation pass later. Define the severity taxonomy once, document it, and enforce it in code review from the start.

↑ Back to top · Next: Where Agentic Data Engineering Is Heading →

---

🔭 Where Agentic Data Engineering Is Heading

Five trends that will define the next evolution of the platform:

Agents as primary pipeline operators. Today’s architecture uses agents alongside deterministic pipeline stages. The emerging pattern is agents that dynamically compose pipeline logic: an agent that observes the incoming data’s schema, selects the appropriate transformation stages from a registry, assembles them into a pipeline, and executes — without a human configuring the stage sequence. The PipelineStage registry and the MCP tool registry are the building blocks for this.

Tighter regulatory frameworks for AI. The EU AI Act, NIST AI RMF, and emerging financial services AI regulations are converging on requirements that look like the platform’s governance architecture: documented risk assessments, human oversight for high-stakes decisions, audit trails for automated decision-making, and regular adversarial testing. The red team exercises, the human approval gate, the governance decision log, and the OPA policies are not ahead of the regulatory curve — they are aligned with where the curve is going.

Real-time lineage. OpenLineage events in this platform are batch: one event per pipeline run. The emerging standard for event-streaming architectures is run-time lineage: a lineage event emitted for each record as it flows through each stage. This enables sub-second provenance queries that the batch model cannot support. Kafka-native OpenLineage emitters are already in development in the OpenLineage project.

Multi-model agent networks. The platform’s LLM provider abstraction supports Ollama (local) and cloud providers with automatic fallback. The next evolution is heterogeneous model selection: the compliance audit agent runs on a large reasoning model because it handles complex policy evaluation; the PII scrubber runs on a smaller, faster model because it handles a bounded NER task. Each agent selects the smallest model that meets its accuracy requirements. The LLMProvider abstraction is the right place to add this routing logic.

Formal verification for agent policies. OPA Rego policies can be formally verified: given a policy and a set of inputs, it is possible to prove whether the policy can ever produce a specific output. The next step is using SMT solvers to prove policy properties — for example, “this policy can never allow a write with contains_pii=true and pii_scrubbed=false“ — rather than testing with finite example inputs.

↑ Back to top · Next: Closing Reflection →

---

🪞 Closing Reflection

The most honest thing I can say about this platform is that it took three iterations to get the architecture right. The first version had governance as a wrapper around the pipeline. The second had it as a stage in the pipeline. The third — the version this series describes — has it as an enforced contract that every stage must satisfy.

The difference between version one and version three is not the code. The Rego policies in version three are not materially more complex than the checks in version one. The difference is where the enforcement happens: in the architecture, not alongside it.

Build it so that the compliant path is the only path.

↑ Back to top · Next: Series Map →

---

🗂️ Series Map

For reference, the complete reading order:

Part	Title	Focus
1	The Agentic Data Platform: Why This, Why Now	Series introduction, architecture overview
2	The Foundation: Local Development Environment	Docker Compose stack, 12-Factor configuration
3	Blueprint: Designing the Pipeline Architecture	TOGAF, PipelineStage contract, failure modes
4	Synthetic Data Engineering at Scale	GenerationAgent, Transaction model, data drift
5	The Core ETL: Implementing Pipeline Stages	All core stages, PipelineRunner, SOX audit trail
6	Policy as Code: OPA-Driven Governance	OPA, Rego, GDPR/SOX/MiFID II mapping, DAMA DMBOK
7	Streaming Integration: Kafka in the Pipeline	Kafka topics, CDC, consumer group scaling
8	Documentation for Agents and Humans	ADRs, agent tool docstrings, OpenMetadata
9	Production Deployment: Kubernetes and Terraform	Multi-stage Docker, HPA, Terraform IaC
10	Testing the Untestable: Agentic System Validation	Unit, integration, property-based, chaos testing
11	Zero-Trust Data Governance	Security architecture, GDPR/SOX, OWASP
12	Multi-Agent Orchestration with LangGraph	StateGraph, A2A protocol, LLM provider abstraction
13	Red Teaming Agentic Systems	Adversarial testing, Garak, NIST AI RMF
14	The Internal MCP Server	Tool registry, execution engine, agent identity
15	Advanced Validation and OpenLineage	Layered validation, FAIR data, GDPR lineage
16	Alert Systems That Think Ahead	AlertEngine, escalation, Prometheus AlertManager
17	Seventeen Parts Later: Lessons Learned	Synthesis, what I’d do differently, future trends

↑ Back to top · Next: Frequently Asked Questions →

---

❓ Frequently Asked Questions

Common questions about agentic data platform architecture answered from real-world implementation experience across this 17-part series.

What is the most important architectural principle for an agentic data platform?

Governance must be architecture, not afterthought. The most critical lesson from building this platform is that advisory governance always fails: flags go into logs, logs are reviewed monthly, violations persist. When governance is architectural — when the pipeline cannot proceed without an OPA policy approval, the human approval gate is a topology constraint in the LangGraph workflow, and authentication middleware runs on every MCP request — it becomes real and enforceable. Build it so that the compliant path is the only path.

How do you prevent alert fatigue in a data pipeline monitoring system?

Alert fatigue prevention comes from strict channel discipline: route each severity level to the channel whose interruption cost matches its urgency. INFO goes only to a dashboard (zero interruption), WARNING to Slack (30-second look), ERROR to email (read within the hour), and CRITICAL to PagerDuty (justifies waking someone at 2am). When operators see only actionable signals in each channel, they stop ignoring alerts — and stop missing the critical governance denials buried in noise.

Why should red team findings become regression tests?

A fixed vulnerability is not truly fixed until an automated regression test confirms it cannot be reproduced. The fix alone is insufficient: a future refactor might quietly revert the fix, or a dependency update might invalidate the behaviour the fix relied on. The red team exercises in this platform found prompt injection via transaction memo fields and an amount-in-words bypass of the approval gate regex — both fixed, both converted to CI tests that run on every merge. The regression test is the durable protection.

What is the difference between advisory governance and architectural governance?

Advisory governance flags policy violations and records them in a log that someone reviews later — the pipeline continues regardless. Architectural governance makes policy enforcement a structural requirement: a governance denial halts the batch, the agent cannot route around it, and there is no code path through the system that bypasses the audit log or the OPA check. The difference is not policy complexity — the Rego rules are similar in both cases — it is where enforcement happens: in the architecture’s critical path, not alongside it.

↑ Back to top · Next: Key Takeaways →

---

---