9. Production-Grade Deployment: Kubernetes, Terraform, and Autonomous Scaling
Series: Building an Agentic Data Platform | Part 9 of 17Reading time: ⏳ ~13 minutesTags: 🏷️ Kubernetes Terraform deployment scaling Docker HPA infrastructure as code 12-Factor App…
8. Documentation as Code: Making Your Agentic Platform Self-Describing
Series: Building an Agentic Data Platform | Part 8 of 17Reading time: ⏳ ~12 minutesTags: 🏷️ documentation as code architecture decision records docstrings OpenMetadata living documentation agentic…
7 Real-Time Intelligence: Kafka Streaming and Event-Driven Agent Triggers
Series: Building an Agentic Data Platform | Part 7 of 17Reading time: ⏳ ~12 minutesTags: 🏷️ Kafka event streaming real-time data CDC change data capture event-driven agents…
6 Governing with Intelligence: OPA Policies and Agent-Driven Compliance
Series: Building an Agentic Data Platform | Part 6 of 17Reading time: ⏳ ~10 minutes 📌 TL;DR Governance is layered: OPA enforces data rules; middleware applies consistent…
5 Building the Engine: Core ETL Stages with Agent Instrumentation
Series: Building an Agentic Data Platform | Part 5 of 17Reading time: ⏳ ~13 minutesTags: 🏷️ ETL stages data ingestion PII scrubbing audit logging pipeline instrumentation PostgreSQL…
4 Synthetic Data Engineering: Teaching Your Platform What Real Data Looks Like
Series: Building an Agentic Data Platform | Part 4 of 17Reading time: ⏳ ~11 minutesTags: 🏷️ synthetic data data generation GDPR financial data Faker Pydantic pipeline testing…
3 Designing for Intelligence: The Agentic Data Pipeline Architecture
Series: Building an Agentic Data Platform | Part 3 of 17Reading time: ⏳ ~13 minutesTags: 🏷️ pipeline architecture stage-based design agentic ETL TOGAF OPA OpenLineage LangGraph 📌…
2. Engineering the Foundation: A Production-Grade Development Environment
Series: Building an Agentic Data Platform | Part 2 of 17Reading time: ⏳ ~11 minutesTags: 🏷️ Docker Compose development environment 12-Factor App Kafka MinIO OPA Prometheus Grafana…
1. From Scripts to Sentience: Building an Agentic Data Platform
Series: Building an Agentic Data Platform | Part 1 of 17Reading time: ⏳ ~12 minutes 📌 TL;DR Most data engineering tutorials teach you to move data from…
4. The Case of the Vanishing Images: A Debugging Journey with AI Agents
Building autonomous AI agents is an exciting frontier in software development. The idea of an agent that can not only write content but also generate its own…