7 Real-Time Intelligence: Kafka Streaming and Event-Driven Agent Triggers
Series: Building an Agentic Data Platform | Part 7 of 17Reading time: ⏳ ~12 minutesTags: 🏷️ Kafka event streaming real-time data CDC change data capture event-driven agents…
6 Governing with Intelligence: OPA Policies and Agent-Driven Compliance
Series: Building an Agentic Data Platform | Part 6 of 17Reading time: ⏳ ~10 minutes 📌 TL;DR Governance is layered: OPA enforces data rules; middleware applies consistent…
5 Building the Engine: Core ETL Stages with Agent Instrumentation
Series: Building an Agentic Data Platform | Part 5 of 17Reading time: ⏳ ~13 minutesTags: 🏷️ ETL stages data ingestion PII scrubbing audit logging pipeline instrumentation PostgreSQL…
4 Synthetic Data Engineering: Teaching Your Platform What Real Data Looks Like
Series: Building an Agentic Data Platform | Part 4 of 17Reading time: ⏳ ~11 minutesTags: 🏷️ synthetic data data generation GDPR financial data Faker Pydantic pipeline testing…
3 Designing for Intelligence: The Agentic Data Pipeline Architecture
Series: Building an Agentic Data Platform | Part 3 of 17Reading time: ⏳ ~13 minutesTags: 🏷️ pipeline architecture stage-based design agentic ETL TOGAF OPA OpenLineage LangGraph 📌…
2. Engineering the Foundation: A Production-Grade Development Environment
Series: Building an Agentic Data Platform | Part 2 of 17Reading time: ⏳ ~11 minutesTags: 🏷️ Docker Compose development environment 12-Factor App Kafka MinIO OPA Prometheus Grafana…
1. From Scripts to Sentience: Building an Agentic Data Platform
Series: Building an Agentic Data Platform | Part 1 of 17Reading time: ⏳ ~12 minutes 📌 TL;DR Most data engineering tutorials teach you to move data from…
4. The Case of the Vanishing Images: A Debugging Journey with AI Agents
Building autonomous AI agents is an exciting frontier in software development. The idea of an agent that can not only write content but also generate its own…
3. The Developer’s Crucible: Debugging, Patience, and the AI Partnership
TL;DR This article reveals the unglamorous but critical reality of software development: debugging. We recount the real-world challenges faced, from frustrating environment setup errors to a cryptic…
2. Deploy Anywhere: A Guide to Cloud-Agnostic, Serverless APIs
TL;DR This article covers the deployment of our API, focusing on achieving true cloud-agnosticism and infinite scalability. We detail our use of the Serverless Framework to define…