Kafka Streaming Platform

The goal was to build a realistic event-driven system that ingests sensor readings, preserves a clear processing path and exposes the resulting operational data through both an API and a dashboard.

The architecture had to demonstrate more than a Kafka installation. It needed application services, persistence, health handling, secret delivery and a repeatable Kubernetes rollout that respects component dependencies.

• ProducersPython services generate IoT-style readings and publish them into Kafka topics.
• Kafka clusterStrimzi manages three combined broker/controller nodes without a ZooKeeper dependency.
• Consumer APIA Python service consumes events, applies the application flow and persists records in PostgreSQL.
• DashboardThe browser interface reads API endpoints for recent events and aggregate data instead of reaching into storage directly.

Separate application and platform ownership

Python services own event production and consumption. Kubernetes manifests define runtime behavior, while ArgoCD owns reconciliation and rollout order. This keeps data-flow logic independent from the deployment mechanism.

Use KRaft for the Kafka control plane

KRaft removes the ZooKeeper dependency and reflects the direction of modern Kafka deployments. Strimzi supplies the Kubernetes operator model and declarative cluster management.

Keep credentials outside images and manifests

Database credentials are mounted at runtime through a secrets integration. Application containers consume files or environment references without embedding secret values in the image.

1. Namespace and prerequisitesCreate the application boundary and establish the resources required by later components.
2. Strimzi operatorInstall the Kafka operator before submitting the Kafka custom resources it manages.
3. Kafka KRaft clusterReconcile the broker/controller layer and wait for a usable streaming endpoint.
4. PostgreSQL and credentialsDeploy persistence and mount runtime credentials before starting the consumer service.
5. Producer, consumer API and dashboardRoll out the application services with dependency-aware readiness checks.
6. IngressExpose the dashboard only after the internal services are available.

• Replicated Kafka stateThe cluster declares replication factor 3, minimum ISR 2 and three 5 GiB persistent broker volumes.
• Dependency orderingSix ArgoCD Applications progress from namespace at wave 0 through external exposure at wave 5.
• Persistent consumer pathPostgreSQL uses a StatefulSet, a 5 GiB volume and a database readiness probe.
• GitOps reconciliationAutomated pruning and self-healing keep declared state aligned after the initial deployment.

The completed platform demonstrates an end-to-end event path from producer to dashboard and the operational work required to deploy that path on Kubernetes. It brings streaming architecture, secrets, health checks, persistence and GitOps ordering into one coherent system.