Kafka Streaming Platform

The goal was to build a realistic event-driven system that ingests sensor readings, preserves a clear processing path and exposes the resulting operational data through both an API and a dashboard.

The architecture had to demonstrate more than a Kafka installation. It needed application services, persistence, health handling, secret delivery and a repeatable Kubernetes rollout that respects component dependencies.

• ProducersPython services generate IoT-style readings and publish them into Kafka topics.
• Kafka clusterStrimzi manages three combined broker/controller nodes without a ZooKeeper dependency.
• Consumer APIA Python service consumes events, applies the application flow and persists records in PostgreSQL.
• DashboardThe browser interface reads API endpoints for recent events and aggregate data instead of reaching into storage directly.

Separate application and platform ownership

Python services own event production and consumption. Kubernetes manifests define runtime behavior, while ArgoCD owns reconciliation and rollout order. This keeps data-flow logic independent from the deployment mechanism.

Use KRaft for the Kafka control plane

KRaft removes the ZooKeeper dependency. Strimzi supplies the Kubernetes operator model and declarative cluster management.

Keep credentials outside images and manifests

Database credentials are mounted at runtime through a secrets integration. Application containers consume files or environment references without embedding secret values in the image.

Keep the demonstration compact

The three KRaft nodes combine broker and controller roles. This keeps the project footprint small while making replication and persistence visible. Separate roles are a design choice to evaluate for a different workload.

Make dependencies part of delivery

Six Argo CD sync waves express the order between prerequisites, the operator, Kafka, persistence, applications and ingress. The rollout can be reviewed alongside the manifests.

State what the evidence establishes

The runtime inventory documents three nodes, replication factor three, minimum ISR two and persistent storage. These are configuration and runtime observations; workload throughput and recovery-time guarantees would require separate testing.

1. Namespace and prerequisitesCreate the application boundary and establish the resources required by later components.
2. Strimzi operatorInstall the Kafka operator before submitting the Kafka custom resources it manages.
3. Kafka KRaft clusterReconcile the broker/controller layer and wait for a usable streaming endpoint.
4. PostgreSQL and credentialsDeploy persistence and mount runtime credentials before starting the consumer service.
5. Producer, consumer API and dashboardRoll out the application services with dependency-aware readiness checks.
6. IngressExpose the dashboard only after the internal services are available.

Runtime capture: Argo CD application health, Kubernetes workloads and the sensor dashboard. Select the image to inspect it.

The deployed system carries events from Python producers through Kafka and PostgreSQL to the API and dashboard. Runtime secrets, health checks, persistent storage and six ordered GitOps waves support that delivery path. The evidence documents the implementation and running workloads; throughput and recovery targets need separate tests.