Kafka Platform
Designing and stabilising a self-service, dual-authenticated Kafka platform for Airtel — kernel-tuned bare-metal brokers carrying high-speed streaming traffic from across India, so product teams get a messaging backbone without each one operating its own cluster.
The problem
Individual teams across Airtel's data platform needed a reliable event backbone, but standing up and operating a production-grade Kafka cluster per team is slow, inconsistent, and a poor use of specialist time. Some teams needed strict Kerberos-based auth tied into existing Active Directory; others just needed a simple username/password (SCRAM) client with no AD dependency. Whatever we built had to serve both without forking the platform in two.
The traffic itself was the harder problem. Producers were streaming in from across India at volumes that exposed default OS network settings as the real bottleneck, long before broker count was. Stabilising the platform meant going deeper than the JVM — into the kernel of the bare-metal hosts underneath it.
I owned the architecture end to end: authentication, cluster operations, kernel-level stability tuning, rebalancing, and the observability that keeps it on-call-friendly.
Architecture
Click a component to see what it does.
Design decisions
Kernel-tuned bare metal, not just more brokers. Streaming ingestion at that speed turned out to be a network-stack problem before it was a Kafka problem. I tuned kernel-level network buffers, NIC interrupt (IRQ) affinity, and file descriptor limits on the bare-metal hosts to sustain high-throughput streaming from producers scattered across India — stabilising the cluster instead of over-provisioning hardware to paper over a tuning problem.
Dual authentication, one cluster. Rather than run separate clusters per auth scheme, I configured the brokers to accept both Kerberos (SASL/GSSAPI, tied into Active Directory for internal services already AD-joined) and SCRAM (SASL/SCRAM, for external or simpler clients that shouldn't need AD integration) on separate listener ports. One cluster, two front doors — teams pick whichever fits their environment instead of the platform dictating it.
Automated rebalancing with Cruise Control. As more teams onboarded, partition placement drifted and a handful of brokers started running hot. Rather than rebalance manually — risky and easy to get wrong on a live cluster — I deployed Cruise Control to continuously monitor broker load and execute safe, incremental partition moves, keeping the cluster balanced as usage patterns changed.
Observability as a first-class citizen, not an afterthought. A shared platform lives or dies by whether the on-call engineer can tell what's wrong at 2am. I built custom Grafana dashboards for broker/topic/consumer-lag metrics, wired in Kafka UI for day-to-day topic and ACL inspection, and routed critical alerts to Telegram so the team gets paged wherever they are, not just when someone happens to be looking at a dashboard.