Day 4 - Kafka Streams: Hands-on, Architecture & First PR

Hands-on with Kafka Streams on Confluent Cloud, diving into architecture, and opening my first PR to the Apache Kafka repository.
Author

Shashank Hosahalli Shivamurthy

Published

July 10, 2025

Day 4 - Kafka Streams Deep Dive & First PR

Today was all about Kafka Streams — and I was pumped.

Yesterday’s investigation into an open issue in the Kafka codebase really motivated me. I literally woke up thinking about it.

Kafka Streams Course (Confluent)

  • Started the Kafka Streams course by Confluent to understand:
    • What Kafka Streams is used for
    • The problem it solves
    • Its overall architecture and workflow
  • Key takeaway:
    • Kafka Streams is a client-side library for real-time stream processing directly within your app
    • It’s used to transform, filter, aggregate, and route data between Kafka topics
    • No separate cluster — lightweight and powerful

First Kafka Streams App

  • Built a simple Kafka Streams application on Confluent Cloud
  • What it did:
    • Consumed messages from a topic
    • Applied basic filtering logic
    • Wrote results to a new topic
  • Explored configurations, checked logs, and observed behavior
  • This helped a lot in understanding the config behavior tied to the open issue I’m working on

Deep Dive into Architecture

  • Watched a Kafka Streams Architecture & Deep Dive talk on Confluent’s YouTube channel
  • Also read through the official documentation
    • It’s surprisingly clean and well-written
    • Helped cement concepts like state stores, repartitioning, topology, and thread model

Revisiting the Issue

  • Returned to the open issue in the Kafka repo
    • Re-read the discussion
    • Found an old PR that someone had started but abandoned midway
  • Carefully traced through the codebase again with more clarity
    • Focused on how StreamsConfig handles internal vs. user-overridden configs
    • Ran tests and validated config behavior

First PR Opened

After a lot of hours, debugging, and local tests — I opened my first-ever pull request to the Apache Kafka GitHub repository.

Need to work towards getting it approved and merged. That’s the next goal.

It was a surreal moment, and I am incredibly excited for what’s ahead.

One small PR for Kafka, one giant leap for my open-source journey, lol

More learning and more contributions to come. Lesssssgoooooo!!!