kafka-stream-processing

What this skill does

# Kafka Stream Processing

A comprehensive skill for building event-driven applications with Apache Kafka. Master producers, consumers, Kafka Streams, connectors, schema registry, and production deployment patterns for real-time data processing at scale.

## When to Use This Skill

Use this skill when:

- Building event-driven microservices architectures
- Processing real-time data streams and event logs
- Implementing publish-subscribe messaging systems
- Creating data pipelines for analytics and ETL
- Building streaming data applications with stateful processing
- Integrating heterogeneous systems with Kafka Connect
- Implementing change data capture (CDC) patterns
- Building real-time dashboards and monitoring systems
- Processing IoT sensor data at scale
- Implementing event sourcing and CQRS patterns
- Building distributed systems requiring guaranteed message delivery
- Creating real-time recommendation engines
- Processing financial transactions with exactly-once semantics
- Building log aggregation and monitoring pipelines

## Core Concepts

### Apache Kafka Architecture

Kafka is a distributed streaming platform designed for high-throughput, fault-tolerant, real-time data processing.

**Key Components:**

1. **Topics**: Named categories for organizing messages
2. **Partitions**: Ordered, immutable sequences of records within topics
3. **Brokers**: Kafka servers that store and serve data
4. **Producers**: Applications that publish records to topics
5. **Consumers**: Applications that read records from topics
6. **Consumer Groups**: Coordinated consumers sharing workload
7. **ZooKeeper/KRaft**: Cluster coordination (ZooKeeper legacy, KRaft modern)

**Design Principles:**

```APIDOC
Kafka Design Philosophy:
  - High Throughput: Millions of messages per second
  - Low Latency: Single-digit millisecond latency
  - Durability: Replicated, persistent storage
  - Scalability: Horizontal scaling via partitions
  - Fault Tolerance: Automatic failover and recovery
  - Message Delivery Semantics: At-least-once, exactly-once support
```

### Topics and Partitions

**Topics** are logical channels for data streams. Each topic is divided into **partitions** for parallelism and scalability.

```bash
# Create a topic with 20 partitions and replication factor 3
$ bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic my_topic_name \
--partitions 20 --replication-factor 3 --config x=y
```

**Partition Characteristics:**

- **Ordered**: Messages within a partition are strictly ordered
- **Immutable**: Records cannot be modified after written
- **Append-only**: New records appended to partition end
- **Retention**: Configurable retention by time or size
- **Replication**: Each partition replicated across brokers

**Adding Partitions:**

```bash
# Increase partition count (cannot decrease)
$ bin/kafka-topics.sh --bootstrap-server localhost:9092 --alter --topic my_topic_name \
--partitions 40
```

**Note**: Adding partitions doesn't redistribute existing data and may affect consumers using custom partitioning.

### Stream Partitions and Tasks

```html
<h3>Stream Partitions and Tasks</h3>
<p> The messaging layer of Kafka partitions data for storing and transporting it. Kafka Streams partitions data for processing it. In both cases, this partitioning is what enables data locality, elasticity, scalability, high performance, and fault tolerance. Kafka Streams uses the concepts of <b>partitions</b> and <b>tasks</b> as logical units of its parallelism model based on Kafka topic partitions. There are close links between Kafka Streams and Kafka in the context of parallelism: </p>
<ul>
<li>Each <b>stream partition</b> is a totally ordered sequence of data records and maps to a Kafka <b>topic partition</b>.</li>
<li>A <b>data record</b> in the stream maps to a Kafka <b>message</b> from that topic.</li>
<li>The <b>keys</b> of data records determine the partitioning of data in both Kafka and Kafka Streams, i.e., how data is routed to specific partitions within topics.</li>
</ul>
<p> An application's processor topology is scaled by breaking it into multiple tasks. More specifically, Kafka Streams creates a fixed number of tasks based on the input stream partitions for the application, with each task assigned a list of partitions from the input streams (i.e., Kafka topics). The assignment of partitions to tasks never changes so that each task is a fixed unit of parallelism of the application. Tasks can then instantiate their own processor topology based on the assigned partitions; they also maintain a buffer for each of its assigned partitions and process messages one-at-a-time from these record buffers. As a result stream tasks can be processed independently and in parallel without manual intervention. </p>
<p> Slightly simplified, the maximum parallelism at which your application may run is bounded by the maximum number of stream tasks, which itself is determined by maximum number of partitions of the input topic(s) the application is reading from. For example, if your input topic has 5 partitions, then you can run up to 5 applications instances. These instances will collaboratively process the topic's data. If you run a larger number of app instances than partitions of the input topic, the "excess" app instances will launch but remain idle; however, if one of the busy instances goes down, one of the idle instances will resume the former's work. </p>
<p> It is important to understand that Kafka Streams is not a resource manager, but a library that "runs" anywhere its stream processing application runs. Multiple instances of the application are executed either on the same machine, or spread across multiple machines and tasks can be distributed automatically by the library to those running application instances. The assignment of partitions to tasks never changes; if an application instance fails, all its assigned tasks will be automatically restarted on other instances and continue to consume from the same stream partitions. </p>
```

### Message Delivery Semantics

```APIDOC
Design:
  - The Producer: Design considerations.
  - The Consumer: Design considerations.
  - Message Delivery Semantics: At-least-once, at-most-once, exactly-once.
  - Using Transactions for atomic operations.
```

**At-Least-Once Delivery:**
- Producer retries until acknowledgment received
- Consumer commits offset after processing
- Risk: Duplicate processing on failures
- Use case: When duplicates are acceptable or idempotent processing

**At-Most-Once Delivery:**
- Consumer commits offset before processing
- No producer retries
- Risk: Message loss on failures
- Use case: When data loss acceptable (e.g., metrics)

**Exactly-Once Semantics (EOS):**
- Transactional writes with idempotent producers
- Consumer reads committed messages only
- Use case: Financial transactions, critical data processing

### Producer Load Balancing

```APIDOC
ProducerClient:
  publish(topic: str, message: bytes, partition_key: Optional[str] = None)
    topic: The topic to publish the message to.
    message: The message payload to send.
    partition_key: Optional key to determine the partition. If None, random partitioning is used.

  get_metadata(topic: str) -> dict
    topic: The topic to get metadata for.
    Returns: A dictionary containing broker information and partition leader details.
```

The producer directs data to the partition leader broker without a routing tier. Kafka nodes provide metadata to producers for directing requests to the correct partition leaders. Producers can implement custom partitioning logic or use random distribution.

## Producers

Kafka producers publish records to topics with configurable reliability and performance characteristics.

### Producer API

```APIDOC
Producer API:
  - send(record): Sends a record to a Kafka topic.
    - Parameters:
      - record: The record to send, including topic, key, and value.
    - Returns: A Future representing the result of the send op