How to Build High-Performance Kafka Streams Applications That Scale Beyond Partition Limits

Every Kafka Streams engineer eventually hits the same wall: your application processes one record at a time, one task at a time, and you have reached the maximum parallelism set by your input topic partitions. The OSO engineers recently faced this exact challenge with a mission-critical deduplication application processing 50+ terabytes of data. Traditional scaling approaches, adding more nodes or increasing partitions, weren’t viable options. This is the story of how they built an asynchronous processing framework that delivered a 6x performance improvement whilst maintaining all processing guarantees.

The fundamental issue lies in Kafka Streams’ sequential processing model. Whilst this design provides excellent consistency guarantees and simplicity, it creates bottlenecks with high-latency operations or once you have exhausted your partition-based parallelism. The challenge is particularly acute for applications making remote state store calls or external API requests, or handling workloads with significant per-record processing overhead. When the OSO engineers met an application that could only process 250 records per second per thread despite running 96 threads, they knew they needed a fundamentally different approach.

Why does Kafka Streams hit a partition limit?

Kafka Streams’ processing model is elegantly simple: each task processes one record at a time, from start to finish, before moving on to the next. This sequential execution guarantees ordering and simplifies debugging, but it creates significant limits when scaling high-performance applications.

The core constraint is the direct relationship between input topic partitions and processing parallelism. Each partition maps to exactly one task, and each task can only be processed by a single stream thread. Your maximum parallelism is therefore capped by the number of partitions in your input topics. Once you have assigned one task per thread, adding more compute becomes pointless: additional threads simply sit idle.

The problem becomes particularly severe with high-latency operations. When a processor makes a remote state store call that takes 50 milliseconds, that entire thread is blocked for the duration. No other records can be processed by that thread until the remote operation completes and the record finishes its journey through the topology. If you have several high-latency operations per record, throughput plummets regardless of available CPU or memory.

Consider a typical deduplication scenario. The OSO engineers met an application that performed a simple putIfAbsent operation: check whether a record exists in the state store and, if not, add it and forward it downstream. With remote state stores, this became a read followed by a write, each potentially taking tens of milliseconds. With very low duplicate rates, nearly every record caused a cache miss and an expensive remote lookup. Because of the sequential model, each thread could only handle one of these expensive operations at a time, severely limiting overall throughput.

How does asynchronous processing scale Kafka Streams?

Faced with an immovable deadline and a production system approaching capacity, the OSO engineers had to think beyond traditional scaling. They couldn’t increase partitions without affecting the entire data pipeline, and they couldn’t add more nodes because they had already reached one task per thread. The solution had to break the assumption of sequential processing whilst preserving every correctness guarantee.

The breakthrough was recognising that true sequential ordering is only required within each key, not across an entire partition. Records with different keys can be processed in parallel without violating any ordering constraint. That opened the door to processing multiple records at once within a single task, scaling parallelism to the number of unique keys rather than the number of partitions.

The architecture centres on augmenting existing stream threads with dedicated worker thread pools. Instead of executing processor logic directly, stream threads schedule work to these async thread pools and immediately return to the next record. Multiple records can be in flight at once, which dramatically improves resource utilisation when individual operations have high latency.

The framework works through async events. When a record enters an async processor, the system creates an async event that tracks everything needed to complete processing. The stream thread hands this event to the async thread pool and carries on with other records. The pool executes the processor logic in parallel, potentially handling dozens of records at once. When processing completes, the results are coordinated back to the stream thread for state store updates and downstream forwarding.

This design needed careful thought about thread safety and coordination. The framework intercepts state store operations and record forwarding so they execute in the stream thread, keeping single-threaded access to Kafka Streams internals. The async processor itself can run arbitrary logic in parallel, but every interaction with the wider Kafka Streams framework stays properly synchronised.

How do you keep processing guarantees in parallel execution?

Moving from sequential to parallel processing adds real complexity around correctness. The OSO engineers had to make sure the framework preserved exactly-once semantics, proper ordering and every existing processing guarantee whilst enabling large performance gains.

The key to ordering was same-key ordering rather than strict sequential ordering. This keeps the semantic correctness applications depend on whilst unlocking parallelism across the key space.

State store synchronisation needed particular attention. Kafka Streams internals aren’t thread-safe, so rather than executing writes immediately, the framework captures them and schedules them for the stream thread. The actual processing logic still runs in parallel.

For exactly-once semantics, coordination happens at commit boundaries. When Kafka Streams commits offsets to indicate successful processing, every corresponding record has genuinely finished. If any async operation fails, the framework can handle rollback within existing transaction boundaries.

Key takeaway: you rarely need strict ordering across a whole partition; ordering per key is enough for most applications, and it is what lets Kafka Streams scale past its partition count.

What performance gains did async processing deliver?

The production deployment delivered results that exceeded initial expectations. The application was a business-critical deduplication service at the head of a major data pipeline, processing events with a 30-day TTL window across 50+ terabytes of state.

Before optimisation, the application struggled badly. Despite running 96 stream threads across multiple nodes, each thread could only process about 250 records per second. The main bottleneck was the high latency of remote state store operations, particularly reads that missed the cache because of the low duplicate rate in the incoming data.

The transformation was immediate. After deploying the async processing framework, the same application achieved identical throughput with only 12 stream threads. This translated into large cost savings and created substantial headroom for future growth.

The gain came from overlapping high-latency operations. Rather than waiting for each remote state store call to finish before processing the next record, the system could keep dozens of concurrent operations in flight, reaching throughput that sequential processing could never achieve.

Just as importantly, the deployment carried minimal risk. The framework wraps existing processors, so teams can enable or disable async processing through configuration without changing application logic or taking downtime. That made it possible to validate the approach in production with a rollback option.

How do you implement async processing in Kafka Streams?

Implementing async processing needs minimal code changes and delivers substantial gains for the right workloads. The OSO engineers designed the framework to integrate with existing applications through configuration changes and processor wrapping.

1. Identify suitable processors. The framework works best for processors with high-latency operations: remote state store calls, external API requests, database queries or CPU-intensive computation. Lightweight operations that complete quickly may not benefit, because of the overhead of thread coordination.
2. Wrap the processor supplier. Converting a processor to async execution means wrapping its processor supplier with an async wrapper, usually a single-line change in your topology definition. The wrapped processor keeps the same interface and behaviour. You can apply async processing to specific processors and leave others unchanged.
3. Size the async thread pools. Each stream thread gets its own dedicated async thread pool, so you can scale processing capacity independently of the number of stream threads. Operations with higher latency typically benefit from larger pools that keep more concurrent operations in flight.
4. Monitor the async layer. Async events track processing state and duration. Watch async thread pool utilisation, average processing times and queue depths to tune performance and spot issues.
5. Roll out safely. The framework can be enabled and disabled through configuration without application restarts or topology changes, and its backwards and forwards compatibility lets you roll it out gradually across your applications.

If your bottleneck is state rather than processing, see how unbundled state stores transform Kafka Streams operations, and for wider tuning advice read our Kafka performance best practices.

What are the trade-offs and limitations?

Like any performance technique, async processing involves trade-offs. Knowing them helps you decide when it adds value and when another approach fits better.

• Partition skew. Tasks are still statically assigned to stream threads. If one partition is heavily loaded while others are idle, async processing won’t redistribute that load. It scales processing within each task but doesn’t change task assignment.
• Cross-key ordering. If your use case depends on processing records in exact offset order across different keys within a partition, same-key ordering may not be enough. Most real-world applications only need ordering within each key, so this is rarely a problem in practice.
• Coordination overhead. Handing records between stream threads and async thread pools has a small cost. For operations that complete in microseconds, it can outweigh the benefit. The framework works best when individual operations take at least several milliseconds.
• Memory and threads. Multiple records may be in flight at once, so allocate enough memory for the extra concurrent load. Async thread pools also add threads beyond the standard stream threads, which you need to account for in your deployment.
• Processor API focus. The framework operates at the processor level, so it suits Processor API applications or those using the process DSL operator. It can work with other DSL operators through processor extraction, but that may need more complex integration.

How can you break through Kafka Streams performance barriers?

The async processing framework developed by the OSO engineers shows that Kafka Streams applications can scale far beyond partition-based limits whilst keeping every correctness guarantee. By recognising that same-key ordering is enough for most use cases, they unlocked parallel processing of multiple records within individual tasks.

This is more than a performance optimisation; it is a shift in how we think about stream processing scalability. Rather than being constrained by partitioning decisions made early in system design, applications can scale processing capacity independently. That flexibility becomes more valuable as systems grow and requirements evolve.

The 6x improvement achieved in production validates the approach, and the framework’s design means those gains come without sacrificing the reliability, simplicity and correctness that make Kafka Streams valuable for mission-critical applications.

Looking ahead, this work points towards broader changes in stream processing architecture. Future versions of Kafka Streams may adopt similar ideas natively, potentially through initiatives such as the processing threads model, which would break the static relationship between consumers and processing threads.

For engineering teams facing similar scaling challenges, async processing offers a practical solution that can be implemented quickly whilst keeping production stable. The key insight, that parallelism can be achieved at the key level rather than the partition level, opens new possibilities for high-performance stream processing as data volumes grow and latency requirements tighten.

Frequently asked questions