From Block to Object Storage: Engineering a Cost-Efficient Data Streaming Platform at Massive Scale

Updated October 2026: We checked this article against current AWS pricing and Apache Kafka and Flink documentation. We corrected the gp3 EBS price ($0.08 per GB-month, not $0.10), the S3 Express One Zone latency (single-digit milliseconds) and the MSK storage scaling timings, and added the 2025 S3 Express One Zone price cuts, KIP-794 adaptive partitioning, KIP-1123 rack-aware producer partitioning (Kafka 4.4) and KIP-1150 diskless topics (accepted March 2026). The 71% figure is a reduction in object storage requests, not total cost.

When you are processing 3.5 terabytes of data per minute and handling 73 billion messages across 30+ Kafka clusters, every architectural decision compounds into millions in annual costs. OSO engineers carried out an extensive migration from EBS-based Kafka deployments to object storage-backed streaming pipelines, and uncovered insights about reliability, cost efficiency and operational simplicity that challenge conventional wisdom about streaming architecture.

This is about real engineering decisions that reduced object storage API requests by 71%, eliminated cross-availability zone charges that were draining budgets, and rethought stateful stream processing for cloud-native environments. The central insight: for high-throughput streaming workloads, disaggregated architectures on object storage beat traditional block storage on cost and operations, but only if you rethink batching, coordination and state management.

What are the hidden costs of block storage for Kafka?

Block storage such as EBS looks like the natural choice for streaming platforms. Kafka was designed for local disks, and block storage latency suits real-time processing. At massive scale, though, it shows weaknesses in both reliability and cost.

The slow volume problem

One of the most painful issues in large Kafka clusters on block storage is the slow volume. Dozens of brokers look healthy, then write latencies spike, pipelines back up and messages queue in producers. The cause is a single EBS volume that has not failed but has become inexplicably slow. It answers health checks and shows no errors in CloudWatch, yet runs at a fraction of its normal throughput.

Engineers at PlanetScale have documented the same behaviour, and OSO engineers hit it repeatedly in production. With latency SLAs measured in seconds, one slow volume can cascade through an entire pipeline.

Kafka has workarounds. Since Kafka 3.3, the default producer partitioner uses adaptive partitioning (KIP-794), sending less data to brokers that respond slowly, and partitioner.availability.timeout.ms can stop it sending to partitions whose leaders stay unresponsive. These help, but they add complexity and do not fix the underlying reliability problem. OSO engineers also built early detection that monitors volume performance and replaces degraded volumes before they hit production traffic. It works, but it is operational overhead that should not need to exist.

Capacity planning and operational complexity

Block storage forces constant capacity planning. Every topic needs retention configured and every workload spike needs headroom. In Amazon MSK, extra broker storage is available as soon as you increase it, but MSK then optimises the volumes, which can take 6 to 24 hours or more, and you must wait at least six hours before the next storage increase.

Operationally, that means that if disk usage passes 80% and you scale up, you cannot scale again quickly if traffic keeps growing or a configuration error causes unexpected data accumulation. Full disks take brokers down. The alternative is over-provisioning: paying for storage you do not need to cover worst cases, multiplied across 30+ clusters with terabytes of mostly empty provisioned storage. On top of that come disk-full runbooks, capacity alerts to tune and emergency procedures to test, all engineering time spent managing storage rather than improving pipelines.

The cross-availability zone tax

EBS volumes live in a single availability zone. High availability needs data replicated across zones, so in Kafka your replicas sit on brokers in different zones and every replicated write incurs cross-AZ data transfer charges. In AWS that is $0.01 per GB in each direction, which sounds trivial until you process terabytes per minute and data transfer outweighs compute and storage combined.

OSO engineers enabled fetch-from-closest-replica for consumers (KIP-392, set client.rack), so reads come from a broker in the same zone. That helps read-heavy workloads significantly. The producer side now has an equivalent: KIP-1123 rack-aware partitioning shipped in Kafka 4.4 (partitioner.rack.aware=true with client.rack) for records without a key. These are still optimisations layered on an inefficient model. Across multiple clusters in different VPCs, transit gateways for cross-cluster replication add yet another layer of charges, and OSO engineers spent substantial time finding and removing cross-VPC traffic.

When does block storage still make sense?

Block storage is not always wrong. For strict latency requirements, such as single-digit millisecond SLAs, it still performs best. EBS write latency is consistently low, and in single-AZ architectures where high availability matters less, block storage remains a valid choice.

Moving from gp2 to gp3 volumes was a significant improvement. gp3 lets you provision throughput independently of capacity, so OSO engineers removed IO bottlenecks without over-provisioning storage. Where gp3 replaced gp2, throughput rose enough to reduce broker counts and consolidate clusters, saving substantial cost while staying on block storage. Block storage is not bad. It is just not universally optimal.

Why is object storage cheaper for high-throughput streaming?

Object storage services such as S3 work differently. Instead of volumes attached to specific instances in specific zones, they provide a regional, multi-AZ service accessed over HTTP APIs.

Prices are AWS list prices for US East (N. Virginia). S3 Express One Zone prices reflect the April 2025 reductions.

Multi-AZ by default

The most immediate benefit is that object storage removes cross-AZ replication costs. Data written to S3 Standard is stored across multiple availability zones in the region at no extra replication charge, and the storage service handles the movement. OSO engineers no longer needed to care which zone a broker was in or tune producers and consumers to minimise cross-AZ traffic. Networking gets simpler too: no transit gateways for cross-VPC replication and no carefully designed peering, because object storage is reached through regional endpoints.

Operational simplicity

With object storage, disk capacity stops being an operational concern. You do not provision storage, you write data and pay for what you use, so there is no disk-full scenario and no emergency capacity increase. OSO engineers never had to replace an S3 bucket for poor performance, something they routinely did with EBS volumes. An S3 outage is effectively a region-level failure, which simplifies disaster recovery planning. Retention becomes a lifecycle rule rather than a capacity calculation.

Cost structure advantages

S3 Standard costs $0.023 per GB-month against $0.08 for gp3 EBS, roughly 3.5 times cheaper for storage alone, before counting the replicas that Kafka keeps on extra volumes. S3 also offers storage classes for different access patterns, and lifecycle policies can move data between them automatically.

S3 Express One Zone was a step change. It delivers single-digit millisecond access and much lower request prices than S3 Standard, although it stores data in a single zone and costs more per GB stored. In April 2025 AWS cut its storage price by 31%, PUT requests by 55% and GET requests by 85%. For some of OSO’s request-heavy pipelines, moving from S3 Standard to S3 Express One Zone cut costs by four to five times for identical workloads, simply by changing storage class. Innovation in object storage moves fast, driven by massive-scale users.

What is the latency trade-off?

Object storage latency is higher than block storage. An EBS read takes single-digit milliseconds, while an S3 Standard request can take tens to hundreds of milliseconds. S3 Express One Zone brings this down to single-digit milliseconds, but each write still has to be batched into larger objects to keep request costs sensible, which adds end-to-end latency.

If you must acknowledge writes in under 10 milliseconds, object storage alone will not meet that SLA. For high-throughput, cost-sensitive workloads where latency is measured in seconds, it enables architectures that are not economically viable on block storage.

When is Kafka tiered storage not enough?

Kafka tiered storage (KIP-405) is the obvious way to use object storage. Data lands on broker disks as normal and older segments move to object storage, and consumers can still read them through Kafka. Broker disks shrink, and adding or removing brokers is faster because there is less local data to move. For long retention with mostly recent reads, tiered storage makes sense. For a hands-on guide, see how to auto-scale Kafka with tiered storage in production.

It does not help every workload:

• Short retention. With 30-minute retention, data is consumed and deleted before it is cold enough to tier, yet you still pay for broker disks.
• Always-hot data. Continuous reprocessing, many consumers with different lag or analytics over recent history keep data hot, so tiering only adds complexity and latency.
• Non-Kafka access. Tiered data is managed by Kafka and read through Kafka APIs. Batch jobs, SQL engines and tools that do not speak the Kafka protocol cannot read it directly.

For the pipelines OSO engineers were optimising, short retention, always-hot data and the need to consume data outside the Kafka protocol all pointed to different architectures.

How does diskless Kafka change the picture?

Diskless Kafka implementations such as WarpStream (acquired by Confluent in 2024) take tiering to its conclusion and remove broker disks altogether. Producers’ data goes straight to object storage, and brokers become stateless agents that coordinate reads and writes. Brokers scale without state migration, storage scales independently of compute, and cross-AZ replication costs disappear because object storage handles durability.

OSO engineers evaluated this approach when WarpStream launched, but they needed a solution that was open source and part of an established foundation, and none existed at the time. That is changing: KIP-1150 (diskless topics) was accepted by the Apache Kafka community in March 2026, with the implementation details being worked through in KIP-1163 and KIP-1164. It was not available when decisions had to be made, and there was a deeper question anyway: if the goal is to write to object storage efficiently, do you need the Kafka protocol at all?

Do you need the Kafka protocol at all?

The Kafka protocol is well designed for reliable, ordered delivery with strong guarantees, but it is complex. Producers discover broker topology, manage partitions and handle retries and idempotency. Consumers handle group coordination, rebalancing and offset tracking. That complexity makes sense when talking to brokers that manage local state.

For some OSO pipelines, both ends were services under OSO’s control, built for the use case, so the protocol’s generality was just overhead. Writing straight to object storage still needs batching, because one S3 write per message would be ruinously expensive and slow, and it still needs consumer coordination: discovering new data, tracking progress and handling failures with exactly-once processing. Those problems need a different abstraction layer, and that is where Apache Flink came in.

How does Apache Flink act as an object storage gateway?

Apache Flink provides connectors for many sources and sinks, built-in stateful processing and strong fault tolerance with exactly-once semantics, which made it the foundation for object storage-backed pipelines without Kafka.

Flink separates compute from state, and it has always checkpointed state to durable storage such as S3. It can consume from Kafka and write to object storage, read from object storage and write back to Kafka, or skip Kafka entirely and use object storage for both ingestion and delivery. Flink supplies the stream processing semantics (windowing, aggregation, joins, exactly-once processing) while you choose the storage backend that suits each pipeline. If you are weighing Flink against Kafka Streams for this, see our Kafka Streams vs Flink comparison.

What does Flink 2.0 disaggregated state add?

Flink 2.0, released in March 2025, introduced disaggregated state management as an experimental feature. Previously, state lived on local disks with periodic checkpoints to remote storage. With the new ForSt state backend, remote storage such as S3 becomes the primary state store and local disk becomes an optional cache.

That removes local disk limits on state size, makes checkpoints much faster and speeds up recovery and rescaling because state does not have to be downloaded first. The Flink community reports up to 94% shorter checkpoint durations and up to 49 times faster recovery or rescaling. ForSt (a fork of RocksDB built for remote storage) handles low-level state, an asynchronous execution model lets state access happen without blocking the main processing thread, and the file system layer is tuned to reduce remote storage calls.

There is still room to improve for object storage workloads. Flink’s file system interface was designed for generic file systems, and operations such as listing directories to discover new files map poorly onto object storage APIs. That is where table formats come in.

Should you use Apache Iceberg or Apache Paimon for streaming?

Table formats organise data in object storage and handle schema evolution, time travel, ACID transactions and efficient queries. They also shape how streaming jobs write and read data.

OSO engineers started with Iceberg. It is mature, widely adopted and its creators (who went on to found Tabular, acquired by Databricks in 2024) built a file IO layer that avoids operations such as directory listing that are cheap on file systems but expensive on object storage. Iceberg worked well for stateful pipelines where data had to be queryable and consolidated, but two issues emerged. Latency rose with throughput, as larger batches meant longer time-to-visibility, and Iceberg had no consumer offset tracking for resuming streaming jobs after failures.

Paimon addressed both. It is built for streaming, includes consumer tracking so jobs resume from the right position, and kept latency low and consistent as throughput grew. It is not perfect: as files accumulate, HEAD requests rise even when no new data arrives, so OSO engineers used aggressive retention to delete processed files quickly. For stateful pipelines that need consolidated, queryable data with schema evolution and time travel, Paimon proved the right choice, and OSO continues to use it extensively. For a related cost guide, see how to reduce Kafka to Iceberg pipeline costs.

How did OSO reduce object storage requests by 71%?

Object storage charges per request as well as for storage, so an inefficient implementation can make millions of API calls per hour. OSO engineers focused on minimising requests while keeping latency low and throughput high, and cut request counts by 71% compared with their initial implementation.

Producer side: keyBy before writing

In a typical Flink pipeline, many parallel instances each write data for whichever destinations (partitions, topics or keys) they happen to see, producing many small files per checkpoint. Request costs are driven by the number of write operations, not the data volume, so many small files cost more than a few large ones.

The fix is to use Flink’s keyBy to route all records for a destination to one instance before writing. With 100 destinations, each checkpoint then writes exactly 100 larger files, one per destination, instead of many fragments from every instance. OSO engineers measured a 71% reduction in request counts from this change, more savings than most other optimisations combined.

The trade-offs are more network traffic inside the Flink cluster, possible data skew when one destination is much busier than others, and harder auto-scaling because destinations must be rebalanced across instances. Compared with the request savings, these were acceptable: traffic within a single availability zone is effectively free, and skew can be managed with careful key choice and monitoring.

Consumer side: discovering new files

Consumers must actively discover new files, because object storage has no subscription mechanism. Listing a bucket every 2 seconds means 43,200 list operations a day, re-checking the same files. Flink’s file source makes this worse: because of how its file system interface works, already-processed files still trigger calls, so 100 files can mean 200 to 300 requests every few seconds just to confirm nothing is new. The interface was built for HDFS and local file systems, where listing is cheap.

OSO engineers reduced polling frequency (every 10 or 30 seconds instead of 2), trading discovery latency for cost, and deleted files as soon as they were processed so list calls stayed small. Aggressive retention is risky: delete too early and a failure before processing completes means data loss, so retention had to be tuned carefully.

Which coordination options avoid list calls?

The most ambitious option, which OSO engineers began building as a next-generation Flink connector, is a lightweight table format for low-latency streaming on object storage. Drawing on lessons from Paimon and SlateDB (an embedded key-value storage engine built on object storage), producers write data files and update a small metadata structure that tracks which files exist, which are processed and what work remains. Consumers read that metadata instead of listing thousands of files, update their progress and coordinate to avoid duplicate processing. It is hard to get right with concurrent producers and failures, but OSO engineers see it as the future of streaming on object storage and are working to open source it.

How do you choose between block storage, tiered storage and object storage?

The move to disaggregated, object storage-backed streaming is not a binary switch. Match the architecture to each workload:

Which architectural patterns emerged?

OSO engineers migrated incrementally rather than overnight:

• Hybrid Kafka plus object storage. Real-time data flows through Kafka for low latency while history is also written to object storage. Consumers read recent data from Kafka and history from object storage, similar to what Pinterest built with client libraries that hide which store the data comes from.
• Kafka-free stateless pipelines. Flink jobs consume from the source, batch data, write to object storage, and downstream Flink jobs consume from object storage. No brokers and no cross-AZ charges.
• Paimon for stateful pipelines. CDC from databases flows into Paimon tables via Flink, processing jobs keep state in Paimon and downstream consumers query the latest state, which is simpler to run than separate Kafka topics, state stores and output databases.

How should you plan the migration?

Start with stateless pipelines. They carry less risk because there is no complex state to migrate and rollback is easier. Measure success with clear metrics: requests per GB processed, latency at p50, p95 and p99, and actual cost per TB processed. Build cost models that compare Kafka on EBS with object storage-backed Flink pipelines, including compute, storage, networking and API requests, across different throughput and retention levels, so decisions rest on economics rather than preference.

What does the future of disaggregated streaming look like?

Moving from block storage to object storage is more than an optimisation. It changes how streaming platforms are designed. OSO’s experience at massive scale showed a 71% reduction in object storage requests from smarter batching with keyBy, the removal of cross-AZ charges that had been the largest single cost component, and four to five times lower costs in specific pipelines through storage class choice, along with faster recovery and compute and storage that scale independently.

Getting there meant rethinking established patterns. Tiered storage suits specific workloads but not all of them. Diskless Kafka improves the model and is now heading into Apache Kafka itself through KIP-1150, while the most efficient option for high-throughput pipelines you control end to end often bypasses the Kafka protocol and lets Flink write to object storage with carefully tuned batching and coordination. As object storage keeps getting faster and cheaper, Flink matures its disaggregated state and table formats such as Paimon target real-time workloads, these advantages will grow.

Key takeaway: Object storage wins for high-throughput, multi-AZ streaming because it removes cross-AZ replication and capacity planning, but only if you design for its request pricing and latency. Batch writes with keyBy, avoid frequent LIST calls, keep block storage for sub-10 ms SLAs and pick tiered storage, diskless Kafka, direct Flink writes or Paimon per workload.

Frequently asked questions