Operating Apache Kafka at scale is complex, and one of the most common issues our clients face is storage. Broker disks have to be sized for retention, replicated three times and rebalanced when the cluster grows. Moving Kafka data to cloud object storage changes that cost model, and this post explains the two main ways to do it.
Traditionally, Kafka brokers store every partition on local disk, which becomes expensive and hard to manage at scale on public cloud. Tiered storage splits the log into two tiers:
| Tier | Where it lives | What it holds |
|---|---|---|
| Local tier | Broker disks | The active segment and recent data, controlled by local.retention.ms / local.retention.bytes |
| Remote tier | Object storage (S3, GCS, Azure Blob) | Older closed segments, kept for the topic’s full retention.ms |
Producers and consumers use the normal Kafka protocol. Consumers reading recent data are served from local disk; consumers reading older offsets are served from the remote tier through the broker. Tiered storage arrived as early access in Apache Kafka 3.6 and was declared production-ready in Kafka 3.9. It is enabled with remote.log.storage.system.enable=true on the brokers and remote.storage.enable=true on each topic, and it needs a RemoteStorageManager plugin for your object store. It does not support compacted topics.
Managed services offer it too: Amazon MSK provides tiered storage on provisioned clusters, and Confluent Cloud uses its own tiered storage under the hood.
Serverless or diskless Kafka designs, popularised by WarpStream (acquired by Confluent in 2024), remove broker disks altogether. The architecture typically looks like this:
The Apache Kafka community has also proposed diskless topics (KIP-1150), which would bring a similar write-to-object-storage model into open-source Kafka.
| Benefit | Tiered storage (KIP-405) | Serverless / diskless Kafka |
|---|---|---|
| Scalability | Retention grows in object storage, not on broker disks | Storage and compute scale independently |
| Cost | Fewer, smaller broker disks for long retention | No broker disks or inter-AZ replication traffic for data |
| Flexibility | Faster broker replacement and rebalancing, as less data sits locally | Agents can be deployed across VPCs, regions or cloud accounts for isolation |
| Load balancing | Partitions still balanced across brokers | Client connections are balanced instead of topic partitions |
Cloud object storage is typically far cheaper per gigabyte than replicated block storage, so organisations can keep much longer retention at lower cost. For managed options that build on these ideas, see our guide to the best managed Kafka services.
While object-storage designs offer many benefits, there are challenges to plan for:
As a rule of thumb, tiered storage suits teams that want cheaper long retention with familiar Kafka operations, while serverless Kafka suits high-throughput workloads that can tolerate higher end-to-end latency in exchange for lower cost.
Moving Kafka data to cloud object storage is one of the most effective ways to reduce the cost of running Kafka. By combining Kafka’s real-time processing with the scalability and cost-effectiveness of object storage, organisations can manage their data pipelines and streaming applications more efficiently. Start by measuring how much of your retention is rarely read, then choose between tiered storage on your existing clusters and a diskless platform for new workloads.
If you are interested in learning more or adopting this approach, contact OSO or explore our Kafka services.
Key takeaway: Kafka tiered storage keeps hot data on brokers and moves older segments to object storage, while serverless Kafka writes everything to object storage through stateless agents. Both lower storage cost; pick based on latency needs and how much operational change you can take on.
Tiered storage (KIP-405) lets Kafka brokers keep recent log segments on local disk and offload older closed segments to remote object storage. Clients keep using the normal Kafka protocol, and the broker fetches older data from the remote tier when a consumer asks for it.
Tiered storage shipped as early access in Apache Kafka 3.6 and was declared production-ready in Apache Kafka 3.9. You also need a RemoteStorageManager plugin for your object store, as Kafka does not bundle one for S3, GCS or Azure Blob Storage.
Tiered storage keeps brokers with local disks and only moves older data to object storage. Serverless or diskless Kafka removes broker disks altogether: stateless agents write directly to object storage and a separate metadata store tracks offsets and ordering.
No. Kafka tiered storage does not support topics with cleanup.policy set to compact, so compacted topics such as changelogs keep all their data on broker disks.
Usually, yes. Writing to object storage adds latency compared with local disks, so end-to-end latency in diskless designs is typically higher. Many analytics, logging and pipeline workloads tolerate this in exchange for lower cost, while latency-critical workloads may not.
Talk to OSO's Kafka engineers about tiered storage, diskless Kafka and right-sizing your clusters for long retention.
CONTACT US