How OSO helped IKEA restore order processing and rebuild Kafka
When Kafka-backed order services failed under live traffic, IKEA brought in OSO. OSO engineers found the root causes, replaced topic deletion with a safe recovery runbook and rebuilt the platform on Strimzi and KRaft.
15 June 20269 mins read
5
days to diagnose the platform and deliver a rebuild plan
130x
headroom above peak order traffic on the rebuilt cluster
<40 ms
p99 producer latency at ten times peak load
Overview
IKEA is one of the world’s best-known home furnishings brands, with EUR 44.6 billion in IKEA retail sales in its 2025 financial year. Behind every order sits a chain of services that has to answer in seconds. In one of IKEA’s regional order platforms, that chain runs through Apache Kafka: requests from the storefront pass through a multi-stage Kafka pipeline before a confirmation comes back to the customer.
In early 2026 that pipeline became the centre of an emergency. The Kafka platform behind regional order processing went down when its brokers could no longer renew their TLS certificates. Weeks later, a separate fault in the order modification service began timing out customer requests every evening. OSO engineers were called in to get order processing back online, find out why it had failed, and make sure it could not fail the same way again.
The strategic goals were clear. IKEA needed an order platform that could:
Process orders reliably again, with the cause of each failure found rather than guessed
Recover from a single bad message in minutes, without deleting topics or losing order data
Run on a supported Kafka platform that renews its own certificates and upgrades through code
Survive the loss of a cloud availability zone without stopping order processing
Give the in-house teams the alerts, runbooks and knowledge to run it themselves
This was emergency support first and platform engineering second, and the platform needed both.
The Technical Challenge and Why They Engaged OSO Engineers
The first outage began with a certificate renewal failure. The Kubernetes platform underneath Kafka had moved to a release that the cluster’s Strimzi operator was never built for, and the operator could no longer rotate broker certificates. When they expired, the brokers went down and order processing stopped. Workarounds brought the cluster back, but it was now impossible to upgrade in place: a supported operator meant a major Kafka version jump, a Strimzi API migration and a move away from ZooKeeper all at once, on a platform the current team had inherited without a detailed handover.
The second problem looked like infrastructure but wasn’t. IKEA’s order modification service accepts an HTTP request, publishes it to Kafka and waits synchronously for a reply from a Kafka Streams pipeline. From April, requests began timing out with HTTP 408 errors, first now and then and then every evening. Suspicion fell on the ingress layer and on Kafka itself. The only recovery that worked was deleting and recreating the affected topics, which restored service but destroyed business data and left no evidence of the cause. Each recurrence risked another major incident, with customers unable to change their orders.
All of this had to happen inside a live production estate. Several teams across time zones owned different layers, network routing sat with a separate team, and every change carried the risk of making an active incident worse.
IKEA engaged OSO for its depth in Apache Kafka and Strimzi under production pressure, its Kafka Streams expertise to separate symptoms from causes, and its experience rebuilding Kafka platforms as code.
The Solution
The OSO engineers worked in two tracks. In the first, they joined live incident calls with IKEA’s platform and application teams to restore order processing and find root causes. In the second, they ran a five-day investigation of the regional Kafka platform and followed it with a blue-green rebuild.
Blue-green rebuild: order services switch their bootstrap endpoint from the legacy cluster to the new three-zone KRaft cluster, with MirrorMaker 2 replication and a one-line rollback.
Emergency Kafka Support: Finding the Poison Pill
The OSO engineers first ruled out the infrastructure. The Kafka cluster was healthy throughout the order modification incidents, and the ingress errors were a symptom, not the cause. The real failure was a poison pill: two malformed messages that a Kafka Streams application could not process. Each one crashed a stream thread, stalled its partitions, and left the consumer group with no active members. With no consumer producing replies, every request for an order on those partitions waited out its timeout and returned a 408.
The engineers traced both failures to the exact class and line in the application code, one a null reference and one an Avro record missing a required field. A review of the application’s change history showed where generic exception handling had been removed from the processing step. IKEA’s application team fixed both defects, and the consumer group returned to a stable state with zero lag.
Recovery in Minutes, Without Losing Orders
Deleting topics had to stop. The OSO engineers built a log alert that fires on any Kafka Streams exception and reports the topic, partition and offset involved, so the next poison message is identified in minutes rather than hours. They then replaced topic deletion in the incident runbook with an offset-skip procedure:
Scale the consuming application to zero, so the consumer group is inactive
Capture the poison message for the application team as forensic evidence
Dry-run, then execute, an offset reset to the message after the poison one
Restart the application and confirm the group is stable with zero lag
Poison pill recovery in four steps: the alert names the offset, the consumer stops, the message is captured, the offset is skipped and the consumer restarts.
Alongside the runbook, the engineers set out a hardening plan for the application layer. It covers:
an uncaught exception handler as a backstop
dead letter queues for both deserialisation and processing failures
per-partition consumer lag alerting
enforced Schema Registry compatibility
a Kafka client upgrade
moving the order modification flow away from synchronous request-reply
Each change came with a test case to prove in UAT before production.
Diagnosing the Platform in Five Days
The OSO engineers spent five days on the regional platform. The investigation covered:
an audit of the Strimzi, Kafka and Kubernetes versions
a TLS and certificate review
an analysis of topics and partitions
a feasibility check on data volumes and replication
a trace of the private network route between the on-premises application platform and the Kafka network
The conclusion was clear. Upgrading the existing cluster in place would stack several high-risk changes onto a fragile production system, so a clean rebuild was the safer path.
A Blue-Green Rebuild on Strimzi and KRaft
The new platform is built entirely from code. Terraform provisions the Kubernetes cluster, Strimzi is installed from a pinned Helm chart, and every change is planned and applied through GitHub Actions. Nobody makes manual changes inside the cluster. The rebuilt stack runs:
Strimzi 0.49.1 and Apache Kafka 4.1.1 in KRaft mode, with no ZooKeeper and Strimzi’s v1 custom resources
Three brokers across three availability zones, with a replication factor of three and a minimum of two in-sync replicas, so the cluster keeps serving reads and writes if a zone is lost
TLS with SCRAM-SHA-512 authentication on the external listener, behind the cloud provider’s native load balancer instead of an ingress passthrough
A newer broker instance type that costs 23% less per node than the one it replaces
Certificate rotation handled by the operator, with expiry alerting planned 30 days ahead
Migration follows a blue-green pattern. The old and new clusters run side by side, with MirrorMaker 2 replicating topics. Applications switch their bootstrap endpoint service by service, and rollback is a single configuration change back to the old cluster. Production is built from the same modules as UAT, so the two differ only in size.
Load Testing Before Cutover
Before any order traffic moved, the OSO engineers load-tested the new cluster from inside the network. They used Kafka’s own performance tooling with production-sized messages and acks=all. Producer latency stayed flat, at 11 to 13 ms on average, from current peak traffic all the way to ten times peak, with p99 under 40 ms. Throughput with no limit applied gave more than 130 times headroom above peak. Consumers ran at nearly 800 times peak, and Schema Registry answered in 20 to 22 ms with zero errors across 1,800 sustained requests.
The Results and Outcomes
IKEA’s order processing is stable again. A failing message can now be identified and skipped in minutes, and the regional Kafka platform has been rebuilt as code with room to grow.
Order processing back online, with the cause found. The order modification service recovered, the consumer group runs stable with zero lag, and both defects behind the outages were fixed at source. Proving the Kafka cluster healthy ended weeks of chasing the wrong layer.
Minutes to recover, and no more deleted orders. The exception alert pinpoints the topic, partition and offset, and the offset-skip runbook replaces topic deletion. Business data and forensic evidence are both preserved.
A Kafka platform with more than 130 times headroom. The rebuilt cluster handles current peak traffic with p99 latency under 40 ms and stays flat at ten times peak. That is a small fraction of the order flow’s end-to-end latency budget.
Resilient to a zone outage, at lower cost. Brokers now span three availability zones, and the newer instance type costs 23% less per node.
Upgrades through code, not hotfixes. Terraform, Helm and GitHub Actions manage every change on a supported Strimzi and KRaft stack. The forced, emergency upgrade that caused the first outage is replaced by routine, reviewed releases.
A zero-downtime path to production. The blue-green design, MirrorMaker 2 replication and endpoint-level rollback let IKEA move order services across without a Kafka outage.
Kafka expertise on call. OSO’s senior Kafka engineers remain available to IKEA’s teams for platform support and incident response.
What OSO Delivered
Joined live incident calls and traced recurring order-service outages to two poison messages in a Kafka Streams application
Built a Kafka Streams exception alert and an offset-skip runbook that replaced data-destroying topic deletion
Set out an application hardening plan covering dead letter queues, exception handlers, lag alerting, schema compatibility and asynchronous request handling
Completed a five-day investigation of the regional Kafka platform and recommended a blue-green rebuild
Built a Strimzi 0.49.1 and Kafka 4.1.1 KRaft cluster across three availability zones, provisioned with Terraform, Helm and GitHub Actions
Load-tested the new cluster and documented scaling thresholds and next steps
Talk to one of our Kafka engineers about stabilising production incidents, finding root causes fast, and rebuilding your platform so the same failure cannot recur.
