Investigate why a Kafka proof of concept helped Dufry prove the merits of its new data architecture

Project overview: Dufry’s data dilemma  

Picture this: it’s the height of the summer travel season, and airports around the world are buzzing with activity. With more than 50,000 products, 5,500 outlets, 75 countries, 1,200 locations, and 20% of the airport retail market share worldwide, Dufry is a multinational retailer with shops and stores around the globe. Thousands of passengers stream through the gates every hour, searching for the perfect souvenir or trinket. Meanwhile, behind the scenes, duty-free retailers like Dufry are working to keep their shops stocked so they can serve millions of travellers around the world. But the company’s batch processing system wasn’t working.

The challenges of batch processing

Challenge: Proving the merits of a Kafka architecture 

Dufry’s Head of Data, CRO, and CTO understood that Apache Kafka could help them address their data pipeline issues, but before adopting a new data architecture, they wanted to prove Kafka’s merits, test its compatibility with their legacy systems, and gather test data to build a case for full Apache Kafka adoption in the future. 

At this point, they hired us to design and build a Kafka proof of concept. (Sometimes, adopting new architecture can take a little bit of work for large organisations to implement, and a trial would allow Dufry to collect data to support a Kafka rollout at a larger scale.) If successful, a proof of concept would validate their assumptions and help prove the value of Kafka architecture to senior leadership.

Designing the technical requirements for Dufry’s proof of concept

In early 2021, we started to design and draft the proof of concept’s requirements. Dufry already had two batch systems in place: sales data from Point of Sale registers and stock data from SAP Enterprise Resource System (ERP). The retailer also provided and managed the systems and software for a variety of global brands under its umbrella: Hellenic Duty-Free in Greece, Duty-Free Uruguay, and Colombian Emeralds. This meant that Dufry had a lot of different global systems, and it made sense that they didn’t want to roll out a brand-new Kafka architecture without testing it first. 

We started out by thinking about how best to design Dufry’s integration architecture. By using the ASAPIO Integration Add-on, the SAP ERP and POS systems could expose their data feeds to publish into Confluent’s event streaming platform. Then Dufry could send streams of data to Dufry Insights, its business analytics tool, and Dufry Digital Channels, its online shopping platform. 

As you can see in the diagram below, Dufry’s Kafka system would capture stock data from its Enterprise Resource Platform and publish the data as Kafka topics, pushing global stock updates to Dufry’s Digital Insights analytics engine. This system would also collect continuous sales data from Dufry’s Point of Sales registers (Kafka producer clients) and publish this data as events to Dufry Insights and Digital Channels, allowing the company to analyse sales trends and quickly replenish popular products.

Solution: Launching a Kafka proof of concept

The Confluent components needed to launch the proof of concept 

Next, we implemented a core set of Confluent components, including Kafka Connect sink and source connectors, Confluent Kubernetes Operator (CFK), ksqlDB, and Confluent Schema Registry to integrate Confluent Platform with Dufry’s multi-cloud infrastructure and legacy systems. 

These components are pretty standard for most implementations, but if your in-house team hasn’t worked with Confluent or Kafka before, you usually work with a Confluent-certified developer or team. (Some of these components also undergo updates from time to time, so you want to make sure you’re using the latest versions.)

The Confluent components we used for Dufry’s proof of concept

Conducting functional tests to assess the proof of concept

When you run a proof of concept, you want to make sure that you come away from it with a better sense of the project’s scope, cost, and concrete examples of the system working for different use cases.

Once we integrated Confluent Platform and Dufry’s systems using Kafka Connect with source and sink connectors, we conducted functional tests. Testing the system using realistic stock and sales scenarios, ones that closely resembled what the retailer would use it for on a daily basis, would show Dufry’s senior leadership how real-time data reports could transform the efficiency of their operations—not just in theory, but in practice. 

To do this, we ran two separate tests to help Dufry’s data leaders prove that Kafka architecture could have a tangible impact on their daily operations, helping them track profit and loss and global stock. As you can see in the tables below, Use Case 1 (Live Stock) and Use Case 2 (Live Sales) illustrate how the Streaming Data Hub Kafka Project connected Dufry’s data sources to its business analytics. 

Dufry’s Streaming Data Hub Kafka Project – Sales and Stock Functional Tests

Project results: Kafka proof of concept experiment pans out 

Now that Dufry’s data leaders have completed their Kafka proof of concept, they have the evidence and data they need to advocate for a wider rollout in the future.

• Concrete tables and data with which to pitch a broader Kafka rollout.

• The knowledge that Kafka successfully integrates with their legacy architecture. 
• The ability to examine bugs or issues in the proof of concept before launching the system on a wider scale.
• Event-driven stock and sales data will let Dufry order new stock at more dynamic intervals and visualise emerging trends, such as the increase in demand for limited edition items, environmentally-friendly souvenirs, and digital products.  

Based on its experiment with Kafka, it looks like Dufry might finally be able to feel confident in its global strategy for stocking and shipping products. Even at the height of the travel season, event-driven data reports will ensure customers have what they want—whether that be a limited edition Italian watch, a bottle of perfume shipped to England, or the perfect souvenir for their friends and family back home.

Curious about a Kafka proof of concept for your organisation?

Connect with us 

Struggling to get senior leadership on board with a new data architecture? Wanting to test Kafka before rolling it out across your entire organisation? 

We’re seeing a larger need for fault-tolerant messaging systems that help businesses react to quick spikes in supply and demand. Beyond retail, other companies use these streams of records to aggregate and send real-time user data to monitor website activity, respond to global sales, and provide user updates. In all of these situations, Kafka enables businesses to increase the scalability of their architecture, while also giving data teams access to the broader Kafka community. 

If you’re curious about how we might adapt the Dufry proof of concept process for your purposes, reach out to us. A lot depends on the legacy architecture you currently have in place and your time scale, but much of the work we do is heavily customised to the needs of individual customers. 

We’re available via email and LinkedIn, and you can always fill out the form on our website to get in touch.

Other content to look at:

How 33N Enabled NHS Real-Time Data Visualisation Successfully

How Drivvn migrated from batch to real-time streaming data with Apache Kafka

How to take your Kafka projects to the next level with a Confluent preferred partner

Event driven Architecture: A Simple Guide

Watch Our Kafka Summit Talk: Offering Kafka as a Service in Your Organisation

Successfully Reduce AWS Costs: 4 Powerful Ways