Multi-message types in Kafka topics: Lessons from a real-world pipeline

When your Kafka pipeline must handle dozens of message types from diverse systems, the conventional “one message type per topic” approach quickly breaks down. This is especially true in complex domains like travel, where data comes in many formats and structures, from room reservations to spa bookings and flight bookings.

At OSO, we recently worked with a travel company serving tens of thousands of hotels globally, and we faced exactly this challenge. What made the use case particularly hard was the diversity of data sources. We had to connect to hotel systems ranging from modern cloud-based APIs to legacy DOS-based platforms, with everything in between. These systems send data through webhooks, APIs, FTP transfers and even email. Some data arrives in real time, while other data comes in batches.

In this post, I’ll share how OSO solved one of our biggest architectural challenges: creating a schema approach that could handle multiple message types while maintaining order, simplifying development and ensuring consistent processing.

Why does one message type per topic break down?

The conventional wisdom in Kafka suggests that each topic should contain just one message type. This works as a form of “strong typing” for Kafka topics, which can prevent errors and simplify processing logic. Initially, we followed this guidance: room reservation messages went into a room reservation topic, restaurant reservation messages into a restaurant reservation topic, and so on.

However, as our schema expanded and grew more complex, we hit several significant problems.

Cross-topic ordering problems

When related messages live in different topics, enforcing message ordering becomes difficult. For example, if a spa reservation (in the spa reservation topic) can only be processed after a specific room reservation (in the room reservation topic), coordinating that dependency becomes unnecessarily complex.

Variations within message types

Even within a single conceptual message type, variations exist. A room reservation update event might contain dozens of fields (reservation ID, account information, guest name, check-in and check-out dates), while a room reservation deletion event might contain only the reservation ID. Both are technically “room reservation events”, but they have different structures and processing needs.

Nested message structures

In hospitality data, complex relationships exist between entities. Room reservations and golf reservations can both contain guest profiles. These profiles are valid messages in their own right and might need separate processing, yet they are nested within other message types. With one message type per topic, should these embedded profiles be extracted and placed in a profile topic? This quickly creates confusion and complexity.

Service configuration overhead

Services often need to consume multiple message types. Adding a new message type in a traditional setup means you have to:

1. Create a new topic for the message type
2. Update your service to consume from the new topic
3. Implement new serialisers and deserialisers
4. Add logic to handle the new message type

This overhead becomes significant when your domain contains dozens of related message types, especially when many services could process new message types without code changes if only they were receiving them.

How does a unified schema handle multiple message types?

To address these challenges, we designed a generalised schema that could be used across all topics. We implemented it with Protocol Buffers (protobuf), which provided strong typing while allowing for flexibility.

The hotel event pattern

At the core of the solution is a “Hotel Event”: a parent message type that can contain any of our specialised message types. In protobuf, we used the oneof field type to implement this pattern:

message HotelEvent {
  oneof event_type {
    RoomReservation room_reservation = 1;
    RestaurantReservation restaurant_reservation = 2;
    GolfReservation golf_reservation = 3;
    Profile profile = 4;
    // ... other event types
  }
}

message RoomReservation {
  oneof event_action {
    RoomReservationUpdated updated = 1;
    RoomReservationDeleted deleted = 2;
    // ... other actions
  }
}

// Define other message types similarly

This approach keeps strong typing (each message is a specific type with a defined structure) while unifying all messages under a common parent type. Every message on any topic is a HotelEvent, regardless of its specific type.

What are the advantages of the unified schema?

Key takeaway: model your Kafka topics on how your domain’s events relate to each other, not on a fixed one-type-per-topic rule.

How do you implement multi-message topics in practice?

While the unified schema approach solved many problems, it introduced some implementation challenges we had to address.

Handling switch statement complexity

With a unified schema and the oneof pattern, your processing code will contain switch statements to handle the different message types:

public void process(HotelEvent event) {
  switch (event.getEventTypeCase()) {
    case ROOM_RESERVATION:
      RoomReservation reservation = event.getRoomReservation();
      switch (reservation.getEventActionCase()) {
        case UPDATED:
          handleRoomReservationUpdate(reservation.getUpdated());
          break;
        case DELETED:
          handleRoomReservationDelete(reservation.getDeleted());
          break;
        // ... other cases
      }
      break;
    case RESTAURANT_RESERVATION:
      // Similar nested switch
      break;
    // ... other event types
  }
}

This can lead to deep nesting and verbose code. We mitigated it by:

1. Using the visitor pattern where appropriate
2. Creating utility functions to handle common processing paths
3. Breaking large switch statements into smaller, focused methods

Serialisation and deserialisation strategies

With a unified schema, serialisation and deserialisation become more consistent. Every service can use the same serialiser and deserialiser for HotelEvent, rather than needing specialised ones for each message type. This significantly reduced boilerplate code.

Schema evolution management

We used the buf CLI tool for protobuf linting, formatting and detecting breaking changes. This was crucial for keeping the schema compatible as the system evolved. The tool helped us:

• Ensure consistent naming conventions
• Detect accidental breaking changes before deployment
• Auto-format protobuf files for readability
• Generate code consistently across services

For more on keeping schemas compatible over time, see how to manage schemas in Apache Kafka.

What results did the unified schema deliver?

After implementing the unified schema approach across our Kafka ecosystem, we saw several positive outcomes.

Development velocity improvements

Adding new message types became significantly easier. Instead of creating new topics and updating multiple services, we simply added new types to the protobuf schema and deployed the updated schema package. Services that needed to process the new types could update their logic, while others kept working without changes.

Onboarding experience

New developers found the system more intuitive. With a consistent message structure across the platform, they understood the data flow more quickly and could focus on business logic rather than the details of Kafka topic configuration.

Performance considerations

We did see minor overheads from the more complex schema structure, mainly in serialisation and deserialisation time and message size. However, these were small compared with the architectural benefits.

When is a unified schema the wrong choice?

The unified schema approach isn’t suitable for every use case. Think twice if:

• Your message types are fundamentally different, with no conceptual relationship
• You have strong security requirements that need physical separation of message types
• Different message types have very different retention or processing requirements
• Your organisation has strict team boundaries where different teams own different message types

How should you plan the move to multi-message topics?

If you’re considering a unified schema approach, these practical steps will help you get started.

Questions to ask before implementation

1. Do your message types share conceptual relationships?
2. Do you need to maintain order across different message types?
3. Are you managing a complex web of topics that is becoming difficult to maintain?
4. Would your services benefit from more flexible consumption patterns?

Transitioning from separate topics

1. Start with a schema audit: document all your current message types and their relationships.
2. Design your unified schema: create a common parent type with specialised subtypes.
3. Build a proof of concept: implement the new schema in a limited scope.
4. Develop a migration strategy: consider running dual systems temporarily.
5. Roll out incrementally: convert one message flow at a time.

Configuration recommendations

• Increase the default message size limit if your unified messages are larger
• Adjust consumer configurations to account for potentially more complex processing
• Configure serialisers to handle the unified schema efficiently
• Consider compression if message size becomes an issue

Managing schema evolution

• Establish clear guidelines for adding new message types
• Use tooling to detect breaking changes
• Create a versioning strategy for your unified schema
• Consider backward and forward compatibility requirements

What did we learn beyond one topic per type?

Moving from a traditional one-topic-per-type Kafka architecture to a unified schema taught us that conventional wisdom doesn’t always apply to complex, real-world systems. By challenging the standard pattern and designing a solution tailored to our domain, we created a more flexible, maintainable and developer-friendly system.

The key insight is that your Kafka architecture should reflect the natural structure of your domain. In hospitality, where data entities are richly interconnected, a unified schema let us model those relationships more accurately and process them more consistently.

As you evaluate your own Kafka architecture, ask whether your current topic structure serves your domain or creates unnecessary complexity. Sometimes the simplest solution is to embrace the complexity of your domain within your schema, rather than flattening it into disconnected topics. If you are designing events from scratch, our guide on how to design Kafka events and event streams covers the wider modelling choices.

Frequently asked questions