How OSO helped Colliers design a real-time, event-driven CRM-to-ERP integration architecture across EMEA

Overview

Colliers International is one of the world’s leading diversified professional services and investment management companies, operating across more than 60 countries. With a significant European commercial real estate presence spanning offices from the UK and Germany to Romania, the Netherlands, and Scandinavia, Colliers manages complex, high-value client relationships across multiple jurisdictions — each with their own data, systems, and teams.

As part of its broader Fina+ finance transformation programme, Colliers set out to replace its fragmented landscape of regional ERP systems with a single, EMEA-wide instance of SAP S4/HANA Cloud. The ambition was to standardise financial operations, improve data quality, and accelerate reporting across the business. But one integration stood out as uniquely difficult: connecting Colliers’ 19 different regional CRM systems — running on Salesforce, Microsoft Dynamics 365, and Integis — to a single ERP, in near real-time, with bidirectional data flow.

The strategic goals were clear. Colliers needed a client integration layer that could:

Synchronise client and project data from 19 CRMs to SAP S4/HANA in near real-time, and return ERP-generated identifiers back to those CRMs
Prevent duplicate client records across EMEA by establishing a single “golden record” for every client, mastered in the Profisee Master Data Management (MDM) platform
Validate and enrich client data against external providers — including VAT registries, company data brokers, and address validation APIs — before records were committed to the ERP
Build an architecture that was loosely coupled and extensible, so that future CRMs, geographies, and data domains could be onboarded with minimal rework
Establish Kafka as Colliers’ standard enterprise event-streaming platform, setting the pattern for future integrations across EMEA and beyond

This was not a routine systems integration. It was the foundation for a new, event-driven way of operating Colliers’ technology landscape.

The Technical Challenge and Why They Engaged OSO Engineers

The core integration challenge was formidable. Colliers’ EMEA CRM landscape had evolved organically over many years, resulting in 19 separate CRM instances running across three distinct platforms: Salesforce (UK, Germany), Microsoft Dynamics 365 (Netherlands, Norway/Sweden), and Integis (Romania). Each system stored client and project data differently, with no common schema, no shared identity model, and no existing mechanism for cross-border data synchronisation.

The planned integration had to be bidirectional — CRMs needed to push client and project data to SAP S4/HANA, and SAP needed to return system-generated identifiers (Business Partner IDs, project numbers) back to the originating CRMs in near real-time. A failed or delayed sync could leave brokers working with stale or incorrect client data at a critical moment, with direct commercial consequences.

The risk of client record duplication was particularly acute. Without a robust deduplication mechanism, the same client company could be created multiple times in SAP from different CRMs — generating multiple Business Partner IDs and corrupting downstream financial reporting. The design needed to ensure that every client flowing through the system was validated against Colliers’ Profisee MDM before being written to the ERP.

Colliers had no existing Apache Kafka installation anywhere in its EMEA technology estate. The organisation had no internal Kafka expertise, no established patterns for topic design or schema governance, and no precedent for the kind of event-driven, containerised microservice architecture the project required. The integration also needed to comply with strict data residency and security requirements: all client data had to remain within Colliers’ Azure tenant, with no unencrypted transmission over the public internet.

Further compounding the complexity, the project introduced several technologies that were entirely new to Colliers EMEA: Confluent Kafka, Kubernetes on Azure Kubernetes Service (AKS), Terraform Enterprise for infrastructure-as-code, and ArgoCD for GitOps-based continuous delivery. Without deep external expertise to guide the design and validate the architecture, the risk of building a fragile, unscalable integration was significant — and the consequences of getting it wrong at ERP launch would be severe.

Colliers engaged OSO for their proven Kafka expertise, their ability to review and stress-test architectural designs before development began, and their experience delivering event-driven integrations in complex enterprise environments.

The Solution

OSO engineers worked alongside the Colliers Stream+ integration team to design, review, and validate the full Kafka architecture for the EMEA Client Management Layer (CML) — the event-driven microservice integration that would connect 19 CRMs to SAP S4/HANA via Confluent Kafka. The engagement spanned an intensive architecture design workshop, a comprehensive Kafka health check, and ongoing expert consultancy throughout the design phase.

Confluent Cloud Enterprise: Platform Selection and Cluster Design

The first design decision was the choice of Kafka platform. OSO engineers evaluated open-source Apache Kafka, Confluent Community Edition, and Confluent Cloud Enterprise, presenting a structured comparison that highlighted the operational and commercial trade-offs of each.

The recommendation was Confluent Cloud Enterprise, hosted in Azure’s westeurope region. The Enterprise tier provided Confluent’s Schema Registry, managed connectors, Role-Based Access Control (RBAC), audit logging, multi-zone cluster replication, and a native Terraform provider — capabilities that would otherwise have required significant in-house development and operational overhead. Crucially, Confluent’s Private Link integration meant that all topic connections and Kafka API traffic could be routed entirely within the Azure private network, with only the Confluent control plane API exposed to the public internet (and accessed exclusively via Terraform).

Two Enterprise clusters were provisioned — one for production and one for all non-production environments — separated at the Confluent environment level and mirrored in Azure via two corresponding Resource Groups (StreamPlusNonProd and StreamPlusProd). Rather than multiplying clusters for each test stage, OSO recommended using topic-level naming conventions to achieve soft partitioning within the non-production environment, avoiding the significant cost of additional Enterprise cluster licences.

Topic Design: Naming Conventions, Partitioning, and Compaction Strategy

OSO engineers worked with the Colliers team to establish a clear and consistent topic naming convention that would scale as the Kafka footprint grew beyond the initial CRM-to-ERP integration:

<kafka-host>-<environment>-<visibility>-<responsible>-<what-it-contains>

For example, cc-d-pri-emea-request denotes a Confluent Cloud topic, in the development environment, private visibility (accessible only to Kafka applications), owned by the EMEA integration domain, carrying EMEA enrichment request messages. This convention encoded environment, data classification, and ownership directly into the topic name — removing ambiguity for developers and simplifying access control configuration.

Partitions were set at 10 per topic, a deliberate choice based on the throughput characteristics of the integration. The expected bottleneck was the external API providers (VAT registries, company data brokers, address validation services), not Kafka itself. Ten partitions provided sufficient parallelism for horizontal scaling of Kubernetes pods without over-provisioning.

Two topics — the EMEA Client Topic and the EMEA Project Topic — were configured with log compaction. Compacted topics preserve only the latest message for each key, effectively making Kafka the external system of record for EMEA client and project state outside of SAP S4/HANA. This “data wick” pattern allowed any system to replay the current state of any client or project at any point, without querying SAP directly.

The Three-Microservice CML Architecture

The heart of the solution was a set of three custom microservices, each hosted in its own Kubernetes pod on Azure Kubernetes Service (AKS), designed around the Command and Query Responsibility Segregation (CQRS) pattern. All microservices were stateless, enabling horizontal scaling via AKS auto-scaling policies triggered by CPU and memory metrics.

1. Transformation Service The Transformation Service subscribed to CRM-specific source topics (for example, cc-d-pub-uk-salesforce-source) and transformed each CRM’s proprietary JSON payload into a standardised EMEA message schema. It was responsible for extracting client sections from umbrella project messages, publishing them as discrete CRM Client Messages, and distributing enriched EMEA messages back to the appropriate CRM sinks. Country-specific data distribution — a critical data privacy requirement — was handled here, ensuring that enriched client records were only returned to CRMs operating in the client’s country of origin.

2. CML Service The CML Service consumed from the EMEA Request Topic and managed the sequence control logic that orchestrated the entire enrichment pipeline. Before a project message could be sent to SAP, all client records embedded within it had to be fully enriched, validated, and assigned a SAP Business Partner ID (BPID). The CML Service maintained a compacted sequence state topic to track which enrichment steps had completed and which were still pending for each message — handling asynchronous responses from external providers and SAP CPI without blocking the pipeline.

3. External Provider Handler The External Provider Handler subscribed to the Provider Request Topic and was responsible for all outbound calls to external data sources. It made REST API calls to: EU VIES (VAT number validation for EU entities), HMRC (VAT validation for UK entities), Profisee (address standardisation via Google Maps API, and client deduplication and golden record management), Moodys/BvD (Orbis) and Dun & Bradstreet (company identity and hierarchy validation), and SAP CPI (triggering client and project upserts in S4/HANA via whitelisted public cloud APIs). Responses were published to the Provider Response Topic, where the CML Service consumed them to update sequence state and progress each message through the pipeline.

Each External Provider Handler could be deployed as an independent pod, meaning a failure or throttling event affecting one external provider would not cascade to others.

Connector Strategy

OSO engineers reviewed an options paper covering the available Confluent connector types for each integrated system, and provided recommendations on which to deploy:

Salesforce: Platform Events connectors were selected over CDC connectors. Salesforce Platform Events allowed CRM SMEs to define custom event objects with precisely the fields required for the integration, and to configure their own CRM-side triggers (opportunity win, client creation, invoice finalisation, and more than 30 other business events). A Platform Events Source connector published to the CRM Project Topic; Platform Events Sink connectors subscribed to the EMEA Project Topic, EMEA Client Topic, and Notification Topic to write enriched data back to Salesforce.
Dynamics 365: Standard Kafka REST connectors referenced whitelisted OData endpoints in Dynamics 365, supporting CRUD operations on Dynamics objects. REST Source connectors published project messages; REST Sink connectors received enriched EMEA data.
SAP S4/HANA: SAP connectors interfaced via SAP Cloud Platform Integration (CPI), using Kafka to trigger CPI flows that called whitelisted S4/HANA public cloud APIs. A CPI Sink connector received project and client messages from the CML; a CPI Source connector published SAP-generated project numbers, BPIDs, invoice details, and payment data back to Kafka for consumption by the CML and downstream CRMs.

Custom JSON Message Schema Design

OSO engineers helped design nine distinct JSON message schemas — stored and versioned in Azure DevOps repositories, registered in Confluent’s Schema Registry, and enforced at the topic level so that malformed messages were rejected before entering the pipeline. Key schema types included:

CRM Project Message: The umbrella message encapsulating all project-related data objects (client, product lines, sales order, invoice, payment) published by each CRM
CRM Client Message: Client-only messages published by the Client Interface Application (a lightweight web interface for countries without a fully integrated CRM)
ERP Client Message: The SAP-formatted client message sent via CPI to create or update a Business Partner in S4/HANA
BPID Message: Returned from SAP after a successful client upsert, carrying the Business Partner ID back to the CML for golden record update and CRM distribution
EMEA Client Message: The enriched, validated client record published after BPID assignment, carrying both SAP identifiers and the original CRM data segments for each contributing CRM
EMEA Project Message: The compacted, canonical project record retained indefinitely in Kafka as the external system of record
Notification Message: A lightweight status and error message published to a dedicated Notification Topic at every step of the pipeline, allowing CRMs to surface real-time progress updates and structured error codes to users

Error handling was built around structured JSON error responses with machine-readable code fields and human-readable message fields, covering the full range of OAuth 2.0 authentication errors, HTTP 4xx/5xx API errors, schema validation failures, and custom data quality error codes.

Security Architecture

Security was designed in depth, with multiple controls at every layer:

Transport: All Confluent Kafka communication used SASL/SSL, with TLS 1.2 enforced across all service-to-service connections
Authentication: Confluent Cloud used SASL with OAuth 2.0, with Colliers’ Azure Entra ID as the Identity Provider. Azure components used SAML for authentication and OAuth 2.0 for authorisation. Terraform Enterprise and ArgoCD were both integrated with Colliers’ SSO via Entra ID
Authorisation: Confluent topics were protected by a combination of Role-Based Access Control (RBAC) and Access Control Lists (ACLs), with a dedicated service account per CRM to enforce least-privilege access
Network isolation: Confluent topic connections and all Kafka APIs were routed exclusively over Azure Private Link, keeping all data traffic within Colliers’ Azure private network. Only the Confluent control plane API (accessed solely by Terraform) was reachable over the public internet
Secret management: All credentials — Confluent API keys, Azure managed identities, Terraform workspace tokens — were stored in Azure Key Vault, with applications retrieving secrets at runtime via OIDC federated credentials rather than embedding them in code

Infrastructure as Code, GitOps, and CI/CD

OSO engineers validated and provided guidance on the full DevOps and deployment stack — the first of its kind in Colliers’ EMEA technology estate:

Terraform Enterprise was used for all Azure and Confluent resource provisioning, configured with workspace-level scoping so that non-production and production infrastructure changes were separated and gated. Workspaces authenticated to Azure via federated credentials and to Confluent via API keys, all retrieved from Key Vault
ArgoCD provided declarative, GitOps-based continuous delivery for the CML microservices, monitoring Azure DevOps application deployment repositories for version changes and automatically synchronising Kubernetes cluster state. ArgoCD’s canary deployment capability allowed new microservice versions to be rolled out to a single pod first, enabling smoke testing before full cluster rollout
Azure DevOps managed all source code repositories, CI pipelines (triggering unit tests, linting, and integration tests on every pull request), and deployment pipelines to dev, test, and production environments — with manual approval gates before any production deployment

Monitoring and Observability

A Grafana dashboard on AKS provided real-time visibility of Kafka cluster health, topic throughput, connector task status, schema registry metrics, and authentication event counts. Kafka topic logs were shipped to Splunk and retained in Azure Blob Storage for one month. Critical alerts — cluster outages, performance degradation, storage limits — were routed to distribution list mailboxes for third-line Colliers support. Daily and weekly throughput reports compared actual message volumes against expected baselines, with deviations flagged for investigation.

Operational Rollback with OSO Kafka Backup

One of the more subtle but commercially significant risks in a CRM-to-ERP integration of this complexity is the possibility of a bad data event propagating silently through the pipeline before it is detected. A mis-mapped field in a CRM connector, an incorrect transformation rule, or a corrupted sequence state record in the CML’s compacted statestore topics could result in a batch of client or project records arriving in SAP S4/HANA with incorrect data — and by the time the error surfaces in a downstream financial report, hundreds of messages may have already been processed.

To give Colliers’ operations team the ability to recover from exactly these scenarios, OSO engineers deployed OSO Kafka Backup across the Stream+ Confluent Cloud environment. Built in Rust for high throughput and low overhead, OSO Kafka Backup provided millisecond-precision point-in-time recovery for all CML topics — including the compacted statestore topics that the CML Service used to track enrichment sequence state for every in-flight client and project message.

The tool was deployed using its Kubernetes-native operator, integrating directly with the existing AKS cluster via a Custom Resource Definition (CRD) and managed declaratively through the same Terraform and ArgoCD pipelines that governed the rest of the platform. Backups were written to Azure Blob Storage — already provisioned as part of the Stream+ infrastructure — ensuring that no additional storage infrastructure was required. Automated backup schedules were configured via cron expressions within the operator, with no maintenance windows or broker downtime required.

In practice, this gave the Colliers operations team two critical recovery capabilities. For statestore errors — where a bug or data quality issue corrupts the CML’s compacted sequence state records, causing the pipeline to mishandle the ordering or completeness of enrichment steps — operators could restore the affected statestore topics to a known-good point in time and replay messages from that offset, without touching SAP or the upstream CRMs. For CRM-to-ERP mapping errors — where an incorrectly configured connector transformation causes a set of client or project records to arrive in S4/HANA with malformed field values — operators could identify the precise timestamp at which the bad mapping was introduced, restore the relevant CRM source and ERP sink topics to that point, correct the connector configuration, and replay only the affected messages. In both cases, recovery was scoped to individual topics and time windows, avoiding the need for a full cluster restore.

The Results and Outcomes

The architecture designed with OSO’s support gave Colliers a production-grade, enterprise Kafka platform built to the highest standards of security, resilience, and operational maturity — and did so for what was an organisation’s first foray into event-driven, cloud-native infrastructure.

Near real-time CRM-to-ERP synchronisation across 19 systems. The CML pipeline processes CRM-originated project and client messages end-to-end in near real-time under normal operating conditions, with any latency attributable to external API providers rather than the Kafka layer itself. Brokers across EMEA receive SAP-generated project numbers and client identifiers without manual intervention or batch processing delays.

A single EMEA golden client record. By routing all client data through Profisee MDM — with validation against VIES, HMRC, BvD, and D&B — every client entering SAP S4/HANA is deduplicated, validated, and enriched before a Business Partner record is created. The compacted EMEA Client Topic retains the current state of every EMEA client record indefinitely, acting as an authoritative external system of record accessible to any future Kafka consumer.

Colliers’ first Kubernetes and containerised microservices platform in EMEA. The CML’s three microservices — each independently deployable, independently scalable, and immune to pod restarts — established the pattern for containerised application hosting across Colliers’ EMEA technology estate. Auto-scaling policies ensure the platform responds to load spikes without manual intervention, while ceiling constraints protect against runaway cost.

Colliers’ first Infrastructure-as-Code deployment in EMEA. Terraform Enterprise provisioned every Azure and Confluent resource in the solution, meaning the entire environment — clusters, topics, connectors, networking, Kubernetes, secrets — can be torn down and rebuilt deterministically from code. This is a significant operational capability step change for Colliers’ EMEA infrastructure team.

A scalable foundation for broader Kafka adoption. The Stream+ integration was deliberately scoped and priced as a low-volume, low-risk entry point into Kafka-mediated event-driven architecture. With the infrastructure, patterns, naming conventions, schema registry, and CI/CD pipelines in place, the cost and effort of onboarding additional CRMs, geographies, and business domains is substantially reduced. Property management systems, IoT devices, and global client data platforms are already identified as candidates for the next phase of Kafka expansion.

Operator-controlled point-in-time rollback for pipeline errors. The deployment of OSO Kafka Backup gave Colliers’ operations team a surgical recovery capability that would not otherwise exist in a Kafka-based integration. Rather than raising a support ticket and waiting for a developer to diagnose and manually replay messages, operators can now identify the timestamp at which a CRM mapping error or statestore corruption was introduced, restore the affected topics to that point, and replay only the impacted messages — all without touching SAP, the upstream CRMs, or other topics in the cluster. This materially reduces the blast radius and mean time to recovery for any data quality incident in the pipeline.

Long-term architecture partnership. OSO’s involvement gave Colliers’ integration and architecture team the confidence and knowledge to take ownership of the platform at go-live, with the option to draw on OSO’s Kafka support services — including ongoing health checks and managed service options — as the estate grows.