Updated November 2026: We have restructured this guide into five clear steps, clarified what dbt does and doesn’t do with Kafka, and added current options for near-real-time dbt models.
dbt (data build tool) focuses on transformation. It doesn’t extract or load data, so in a streaming pipeline something else moves events out of Kafka: typically Kafka Connect sink connectors, a warehouse’s streaming ingestion such as Snowpipe Streaming, or a streaming database that reads topics directly.
Once the data has landed, dbt models turn raw events into clean, tested tables. That combination, streaming ingestion plus frequent or continuous dbt runs, is what most teams mean by dbt streaming. How fresh those tables are depends on how you run dbt:
| Approach | How it keeps data fresh | Good fit |
|---|---|---|
| Scheduled incremental models | Each run processes only new rows since the last run | Minutes-level freshness on any warehouse |
| Microbatch incremental strategy | Processes time-based batches independently (dbt 1.9 and later) | Large event tables that need reliable backfills |
| Dynamic tables | The warehouse refreshes the model to a target lag (Snowflake) | Near-real-time models without a tight scheduler |
| Streaming database adapters | Models become continuously updated views, for example with Materialize or RisingWave adapters | Sub-second freshness straight from Kafka |
Whichever route you choose, the hard problems sit upstream of dbt: duplicates, schema changes and schema ownership. The five steps below tackle them.
Pick the ingestion path first, because it decides your freshness ceiling. A sink connector or streaming ingestion API writes events into raw tables continuously; dbt then builds staging and mart models on top.
Keep the raw layer append-only and include the Kafka topic, partition, offset and event timestamp in each row. Those columns let your dbt models deduplicate reliably, rebuild from any point and trace a bad row back to its source event.
One common problem with dbt for real-time data is duplication. When you move a stream to a new topic, for example after a breaking schema change, avoid producing the same events to both topics.
As soon as the producer stops writing to the old topic, it should start writing to the new one. Consumers subscribe to both topics during the switch, so they keep reading the tail of the old topic while new data arrives on the new topic. Once the producer has switched over and the old topic is drained, consumers read only from the new topic. This gives a seamless transition without data loss or interruption.
Subscribing to more than one topic also helps you:
Partitions, not extra topics, are Kafka’s main way to spread load and add parallelism, so use multiple topics for separation and migration rather than as a scaling tool.
Schema translation converts data from one schema version to another. Sometimes two versions aren’t directly compatible under the registry’s forward or backward compatibility rules. That is where upcasters and downcasters come in.
| Technique | Converts | Typical use |
|---|---|---|
| Upcaster | Older schema version to a newer one | Let new consumers and dbt models read historical events |
| Downcaster | Newer schema version to an older one | Keep existing consumers working while producers move ahead |
Translation with upcasters and downcasters needs human knowledge and custom logic for each version pair. It isn’t the right approach for every scenario: maintaining casting rules for many versions is complex and time-consuming, so prefer compatible schema evolution and reserve casters for genuinely breaking changes.
The schema registry is valuable for managing schema definitions, but it isn’t always the best home for the master copy. Keep the source of truth, the golden record of your schema definitions, in a separate Git repository.
Holding schemas in version control gives you more control over versioning, compatibility checks and DataOps processes. Define build rules that check compatibility between versions, and let the schema registry validate compatibility during the merge, so the main branch always builds and can package the new version. For more on this pattern, see our guide on how to manage schemas in Apache Kafka and how to back up a Kafka schema registry.
A separate schema repository also lets developers generate language-specific code. By pulling the repository and running tools such as avro-tools for Avro or protoc for Protobuf, they can render typed objects in their preferred language.
A central schema repository also makes integration with data discovery tools easier. Discoverability is a major challenge in data management, and one well-structured home for schemas makes it simpler to catalogue data, share it and collaborate across teams. dbt adds to this with its own documentation and lineage for the models built on top.
To keep schema and configuration changes consistent across environments, see how to avoid configuration drift across multiple Kafka environments using GitOps, and avoid the pitfalls in Apache Kafka common mistakes.
Key takeaway: dbt transforms streaming data once it has landed; it doesn’t read Kafka itself. Get real-time results by landing events continuously, switching topics without double posting, translating breaking schema changes with upcasters and downcasters, keeping schemas in Git with registry validation, and running dbt incrementally or on a streaming adapter.
Not on its own. dbt runs SQL against a warehouse or database. Use Kafka Connect, warehouse streaming ingestion or a streaming database that reads topics, then point dbt at the landed data.
dbt is suitable for near-real-time analytics. Incremental and microbatch models, Snowflake dynamic tables and streaming database adapters can keep models fresh within minutes or seconds, depending on the platform.
An upcaster converts an event from an older schema version to a newer one. A downcaster converts a newer version back to an older one so existing consumers keep working.
It works better as a validation gate than as the master copy. Keep schemas in Git, review changes there, and have the registry check compatibility before new versions are merged and registered.
Switch the producer from the old topic to the new one in a single step rather than writing to both. Let consumers read both topics until the old one is drained, then move them to the new topic only.
Talk to OSO's Kafka engineers about landing streaming data reliably, managing schemas across teams, and building real-time pipelines your analysts can trust.
CONTACT US