blog by OSO

Kafka Connect S3: Sink and Source Connector Guide

Sion Smith
Sion Smith 19 November 2026
OSO guide to the Kafka Connect S3 sink and source connectors, with white title text on a navy panel and a lime Blogs label.

What is the Kafka Connect S3 sink?

The S3 sink is a Kafka Connect connector that exports data from Kafka topics to S3 objects. Confluent’s Amazon S3 Sink connector uses the class io.confluent.connect.s3.S3SinkConnector and is the one most teams reach for. Confluent’s documentation lists it under the Confluent Community Licence.

A connector beats a hand-written consumer here because Connect already handles offsets, task scaling and retries. The sink also supports a dead letter queue, multiple tasks, and exactly-once delivery when you use a deterministic partitioner.

Confluent’s own documentation notes that self-managed connector versions below its supported minimum have been removed from Confluent Marketplace. Check the supported versions page before you pin a version.

If you are still deciding where Connect should run, our MSK Connect guide covers the managed option in more depth.

How do you configure the S3 sink connector?

A working sink needs a connector class, topics, a bucket, a region, a format and a flush rule. For credentials, prefer an IAM role: by default the connector uses the AWS default credentials provider chain, so no access keys need to sit in the config.

SettingWhat it does
connector.classSet to io.confluent.connect.s3.S3SinkConnector
s3.bucket.name and s3.regionThe target bucket and its AWS region
format.classOutput format: Avro, JSON, byte array or Parquet
flush.sizeNumber of records written before a file is committed
rotate.interval.msCommits files at a set interval, based on record timestamps
rotate.schedule.interval.msCommits files on a wall-clock schedule, for example hourly
partitioner.classHow objects are split into paths in the bucket
storage.classStorage implementation, for example io.confluent.connect.s3.storage.S3Storage

A minimal example in properties form:

name=orders-s3-sink
connector.class=io.confluent.connect.s3.S3SinkConnector
tasks.max=2
topics=orders
s3.bucket.name=example-orders-archive
s3.region=eu-west-2
storage.class=io.confluent.connect.s3.storage.S3Storage
format.class=io.confluent.connect.s3.format.json.JsonFormat
flush.size=10000
rotate.schedule.interval.ms=3600000
partitioner.class=io.confluent.connect.storage.partitioner.DefaultPartitioner

For failures, set errors.tolerance and a dead letter queue topic so one bad record does not stop the task. Note that the connector’s null-value handling (behavior.on.null.values) interacts with DLQ reporting, so test tombstone records before production.

Key takeaway: Get the sink’s partitioner, format and flush settings right first; treat the S3 source as a separate connector with its own failure modes.

How should you partition and format data in S3?

The default partitioner preserves Kafka partitioning. The field partitioner writes to directories based on a record field named in partition.field.name. The time-based partitioner groups data by time using path.format, timezone, locale and a timestamp extractor of Wallclock, Record or RecordField.

For analytics, a Hive-style path such as 'year'=YYYY/'month'=MM/'day'=dd/'hour'=HH lets Athena or Spark prune partitions. Confluent’s documentation uses exactly this form as its path.format example.

The sink’s format.class options are Avro, JSON, byte array and Parquet. Parquet needs io.confluent.connect.s3.format.parquet.ParquetFormat and a schema, so plan your converters first. Compression is set separately, for example s3.compression.type accepts none or gzip.

This is not the same thing as Kafka tiered storage. Connect copies data into S3 as a separate dataset for other tools to query. Tiered storage (KIP-405) offloads old log segments while Kafka keeps serving them to consumers. Our post on tiered storage in Apache Kafka explains that side.

How does the S3 source connector differ?

The S3 source connector reads objects from a bucket and produces them back into Kafka topics. Its class is io.confluent.connect.s3.source.S3SourceConnector. It is a separate connector from the sink, not a mode of it.

The licensing also differs. Confluent documents the S3 source as proprietary, needing a Confluent licence, while the sink is under the Confluent Community Licence.

Confluent’s reference describes two modes: RESTORE_BACKUP, which is the default and re-reads data the sink wrote, and GENERIC, which maps object paths to topics with regular expressions. Its format.class options are Avro, JSON, byte array and string, so Parquet is not in that list.

Because the failure modes differ (bucket listing, schema handling, topic mapping), test the source on its own before relying on a round trip. Do not assume exactly-once behaviour for the source unless the connector version you run documents it.

When should you run this on MSK Connect?

MSK Connect suits teams that want AWS to run the Connect workers. You package the connector as a ZIP of JARs or an uber JAR, upload it to S3 and create a custom plugin from it. AWS notes that plugins cannot be updated in place, so a new version means a new plugin and updated connectors.

Self-hosting suits teams that already run Kubernetes and want control over versions, networking and monitoring. See how we approached that in our case study on self-hosted Kafka Connect on Strimzi.

Whichever you choose, give the workers an IAM role that can write to the bucket and a network path to S3. If you want help choosing, our Kafka consulting services and Kafka support teams do this work every week.

Frequently asked questions

What is the Kafka Connect S3 sink connector?

It is a Kafka Connect connector that writes records from Kafka topics to objects in Amazon S3. Confluent’s version uses the class io.confluent.connect.s3.S3SinkConnector and supports Avro, JSON, byte array and Parquet output.

How often does the S3 sink write objects?

It commits a file when flush.size records have been written, or when a rotation setting fires. rotate.interval.ms works from record timestamps, while rotate.schedule.interval.ms works from the wall clock. Both are disabled by default.

Can I write Parquet to S3 from Kafka?

Yes, with the sink. Set format.class to io.confluent.connect.s3.format.parquet.ParquetFormat, and make sure your records carry a schema. The S3 source connector’s documented formats do not include Parquet.

Is the S3 source connector the same as the sink?

No. It is a separate connector that reads objects from S3 into Kafka topics, and Confluent documents it as proprietary and licensed. The sink is documented under the Confluent Community Licence.

Can I run the S3 connector on MSK Connect?

Yes, as a custom plugin. Upload the connector as a ZIP of JARs or an uber JAR to S3, create the plugin, then create a connector from it. Check the Java runtime for your chosen Connect version.

How is this different from Kafka tiered storage?

Connect copies topic data into S3 as a separate dataset for analytics. Tiered storage offloads old log segments of the Kafka log itself, and consumers still read them through Kafka.

Shipping Kafka data to S3?

OSO engineers run Apache Kafka in production for banks, retailers and the public sector. Tell us what you are weighing up and we will give you a straight answer.

CONTACT US
OSO
Privacy Overview

This website uses cookies so that we can provide you with the best user experience possible. Cookie information is stored in your browser and performs functions such as recognising you when you return to our website and helping our team to understand which sections of the website you find most interesting and useful.