Kafka

This page guides you through the process of setting up the Kafka source connector.

Set up guide

Step 1: Set up Kafka​

To use the Kafka source connector, you'll need:

• A Kafka cluster 1.0 or above
• Seek Connect user should be allowed to read messages from topics, and these topics should be created before reading from Kafka.

Step 2: Setup the Kafka source in Seek Connect​

You'll need the following information to configure the Kafka source:

• Group ID - The Group ID is how you distinguish different consumer groups. (e.g. group.id)
• Protocol - The Protocol used to communicate with brokers.
• Client ID - An ID string to pass to the server when making requests. The purpose of this is to be able to track the source of requests beyond just ip/port by allowing a logical application name to be included in server-side request logging. (e.g. Seek Connect-consumer)
• Test Topic - The Topic to test in case the Seek Connect can consume messages. (e.g. test.topic)
• Subscription Method - You can choose to manually assign a list of partitions, or subscribe to all topics matching specified pattern to get dynamically assigned partitions.
• List of topic
• Bootstrap Servers - A list of host/port pairs to use for establishing the initial connection to the Kafka cluster.
• Schema Registry - Host/port to connect schema registry server. Note: It supports for AVRO format only.

For Seek Connect:​

1. Go to the  UI and in the left navigation bar, click Sources. In the top-right corner, click +new source.
2. On the Set up the source page, enter the name for the Kafka connector and select Kafka from the Source type dropdown.
3. Follow the Setup the Kafka source

Supported sync modes​

The Kafka source connector supports the following sync modes:

Supported Format​

JSON - Json value messages. It does not support schema registry now.

AVRO - deserialize Using confluent API. Please refer (https://docs.confluent.io/platform/current/schema-registry/serdes-develop/serdes-avro.html)