Kafka Sink
Provided by: "Apache Software Foundation"
Support Level for this Kamelet is: "Stable"
Sends each message as a record to a topic in Apache Kafka, or in a Kafka-compatible service such as Amazon MSK, Confluent Cloud or Redpanda.
Use it to publish events or data for other applications to read, such as orders, sensor readings or changes from a database. To write the records with a schema kept in a schema registry, such as Confluent Schema Registry, Redpanda or Apicurio Registry, set schemaRegistryVendor.
Input: the body becomes the value of the record, as text. Turn structured data such as a Map into JSON text first, for example with json-serialize-action. With schemaRegistryVendor and schemaFormat json, the body must be JSON text (or a Map), and the record is written with a JSON Schema; with avro, confluent and apicurio write a text body as an Avro string, while aws-glue needs an Avro record. With azure, set schemaGroup. The optional key header (or kafka-key, ce-key, ce-kafkakey) becomes the key of the record, and the optional partition-key header (or ce-partitionkey) is the number of the partition to write to. The other headers of the message, except the Camel ones, are sent as record headers.
Output: the message is unchanged.
Good to know: saslAuthType has no default and must be set; NONE connects with no login and no TLS, as a local Kafka expects. A managed Kafka service needs one of the other types. With schemaRegistryVendor, the integration needs the serde jars of the vendor, which Camel JBang downloads by itself; for confluent add the Confluent Maven repository, as in camel run --repos=confluent. With aws-glue and azure, the serde logs in with the credentials of the environment the integration runs in: the AWS default credentials chain, or the Azure DefaultAzureCredential (such as the AZURE_CLIENT_ID, AZURE_TENANT_ID and AZURE_CLIENT_SECRET variables). A confluent or apicurio registry that requires a login is not supported yet. The sink registers a schema made from the message, which with json accepts any JSON object; it does not check the message against a schema that is already registered for the topic. Headers that start with CamelKafka, except CamelKafkaKey and CamelKafkaOverrideTopic, are removed from the message first, so they cannot change what the Kamelet does.
Configuration Options
The following table summarizes the configuration options available for the kafka-sink Kamelet:
| Property | Name | Description | Type | Default | Example |
|---|---|---|---|---|---|
Bootstrap Servers | Required The address of one or more Kafka brokers to connect to first, as | string | broker1:9092,broker2:9092 | ||
Authentication Type | Required How to log in to Kafka. It has no default and must be set. Enum values: * NONE * PLAIN * SCRAM_SHA_256 * SCRAM_SHA_512 * SSL * OAUTH * AWS_MSK_IAM * KERBEROS | string | |||
Topic Name | Required The name of the topic to write the records to. | string | orders | ||
OAuth Client ID | OAuth client ID. Required when saslAuthType is OAUTH. | string | |||
OAuth Client Secret | OAuth client secret. Required when saslAuthType is OAUTH. | string | |||
OAuth Scope | OAuth scope. Optional when saslAuthType is OAUTH. | string | |||
OAuth Token Endpoint | OAuth token endpoint URI. Required when saslAuthType is OAUTH. | string | |||
Password | Password for SASL authentication. Required when saslAuthType is PLAIN, SCRAM_SHA_256, or SCRAM_SHA_512. | string | |||
Username | Username for SASL authentication. Required when saslAuthType is PLAIN, SCRAM_SHA_256, or SCRAM_SHA_512. | string | |||
Schema Format | The format of the record values in the schema registry: Enum values: * avro * json | string | avro | ||
Schema Group | The schema group in the Azure Schema Registry that the schemas of the records are registered in. Required when | string | |||
Schema Registry Name | The name of the registry in AWS Glue. When not set, it is | string | |||
Schema Registry Region | The AWS region of the AWS Glue Schema Registry, such as | string | |||
Schema Registry URL | The URL of the schema registry: for | string | |||
Schema Registry Vendor | The schema registry that holds the schemas of the record values: Enum values: * confluent * apicurio * aws-glue * azure | string | |||
SSL Key Password | The password of the private key in the key store file. | string | |||
SSL Keystore Location | The location of the key store file. Used for mTLS authentication. | string | |||
SSL Keystore Password | The password for the key store file. | string | |||
SSL Truststore Location | The location of the trust store file. | string | |||
SSL Truststore Password | The password for the trust store file. | string |
Dependencies
At runtime, the kafka-sink Kamelet relies upon the presence of the following dependencies:
-
camel:jackson3
-
camel:kafka
-
camel:kamelet
Camel JBang usage
Prerequisites
-
You’ve installed JBang.
-
You have executed the following command:
jbang app install camel@apache/camel Supposing you have a file named route.yaml with this content:
- route:
from:
uri: "kamelet:timer-source"
parameters:
period: 10000
message: 'test'
steps:
- to:
uri: "kamelet:kafka-sink" You can now run it directly through the following command
camel run route.yaml Kafka Sink Kamelet Description
Authentication
The saslAuthType option chooses how the Kamelet logs in to Kafka. The options that each type needs are listed below; leave the others empty.
saslAuthType | Options to set |
|---|---|
| None. No login and no TLS, as a local Kafka expects. |
|
|
|
|
|
|
| None in the Kamelet: the AWS credentials are taken from the environment the integration runs in. |
| None in the Kamelet: Kerberos is configured in the environment the integration runs in. |
Schema registry
To write the records with a schema kept in a schema registry, set schemaRegistryVendor, schemaRegistryUrl and schemaFormat (the default is avro). With json, the body must be JSON text (or a Map); the Kamelet turns it into an object, and the record is written with a JSON Schema made from it, which accepts any JSON object. With avro, confluent and apicurio write a text body as an Avro string, while aws-glue needs an Avro record, which a step must create in code. With azure, also set schemaGroup. The sink registers the schema it makes, and does not check the message against a schema that is already registered for the topic.
schemaRegistryVendor | schemaFormat | Registry |
|---|---|---|
|
| Confluent Schema Registry, and the registries with the same API, such as the one of Redpanda and Karapace. |
|
| Apicurio Registry 3.x. |
|
| AWS Glue Schema Registry. Set |
|
| Azure Schema Registry of Event Hubs. |
The serde jars of the vendor are not part of the Kamelet. Camel JBang downloads them by itself; the Confluent jars are in the Confluent Maven repository, not in Maven Central, so run with --repos=confluent. In a Maven project, add the jar of the vendor and format, such as io.confluent:kafka-avro-serializer, and for Confluent the repository https://packages.confluent.io/maven/. A confluent or apicurio registry that requires a login is not supported yet.
- route:
from:
uri: "kamelet:timer-source"
parameters:
period: 10000
message: '{"id": 42, "item": "book"}'
contentType: application/json
steps:
- to:
uri: "kamelet:kafka-sink"
parameters:
bootstrapServers: "localhost:9092"
topic: orders
saslAuthType: NONE
schemaRegistryVendor: confluent
schemaRegistryUrl: "http://localhost:8081"
schemaFormat: json Usage example
The route below sends a JSON order every ten seconds to the orders topic, with the order number as the record key.
- route:
from:
uri: "kamelet:timer-source"
parameters:
period: 10000
message: '{"id": 42, "item": "book"}'
contentType: application/json
steps:
- setHeader:
name: key
constant: "42"
- to:
uri: "kamelet:kafka-sink"
parameters:
bootstrapServers: "localhost:9092"
topic: orders
saslAuthType: NONE