Kafka Source
Provided by: "Apache Software Foundation"
Support Level for this Kamelet is: "Stable"
Receives each record from one or more topics in Apache Kafka, or in a Kafka-compatible service such as Amazon MSK, Confluent Cloud or Redpanda.
Use it to react to events or data that other applications publish, such as orders, sensor readings or changes from a database. To get many records in one message use kafka-batch-source. When the records are written with a schema kept in a schema registry, such as Confluent Schema Registry, Redpanda or Apicurio Registry, set schemaRegistryVendor.
Output: the body is the value of the record, as text. With schemaRegistryVendor, the value is read with its schema from the registry and the body is JSON text, such as {"id": 7, "item": "camel"} for an Avro record. A record without a value (a tombstone) gives a message with no body. The kafka-topic, kafka-key, kafka-partition, kafka-offset and kafka-timestamp (in milliseconds since 1970) headers say where the record came from. They are also set as ce-kafkatopic, ce-kafkakey and so on, and as the CamelKafka* headers that the Kafka actions read. The record headers become message headers, as text unless deserializeHeaders is false.
Good to know: saslAuthType has no default and must be set; NONE connects with no login and no TLS. A new consumer group starts at the newest records, unless autoOffsetReset is earliest. Without consumerGroup, every start uses a new group, so the records sent while the integration was stopped are skipped. With schemaRegistryVendor, the integration needs the serde jars of the vendor, which Camel JBang downloads by itself; for confluent add the Confluent Maven repository, as in camel run --repos=confluent. With aws-glue and azure, the serde logs in with the credentials of the environment the integration runs in: the AWS default credentials chain, or the Azure DefaultAzureCredential (such as the AZURE_CLIENT_ID, AZURE_TENANT_ID and AZURE_CLIENT_SECRET variables). A confluent or apicurio registry that requires a login is not supported yet.
Configuration Options
The following table summarizes the configuration options available for the kafka-source Kamelet:
| Property | Name | Description | Type | Default | Example |
|---|---|---|---|---|---|
Bootstrap Servers | Required The address of one or more Kafka brokers to connect to first, as | string | broker1:9092,broker2:9092 | ||
Authentication Type | Required How to log in to Kafka. It has no default and must be set. Enum values: * NONE * PLAIN * SCRAM_SHA_256 * SCRAM_SHA_512 * SSL * OAUTH * AWS_MSK_IAM * KERBEROS | string | |||
Topic Names | Required The topics to read from, separated by commas. When | string | orders | ||
Allow Manual Commit | If true, each message carries what | boolean | false | ||
Auto Commit | If true, the position reached in each partition (the offset) is saved in Kafka automatically every 5 seconds, so that a restarted integration goes on from there. Set it to false together with | boolean | true | ||
Auto Offset Reset | Where to start when the consumer group has no saved position yet: | string | latest | earliest | |
Consumer Group | The name of the consumer group. Integrations with the same group share the records of the topics between them, and the group remembers the position reached, so that a restarted integration goes on where it stopped. When it is not set, a new random group is used on every start. | string | my-group-id | ||
Automatically Deserialize Headers | If true, the values of the record headers are turned into text. If false, they stay bytes. | boolean | true | ||
OAuth Client ID | The ID of the client registered with the OAuth 2.0 provider. Required when | string | |||
OAuth Client Secret | The secret of the OAuth client | string | |||
OAuth Scope | The scope to ask the OAuth 2.0 provider for, when it requires one. Only used when | string | kafka | ||
OAuth Token URL | The URL of the OAuth 2.0 provider where the Kamelet gets its access token. Required when | string | https://auth.example.com/oauth2/token | ||
Poll On Error Behavior | What to do when reading from Kafka fails: Enum values: * DISCARD * ERROR_HANDLER * RECONNECT * RETRY * STOP | string | ERROR_HANDLER | ||
Password | The password of | string | |||
Username | The user name to log in to Kafka with. Required when | string | |||
Schema Format | The format of the record values in the schema registry: Enum values: * avro * json | string | avro | ||
Schema Registry Name | The name of the registry in AWS Glue. When not set, it is | string | |||
Schema Registry Region | The AWS region of the AWS Glue Schema Registry, such as | string | |||
Schema Registry URL | The URL of the schema registry: for | string | |||
Schema Registry Vendor | The schema registry that holds the schemas of the record values: Enum values: * confluent * apicurio * aws-glue * azure | string | |||
SSL Key Password | The password of the private key in the key store file in | string | |||
SSL Keystore Location | The path of a key store file, in JKS format, with the certificate and private key of the client. Needed only to log in with a client certificate (mutual TLS), usually with | string | /etc/kafka/keystore.jks | ||
SSL Keystore Password | The password of the key store file in | string | |||
SSL Truststore Location | The path of a trust store file, in JKS format, with the certificates to trust the brokers by. Needed only when the brokers use a certificate that Java does not trust by default, such as a self-signed one. | string | /etc/kafka/truststore.jks | ||
SSL Truststore Password | The password of the trust store file in | string | |||
Topic Is Pattern | If true, | boolean | false |
Dependencies
At runtime, the kafka-source Kamelet relies upon the presence of the following dependencies:
-
camel:jackson3
-
camel:kafka
-
camel:kamelet
Camel JBang usage
Prerequisites
-
You’ve installed JBang.
-
You have executed the following command:
jbang app install camel@apache/camel Supposing you have a file named route.yaml with this content:
- route:
from:
uri: "kamelet:kafka-source"
parameters:
.
.
.
steps:
- to:
uri: "kamelet:log-sink" You can now run it directly through the following command
camel run route.yaml Kafka Source Kamelet Description
Authentication
Authentication is selected with saslAuthType, which is required and has no default, so the choice is always explicit. The accepted values are NONE, PLAIN, SCRAM_SHA_256, SCRAM_SHA_512, SSL, OAUTH, AWS_MSK_IAM and KERBEROS. NONE is a valid choice and connects to an unauthenticated broker over a plaintext connection — it just has to be asked for rather than inherited.
Which other properties are needed depends on that choice:
-
PLAIN,SCRAM_SHA_256andSCRAM_SHA_512takesaslUsernameandsaslPassword. -
OAUTHtakesoauthClientId,oauthClientSecret,oauthTokenEndpointUriandoauthScope. -
SSLtakes the keystore and truststore properties below. -
AWS_MSK_IAMandKERBEROSrely on the surrounding environment rather than on Kamelet properties.
Resolve credentials through a Camel vault rather than plaintext properties wherever the deployment allows it.
Configuration
topic, bootstrapServers and saslAuthType are required. The Kamelet supports:
-
topic: Comma-separated list of Kafka topic names to consume from (required)
-
bootstrapServers: Comma-separated list of Kafka bootstrap servers (required)
-
saslAuthType: Authentication mechanism, no default (required)
-
saslUsername / saslPassword: Credentials for the username and password mechanisms
-
oauthClientId / oauthClientSecret / oauthTokenEndpointUri / oauthScope: OAuth 2.0 settings
-
sslTruststoreLocation / sslTruststorePassword / sslKeystoreLocation / sslKeystorePassword / sslKeyPassword: TLS material
-
consumerGroup: Kafka consumer group ID for managing offsets
-
autoOffsetReset: What to do when there is no initial offset -
earliest,latestornone, defaultlatest -
autoCommitEnable: Commit offsets automatically, default
true -
allowManualCommit: Enable manual commit for control over when offsets advance, default
false -
pollOnError: What to do when polling fails, default
ERROR_HANDLER -
deserializeHeaders: Deserialize the Kafka record headers onto the exchange, default
true -
topicIsPattern: Treat
topicas a regular expression rather than a literal list, defaultfalse -
schemaRegistryVendor / schemaRegistryUrl / schemaFormat / schemaRegistryRegion / schemaRegistryName: Read the records with their schema from a schema registry, see below
Output Format
The body is the Kafka record value. The record metadata is surfaced as headers under names that are not Camel internals, so a downstream consumer does not have to read the CamelKafka* headers directly. Each has a ce- prefixed CloudEvents counterpart as well.
-
kafka-topicfromCamelKafkaTopic- the topic the record was consumed from, which is worth having when the Kamelet subscribes to several topics or to a pattern. -
kafka-keyfromCamelKafkaKey- the record key, absent for records produced without one. -
kafka-partitionfromCamelKafkaPartition- the partition the record came from. -
kafka-offsetfromCamelKafkaOffset- the offset within that partition. -
kafka-timestampfromCamelKafkaTimestamp- the record timestamp, in milliseconds since the epoch.
The CamelKafka* headers are left on the exchange as well, so consumers already reading them keep working. The Kafka record headers are passed through separately, controlled by the deserializeHeaders property.
Schema registry
When the records are written with a schema kept in a schema registry, set schemaRegistryVendor, schemaRegistryUrl and, for JSON Schema, schemaFormat: json (the default is avro). The Kamelet reads each value with its schema and turns it into JSON text, so the next step gets text as without a registry: an Avro record becomes, for example, {"id": 7, "item": "camel"}.
schemaRegistryVendor | schemaFormat | Registry |
|---|---|---|
|
| Confluent Schema Registry, and the registries with the same API, such as the one of Redpanda and Karapace. |
|
| Apicurio Registry 3.x. |
|
| AWS Glue Schema Registry. Set |
|
| Azure Schema Registry of Event Hubs. |
The serde jars of the vendor are not part of the Kamelet. Camel JBang downloads them by itself; the Confluent jars are in the Confluent Maven repository, not in Maven Central, so run with --repos=confluent. In a Maven project, add the jar of the vendor and format, such as io.confluent:kafka-avro-serializer, and for Confluent the repository https://packages.confluent.io/maven/. A confluent or apicurio registry that requires a login is not supported yet.
- route:
from:
uri: "kamelet:kafka-source"
parameters:
topic: orders
bootstrapServers: "localhost:9092"
saslAuthType: NONE
schemaRegistryVendor: confluent
schemaRegistryUrl: "http://localhost:8081"
steps:
- to:
uri: "kamelet:log-sink" Usage Example
- route:
from:
uri: "kamelet:kafka-source"
parameters:
topic: "orders,payments"
bootstrapServers: "kafka.example.com:9092"
saslAuthType: "PLAIN"
saslUsername: "kafka-user"
saslPassword: "kafka-password"
steps:
- to:
uri: "kamelet:log-sink" Example with Consumer Group
- route:
from:
uri: "kamelet:kafka-source"
parameters:
topic: "user-events"
bootstrapServers: "kafka1.example.com:9092,kafka2.example.com:9092"
saslAuthType: "PLAIN"
saslUsername: "kafka-user"
saslPassword: "kafka-password"
consumerGroup: "my-consumer-group"
autoOffsetReset: "earliest"
steps:
- to:
uri: "kamelet:log-sink" Security
saslAuthType has no default, so the security of the connection is always chosen explicitly. With NONE the Kamelet talks to the broker in the clear, with no login. All the other types connect over TLS; supply the truststore properties when the brokers use a certificate that Java does not trust by default, and the keystore properties to log in with a client certificate.