Kafka Batch Source
Provided by: "Apache Software Foundation"
Support Level for this Kamelet is: "Stable"
Receives records from one or more topics in Apache Kafka in batches, as one message that holds a list of up to batchSize records.
Use it to handle many records at once, for example to write them to a database in one go, or to save the position in Kafka only once a whole batch was handled. To handle one record at a time use kafka-source.
Output: the body is a List with one entry per record, up to batchSize. Each entry is a message of its own, with the value of the record as its body (text), and the key, offset and timestamp of the record in its CamelKafkaKey, CamelKafkaOffset and CamelKafkaTimestamp headers. The record headers are in the entries too, as bytes. The message itself has the CamelKafkaTopic and CamelKafkaPartition headers. The kafka-topic, kafka-key and other headers of kafka-source are not set.
Good to know: saslAuthType has no default and must be set; NONE connects with no login and no TLS. To save the position in Kafka only once a batch was handled, set allowManualCommit to true and autoCommitEnable to false, and end the flow with kafka-batch-manual-commit-action. A new consumer group starts at the newest records, unless autoOffsetReset is earliest. Without consumerGroup, every start uses a new group, so the records sent while the integration was stopped are skipped.
Configuration Options
The following table summarizes the configuration options available for the kafka-batch-source Kamelet:
| Property | Name | Description | Type | Default | Example |
|---|---|---|---|---|---|
Bootstrap Servers | Required The address of one or more Kafka brokers to connect to first, as | string | broker1:9092,broker2:9092 | ||
Authentication Type | Required How to log in to Kafka. It has no default and must be set. Enum values: * NONE * PLAIN * SCRAM_SHA_256 * SCRAM_SHA_512 * SSL * OAUTH * AWS_MSK_IAM * KERBEROS | string | |||
Topic Names | Required The topics to read from, separated by commas. When | string | orders | ||
Allow Manual Commit | If true, each message carries what | boolean | false | ||
Auto Commit | If true, the position reached in each partition (the offset) is saved in Kafka automatically every 5 seconds, so that a restarted integration goes on from there. Set it to false together with | boolean | true | ||
Auto Offset Reset | Where to start when the consumer group has no saved position yet: | string | latest | earliest | |
Batch Size | The largest number of records in one message. A message holds fewer when fewer records are waiting. | integer | 500 | ||
Batching Interval | When set, a batch that has not reached | integer | 10000 | ||
Consumer Group | The name of the consumer group. Integrations with the same group share the records of the topics between them, and the group remembers the position reached, so that a restarted integration goes on where it stopped. When it is not set, a new random group is used on every start. | string | my-group-id | ||
Automatically Deserialize Headers | Meant to turn the values of the record headers into text. In a batch it has no effect: the record headers of the entries of the list stay bytes. | boolean | true | ||
Max Poll Interval | The longest time, in milliseconds, that the integration may take to handle one batch. When it takes longer, Kafka considers it failed and gives its partitions to another member of the consumer group. When it is not set, the Kafka default of 300000 (5 minutes) applies. | integer | 600000 | ||
OAuth Client ID | The ID of the client registered with the OAuth 2.0 provider. Required when | string | |||
OAuth Client Secret | The secret of the OAuth client | string | |||
OAuth Scope | The scope to ask the OAuth 2.0 provider for, when it requires one. Only used when | string | kafka | ||
OAuth Token URL | The URL of the OAuth 2.0 provider where the Kamelet gets its access token. Required when | string | https://auth.example.com/oauth2/token | ||
Poll On Error Behavior | What to do when reading from Kafka fails: Enum values: * DISCARD * ERROR_HANDLER * RECONNECT * RETRY * STOP | string | ERROR_HANDLER | ||
Poll Timeout | How long one read from Kafka waits for new records, in milliseconds. | integer | 5000 | ||
Password | The password of | string | |||
Username | The user name to log in to Kafka with. Required when | string | |||
SSL Key Password | The password of the private key in the key store file in | string | |||
SSL Keystore Location | The path of a key store file, in JKS format, with the certificate and private key of the client. Needed only to log in with a client certificate (mutual TLS), usually with | string | /etc/kafka/keystore.jks | ||
SSL Keystore Password | The password of the key store file in | string | |||
SSL Truststore Location | The path of a trust store file, in JKS format, with the certificates to trust the brokers by. Needed only when the brokers use a certificate that Java does not trust by default, such as a self-signed one. | string | /etc/kafka/truststore.jks | ||
SSL Truststore Password | The password of the trust store file in | string | |||
Topic Is Pattern | If true, | boolean | false |
Dependencies
At runtime, the kafka-batch-source Kamelet relies upon the presence of the following dependencies:
-
camel:kafka
-
camel:kamelet
Camel JBang usage
Prerequisites
-
You’ve installed JBang.
-
You have executed the following command:
jbang app install camel@apache/camel Supposing you have a file named route.yaml with this content:
- route:
from:
uri: "kamelet:kafka-batch-source"
parameters:
.
.
.
steps:
- to:
uri: "kamelet:log-sink" You can now run it directly through the following command
camel run route.yaml Kafka-batch-source Kamelet Description
Authentication methods
The saslAuthType option chooses how the Kamelet logs in to Kafka, as for kafka-source. It has no default and must be set; NONE connects with no login and no TLS, as a local Kafka expects.
Output format
Each message holds a batch: its body is a List with one entry per record, up to batchSize. Each entry has the value of the record as its body, and the record’s key, offset, timestamp and headers as its headers. To save the position in Kafka only once a batch was handled, set allowManualCommit to true and autoCommitEnable to false, and end the flow with kafka-batch-manual-commit-action.
Configuration
The Kamelet requires Kafka connection parameters:
-
topic: The Kafka topic to consume from -
bootstrapServers: Comma separated list of Kafka Broker URLs -
Security-specific parameters based on the authentication method
Usage example
apiVersion: camel.apache.org/v1alpha1
kind: KameletBinding
metadata:
name: kafka-batch-source-binding
spec:
source:
ref:
kind: Kamelet
apiVersion: camel.apache.org/v1alpha1
name: kafka-batch-source
properties:
topic: "my-topic"
bootstrapServers: "kafka-broker1:9092,kafka-broker2:9092"
sink:
ref:
kind: Service
apiVersion: v1
name: my-service