AWS Kinesis Source
Provided by: "Apache Software Foundation"
Support Level for this Kamelet is: "Stable"
Reads the records of an Amazon Kinesis data stream, one message per record.
Use it to process a stream of events that other applications write to Kinesis, such as clicks, logs or readings from devices, for example with aws-kinesis-sink.
Output: the body is the data of the record, as bytes. The headers give the partition key, the sequence number, the shard ID and the arrival time of the record.
Good to know: by default it reads the stream from its oldest record, and again from the oldest record each time the integration starts: it does not remember where it stopped, so records are read again after a restart. Set iteratorType to LATEST to read only the records written after it started; the records written while it was stopped are then not read. iteratorType can also start at a sequence number or at a time, again on each start. Records are not removed: Kinesis keeps them for the retention period of the stream. It reads all the shards of the stream, up to 1000 records from each in one poll: set shardId to read one shard only, and maxResultsPerRequest to read fewer records in one poll. When the stream is resharded, it goes on with the new shards; with shardClosed set to fail, every poll also fails with an error in the log, but the records are still read. With useKclConsumers it reads through the Kinesis Client Library instead, which keeps where it stopped in a DynamoDB table named after applicationName, or after the stream, and does not use iteratorType, shardId, maxResultsPerRequest or shardClosed; the body is then a ByteBuffer rather than bytes. To reach AWS through an HTTP proxy, set proxyHost and proxyPort, and set proxyProtocol to HTTP for a proxy that takes plain HTTP, such as Squid. With useKclConsumers, the calls of the KCL to DynamoDB and CloudWatch go straight to AWS, without the proxy and without uriEndpointOverride.
Configuration Options
The following table summarizes the configuration options available for the aws-kinesis-source Kamelet:
| Property | Name | Description | Type | Default | Example |
|---|---|---|---|---|---|
AWS Region | Required The AWS region of the stream, as a region code such as Enum values: * ap-south-1 * eu-south-1 * us-gov-east-1 * me-central-1 * ca-central-1 * eu-central-1 * us-iso-west-1 * us-west-1 * us-west-2 * af-south-1 * eu-north-1 * eu-west-3 * eu-west-2 * eu-west-1 * ap-northeast-3 * ap-northeast-2 * ap-northeast-1 * me-south-1 * sa-east-1 * ap-east-1 * cn-north-1 * us-gov-west-1 * ap-southeast-1 * ap-southeast-2 * us-iso-east-1 * ap-southeast-3 * us-east-1 * us-east-2 * cn-northwest-1 * us-isob-east-1 * aws-global * aws-cn-global * aws-us-gov-global * aws-iso-global * aws-iso-b-global | string | eu-west-1 | ||
Stream Name | Required The name of the Kinesis data stream, such as | string | clickstream | ||
Access Key | The access key ID of an IAM user, which you create in the AWS console under IAM, Users, Security credentials. Use it together with | string | |||
KCL Application Name | The name of the application for the Kinesis Client Library. The KCL keeps where it stopped, and which copy of the integration reads which shard, in a DynamoDB table of this name, which it creates. Copies of the integration with the same name share the shards of the stream; give another integration that reads the same stream its own name, so that it reads all the records too. When it is not set, it is the name of the stream. Only used when | string | orders-processor | ||
Async Client | If true, it uses the asynchronous Kinesis client of the AWS SDK. | boolean | false | ||
Delay | How long to wait between two polls of the stream, in milliseconds. | integer | 500 | ||
Start Position | Where in the stream it starts reading each time the integration starts. Enum values: * TRIM_HORIZON * LATEST * AT_SEQUENCE_NUMBER * AFTER_SEQUENCE_NUMBER * AT_TIMESTAMP | string | TRIM_HORIZON | ||
KCL Disable CloudWatch Metrics | If true, the Kinesis Client Library does not send its metrics to CloudWatch. Only used when | boolean | false | ||
Max Records Per Poll | The most records it reads from each shard in one poll, from 1 to 10000. When it is not set, it is 1000. Not used when | integer | 100 | ||
Start Time | The time to start reading at, either as an ISO 8601 date and time in UTC, such as | string | 2026-11-17T08:00:00Z | ||
Endpoint Overwrite | If true, it connects to the address in | boolean | false | ||
Profile Credentials Name | The name of the profile in the AWS credentials file. Only used when | string | production | ||
Proxy Host | The host name or IP address of an HTTP proxy to send the calls to AWS through, such as the proxy of a company network. Used only together with | string | proxy.example.com | ||
Proxy Port | The port of the proxy in | integer | 3128 | ||
Proxy Protocol | How to talk to the proxy itself: Enum values: * HTTP * HTTPS | string | HTTP | ||
Secret Key | The secret access key that belongs to | string | |||
Sequence Number | The sequence number of the record to start at, as Kinesis gives it, such as the value of the | string | 49655467183424123548936502291338416434578112145614569474 | ||
Session Token | The session token of temporary credentials, which comes with their access key and secret key, for example from AWS STS when you assume an IAM role. Only used when | string | |||
Closed Shard | What to do when it has read the last record of a closed shard. A shard is closed when the stream is resharded, for example when the number of shards is changed, and new shards take its place. Enum values: * ignore * silent * fail | string | ignore | ||
Shard ID | The ID of the one shard to read, such as | string | shardId-000000000000 | ||
Overwrite Endpoint URI | The address to connect to instead of AWS, as a URL such as | string | http://localhost:4566 | ||
Default Credentials Provider | If true, the credentials are found in the environment instead of in | boolean | false | ||
KCL Consumer | If true, it reads the stream with the Kinesis Client Library (KCL), which shares the shards between the running copies of the integration. The KCL also uses DynamoDB and CloudWatch in the same account, so the credentials need access to them. Set | boolean | false | ||
Profile Credentials Provider | If true, the credentials are read from a named profile in the AWS credentials file ( | boolean | false | ||
Session Credentials | If true, it logs in with temporary credentials: | boolean | false |
Dependencies
At runtime, the aws-kinesis-source Kamelet relies upon the presence of the following dependencies:
-
camel:aws2-kinesis
-
camel:kamelet
Camel JBang usage
Prerequisites
-
You’ve installed JBang.
-
You have executed the following command:
jbang app install camel@apache/camel Supposing you have a file named route.yaml with this content:
- route:
from:
uri: "kamelet:aws-kinesis-source"
parameters:
.
.
.
steps:
- to:
uri: "kamelet:log-sink" You can now run it directly through the following command
camel run route.yaml AWS Kinesis Source Kamelet Description
Authentication methods
In this Kamelet you can avoid using explicit static credentials by specifying the useDefaultCredentialsProvider option and set it to true.
The order of evaluation for Default Credentials Provider is the following:
-
Java system properties -
aws.accessKeyIdandaws.secretKey. -
Environment variables -
AWS_ACCESS_KEY_IDandAWS_SECRET_ACCESS_KEY. -
Web Identity Token from AWS STS.
-
The shared credentials and config files.
-
Amazon ECS container credentials - loaded from the Amazon ECS if the environment variable
AWS_CONTAINER_CREDENTIALS_RELATIVE_URIis set. -
Amazon EC2 Instance profile credentials.
You can also use the Profile Credentials Provider, by setting the useProfileCredentialsProvider option to true and profileCredentialsName to the profile name.
Only one of access key/secret key or default credentials provider could be used
For more information, see the AWS credentials documentation
Usage example with plain consumer
You could consume the stream content directly
- route:
from:
uri: "kamelet:aws-kinesis-source"
parameters:
useDefaultCredentialsProvider: true
region: "eu-west-1"
stream: "kamelets"
steps:
- to:
uri: "kamelet:log-sink" Where it starts reading
By default (iteratorType is TRIM_HORIZON) the Kamelet reads the stream from its oldest record each time the integration starts, so the records are read again after a restart. To read only the records written after the integration started, set iteratorType to LATEST:
- route:
from:
uri: "kamelet:aws-kinesis-source"
parameters:
useDefaultCredentialsProvider: true
region: "eu-west-1"
stream: "kamelets"
iteratorType: "LATEST"
steps:
- to:
uri: "kamelet:log-sink" The records written while the integration was stopped are then not read. To start at a given record, set iteratorType to AT_SEQUENCE_NUMBER or AFTER_SEQUENCE_NUMBER and sequenceNumber to its sequence number; to start at a given time, set iteratorType to AT_TIMESTAMP and messageTimestamp to the time, such as 2026-11-17T08:00:00Z. The Kamelet does not remember where it stopped: it starts at the same place on each start. The KCL consumer below does not use iteratorType.
One shard, and smaller polls
By default the Kamelet reads all the shards of the stream, and up to 1000 records from each shard in one poll. To read one shard only, set shardId to its ID, as aws kinesis list-shards shows it. To read fewer records in one poll, set maxResultsPerRequest: the rest of the records are read in the next polls, delay apart.
- route:
from:
uri: "kamelet:aws-kinesis-source"
parameters:
useDefaultCredentialsProvider: true
region: "eu-west-1"
stream: "kamelets"
shardId: "shardId-000000000001"
maxResultsPerRequest: 100
steps:
- to:
uri: "kamelet:log-sink" When the stream is resharded, for example when the number of shards is changed, the old shards are closed and new ones take their place. Without shardId, the Kamelet goes on with the new shards, and shardClosed says what it does about a closed shard once it has read its last record: ignore (the default) logs a warning once, and silent logs nothing. With fail, each later poll fails with an error and a stack trace in the log, for as long as the closed shard is listed, which is 24 hours by default; the records of the other shards are still read.
Usage example with KCL Consumer
You could consume the stream content with the KCL support
- route:
from:
uri: "kamelet:aws-kinesis-source"
parameters:
stream: "kamelets"
useDefaultCredentialsProvider: true
region: "eu-west-1"
asyncClient: true
useKclConsumers: true
applicationName: "orders-processor"
steps:
- to:
uri: "kamelet:log-sink"
parameters:
showHeaders: true With the useKclConsumers enabled, you won’t have to deal with shard iteration directly. Everything is managed by the AWS Kinesis client library and the KCL layer.
As a side note you need to remember that the KCL consumer will need access to DynamoDB and Cloudwatch services from AWS, so it will create clients to these services under the hood and it will use them.
The KCL keeps its state in a DynamoDB table named after applicationName, orders-processor above, which it creates; without applicationName, the table is named after the stream. Copies of the integration with the same applicationName share the shards of the stream. Give each integration that reads the same stream its own applicationName, so that each one reads all the records.
Through an HTTP Proxy
Where AWS can be reached only through an HTTP proxy, as in many company networks, set proxyHost and proxyPort to the proxy. proxyProtocol is how the Kamelet talks to the proxy itself, not to AWS: it is HTTPS when it is not set, so set it to HTTP for a proxy that takes plain HTTP, such as Squid. The calls to AWS stay encrypted, as the proxy only passes them on. The Kamelet has no option for a proxy user or password.
- route:
from:
uri: "kamelet:aws-kinesis-source"
parameters:
stream: "clickstream"
region: "eu-west-1"
useDefaultCredentialsProvider: true
proxyHost: "proxy.example.com"
proxyPort: 3128
proxyProtocol: "HTTP"
steps:
- to:
uri: "kamelet:log-sink" With useKclConsumers, the calls of the KCL to DynamoDB and CloudWatch go straight to AWS, without the proxy: the network must let them through.