AWS Kinesis Sink
Provided by: "Apache Software Foundation"
Support Level for this Kamelet is: "Stable"
Writes each message as a record to an Amazon Kinesis data stream.
Use it to feed a stream of events, such as clicks, logs or sensor readings, to the applications that read the stream, for example with aws-kinesis-source. To deliver data to S3 or a data warehouse without writing a reader, use aws-kinesis-firehose-sink.
Input: the body becomes the data of the record, as text, bytes or a stream. The optional partition header (or ce-partition) is the partition key: the records with the same key go to the same shard, in order. Without it, the key is a unique ID, so the records spread over the shards. The optional sequence-number header (or ce-sequencenumber) is the sequence number of the record that must come before this one, for a strict order within a partition key.
Output: the message is unchanged. The CamelAwsKinesisSequenceNumber and CamelAwsKinesisShardId headers give the sequence number and the shard of the record.
Good to know: the stream must exist. The other headers are not sent: a Kinesis record has only data and a partition key. To reach AWS through an HTTP proxy, set proxyHost and proxyPort, and set proxyProtocol to HTTP for a proxy that takes plain HTTP, such as Squid. Headers that start with CamelAwsKinesis are removed from the message first, so they cannot change what the Kamelet does.
Configuration Options
The following table summarizes the configuration options available for the aws-kinesis-sink Kamelet:
| Property | Name | Description | Type | Default | Example |
|---|---|---|---|---|---|
AWS Region | Required The AWS region of the stream, as a region code such as Enum values: * ap-south-1 * eu-south-1 * us-gov-east-1 * me-central-1 * ca-central-1 * eu-central-1 * us-iso-west-1 * us-west-1 * us-west-2 * af-south-1 * eu-north-1 * eu-west-3 * eu-west-2 * eu-west-1 * ap-northeast-3 * ap-northeast-2 * ap-northeast-1 * me-south-1 * sa-east-1 * ap-east-1 * cn-north-1 * us-gov-west-1 * ap-southeast-1 * ap-southeast-2 * us-iso-east-1 * ap-southeast-3 * us-east-1 * us-east-2 * cn-northwest-1 * us-isob-east-1 * aws-global * aws-cn-global * aws-us-gov-global * aws-iso-global * aws-iso-b-global | string | eu-west-1 | ||
Stream Name | Required The name of the Kinesis data stream, such as | string | clickstream | ||
Access Key | The access key ID of an IAM user, which you create in the AWS console under IAM, Users, Security credentials. Use it together with | string | |||
Endpoint Overwrite | If true, it connects to the address in | boolean | false | ||
Profile Credentials Name | The name of the profile in the AWS credentials file. Only used when | string | production | ||
Proxy Host | The host name or IP address of an HTTP proxy to send the calls to AWS through, such as the proxy of a company network. Used only together with | string | proxy.example.com | ||
Proxy Port | The port of the proxy in | integer | 3128 | ||
Proxy Protocol | How to talk to the proxy itself: Enum values: * HTTP * HTTPS | string | HTTP | ||
Secret Key | The secret access key that belongs to | string | |||
Session Token | The session token of temporary credentials, which comes with their access key and secret key, for example from AWS STS when you assume an IAM role. Only used when | string | |||
Overwrite Endpoint URI | The address to connect to instead of AWS, as a URL such as | string | http://localhost:4566 | ||
Default Credentials Provider | If true, the credentials are found in the environment instead of in | boolean | false | ||
Profile Credentials Provider | If true, the credentials are read from a named profile in the AWS credentials file ( | boolean | false | ||
Session Credentials | If true, it logs in with temporary credentials: | boolean | false |
Dependencies
At runtime, the aws-kinesis-sink Kamelet relies upon the presence of the following dependencies:
-
camel:aws2-kinesis
-
camel:kamelet
Camel JBang usage
Prerequisites
-
You’ve installed JBang.
-
You have executed the following command:
jbang app install camel@apache/camel Supposing you have a file named route.yaml with this content:
- route:
from:
uri: "kamelet:timer-source"
parameters:
period: 10000
message: 'test'
steps:
- to:
uri: "kamelet:aws-kinesis-sink" You can now run it directly through the following command
camel run route.yaml AWS Kinesis Sink Kamelet Description
Authentication methods
In this Kamelet you can avoid using explicit static credentials by specifying the useDefaultCredentialsProvider option and set it to true.
The order of evaluation for Default Credentials Provider is the following:
-
Java system properties -
aws.accessKeyIdandaws.secretKey. -
Environment variables -
AWS_ACCESS_KEY_IDandAWS_SECRET_ACCESS_KEY. -
Web Identity Token from AWS STS.
-
The shared credentials and config files.
-
Amazon ECS container credentials - loaded from the Amazon ECS if the environment variable
AWS_CONTAINER_CREDENTIALS_RELATIVE_URIis set. -
Amazon EC2 Instance profile credentials.
You can also use the Profile Credentials Provider, by setting the useProfileCredentialsProvider option to true and profileCredentialsName to the profile name.
Only one of access key/secret key or default credentials provider could be used
For more information, see the AWS credentials documentation
Optional Headers
In the header, you can optionally set the partition / ce-partition property to set the Kinesis partition key.
If you do not set the property in the header, the Kamelet uses a generated unique ID for the partition key.
You can also set the sequence-number / ce-sequencenumber property in the header to specify the Sequence number.
Through an HTTP Proxy
Where AWS can be reached only through an HTTP proxy, as in many company networks, set proxyHost and proxyPort to the proxy. proxyProtocol is how the Kamelet talks to the proxy itself, not to AWS: it is HTTPS when it is not set, so set it to HTTP for a proxy that takes plain HTTP, such as Squid. The calls to AWS stay encrypted, as the proxy only passes them on. The Kamelet has no option for a proxy user or password.
The step that sends to the Kamelet, in a route:
- to:
uri: "kamelet:aws-kinesis-sink"
parameters:
stream: "clickstream"
region: "eu-west-1"
useDefaultCredentialsProvider: true
proxyHost: "proxy.example.com"
proxyPort: 3128
proxyProtocol: "HTTP"