AWS DynamoDB Streams Source
Provided by: "Apache Software Foundation"
Support Level for this Kamelet is: "Stable"
Receives the changes made to the items of an Amazon DynamoDB table from its stream, one message per inserted, modified or removed item.
Use it to react when data changes in a table, for example to copy new orders to another system, keep a search index or a cache up to date, or record an audit trail. To write items into a table, use aws-ddb-sink.
Output: the body is a JSON object, as JSON text, that describes one change: eventName is INSERT, MODIFY or REMOVE, and dynamodb holds the key of the item (keys) and, depending on the view type of the stream, the item before (oldImage) and after (newImage) the change, with the type of each attribute. The headers give the ID and the name of the event.
Good to know: DynamoDB Streams must be enabled on the table, with the view type that holds the images you need, such as new and old images. By default it reads only the changes made after it starts, up to 100 in one poll; set maxResultsPerRequest to read fewer at a time. To reach AWS through an HTTP proxy, set proxyHost and proxyPort, and set proxyProtocol to HTTP for a proxy that takes plain HTTP, such as Squid.
Configuration Options
The following table summarizes the configuration options available for the aws-ddb-streams-source Kamelet:
| Property | Name | Description | Type | Default | Example |
|---|---|---|---|---|---|
AWS Region | Required The AWS region of the table, such as Enum values: * ap-south-1 * eu-south-1 * us-gov-east-1 * me-central-1 * ca-central-1 * eu-central-1 * us-iso-west-1 * us-west-1 * us-west-2 * af-south-1 * eu-north-1 * eu-west-3 * eu-west-2 * eu-west-1 * ap-northeast-3 * ap-northeast-2 * ap-northeast-1 * me-south-1 * sa-east-1 * ap-east-1 * cn-north-1 * us-gov-west-1 * ap-southeast-1 * ap-southeast-2 * us-iso-east-1 * ap-southeast-3 * us-east-1 * us-east-2 * cn-northwest-1 * us-isob-east-1 * aws-global * aws-cn-global * aws-us-gov-global * aws-iso-global * aws-iso-b-global | string | eu-west-1 | ||
Table Name | Required The name of the DynamoDB table to receive the changes of. DynamoDB Streams must be enabled on the table. | string | orders | ||
Access Key | The access key ID of an IAM user, which AWS gives when you create an access key for the user (in the AWS console: IAM, Users, Security credentials). Use it together with | string | |||
Delay | How long to wait between two reads of the stream, in milliseconds. | integer | 500 | ||
Max Changes Per Poll | The most changes it reads in one poll, from 1 to 1000. The number is shared between the shards of the stream that it reads, at least 1 each. When it is not set, it is 100. With a lower number, a burst of changes takes more polls, | integer | 50 | ||
Endpoint Overwrite | If true, the requests go to the address in | boolean | false | ||
Profile Credentials Name | The name of the profile to use from the AWS credentials file. Only used when | string | |||
Proxy Host | The host name or IP address of an HTTP proxy to send the calls to AWS through, such as the proxy of a company network. Used only together with | string | proxy.example.com | ||
Proxy Port | The port of the proxy in | integer | 3128 | ||
Proxy Protocol | How to talk to the proxy itself: Enum values: * HTTP * HTTPS | string | HTTP | ||
Secret Key | The secret access key that goes with | string | |||
Session Token | The session token of temporary credentials, which AWS gives together with their access key and secret key. Only used when | string | |||
Stream Iterator Type | Where to start reading: Enum values: * FROM_LATEST * FROM_START | string | FROM_LATEST | ||
Overwrite Endpoint URI | The address to send the requests to instead of the usual AWS address, such as | string | http://localhost:4566 | ||
Default Credentials Provider | If true, the credentials are found the way the AWS SDK finds them: from the | boolean | false | ||
Profile Credentials Provider | If true, the credentials are read from a profile of the AWS credentials file ( | boolean | false | ||
Session Credentials | If true, it logs in with temporary credentials, such as those of an assumed IAM role: | boolean | false |
Dependencies
At runtime, the aws-ddb-streams-source Kamelet relies upon the presence of the following dependencies:
-
camel:gson
-
camel:aws2-ddb
-
camel:kamelet
Camel JBang usage
Prerequisites
-
You’ve installed JBang.
-
You have executed the following command:
jbang app install camel@apache/camel Supposing you have a file named route.yaml with this content:
- route:
from:
uri: "kamelet:aws-ddb-streams-source"
parameters:
.
.
.
steps:
- to:
uri: "kamelet:log-sink" You can now run it directly through the following command
camel run route.yaml AWS DynamoDB Streams Source Kamelet Description
Authentication methods
In this Kamelet you can avoid using explicit static credentials by specifying the useDefaultCredentialsProvider option and set it to true.
The order of evaluation for Default Credentials Provider is the following:
-
Java system properties -
aws.accessKeyIdandaws.secretKey. -
Environment variables -
AWS_ACCESS_KEY_IDandAWS_SECRET_ACCESS_KEY. -
Web Identity Token from AWS STS.
-
The shared credentials and config files.
-
Amazon ECS container credentials - loaded from the Amazon ECS if the environment variable
AWS_CONTAINER_CREDENTIALS_RELATIVE_URIis set. -
Amazon EC2 Instance profile credentials.
You can also use the Profile Credentials Provider, by setting the useProfileCredentialsProvider option to true and profileCredentialsName to the profile name.
Only one of access key/secret key or default credentials provider could be used
For more information, see the AWS credentials documentation
Fewer Changes in One Poll
The Kamelet reads up to 100 changes in one poll, and polls again delay milliseconds later. To read fewer at a time, for example so that a slow route behind it is not handed a large burst at once, set maxResultsPerRequest. The number is shared between the shards of the stream. Here a burst of 1000 changes is read 50 at a time, one poll a second:
- route:
from:
uri: "kamelet:aws-ddb-streams-source"
parameters:
table: "orders"
region: "eu-west-1"
useDefaultCredentialsProvider: true
maxResultsPerRequest: 50
delay: 1000
steps:
- to:
uri: "kamelet:log-sink" Through an HTTP Proxy
Where AWS can be reached only through an HTTP proxy, as in many company networks, set proxyHost and proxyPort to the proxy. proxyProtocol is how the Kamelet talks to the proxy itself, not to AWS: it is HTTPS when it is not set, so set it to HTTP for a proxy that takes plain HTTP, such as Squid. The calls to AWS stay encrypted, as the proxy only passes them on. The Kamelet has no option for a proxy user or password.
- route:
from:
uri: "kamelet:aws-ddb-streams-source"
parameters:
table: "orders"
region: "eu-west-1"
useDefaultCredentialsProvider: true
proxyHost: "proxy.example.com"
proxyPort: 3128
proxyProtocol: "HTTP"
steps:
- to:
uri: "kamelet:log-sink"