AWS SQS Sink
Provided by: "Apache Software Foundation"
Support Level for this Kamelet is: "Stable"
Sends each message as an SQS message to an Amazon SQS (Simple Queue Service) queue.
Use it to hand work to other systems that read the queue, such as orders to process or jobs to run, and to buffer it when they are slow or down. For a FIFO queue use aws-sqs-fifo-sink, to send many parts in one call use aws-sqs-batch-sink, and to notify many receivers at once use aws-sns-sink.
Input: the body becomes the body of the SQS message, as text; bytes and streams are read as text. Turn structured data such as a Map into JSON text first, for example with json-serialize-action. The other headers of the message, except the Camel ones, are sent as SQS message attributes.
Output: the message is unchanged. The CamelAwsSqsMessageId header gives the ID of the SQS message.
Good to know: unless queueURL, queueOwnerAWSAccountId or an ARN in queueNameOrArn gives the address of the queue, the queue is looked up by listing the queues, which needs the sqs:ListQueues permission besides sqs:SendMessage. delaySeconds keeps each message hidden for a while after it is sent. serverSideEncryptionEnabled with kmsMasterKeyId changes the queue itself when the integration starts, so that SQS encrypts its messages with that KMS key. To reach a local stand-in of AWS, use overrideEndpoint and uriEndpointOverride. To reach AWS through an HTTP proxy, set proxyHost and proxyPort, and set proxyProtocol to HTTP for a proxy that takes plain HTTP, such as Squid. Headers that start with CamelAwsSqs are removed from the message first, so they cannot change what the Kamelet does.
Configuration Options
The following table summarizes the configuration options available for the aws-sqs-sink Kamelet:
| Property | Name | Description | Type | Default | Example |
|---|---|---|---|---|---|
Queue Name | Required The name of the queue, such as | string | orders | ||
AWS Region | Required The AWS region of the queue, as a region code such as Enum values: * ap-south-1 * eu-south-1 * us-gov-east-1 * me-central-1 * ca-central-1 * eu-central-1 * us-iso-west-1 * us-west-1 * us-west-2 * af-south-1 * eu-north-1 * eu-west-3 * eu-west-2 * eu-west-1 * ap-northeast-3 * ap-northeast-2 * ap-northeast-1 * me-south-1 * sa-east-1 * ap-east-1 * cn-north-1 * us-gov-west-1 * ap-southeast-1 * ap-southeast-2 * us-iso-east-1 * ap-southeast-3 * us-east-1 * us-east-2 * cn-northwest-1 * us-isob-east-1 * aws-global * aws-cn-global * aws-us-gov-global * aws-iso-global * aws-iso-b-global | string | eu-west-1 | ||
Access Key | The access key ID of an IAM user, which you create in the AWS console under IAM, Users, Security credentials. Use it together with | string | |||
AWS Host | The host name of AWS. Leave it as | string | amazonaws.com | ||
Autocreate Queue | If true, the queue is created when it does not exist. A name that ends in | boolean | false | ||
Delay Seconds | How long SQS keeps each message hidden after it is sent, before a reader can receive it, in seconds, from 0 to 900 (15 minutes). Use it to process a message later, such as a retry in a minute. When it is not set, the delay of the queue is used, which is 0 for a new queue. | integer | 60 | ||
KMS Key ID | The key of AWS KMS for the queue to encrypt its messages with, as its key ID, such as | string | alias/aws/sqs | ||
Too Many Headers | What to do when a message has more than 10 headers to send as message attributes, the most that SQS takes: Enum values: * WARN * WARN_ONCE * IGNORE * FAIL | string | FAIL | ||
Endpoint Overwrite | If true, it connects to the address in | boolean | false | ||
Profile Credentials Name | The name of the profile in the AWS credentials file. Only used when | string | production | ||
Protocol |
| string | https | http | |
Proxy Host | The host name or IP address of an HTTP proxy to send the calls to AWS through, such as the proxy of a company network. Used only together with | string | proxy.example.com | ||
Proxy Port | The port of the proxy in | integer | 3128 | ||
Proxy Protocol | How to talk to the proxy itself: Enum values: * HTTP * HTTPS | string | HTTP | ||
Queue Owner AWS Account ID | The 12-digit ID of the AWS account that owns the queue, such as | string | 123456789012 | ||
Queue URL | The full address of the queue, as the SQS console shows it. When it is set, the messages are sent to it, and the queue is not looked up by listing the queues, which needs the | string | https://sqs.eu-west-1.amazonaws.com/123456789012/orders | ||
Secret Key | The secret access key that belongs to | string | |||
Server-Side Encryption | If true, the queue is set to encrypt its messages with the key of AWS KMS in | boolean | |||
Session Token | The session token of temporary credentials, which comes with their access key and secret key, for example from AWS STS when you assume an IAM role. Only used when | string | |||
Overwrite Endpoint URI | The address to connect to instead of AWS, as a URL such as | string | http://localhost:4566 | ||
Default Credentials Provider | If true, the credentials are found in the environment instead of in | boolean | false | ||
Profile Credentials Provider | If true, the credentials are read from a named profile in the AWS credentials file ( | boolean | false | ||
Session Credentials | If true, it logs in with temporary credentials: | boolean | false |
Dependencies
At runtime, the aws-sqs-sink Kamelet relies upon the presence of the following dependencies:
-
camel:aws2-sqs
-
camel:kamelet
Camel JBang usage
Prerequisites
-
You’ve installed JBang.
-
You have executed the following command:
jbang app install camel@apache/camel Supposing you have a file named route.yaml with this content:
- route:
from:
uri: "kamelet:timer-source"
parameters:
period: 10000
message: 'test'
steps:
- to:
uri: "kamelet:aws-sqs-sink" You can now run it directly through the following command
camel run route.yaml AWS SQS Sink Kamelet Description
Authentication methods
In this Kamelet you can avoid using explicit static credentials by specifying the useDefaultCredentialsProvider option and set it to true.
The order of evaluation for Default Credentials Provider is the following:
-
Java system properties -
aws.accessKeyIdandaws.secretKey. -
Environment variables -
AWS_ACCESS_KEY_IDandAWS_SECRET_ACCESS_KEY. -
Web Identity Token from AWS STS.
-
The shared credentials and config files.
-
Amazon ECS container credentials - loaded from the Amazon ECS if the environment variable
AWS_CONTAINER_CREDENTIALS_RELATIVE_URIis set. -
Amazon EC2 Instance profile credentials.
You can also use the Profile Credentials Provider, by setting the useProfileCredentialsProvider option to true and profileCredentialsName to the profile name.
Only one of access key/secret key or default credentials provider could be used
For more information, see the AWS credentials documentation
Delaying the Messages
delaySeconds keeps each message hidden for that many seconds after it is sent, up to 900 (15 minutes), before a reader can receive it. Use it to process a message later, such as a retry in a minute.
Without Permission to List the Queues
By default the Kamelet finds the address of the queue by listing the queues, which needs the sqs:ListQueues permission. It does not list the queues when it is given the address:
-
Queue URL (
queueURL): the full address of the queue, as the SQS console shows it. -
Queue Owner AWS Account ID (
queueOwnerAWSAccountId): the 12-digit ID of the account that owns the queue. The address is then built from the account, the region and the queue name. Use it for a queue of another AWS account too. -
An ARN in Queue Name (
queueNameOrArn), which holds the account.
Too Many Headers
An SQS message takes at most 10 message attributes. When a message has more than 10 headers to send, the first 10 are sent, and messageHeaderExceededLimit sets what happens to the others: WARN (the default) logs a warning for each one, WARN_ONCE logs one warning, IGNORE logs nothing, and FAIL fails the message, which is then not sent.
Encrypting the Queue
serverSideEncryptionEnabled and kmsMasterKeyId set the queue to encrypt its messages with a key of AWS KMS (SSE-KMS). They change the queue itself, not only the messages of this Kamelet: when the integration starts, an existing queue is updated, which needs the sqs:SetQueueAttributes permission, and a queue that autoCreateQueue creates is created with the key. serverSideEncryptionEnabled without kmsMasterKeyId has no effect.
For example, to send to a queue without listing the queues, with each message delayed by one minute, and failing a message with too many headers:
- to:
uri: "kamelet:aws-sqs-sink"
parameters:
queueNameOrArn: "orders"
region: "eu-west-1"
queueURL: "https://sqs.eu-west-1.amazonaws.com/123456789012/orders"
delaySeconds: 60
messageHeaderExceededLimit: "FAIL" Through an HTTP Proxy
Where AWS can be reached only through an HTTP proxy, as in many company networks, set proxyHost and proxyPort to the proxy. proxyProtocol is how the Kamelet talks to the proxy itself, not to AWS: it is HTTPS when it is not set, so set it to HTTP for a proxy that takes plain HTTP, such as Squid. The calls to AWS stay encrypted, as the proxy only passes them on. The Kamelet has no option for a proxy user or password.
The step that sends to the Kamelet, in a route:
- to:
uri: "kamelet:aws-sqs-sink"
parameters:
queueNameOrArn: "orders"
region: "eu-west-1"
useDefaultCredentialsProvider: true
proxyHost: "proxy.example.com"
proxyPort: 3128
proxyProtocol: "HTTP"