Azure CosmosDB Source
Provided by: "Apache Software Foundation"
Support Level for this Kamelet is: "Stable"
Receives the items created or changed in a container of Azure Cosmos DB, through its change feed, as a JSON array per batch.
Use it to react to new or updated documents, for example to copy them to a search index, a cache or Kafka. To create items, use azure-cosmosdb-sink.
Output: the body is a JSON array of the items of one batch, as bytes, each item with the fields that Cosmos DB adds, such as _etag.
Good to know: it reads only the changes made after it first starts, unless startFromBeginning is true, and it does not see deleted items. It keeps its position in a lease container, camel-lease in the database databaseName unless set otherwise, which must exist unless createLeaseContainerIfNotExists is true. Two integrations that read the same container with the same lease container share one position, and each change goes to only one of them, unless each has its own leasePrefix. It checks for new changes every 5 seconds unless feedPollDelay is set. Set hostName so that after a crash, the integration continues at once instead of after up to a minute.
Configuration Options
The following table summarizes the configuration options available for the azure-cosmosdb-source Kamelet:
| Property | Name | Description | Type | Default | Example |
|---|---|---|---|---|---|
Container Name | Required The name of the container in the database to receive the changes of. | string | orders | ||
Database Endpoint | Required The address of the Azure Cosmos DB account, shown as URI on the Overview page of the account in the Azure portal. | string | https://my-account.documents.azure.com:443/ | ||
Database Name | Required The name of the database in the Azure Cosmos DB account. | string | shop | ||
Account Key | The primary or secondary key of the Azure Cosmos DB account, from the Azure portal: the account, then Settings, Keys. Used when | string | |||
Autocreate Lease Container | If true, the lease container is created when it does not exist. If false, a missing lease container stops the source from starting. | boolean | false | ||
Autocreate Lease Database | If true, the lease database is created when it does not exist. If false, a missing lease database stops the source from starting. | boolean | false | ||
Credential Type | How to log in to Azure Cosmos DB. Enum values: * SHARED_ACCOUNT_KEY * AZURE_IDENTITY | string | SHARED_ACCOUNT_KEY | ||
Feed Poll Delay | How long to wait before checking again for new changes, when there were none, in milliseconds. A smaller value receives changes sooner, but makes more requests to Cosmos DB, which use request units. When not set, 5000 (5 seconds). | integer | 1000 | ||
Host Name | A name for this running copy of the integration, which the source writes into the lease container to mark the leases it holds. When it is not set, the source takes a new random name at each start, so after a crash it cannot take back the leases of its previous run until they expire, and receives no changes for up to a minute. With a name that stays the same across restarts, it continues at once. Give each copy that runs at the same time its own name. | string | orders-sync-1 | ||
Lease Container Name | The name of the container where the source keeps its leases, which record how far it has read the change feed. When it is not set, | string | camel-lease | ||
Lease Database Name | The name of the database of the lease container. When it is not set, the lease container is in | string | shop | ||
Lease Prefix | A prefix for the names of the leases of this source in the lease container, such as | string | search- | ||
Max Items Per Batch | The largest number of items in one message, the JSON array of a batch. When not set, 100. | integer | 10 | ||
Start From Beginning | If true, the first start reads all the items of the container, from the oldest change, instead of only the changes made after it starts. Used only when the lease container holds no position yet for this source; afterwards the source continues from its saved position. | boolean |
Dependencies
At runtime, the azure-cosmosdb-source Kamelet relies upon the presence of the following dependencies:
-
camel:azure-cosmosdb
-
camel:kamelet
-
camel:jackson3
Camel JBang usage
Prerequisites
-
You’ve installed JBang.
-
You have executed the following command:
jbang app install camel@apache/camel Supposing you have a file named route.yaml with this content:
- route:
from:
uri: "kamelet:azure-cosmosdb-source"
parameters:
.
.
.
steps:
- to:
uri: "kamelet:log-sink" You can now run it directly through the following command
camel run route.yaml Azure CosmosDB Source Kamelet Description
Authentication methods
In this Kamelet, you can use these Azure authentication methods:
-
Azure Identity mechanism:
AZURE_IDENTITY -
Plain Shared Account Key:
SHARED_ACCOUNT_KEY
The order of evaluation for AZURE_IDENTITY is the following:
-
Enviroment
-
Workload Identity
-
Managed Identity
-
Azure Developer CLI
-
IntelliJ
-
Azure CLI
-
Azure Powershell
For more information, see the Azure Identity documentation
Reading the change feed
The Kamelet keeps its position in the lease container. These options change how it reads:
-
startFromBeginning: if true, the first start reads all the items of the container. Once a position is saved, the source continues from it. -
maxItemCount: the largest number of items in one message, 100 by default. -
feedPollDelay: how long to wait before checking again for new changes, in milliseconds, 5000 by default. A smaller value receives changes sooner, with more requests to Cosmos DB. -
leasePrefix: two integrations that read the same container with the same lease container share one position, and each change goes to only one of them. Give each integration its ownleasePrefixto have each receive every change. -
hostName: the name the source marks its leases with. Without it, the source takes a new random name at each start, and after a crash waits up to a minute for the leases of its previous run to expire. With a name that stays the same, it continues at once. Give each copy that runs at the same time its own name.
- route:
from:
uri: "kamelet:azure-cosmosdb-source"
parameters:
databaseEndpoint: "https://my-account.documents.azure.com:443/"
accountKey: "{{cosmos.key}}"
databaseName: "shop"
containerName: "orders"
createLeaseContainerIfNotExists: true
startFromBeginning: true
maxItemCount: 10
feedPollDelay: 1000
leasePrefix: "search-"
hostName: "orders-search-1"
steps:
- to:
uri: "kamelet:log-sink"