Azure Storage Blob Event-based Source
Provided by: "Apache Software Foundation"
Support Level for this Kamelet is: "Preview"
Receives the Event Grid events of an Azure Storage account from a Service Bus queue or topic, and can read the content of each new blob in a container.
Use it to react quickly to new blobs, such as uploaded files, without reading the container over and over. It needs an Event Grid subscription on the storage account that sends the events, in the Event Grid schema, to a Service Bus queue, or to a topic so that other applications can receive the same events from subscriptions of their own. To read the container on a timer instead, use azure-storage-blob-source.
Output: when getBlob is false, the default, the body is the Event Grid event as JSON (an Azure BinaryData, which converts to text or bytes). When getBlob is true, the body of a Microsoft.Storage.BlobCreated event for a blob of containerName is the content of that blob, as a stream of bytes; the other events are passed on as parsed JSON (a Jackson JSON node).
Good to know: with the default serviceBusReceiveMode of RECEIVE_AND_DELETE, an event is removed from the queue, or from the subscription of the topic, as soon as it is received, so an event whose processing fails is lost. With getBlob, only blobs at the top of the container can be read: an event for a blob in a virtual folder, such as in/a.txt, fails. Events are processed one at a time, in order, unless maxConcurrentCalls is set.
Configuration Options
The following table summarizes the configuration options available for the azure-storage-blob-event-based-source Kamelet:
| Property | Name | Description | Type | Default | Example |
|---|---|---|---|---|---|
Account Name | Required The name of the storage account, as shown in the Azure portal under Storage accounts. It is the first part of the address of the account, | string | mystorageaccount | ||
Connection String | Required The connection string of the Service Bus namespace, from the Azure portal: the namespace, then Settings, Shared access policies, a policy, Primary Connection String. It starts with | string | |||
Container Name | Required The name of the container whose new blobs are read when | string | uploads | ||
Queue or Topic Name | Required The name of the Service Bus queue, or of the topic when | string | blob-events | ||
Access Key | An access key of the storage account, from the Azure portal: the storage account, then Security + networking, then Access keys. Used when | string | |||
Client ID | The application (client) ID of an app registration (service principal) in Microsoft Entra ID that has access to the storage account, from the Overview page of the app registration in the Azure portal. Used with | string | |||
Client Secret | A client secret of the app registration, from the Azure portal: the app registration, then Certificates & secrets. Used with | string | |||
Credential Type | How to log in to the storage account to read the new blobs. Enum values: * SHARED_ACCOUNT_KEY * SHARED_KEY_CREDENTIAL * AZURE_IDENTITY * CLIENT_SECRET | string | SHARED_ACCOUNT_KEY | ||
Get Blob | If true, the body of each event about a new blob in | boolean | false | ||
Max Concurrent Events | How many events are processed at the same time. When it is not set, one at a time, in order. With a higher number, events are processed in parallel and can finish out of order. | integer | 4 | ||
Service Bus Receive Mode |
Enum values: * RECEIVE_AND_DELETE * PEEK_LOCK | string | RECEIVE_AND_DELETE | ||
Service Bus Type |
Enum values: * queue * topic | string | |||
Subscription Name | The name of the subscription of the topic to read the events from. Required when | string | |||
Tenant ID | The directory (tenant) ID of the app registration, from its Overview page in the Azure portal. Used with | string |
Dependencies
At runtime, the azure-storage-blob-event-based-source Kamelet relies upon the presence of the following dependencies:
-
camel:azure-servicebus
-
camel:azure-storage-blob
-
camel:kamelet
-
camel:jsonpath
-
camel:jackson3
Camel JBang usage
Prerequisites
-
You’ve installed JBang.
-
You have executed the following command:
jbang app install camel@apache/camel Supposing you have a file named route.yaml with this content:
- route:
from:
uri: "kamelet:azure-storage-blob-event-based-source"
parameters:
.
.
.
steps:
- to:
uri: "kamelet:log-sink" You can now run it directly through the following command
camel run route.yaml Azure Storage Blob Event Based Source Kamelet Description
Authentication methods
For this Kamelet the Connection string is the basic method for authenticating to the Azure Servicebus Queue.
To use this Kamelet you’ll need to set up Events on your Azure Storage Blob account and select as an endpoint an Azure Servicebus Queue or Topic, with the Event Grid event schema.
For the Azure Storage Blob access, you can use these Azure authentication methods:
-
Azure Identity mechanism:
AZURE_IDENTITY -
Plain Shared Account Key:
SHARED_ACCOUNT_KEY -
Shared Key Credential:
SHARED_KEY_CREDENTIAL
To log in as an app registration (a service principal), set credentialType to AZURE_IDENTITY and provide clientId, clientSecret, and tenantId: when all three are set, they are used instead of the identity of the environment. CLIENT_SECRET does not use these three properties.
For more information, see the Azure Identity documentation
Example: Read the Events from a Topic
When the events go to a Service Bus topic, so that other applications can receive them from subscriptions of their own, set serviceBusType to topic, topicOrQueueName to the name of the topic, and subscriptionName to the subscription this Kamelet reads from:
- route:
from:
uri: "kamelet:azure-storage-blob-event-based-source"
parameters:
connectionString: "Endpoint=sb://my-namespace.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=..."
serviceBusType: "topic"
topicOrQueueName: "blob-events"
subscriptionName: "invoice-processor"
maxConcurrentCalls: 4
accountName: "mystorageaccount"
containerName: "invoices"
accessKey: "..."
getBlob: true
steps:
- to: "log:blob" Without subscriptionName the Kamelet does not start. maxConcurrentCalls processes up to four events at the same time; when it is not set, events are processed one at a time, in order.