Azure Storage Blob Data Lake Source
Provided by: "Apache Software Foundation"
Support Level for this Kamelet is: "Stable"
Reads the files in a file system of Azure Data Lake Storage Gen2, one message per file, on every poll.
Use it to process files that land in a data lake, such as exports or reports. For plain blob containers, use azure-storage-blob-source, which can also delete what it reads.
Output: the body is the content of the file, as a stream of bytes. The CamelAzureStorageDataLakeFileName header is the name of the file, and other headers give its size, content type, ETag and last-modified time.
Good to know: it neither deletes nor moves the files, so it reads all of them again on every poll, every 500 milliseconds by default. It reads the files at the top of the file system, or of directoryName; the files in directories below are read only when recursive is true. regex keeps only the files whose path matches, and fileName reads one file. Log in as an app registration with clientId, clientSecret and tenantId (the default), or with an account key (accountKey) or a SAS token (sasSignature).
Configuration Options
The following table summarizes the configuration options available for the azure-storage-datalake-source Kamelet:
| Property | Name | Description | Type | Default | Example |
|---|---|---|---|---|---|
Account Name | Required The name of the storage account, an Azure Data Lake Storage Gen2 account (a storage account with hierarchical namespace), as shown in the Azure portal under Storage accounts. | string | mydatalake | ||
File System Name | Required The name of the file system (the container) in the account to read the files from. | string | raw-data | ||
Account Key | An access key of the storage account, from the Azure portal: the storage account, then Security + networking, then Access keys. Used when | string | |||
Client ID | The application (client) ID of an app registration (service principal) in Microsoft Entra ID that has access to the account, from the Overview page of the app registration in the Azure portal. Required when | string | |||
Client Secret | A client secret of the app registration, from the Azure portal: the app registration, then Certificates & secrets. Required when | string | |||
Credential Type | How to log in to the account. Enum values: * CLIENT_SECRET * SHARED_KEY_CREDENTIAL * AZURE_IDENTITY * AZURE_SAS * SERVICE_CLIENT_INSTANCE | string | CLIENT_SECRET | ||
Delay | How long to wait between two polls of the file system, in milliseconds. | integer | 500 | ||
Directory Name | The directory of the file system to read the files from, such as | string | exports/2026 | ||
File Name | The path of one file to read, from the top of the file system, such as | string | exports/sales.csv | ||
Recursive | If true, also reads the files in the directories below the top of the file system, or below | boolean | |||
File Name Pattern | A regular expression that the path of a file, from the top of the file system, must match as a whole to be read, such as | string | .*\.csv | ||
SAS Token | A shared access signature (SAS) token that gives access to the account or the file system, such as | string | |||
Tenant ID | The directory (tenant) ID of the app registration, from its Overview page in the Azure portal. Required when | string |
Dependencies
At runtime, the azure-storage-datalake-source Kamelet relies upon the presence of the following dependencies:
-
camel:azure-storage-datalake
-
camel:kamelet
Camel JBang usage
Prerequisites
-
You’ve installed JBang.
-
You have executed the following command:
jbang app install camel@apache/camel Supposing you have a file named route.yaml with this content:
- route:
from:
uri: "kamelet:azure-storage-datalake-source"
parameters:
.
.
.
steps:
- to:
uri: "kamelet:log-sink" You can now run it directly through the following command
camel run route.yaml Azure Storage Blob Data Lake Source Kamelet Description
Authentication methods
In this Kamelet, you can use these Azure authentication methods:
-
Client secret:
CLIENT_SECRET(default), with clientId, clientSecret and tenantId -
Shared key credentials:
SHARED_KEY_CREDENTIAL, with accountKey -
Azure SAS:
AZURE_SAS, with sasSignature -
Azure Identity mechanism:
AZURE_IDENTITY -
Service client instance:
SERVICE_CLIENT_INSTANCE
For more information, see the Azure Identity documentation
Example with a Directory and a File Name Pattern
- route:
from:
uri: "kamelet:azure-storage-datalake-source"
parameters:
accountName: "mydatalake"
fileSystemName: "raw-data"
credentialType: "SHARED_KEY_CREDENTIAL"
accountKey: "the-account-key"
directoryName: "exports"
recursive: true
regex: ".*\\.csv"
delay: 60000
steps:
- to:
uri: "kamelet:log-sink" Every minute, all the CSV files in the exports directory and the directories below it are read, such as exports/a.csv and exports/2026/c.csv. regex must match the whole path of the file from the top of the file system. The files are not deleted or moved, so they are read again at every poll. To read one file only, give its path in fileName.