Azure Storage File Source
Provided by: "Apache Software Foundation"
Support Level for this Kamelet is: "Stable"
Reads the files in a directory of a file share in Azure Files, one message per file, and can delete each file once it has been read.
Use it to process files that people or applications put on a share, such as scanned documents or exports. For blob containers, use azure-storage-blob-source.
Output: the body is the content of the file, which converts to text, bytes or a stream. The CamelFileName header is the path of the file from directoryName, CamelFileNameOnly its name, and CamelFileLength and CamelFileLastModified its size and last-modified time.
Good to know: files are kept after they are read, so the same files are read again on every poll, every 500 milliseconds by default. To read each file only once, set deleteAfterRead to true to delete it, set move to a directory such as .done to move it, or set noop to true to leave it in place and remember the files read, in memory or, with idempotentRepository, across restarts. Subdirectories are read only when recursive is true. include and antInclude read only the files whose names match.
Configuration Options
The following table summarizes the configuration options available for the azure-storage-files-source Kamelet:
| Property | Name | Description | Type | Default | Example |
|---|---|---|---|---|---|
Account Name | Required The name of the storage account, as shown in the Azure portal under Storage accounts. It is the first part of the address of the account, | string | mystorageaccount | ||
Share Name | Required The name of the file share in the storage account to read the files from. | string | documents | ||
Ant Include | Wildcard patterns, separated by commas, that the path of a file below | string | */.csv | ||
Credential Type | How to log in to the storage account. Enum values: * SHARED_ACCOUNT_KEY * AZURE_IDENTITY * SHARED_KEY_CREDENTIAL * AZURE_SAS | string | SHARED_ACCOUNT_KEY | ||
Delay | How long to wait between two polls of the directory, in milliseconds. | integer | 500 | ||
Auto-delete File | If true, each file is deleted from the share once it has been processed. When false, the files stay and are read again on every poll, unless | boolean | false | ||
Directory Name | The directory of the share to read the files from, such as | string | . | inbox/orders | |
Files Read Store | The name of a bean that keeps the list of the files read, so that the files left on the share are not read again after a restart, such as a | string | readFiles | ||
Include | A regular expression that the name of a file must match, as a whole and ignoring case, to be read, such as | string | .*\.csv | ||
Max Files Per Poll | The most files to read at each poll, such as 100. The other files are read at the next polls. Use it to start slowly on a share that holds many files. When it is not set, all files are read at each poll. | integer | 100 | ||
Move After Read | Where to move each file on the share once it has been read and handed on successfully, so that it is not read again, also after a restart. A directory name, such as | string | .done | ||
Leave Files Untouched | If true, the files are left where they are, and each file is read only once while the integration runs. The list of the files read is kept in memory, so the files are all read again after a restart, unless | boolean | |||
Recursive Mode | If true, also reads the files in the subdirectories of | boolean | false | ||
Shared Access Key | An access key of the storage account, from the Azure portal: the storage account, then Security + networking, then Access keys. Used when | string |
Dependencies
At runtime, the azure-storage-files-source Kamelet relies upon the presence of the following dependencies:
-
camel:azure-files
-
camel:kamelet
Camel JBang usage
Prerequisites
-
You’ve installed JBang.
-
You have executed the following command:
jbang app install camel@apache/camel Supposing you have a file named route.yaml with this content:
- route:
from:
uri: "kamelet:azure-storage-files-source"
parameters:
.
.
.
steps:
- to:
uri: "kamelet:log-sink" You can now run it directly through the following command
camel run route.yaml Azure Storage Files Source Kamelet Description
Authentication methods
In this Kamelet, you can use these Azure authentication methods:
-
Plain Shared Account Key:
SHARED_ACCOUNT_KEY(default) -
Azure Identity mechanism:
AZURE_IDENTITY -
Shared key credentials:
SHARED_KEY_CREDENTIAL -
Azure SAS:
AZURE_SAS
The order of evaluation for AZURE_IDENTITY is the following:
-
Enviroment
-
Workload Identity
-
Managed Identity
-
Azure Developer CLI
-
IntelliJ
-
Azure CLI
-
Azure Powershell
For more information, see the Azure Identity documentation
Example with Files Moved After Reading
- route:
from:
uri: "kamelet:azure-storage-files-source"
parameters:
accountName: "mystorageaccount"
shareName: "documents"
sharedKey: "the-access-key"
directoryName: "inbox"
include: ".*\\.csv"
move: ".done"
maxMessagesPerPoll: 100
steps:
- to:
uri: "kamelet:log-sink" Only the CSV files are read, at most 100 at each poll. Each file is moved to inbox/.done once it has been read, so it is not read again, also after a restart. Directories whose names start with a dot are never read, also when recursive is true. A path that ends with the name of the file, such as done/$\{date:now:yyyyMMdd}/$\{file:name}, moves the file to a directory of its own day below directoryName.
Example with Files Remembered Across Restarts
To keep the files on the share and still read each one only once, also across restarts, set noop to true and keep the list of the files read in a file. Declare a FileIdempotentRepository bean that writes the list to a file, and give its name in idempotentRepository:
- beans:
- name: readFiles
type: "#class:org.apache.camel.support.processor.idempotent.FileIdempotentRepository"
properties:
fileStore: "#class:java.io.File('/var/data/read-files.dat')"
- route:
from:
uri: "kamelet:azure-storage-files-source"
parameters:
accountName: "mystorageaccount"
shareName: "documents"
sharedKey: "the-access-key"
directoryName: "inbox"
noop: true
idempotentRepository: "readFiles"
steps:
- to:
uri: "kamelet:log-sink" The file must be on storage that survives a restart of the integration, such as a persistent volume. Without noop, idempotentRepository is not used.