FTP Source
Provided by: "Apache Software Foundation"
Support Level for this Kamelet is: "Stable"
Reads the files in a directory on an FTP server, one message per file, and by default leaves them on the server.
Use it to collect the files that a partner or a legacy system drops on an FTP server, such as orders, invoices or exports. For an encrypted connection use ftps-source, and for a server you reach with SSH use sftp-source.
Output: the body is the content of the file, as a stream. The file and ce-file headers hold the name of the file, with its path below directoryName when recursive is true, such as 2026/orders.csv. CamelFileName holds the same name, and CamelFileLength the size of the file in bytes.
Good to know: files are not deleted or moved after they are read, unless delete is true or move is set. A file is read only once while the integration runs (idempotent), but the files left on the server are all read again after a restart; to read each file only once, also across restarts, set move to a directory such as .done, set delete to true, or keep the list of the files read in a file with idempotentRepository. To read only some of the files, set include, exclude or antInclude. To leave a file alone while another program is still writing it, set readLock. The connection and the login are not encrypted. Files are transferred in ASCII (text) mode unless binary is true.
Configuration Options
The following table summarizes the configuration options available for the ftp-source Kamelet:
| Property | Name | Description | Type | Default | Example |
|---|---|---|---|---|---|
Connection Host | Required The host name or IP address of the FTP server, without | string | ftp.example.com | ||
Connection Port | Required The port of the FTP server. FTP servers usually listen on port 21. | string | 21 | ||
Directory Name | Required The directory on the server to read the files from. A path without a leading | string | outgoing/orders | ||
Password | Required The password of | string | |||
Username | Required The user to log in to the FTP server as. | string | |||
Include Files (Wildcards) | Wildcard patterns, separated by commas, that the path of a file must match to be read, such as | string | */.csv | ||
Autocreate Missing Directories | If true, | boolean | true | ||
Binary Mode | If true, files are transferred in binary mode, byte for byte. When false, they are transferred in ASCII mode, which is meant for text. Set it to true for files that are not text, such as images, archives or PDF files. | boolean | false | ||
Connect Timeout | How long to wait for the connection to the FTP server, in milliseconds, and then for its first answer. A connection that fails is tried 3 times before the poll is skipped. When it is not set, 10000. | integer | 10000 | ||
Data Timeout | How long to wait for data while a file or the list of the files is transferred, in milliseconds, before the poll fails. When it is not set, 30000. | integer | 30000 | ||
Poll Interval | How long to wait between two polls of the directory, in milliseconds. For example, 60000 polls once a minute. When it is not set, 500. | integer | 60000 | ||
Delete | If true, each file is deleted from the server once it has been read and handed on successfully. Use it to read each file only once, also across restarts. To keep the files, set | boolean | false | ||
Exclude Files | A regular expression for the names of the files not to read, such as | string | .*\.(tmp|part) | ||
Skip Files Already Read | If true, a file is read only once while the integration runs, even when it stays on the server. The list of the files read is kept in memory, so after a restart the files still on the server are read again, unless | boolean | true | ||
Files Read Store | The name of a bean that keeps the list of the files read, so that the files left on the server are not read again after a restart, such as a | string | readFiles | ||
Include Files | A regular expression that the name of a file must match to be read, such as | string | .*\.csv | ||
Max Files Per Poll | The largest number of files to read in one poll, such as 100. The other files are read at the next polls. The poll takes the first files the server lists, and | integer | 100 | ||
Move After Read | Where to move each file on the server once it has been read and handed on successfully, so that it is not read again, also after a restart. A directory name, such as | string | .done | ||
Move Failed Files To | Where to move a file on the server when it could not be handed on, for example because the next step failed, so that it is not read over and over. The same format as | string | .error | ||
Passive Mode | If true, the transfers use passive mode, in which the integration opens all the connections to the server. Set it to true when the integration or the server is behind a firewall or NAT, where the default active mode often fails. | boolean | false | ||
Move Before Read | Where to move each file on the server before it is read, so that another reader of the same directory does not take it too. The same format as | string | .inprogress | ||
Wait Until Written | How to make sure that another program has finished writing a file before it is read. Enum values: * none * changed * rename | string | changed | ||
Minimum File Age | Only used when | integer | 60000 | ||
Recursive | If true, the files in the subdirectories of | boolean | false | ||
Socket Timeout | How long to wait for the answer of the server to a command, in milliseconds, before the poll fails. When it is not set, 300000 (5 minutes). | integer | 60000 | ||
Sort By | The order in which the files of a poll are read. | string | file:modified | ||
Change Directory Stepwise | If true, the integration changes into the directory of a file, one level at a time, to read it. If false, it uses the full path of the file instead, which some servers need. It must be false when | boolean | |||
Stream Download | If true, each file is read from the server while it is handed on, instead of being downloaded into memory first, and the body is a stream. Use it for files too large for memory. It saves memory only when the integration does not keep the stream in memory either: set | boolean |
Dependencies
At runtime, the ftp-source Kamelet relies upon the presence of the following dependencies:
-
camel:ftp
-
camel:kamelet
Camel JBang usage
Prerequisites
-
You’ve installed JBang.
-
You have executed the following command:
jbang app install camel@apache/camel Supposing you have a file named route.yaml with this content:
- route:
from:
uri: "kamelet:ftp-source"
parameters:
.
.
.
steps:
- to:
uri: "kamelet:log-sink" You can now run it directly through the following command
camel run route.yaml FTP Source Kamelet Description
Authentication
This Kamelet requires username and password authentication to access the FTP server. The credentials are configured through the username and password properties.
Configuration
The FTP Source Kamelet supports the following configurations:
-
Connection Host: The hostname or IP address of the FTP server (required)
-
Connection Port: The port number of the FTP server (default: 21)
-
Username: Username for FTP authentication (required)
-
Password: Password for FTP authentication (required)
-
Directory Name: The starting directory path on the FTP server (required)
-
Passive Mode: Use passive mode for FTP connections (default: false)
-
Recursive: Process files in subdirectories (default: false)
-
Idempotent: Skip already-processed files (default: true)
-
Binary: Use binary transfer mode instead of ASCII (default: false)
-
Auto Create: Automatically create the starting directory if it doesn’t exist (default: true)
-
Delete: Delete files after successful processing (default: false). When false, and Move is not set, files are left on the server; with Idempotent, they are skipped while the integration runs but read again after a restart
-
Move: Where to move each file on the server after it has been read, such as
.done, so that it is not read again, also after a restart (not set by default). It cannot be combined with Delete -
Move Failed: Where to move a file on the server when it could not be handed on, such as
.error(not set by default). When not set, a file that fails is read again at every poll -
Include Files: A regular expression that the name of a file must match to be read, such as
.*\.csv; upper and lower case count as the same (not set by default) -
Exclude Files: A regular expression for the names of the files not to read; it wins over Include Files and Include Files (Wildcards) (not set by default)
-
Include Files (Wildcards): Wildcard patterns for the path below the directory, such as
.csvor*/*.csv; upper and lower case differ (not set by default) -
Move Before Read: Where to move each file before it is read, such as
.inprogress(not set by default) -
Wait Until Written:
changedreads a file only once it has stopped changing,renameonce it can be renamed (default: none) -
Minimum File Age: With
changed, how old a file must be before it is read, in milliseconds (not set by default) -
Poll Interval: How long to wait between two polls, in milliseconds (default: 500)
-
Max Files Per Poll: The largest number of files read in one poll (not set by default: all)
-
Sort By: The order of the files in a poll, such as
file:modifiedorreverse:file:name(not set by default: the order of the server) -
Stream Download: Read each file from the server while it is handed on, instead of into memory first (default: false)
-
Change Directory Stepwise: Change into the directory of a file one level at a time; must be false with Stream Download (default: true)
-
Connect Timeout: How long to wait for the connection, in milliseconds (default: 10000)
-
Socket Timeout: How long to wait for the answer to a command, in milliseconds (default: 300000)
-
Data Timeout: How long to wait for data while a file or a listing is transferred, in milliseconds (default: 30000)
Output Format
The Kamelet outputs file content as an InputStream and sets headers with file information: - file: The name of the processed file - ce-file: Cloud Events compatible file name header
Usage Example
- route:
from:
uri: "kamelet:ftp-source"
parameters:
connectionHost: "ftp.example.com"
connectionPort: "21"
username: "ftpuser"
password: "ftppass"
directoryName: "/incoming"
steps:
- to:
uri: "kamelet:log-sink" Example with Passive Mode and Recursive Processing
- route:
from:
uri: "kamelet:ftp-source"
parameters:
connectionHost: "ftp.example.com"
connectionPort: "21"
username: "ftpuser"
password: "ftppass"
directoryName: "/data"
passiveMode: true
recursive: true
binary: true
steps:
- to:
uri: "kamelet:log-sink" Example with Files Moved After Reading
- route:
from:
uri: "kamelet:ftp-source"
parameters:
connectionHost: "ftp.example.com"
username: "ftpuser"
password: "ftppass"
directoryName: "/incoming"
move: ".done"
moveFailed: ".error"
steps:
- to:
uri: "kamelet:log-sink" Each file is moved to the .done directory next to it once it has been read, and to .error when it could not be handed on, so a restart does not read the same files again while they are kept on the server. Directories whose names start with a dot are never read, also when Recursive is true. The user needs the right to create directories and rename files on the server.
Example with Files Remembered Across Restarts
To keep the files on the server and still read each one only once, also across restarts, keep the list of the files read in a file. Declare a FileIdempotentRepository bean that writes the list to a file, and give its name in idempotentRepository:
- beans:
- name: readFiles
type: "#class:org.apache.camel.support.processor.idempotent.FileIdempotentRepository"
properties:
fileStore: "#class:java.io.File('/var/data/read-files.dat')"
- route:
from:
uri: "kamelet:ftp-source"
parameters:
connectionHost: "ftp.example.com"
username: "ftpuser"
password: "ftppass"
directoryName: "/incoming"
idempotentRepository: "readFiles"
steps:
- to:
uri: "kamelet:log-sink" The file must be on storage that survives a restart of the integration, such as a persistent volume. A file that is put on the server again under the same name is not read again.
Example with File Deletion
- route:
from:
uri: "kamelet:ftp-source"
parameters:
connectionHost: "ftp.example.com"
connectionPort: "21"
username: "ftpuser"
password: "ftppass"
directoryName: "/processed"
delete: true
idempotent: false
steps:
- to:
uri: "kamelet:log-sink" This example deletes files after processing and disables idempotency to allow re-processing of files with the same name.
Example with File Filters and Files Still Being Written
- route:
from:
uri: "kamelet:ftp-source"
parameters:
connectionHost: "ftp.example.com"
username: "ftpuser"
password: "ftppass"
directoryName: "/incoming"
recursive: true
antInclude: "**/*.csv"
exclude: "draft-.*"
readLock: "changed"
sortBy: "file:modified"
maxMessagesPerPoll: 100
delay: 60000
steps:
- to:
uri: "kamelet:log-sink" Once a minute, this reads the CSV files in /incoming and its subdirectories, except those whose names start with draft-, at most 100 of them in each poll, the oldest first. Include Files (Wildcards) matches the path below the directory, and upper and lower case differ: *.csv alone would match only the files directly in /incoming. Include Files and Exclude Files are regular expressions that match the name of the file alone, whatever the case. With Max Files Per Poll, a poll takes the first files the server lists and sorts only these.
readLock: changed reads a file only once its size and modification time have not changed for 5 seconds, so a file that another program is still uploading is left alone. It never reads an empty file, and while it checks a file the other files wait. With Minimum File Age, a file is read only once it is that old instead. FTP servers list the modification time only to the minute and in their own time zone, so use Minimum File Age only when the server and the integration are in the same time zone; otherwise every file looks older or younger by the difference. Sort By file:modified is also only to the minute.
The files a poll reads can be moved out of the way first with Move Before Read, so that a second integration reading the same directory does not take them too:
- route:
from:
uri: "kamelet:ftp-source"
parameters:
connectionHost: "ftp.example.com"
username: "ftpuser"
password: "ftppass"
directoryName: "/incoming"
preMove: ".inprogress"
move: ".done"
steps:
- to:
uri: "kamelet:log-sink" Each file is moved to .inprogress before it is read, and from there to .done once it has been handed on, so it ends up in /incoming/.inprogress/.done.
Example with Large Files
By default each file is downloaded into memory before it is handed on. For files too large for that, set Stream Download:
- route:
from:
uri: "kamelet:ftp-source"
parameters:
connectionHost: "ftp.example.com"
username: "ftpuser"
password: "ftppass"
directoryName: "/exports"
streamDownload: true
stepwise: false
binary: true
steps:
- to:
uri: "kamelet:log-sink" Camel keeps a message body that is a stream in memory too, unless told otherwise, so also set camel.main.streamCachingSpoolEnabled=true in the application properties: large bodies are then written to a temporary file. In a test with 256 MB of memory, a 400 MB file failed with an out of memory error without Stream Download, and with it but without spooling, and was read with both. Stream Download needs Change Directory Stepwise set to false, or every file fails.
Example with Timeouts
- route:
from:
uri: "kamelet:ftp-source"
parameters:
connectionHost: "ftp.example.com"
username: "ftpuser"
password: "ftppass"
directoryName: "/incoming"
connectTimeout: 5000
socketTimeout: 60000
dataTimeout: 30000
steps:
- to:
uri: "kamelet:log-sink" Connect Timeout is how long to wait for the connection and the first answer of the server; a connection that fails is tried 3 times before the poll is skipped. Socket Timeout is how long to wait for the answer to a command, 5 minutes by default, and Data Timeout how long to wait for data while a file or the list of the files is transferred. When one of them runs out, the poll fails and the next poll connects again.