FTPS Source
Provided by: "Apache Software Foundation"
Support Level for this Kamelet is: "Stable"
Reads the files in a directory on an FTPS server (FTP with TLS encryption), one message per file, and by default leaves them on the server.
Use it to collect the files that a partner or a legacy system drops on an FTPS server, such as orders, invoices or exports. For a server without TLS use ftp-source, and for a server you reach with SSH use sftp-source.
Output: the body is the content of the file, as a stream. The file and ce-file headers hold the name of the file, with its path below directoryName when recursive is true, such as 2026/orders.csv. CamelFileName holds the same name, and CamelFileLength the size of the file in bytes.
Good to know: files are not deleted or moved after they are read, unless delete is true or move is set. A file is read only once while the integration runs (idempotent), but the files left on the server are all read again after a restart; to read each file only once, also across restarts, set move to a directory such as .done, set delete to true, or keep the list of the files read in a file with idempotentRepository. To read only some of the files, set include, exclude or antInclude. To leave a file alone while another program is still writing it, set readLock. By default it uses explicit TLS: it connects to the normal FTP port, usually 21, and then switches to TLS. For a server that uses implicit TLS, usually on port 990, set implicit to true. Files are transferred in ASCII (text) mode unless binary is true.
Configuration Options
The following table summarizes the configuration options available for the ftps-source Kamelet:
| Property | Name | Description | Type | Default | Example |
|---|---|---|---|---|---|
Connection Host | Required The host name or IP address of the FTPS server, without | string | ftps.example.com | ||
Connection Port | Required The port of the FTPS server, usually 21, or 990 when | string | 21 | ||
Directory Name | Required The directory on the server to read the files from. A path without a leading | string | outgoing/orders | ||
Password | Required The password of | string | |||
Username | Required The user to log in to the FTPS server as. | string | |||
Include Files (Wildcards) | Wildcard patterns, separated by commas, that the path of a file must match to be read, such as | string | */.csv | ||
Autocreate Missing Directories | If true, | boolean | true | ||
Binary Mode | If true, files are transferred in binary mode, byte for byte. When false, they are transferred in ASCII mode, which is meant for text. Set it to true for files that are not text, such as images, archives or PDF files. | boolean | false | ||
Connect Timeout | How long to wait for the connection to the FTPS server, in milliseconds, and then for its first answer. A connection that fails is tried 3 times before the poll is skipped. When it is not set, 10000. | integer | 10000 | ||
Data Timeout | How long to wait for data while a file or the list of the files is transferred, in milliseconds, before the poll fails. When it is not set, 30000. | integer | 30000 | ||
Poll Interval | How long to wait between two polls of the directory, in milliseconds. For example, 60000 polls once a minute. When it is not set, 500. | integer | 60000 | ||
Delete | If true, each file is deleted from the server once it has been read and handed on successfully. Use it to read each file only once, also across restarts. To keep the files, set | boolean | false | ||
Exclude Files | A regular expression for the names of the files not to read, such as | string | .*\.(tmp|part) | ||
Skip Files Already Read | If true, a file is read only once while the integration runs, even when it stays on the server. The list of the files read is kept in memory, so after a restart the files still on the server are read again, unless | boolean | true | ||
Files Read Store | The name of a bean that keeps the list of the files read, so that the files left on the server are not read again after a restart, such as a | string | readFiles | ||
Implicit TLS | If true, the connection uses implicit TLS: it is encrypted from the start, usually on port 990. If false, it uses explicit TLS: it connects to the normal FTP port, usually 21, and then switches to TLS. Use the mode the server expects; the other one fails to connect. When it is not set, false. | boolean | |||
Include Files | A regular expression that the name of a file must match to be read, such as | string | .*\.csv | ||
Max Files Per Poll | The largest number of files to read in one poll, such as 100. The other files are read at the next polls. The poll takes the first files the server lists, and | integer | 100 | ||
Move After Read | Where to move each file on the server once it has been read and handed on successfully, so that it is not read again, also after a restart. A directory name, such as | string | .done | ||
Move Failed Files To | Where to move a file on the server when it could not be handed on, for example because the next step failed, so that it is not read over and over. The same format as | string | .error | ||
Passive Mode | If true, the transfers use passive mode, in which the integration opens all the connections to the server. Set it to true when the integration or the server is behind a firewall or NAT, where the default active mode often fails. | boolean | false | ||
Move Before Read | Where to move each file on the server before it is read, so that another reader of the same directory does not take it too. The same format as | string | .inprogress | ||
Wait Until Written | How to make sure that another program has finished writing a file before it is read. Enum values: * none * changed * rename | string | changed | ||
Minimum File Age | Only used when | integer | 60000 | ||
Recursive | If true, the files in the subdirectories of | boolean | false | ||
Socket Timeout | How long to wait for the answer of the server to a command, in milliseconds, before the poll fails. When it is not set, 300000 (5 minutes). | integer | 60000 | ||
Sort By | The order in which the files of a poll are read. | string | file:modified | ||
Change Directory Stepwise | If true, the integration changes into the directory of a file, one level at a time, to read it. If false, it uses the full path of the file instead, which some servers need. It must be false when | boolean | |||
Stream Download | If true, each file is read from the server while it is handed on, instead of being downloaded into memory first, and the body is a stream. Use it for files too large for memory. It saves memory only when the integration does not keep the stream in memory either: set | boolean |
Dependencies
At runtime, the ftps-source Kamelet relies upon the presence of the following dependencies:
-
camel:ftp
-
camel:kamelet
Camel JBang usage
Prerequisites
-
You’ve installed JBang.
-
You have executed the following command:
jbang app install camel@apache/camel Supposing you have a file named route.yaml with this content:
- route:
from:
uri: "kamelet:ftps-source"
parameters:
.
.
.
steps:
- to:
uri: "kamelet:log-sink" You can now run it directly through the following command
camel run route.yaml FTPS Source Kamelet Description
Authentication
This Kamelet requires username and password authentication to access the FTPS server. FTPS provides FTP with SSL/TLS encryption for secure file transfer. The credentials are configured through the username and password properties.
Configuration
The FTPS Source Kamelet supports the following configurations:
-
Connection Host: The hostname or IP address of the FTPS server (required)
-
Connection Port: The port number of the FTPS server (default: 21)
-
Username: Username for FTPS authentication (required)
-
Password: Password for FTPS authentication (required)
-
Directory Name: The starting directory path on the FTPS server (required)
-
Files are left on the server after they are read unless Delete is true or Move is set; with Idempotent, they are skipped while the integration runs but read again after a restart
-
Passive Mode: Use passive mode for FTPS connections (default: false)
-
Recursive: Process files in subdirectories (default: false)
-
Idempotent: Skip already-processed files (default: true)
-
Binary: Use binary transfer mode instead of ASCII (default: false)
-
Auto Create: Automatically create the starting directory if it doesn’t exist (default: true)
-
Delete: Delete files after successful processing (default: false)
-
Move: Where to move each file on the server after it has been read, such as
.done, so that it is not read again, also after a restart (not set by default). It cannot be combined with Delete -
Move Failed: Where to move a file on the server when it could not be handed on, such as
.error(not set by default). When not set, a file that fails is read again at every poll -
Include Files: A regular expression that the name of a file must match to be read, such as
.*\.csv; upper and lower case count as the same (not set by default) -
Exclude Files: A regular expression for the names of the files not to read; it wins over Include Files and Include Files (Wildcards) (not set by default)
-
Include Files (Wildcards): Wildcard patterns for the path below the directory, such as
.csvor*/*.csv; upper and lower case differ (not set by default) -
Move Before Read: Where to move each file before it is read, such as
.inprogress(not set by default) -
Wait Until Written:
changedreads a file only once it has stopped changing,renameonce it can be renamed (default: none) -
Minimum File Age: With
changed, how old a file must be before it is read, in milliseconds (not set by default) -
Poll Interval: How long to wait between two polls, in milliseconds (default: 500)
-
Max Files Per Poll: The largest number of files read in one poll (not set by default: all)
-
Sort By: The order of the files in a poll, such as
file:modifiedorreverse:file:name(not set by default: the order of the server) -
Stream Download: Read each file from the server while it is handed on, instead of into memory first (default: false)
-
Change Directory Stepwise: Change into the directory of a file one level at a time; must be false with Stream Download (default: true)
-
Connect Timeout: How long to wait for the connection, in milliseconds (default: 10000)
-
Socket Timeout: How long to wait for the answer to a command, in milliseconds (default: 300000)
-
Data Timeout: How long to wait for data while a file or a listing is transferred, in milliseconds (default: 30000)
-
Implicit TLS: Use implicit TLS, usually on port 990, instead of explicit TLS (default: false)
Output Format
The Kamelet outputs file content as an InputStream and sets headers with file information: - file: The name of the processed file - ce-file: Cloud Events compatible file name header
Security
FTPS encrypts the connection of standard FTP with TLS. By default the Kamelet uses explicit FTPS: it switches to TLS on the normal FTP port. Set Implicit TLS for a server that uses implicit FTPS, usually on port 990.
Usage Example
- route:
from:
uri: "kamelet:ftps-source"
parameters:
connectionHost: "ftps.example.com"
connectionPort: "21"
username: "ftpsuser"
password: "ftpspass"
directoryName: "/secure-incoming"
steps:
- to:
uri: "kamelet:log-sink" Example with Passive Mode and Binary Transfer
- route:
from:
uri: "kamelet:ftps-source"
parameters:
connectionHost: "ftps.example.com"
connectionPort: "21"
username: "ftpsuser"
password: "ftpspass"
directoryName: "/data"
passiveMode: true
binary: true
recursive: true
steps:
- to:
uri: "kamelet:log-sink" This example enables passive mode, binary transfer, and recursive directory processing. The Kamelet uses explicit FTPS here: it connects to the normal FTP port and then switches to TLS.
Example with Files Moved After Reading
- route:
from:
uri: "kamelet:ftps-source"
parameters:
connectionHost: "ftps.example.com"
username: "ftpsuser"
password: "ftpspass"
directoryName: "/incoming"
move: ".done"
moveFailed: ".error"
steps:
- to:
uri: "kamelet:log-sink" Each file is moved to the .done directory next to it once it has been read, and to .error when it could not be handed on, so a restart does not read the same files again while they are kept on the server. Directories whose names start with a dot are never read, also when Recursive is true. The user needs the right to create directories and rename files on the server.
Example with Files Remembered Across Restarts
To keep the files on the server and still read each one only once, also across restarts, keep the list of the files read in a file. Declare a FileIdempotentRepository bean that writes the list to a file, and give its name in idempotentRepository:
- beans:
- name: readFiles
type: "#class:org.apache.camel.support.processor.idempotent.FileIdempotentRepository"
properties:
fileStore: "#class:java.io.File('/var/data/read-files.dat')"
- route:
from:
uri: "kamelet:ftps-source"
parameters:
connectionHost: "ftps.example.com"
username: "ftpsuser"
password: "ftpspass"
directoryName: "/incoming"
idempotentRepository: "readFiles"
steps:
- to:
uri: "kamelet:log-sink" The file must be on storage that survives a restart of the integration, such as a persistent volume. A file that is put on the server again under the same name is not read again.
Example with File Filters and Files Still Being Written
- route:
from:
uri: "kamelet:ftps-source"
parameters:
connectionHost: "ftps.example.com"
username: "ftpsuser"
password: "ftpspass"
directoryName: "/incoming"
recursive: true
antInclude: "**/*.csv"
exclude: "draft-.*"
readLock: "changed"
sortBy: "file:modified"
maxMessagesPerPoll: 100
delay: 60000
steps:
- to:
uri: "kamelet:log-sink" Once a minute, this reads the CSV files in /incoming and its subdirectories, except those whose names start with draft-, at most 100 of them in each poll, the oldest first. Include Files (Wildcards) matches the path below the directory, and upper and lower case differ: *.csv alone would match only the files directly in /incoming. Include Files and Exclude Files are regular expressions that match the name of the file alone, whatever the case. With Max Files Per Poll, a poll takes the first files the server lists and sorts only these.
readLock: changed reads a file only once its size and modification time have not changed for 5 seconds, so a file that another program is still uploading is left alone. It never reads an empty file, and while it checks a file the other files wait. With Minimum File Age, a file is read only once it is that old instead. FTP servers list the modification time only to the minute and in their own time zone, so use Minimum File Age only when the server and the integration are in the same time zone; otherwise every file looks older or younger by the difference. Sort By file:modified is also only to the minute.
The files a poll reads can be moved out of the way first with Move Before Read, so that a second integration reading the same directory does not take them too:
- route:
from:
uri: "kamelet:ftps-source"
parameters:
connectionHost: "ftps.example.com"
username: "ftpsuser"
password: "ftpspass"
directoryName: "/incoming"
preMove: ".inprogress"
move: ".done"
steps:
- to:
uri: "kamelet:log-sink" Each file is moved to .inprogress before it is read, and from there to .done once it has been handed on, so it ends up in /incoming/.inprogress/.done.
Example with Large Files
By default each file is downloaded into memory before it is handed on. For files too large for that, set Stream Download:
- route:
from:
uri: "kamelet:ftps-source"
parameters:
connectionHost: "ftps.example.com"
username: "ftpsuser"
password: "ftpspass"
directoryName: "/exports"
streamDownload: true
stepwise: false
binary: true
steps:
- to:
uri: "kamelet:log-sink" Camel keeps a message body that is a stream in memory too, unless told otherwise, so also set camel.main.streamCachingSpoolEnabled=true in the application properties: large bodies are then written to a temporary file. In a test with 256 MB of memory, a 400 MB file failed with an out of memory error without Stream Download, and with it but without spooling, and was read with both. Stream Download needs Change Directory Stepwise set to false, or every file fails.
Example with Timeouts
- route:
from:
uri: "kamelet:ftps-source"
parameters:
connectionHost: "ftps.example.com"
username: "ftpsuser"
password: "ftpspass"
directoryName: "/incoming"
connectTimeout: 5000
socketTimeout: 60000
dataTimeout: 30000
steps:
- to:
uri: "kamelet:log-sink" Connect Timeout is how long to wait for the connection and the first answer of the server; a connection that fails is tried 3 times before the poll is skipped. Socket Timeout is how long to wait for the answer to a command, 5 minutes by default, and Data Timeout how long to wait for data while a file or the list of the files is transferred. When one of them runs out, the poll fails and the next poll connects again.
Example with Implicit TLS
- route:
from:
uri: "kamelet:ftps-source"
parameters:
connectionHost: "ftps.example.com"
connectionPort: "990"
username: "ftpsuser"
password: "ftpspass"
directoryName: "/incoming"
implicit: true
passiveMode: true
steps:
- to:
uri: "kamelet:log-sink" With implicit TLS the connection is encrypted from the start, usually on port 990. Use the mode the server expects: explicit TLS against an implicit server waits for an answer until Connect Timeout runs out, and implicit TLS against an explicit server fails at once.