LangChain4j Ingest File Source
Provided by: "Apache Software Foundation"
Support Level for this Kamelet is: "Preview"
Reads the documents in a directory and its subdirectories, one message per file, leaves them in place, and reads a file again when it changes.
Use it to feed a knowledge base for an AI assistant: put it in front of langchain4j-ingest-sink, with tika-extract-text-action or docling-convert-action in between for PDF or office documents.
Output: the body is the file, read as bytes unless charset is set. The CamelLangChain4jIngestDocumentId header is the path of the file relative to directory, such as manuals/pump.pdf, which langchain4j-ingest-sink uses as the document id. The usual file headers, such as CamelFileName, are set too.
Good to know: the files are never deleted or moved. A file is read again when its modification time or size changes, and langchain4j-ingest-sink then adds the new version next to the old one, unless it has an idempotentRepository, which skips it. The files already read are remembered in memory only, by default the last 1000, so after a restart every file is read again; set idempotentRepository to keep them. A file that is still being copied in is read only once it stops changing; for a slow copy, which can pause for more than a second, set readLockMinAge.
Configuration Options
The following table summarizes the configuration options available for the langchain4j-ingest-file-source Kamelet:
| Property | Name | Description | Type | Default | Example |
|---|---|---|---|---|---|
Directory | Required The path of the directory to read the documents from, on the machine where the integration runs. | string | /data/knowledge-base | ||
Charset | The character set to read text files with, such as | string | UTF-8 | ||
Delay | How long to wait between two looks at the directory, in milliseconds. When it is not set, the directory is looked at every 500 milliseconds; a longer delay, such as one minute, is usually enough for a knowledge base. | integer | 60000 | ||
Exclude | Ant-style patterns of the files to leave out, separated by commas, in the same form as | string | */draft- | ||
Idempotent Repository | A Camel | string | #bean:readFiles | ||
Include | Ant-style patterns of the files to read, separated by commas, matched on the path relative to | string | /.pdf,/.md | ||
Initial Delay | How long to wait after the integration starts before the first look at the directory, in milliseconds. When it is not set, 1000 (one second) is used. | integer | 30000 | ||
Max Messages Per Poll | The largest number of files to read in one look at the directory; the others are read in the next looks, one | integer | 100 | ||
Read Lock Min Age | How long ago a file must have last changed before it is read, in milliseconds, such as 60000 for a file copied in slowly. It replaces the usual check, which reads each file only once its size and modification time have not changed for one second, so that an older file is read at once, and a newer one once it is old enough. | integer | 60000 | ||
Recursive | If true, the files in the subdirectories are read too. | boolean | true |
Dependencies
At runtime, the langchain4j-ingest-file-source Kamelet relies upon the presence of the following dependencies:
-
camel:file
-
camel:kamelet
Camel JBang usage
Prerequisites
-
You’ve installed JBang.
-
You have executed the following command:
jbang app install camel@apache/camel Supposing you have a file named route.yaml with this content:
- route:
from:
uri: "kamelet:langchain4j-ingest-file-source"
parameters:
.
.
.
steps:
- to:
uri: "kamelet:log-sink" You can now run it directly through the following command
camel run route.yaml LangChain4j Ingest File Source Kamelet Description
When Files Are Read
The Kamelet looks at the directory one second after it starts, or initialDelay milliseconds, and then every delay milliseconds.
By default, it reads each file only once its size and modification time have not changed for one second, so that a file still being copied in is not read half-written. That check costs one second per file, also for a file that has not changed in months. With readLockMinAge, a file is read when it last changed at least that many milliseconds ago: an older file is read at once, and a newer one once it is old enough. Set it for a copy that can pause for more than a second, such as over a slow network share.
maxMessagesPerPoll limits how many files are read in one look at the directory; the others are read in the next looks. It spreads the first load of a large directory over time, such as for an embedding model with a rate limit.
Example
The following route loads the Markdown and PDF files of a directory into a vector store. It starts 30 seconds after the application, reads at most 100 files a minute, and reads a file only one minute after it last changed.
- route:
from:
uri: "kamelet:langchain4j-ingest-file-source"
parameters:
directory: "/data/knowledge-base"
include: "**/*.md,**/*.pdf"
initialDelay: 30000
delay: 60000
maxMessagesPerPoll: 100
readLockMinAge: 60000
steps:
- to:
uri: "kamelet:tika-extract-text-action"
- to:
uri: "kamelet:langchain4j-ingest-sink" The embedding store and the embedding model of langchain4j-ingest-sink are beans of the application.