LangChain4j Ingest Sink
Provided by: "Apache Software Foundation"
Support Level for this Kamelet is: "Preview"
Splits each document into small pieces of text, turns each piece into an embedding (a vector of numbers) with an AI embedding model, and writes them to a LangChain4j embedding store.
Use it to load documents into a vector store, so that an AI assistant can later find the pieces that answer a question (RAG). Read PDF or office documents with tika-extract-text-action or docling-convert-action first, and a directory with langchain4j-ingest-file-source.
Input: the document, as text, bytes, a stream or a file, read as text. Every document needs an id, such as its file name, in the CamelLangChain4jIngestDocumentId header (or the one named in documentIdHeader); an id kept by tika-extract-text-action or docling-convert-action comes first.
Output: when it is used inside a route, the body afterwards is a result object with the pipeline name, the document id, the number of pieces written, and the outcome: INGESTED, EMPTY (an empty document, nothing written), SKIPPED (an id seen before, with idempotentRepository) or FILTERED (left out by includeId, excludeId, minDocumentSize or documentFilter).
Good to know: the embedding store and the embedding model are objects you define in the application, such as in the beans section of a YAML route; when there is one of each, the options can be left out. Writing a document again adds new pieces, and does not replace the ones written before. With idempotentRepository, only the first document with a given id is written, so a changed version of a file is skipped. A message without a document id, or larger than maxDocumentSize, fails.
Configuration Options
The following table summarizes the configuration options available for the langchain4j-ingest-sink Kamelet:
| Property | Name | Description | Type | Default | Example |
|---|---|---|---|---|---|
Content Type | The kind of content of the documents, as a MIME type, handed to the embedding model. Only with | string | image/png | ||
Document Filter | A filter object defined in the application (a Camel | string | #bean:myFilter | ||
Document ID Header | The name of the header that holds the id of the document, such as its file name or path. Set it to the header your source fills, such as | string | CamelLangChain4jIngestDocumentId | CamelAwsS3Key | |
Document Splitter | A LangChain4j | string | #bean:mySplitter | ||
Embedding Batch Size | How many pieces are sent to the embedding model in one request. | integer | 32 | ||
Embedding Model | The LangChain4j | string | #bean:model | ||
Embedding Store | The LangChain4j | string | #bean:store | ||
Exclude ID | Patterns for the document ids to leave out, separated by commas, in the same form as | string | */draft- | ||
Idempotent Repository | A Camel | string | #bean:ingestedIds | ||
Include ID | Ant-style patterns that the document id must match to be written, separated by commas, such as | string | docs/*,.md | ||
Max Document Size | The largest document to write, in characters, or in bytes with | integer | 1000000 | ||
Max Overlap Size | How many characters at the end of a piece are repeated at the start of the next one, so that a sentence cut in two is still found. | integer | 50 | ||
Max Segment Size | The longest piece of text to embed, in characters. Smaller pieces give more precise search results; larger ones keep more context together. | integer | 500 | ||
Min Document Size | The smallest document to write, in characters, or in bytes with | integer | 100 | ||
Modality | What the documents are. Enum values: * text * media | string | |||
Pipeline Name | A name for this ingestion, written on every piece in the store, so that the pieces of several pipelines can be told apart. Letters, digits, dots, underscores and dashes only. | string | ingest | product-manuals |
Dependencies
At runtime, the langchain4j-ingest-sink Kamelet relies upon the presence of the following dependencies:
-
camel:langchain4j-ingest
-
camel:kamelet
Camel JBang usage
Prerequisites
-
You’ve installed JBang.
-
You have executed the following command:
jbang app install camel@apache/camel Supposing you have a file named route.yaml with this content:
- route:
from:
uri: "kamelet:timer-source"
parameters:
period: 10000
message: 'test'
steps:
- to:
uri: "kamelet:langchain4j-ingest-sink" You can now run it directly through the following command
camel run route.yaml