MongoDB Source
Provided by: "Apache Software Foundation"
Support Level for this Kamelet is: "Stable"
Reads the documents of a capped collection in a MongoDB database, first those already there and then each new one as it is added, one message per document, as JSON text.
Use it to pass on records that an application appends to a capped collection, such as log entries or events, as they arrive. For a normal collection, or to also see updates and deletes, use mongodb-changes-stream-source. To write documents, use mongodb-sink.
Output: the body is one document as JSON text. An _id that is an ObjectId comes out as an object with only its time (timestamp and date), not as the ID text, and the format of dates depends on the Camel version. The CamelMongoDbDatabase and CamelMongoDbCollection headers name where the document comes from.
Good to know: the collection must be capped (created with a fixed size), or the Kamelet fails to start. Each time the integration starts, it reads the whole collection again from the start, unless persistentTailTracking is true. It stores its place only when the integration stops, so after a crash it reads again the documents since the last stop. Set either hosts or connectionString. With hosts, it connects with TLS by default; set ssl to false for a MongoDB without TLS, such as a local one. With connectionString, such as the mongodb+srv:// string of MongoDB Atlas, it connects only as that string says, and hosts, username, password, ssl and sslValidationEnabled are not used. For a user defined in a database other than admin, use connectionString with authSource. To read from a secondary member of a replica set, use connectionString with readPreference=secondaryPreferred.
Configuration Options
The following table summarizes the configuration options available for the mongodb-source Kamelet:
| Property | Name | Description | Type | Default | Example |
|---|---|---|---|---|---|
MongoDB Collection | Required The name of the capped collection to read. | string | events | ||
MongoDB Database | Required The name of the database that holds the collection. | string | shop | ||
Connection String | The MongoDB connection string, which starts with | string | mongodb+srv://myuser:mypassword@cluster0.abcde.mongodb.net/?retryWrites=true&w=majority | ||
MongoDB Hosts | The address of one or more MongoDB servers, as | string | mongo1.example.com:27017,mongo2.example.com:27017 | ||
MongoDB Password | The password of the MongoDB user | string | |||
Tail Tracking Name | The name under which the place of this Kamelet is stored when | string | events-to-kafka | ||
Remember Last Document | If true, the Kamelet stores the | boolean | false | ||
Use TLS | Whether to connect with TLS (SSL). Set it to false for a MongoDB that does not offer TLS. Not used with | boolean | true | ||
Check TLS Certificate | Whether to check the certificate and the host name of the server when | boolean | true | ||
Tail Tracking Collection | The collection to store the place in when | string | trackers | ||
Tail Tracking Database | The database to store the place in when | string | camelstate | ||
Increasing Field | The name of a field whose value grows with each new document, such as a sequence number or a time, used to find the place to go on reading from. Required when | string | seq | ||
MongoDB Username | The MongoDB user to log in as, defined in the | string |
Dependencies
At runtime, the mongodb-source Kamelet relies upon the presence of the following dependencies:
-
camel:kamelet
-
camel:mongodb
-
camel:jackson3
Camel JBang usage
Prerequisites
-
You’ve installed JBang.
-
You have executed the following command:
jbang app install camel@apache/camel Supposing you have a file named route.yaml with this content:
- route:
from:
uri: "kamelet:mongodb-source"
parameters:
.
.
.
steps:
- to:
uri: "kamelet:log-sink" You can now run it directly through the following command
camel run route.yaml MongoDB Source Kamelet Description
Authentication
This Kamelet supports MongoDB authentication using username and password credentials. The connection can be configured with or without authentication depending on your MongoDB setup.
Configuration
The MongoDB Source Kamelet supports the following configurations:
-
Hosts: MongoDB server hosts and ports; set either this or the connection string
-
Connection String: a MongoDB connection string, such as the
mongodb+srv://string of MongoDB Atlas, used instead of the hosts, username, password and SSL settings -
Collection: MongoDB collection name to read; it must be a capped collection (required)
-
Database: MongoDB database name (required)
-
Username: Username for authentication (optional)
-
Password: Password for authentication (optional)
-
SSL: Whether to connect with TLS (default: true); set it to false for a MongoDB without TLS
-
Tail Track Increasing Field: A field whose value grows with each new document
-
Persistent Tail Tracking: Remember the last document read across restarts (needs the increasing field)
-
Tail Tracking Name (
persistentId): The name the place is stored under; give each source its own -
Tail Tracking Database and Collection (
tailTrackDb,tailTrackCollection): Where the place is stored
Output Format
The Kamelet reads the documents of a capped collection with a tailable cursor: first the documents already there, then each new one as it is added. Each document is sent as one message, as JSON text.
Usage Example
- route:
from:
uri: "kamelet:mongodb-source"
parameters:
hosts: "mongodb.example.com:27017"
database: "logs"
collection: "app_logs"
username: "mongouser"
password: "mongopass"
steps:
- to:
uri: "kamelet:log-sink" Example with Persistent Tail Tracking
The Kamelet stores the seq value of the last document read, and after a restart goes on from there. It stores it when the integration stops, so after a crash it reads again the documents since the last stop.
The place is stored in a document named after persistentId, in the camelTailTracking collection of the same database. Give each source its own persistentId: sources without one share a single place, and a restarted source then reads documents again or skips them. Set tailTrackDb and tailTrackCollection to store the place elsewhere, for example when the user may only read the database of the events:
- route:
from:
uri: "kamelet:mongodb-source"
parameters:
hosts: "mongo1.example.com:27017,mongo2.example.com:27017,mongo3.example.com:27017"
database: "production"
collection: "events"
username: "mongouser"
password: "mongopass"
tailTrackIncreasingField: "seq"
persistentTailTracking: true
persistentId: "events-to-kafka"
tailTrackDb: "camelstate"
tailTrackCollection: "trackers"
steps:
- to:
uri: "kamelet:log-sink" The place is then the document { persistentId: "events-to-kafka", lastTrackingValue: 42 } in camelstate.trackers.
Capped Collections
The collection must be capped, or the Kamelet fails to start. A capped collection is created with a fixed size, for example in mongosh:
db.createCollection("events", { capped: true, size: 100000 }) Connecting with a connection string
Set either hosts or connectionString. Use connectionString to connect to MongoDB Atlas with its mongodb+srv:// string, to log in as a user defined in a database other than admin (with authSource), or to give other settings of the MongoDB connection string, such as replicaSet or tls. When it is set, hosts, username, password, ssl and sslValidationEnabled are not used, and TLS is used as the string says: always with mongodb+srv://, and with mongodb:// only when it has tls=true.
Write @, /, : and % in the user name or the password as %40, %2F, %3A and %25. The connection string holds the password, so keep it in a secret, like the password option.
With MongoDB Atlas, copy the string from the Connect dialog of the cluster:
- route:
from:
uri: "kamelet:mongodb-source"
parameters:
connectionString: "mongodb+srv://myuser:mypassword@cluster0.abcde.mongodb.net/?retryWrites=true&w=majority"
database: "logs"
collection: "app_logs"
steps:
- to:
uri: "kamelet:log-sink" For a user defined in the shop database, on a MongoDB without TLS:
connectionString: "mongodb://shopuser:shoppass@mongo1.example.com:27017,mongo2.example.com:27017/?authSource=shop&replicaSet=rs0" To read from a secondary member of a replica set, and so take load off the primary, add readPreference:
connectionString: "mongodb://mongouser:mongopass@mongo1.example.com:27017,mongo2.example.com:27017,mongo3.example.com:27017/?replicaSet=rs0&readPreference=secondaryPreferred"