MongoDB Changes Stream Source
Provided by: "Apache Software Foundation"
Support Level for this Kamelet is: "Stable"
Sends a message for each change to a collection of a MongoDB database, such as a document inserted, updated or deleted, as it happens.
Use it to react to changes in MongoDB, for example to copy new orders to another system or to keep a cache or a search index up to date. To read the documents of a capped collection from the start, use mongodb-source. To write documents, use mongodb-sink.
Output: the body is JSON text. For an insert or a replace, it is the whole new document. For an update, it is the whole document as it is when the change is sent, read again from the collection, so it can already hold later changes, and it is the text null when the document was deleted meanwhile; fullDocument changes this. For a delete, it is an object with only the _id of the deleted document. The CamelMongoDbStreamOperationType header says what happened (insert, update, replace or delete), and the _id header holds the _id of the document.
Good to know: only changes made after the Kamelet starts are sent. It does not save where it stopped, so the changes made while it is stopped, such as during a restart, are never sent. The collection must exist when the Kamelet starts; otherwise it fails to start with Collection … does not exist on the database. Reading the document again costs MongoDB one more read for each update; set fullDocument to default to send null for an update instead. MongoDB offers change streams only on a replica set or a sharded cluster, not on a single standalone server. Set either hosts or connectionString. With hosts, it connects with TLS by default; set ssl to false for a MongoDB without TLS, such as a local one. With connectionString, such as the mongodb+srv:// string of MongoDB Atlas, it connects only as that string says, and hosts, username, password, ssl and sslValidationEnabled are not used. For a user defined in a database other than admin, use connectionString with authSource.
Configuration Options
The following table summarizes the configuration options available for the mongodb-changes-stream-source Kamelet:
| Property | Name | Description | Type | Default | Example |
|---|---|---|---|---|---|
MongoDB Collection | Required The name of the collection to watch for changes. | string | orders | ||
MongoDB Database | Required The name of the database that holds the collection. | string | shop | ||
Connection String | The MongoDB connection string, which starts with | string | mongodb+srv://myuser:mypassword@cluster0.abcde.mongodb.net/?retryWrites=true&w=majority | ||
Document of an Update | What the body is for an update; inserts, replaces and deletes are sent the same with any value. Enum values: * updateLookup * default * whenAvailable * required | string | updateLookup | ||
MongoDB Hosts | The address of one or more members of the MongoDB replica set or sharded cluster, as | string | mongo1.example.com:27017,mongo2.example.com:27017 | ||
MongoDB Password | The password of the MongoDB user | string | |||
Use TLS | Whether to connect with TLS (SSL). Set it to false for a MongoDB that does not offer TLS. Not used with | boolean | true | ||
Check TLS Certificate | Whether to check the certificate and the host name of the server when | boolean | true | ||
Stream Filter | A | string | { '$match':{'$or':[{'fullDocument.stringValue': 'specificValue'}]} } | ||
MongoDB Username | The MongoDB user to log in as, defined in the | string |
Dependencies
At runtime, the mongodb-changes-stream-source Kamelet relies upon the presence of the following dependencies:
-
camel:kamelet
-
camel:mongodb
-
camel:jackson3
Camel JBang usage
Prerequisites
-
You’ve installed JBang.
-
You have executed the following command:
jbang app install camel@apache/camel Supposing you have a file named route.yaml with this content:
- route:
from:
uri: "kamelet:mongodb-changes-stream-source"
parameters:
.
.
.
steps:
- to:
uri: "kamelet:log-sink" You can now run it directly through the following command
camel run route.yaml Mongodb Changes Stream Source Kamelet Description
Authentication methods
This Kamelet connects to MongoDB with:
-
hosts, with username and password authentication (optional) for a user defined in theadmindatabase, and TLS on by default (ssl); setsslto false for a MongoDB without TLS -
or a MongoDB connection string (
connectionString), such as themongodb+srv://string of MongoDB Atlas; see below
Output format
The Kamelet watches a collection with a MongoDB change stream, and sends one message for each change made after it starts, as JSON text: the new document for an insert or a replace, the whole document for an update (see below), and an object with the _id for a delete. The CamelMongoDbStreamOperationType header says which kind of change it is.
Change streams need a replica set or a sharded cluster.
Restarts
The Kamelet does not save where it stopped. Each time it starts, it sends only the changes made from then on, so the changes made while it is stopped, such as during a restart or an upgrade, are never sent. A system that must not miss a change has to catch up from the collection itself after a restart.
The collection must exist when the Kamelet starts. Otherwise the Kamelet fails to start with Collection … does not exist on the database and createCollection is false.
The document of an update
MongoDB records an update as the fields that changed, not as the document. The fullDocument option says what the Kamelet sends as the body for an update:
fullDocument | Body for an update |
|---|---|
| The whole document, read again from the collection when the change is sent. It can already hold later changes: after three quick updates of |
|
|
| The document as it was right after this update, on a collection with change stream post-images turned on (see below); |
| As |
Inserts, replaces and deletes are sent the same with any value. A streamFilter on a field under fullDocument matches an update only when the body of the update is a document, so never with default.
With updateLookup, an update whose document and changed fields are together larger than about 16 MB cannot be sent (MongoDB fails with BSONObjectTooLarge): the Kamelet retries it every second and sends no later change. Use default for a collection with such documents.
To send each update with the document as it was right after that update, turn on post-images for the collection, as a user allowed to run collMod:
db.runCommand({ collMod: "orders", changeStreamPreAndPostImages: { enabled: true } }) and set fullDocument to whenAvailable:
- route:
from:
uri: "kamelet:mongodb-changes-stream-source"
parameters:
hosts: "mongo1.example.com:27017,mongo2.example.com:27017"
database: "shop"
collection: "orders"
username: "mongouser"
password: "mongopass"
fullDocument: "whenAvailable"
steps:
- to:
uri: "kamelet:log-sink" With post-images turned on, MongoDB saves a copy of the document for each update in its config.system.preimages collection, which takes disk space until MongoDB removes it.
Configuration
The Kamelet requires database connection parameters:
-
hosts(the members of the replica set, ashost:port) orconnectionString -
databaseandcollection: the collection to watch -
streamFilter(optional): a$matchstage that chooses the changes to send -
fullDocument(optional): what the body is for an update; by default the whole document
Usage example
The route below logs each document inserted into the orders collection.
- route:
from:
uri: "kamelet:mongodb-changes-stream-source"
parameters:
hosts: "mongo1.example.com:27017,mongo2.example.com:27017"
database: "shop"
collection: "orders"
username: "mongouser"
password: "mongopass"
streamFilter: "{ '$match': { 'operationType': 'insert' } }"
steps:
- to:
uri: "kamelet:log-sink" Connecting with a connection string
Set either hosts or connectionString. Use connectionString to connect to MongoDB Atlas with its mongodb+srv:// string, to log in as a user defined in a database other than admin (with authSource), or to give other settings of the MongoDB connection string, such as replicaSet or tls. When it is set, hosts, username, password, ssl and sslValidationEnabled are not used, and TLS is used as the string says: always with mongodb+srv://, and with mongodb:// only when it has tls=true.
Write @, /, : and % in the user name or the password as %40, %2F, %3A and %25. The connection string holds the password, so keep it in a secret, like the password option.
With MongoDB Atlas, copy the string from the Connect dialog of the cluster:
- route:
from:
uri: "kamelet:mongodb-changes-stream-source"
parameters:
connectionString: "mongodb+srv://myuser:mypassword@cluster0.abcde.mongodb.net/?retryWrites=true&w=majority"
database: "shop"
collection: "orders"
steps:
- to:
uri: "kamelet:log-sink" For a user defined in the shop database, on a MongoDB without TLS:
connectionString: "mongodb://shopuser:shoppass@mongo1.example.com:27017,mongo2.example.com:27017/?authSource=shop&replicaSet=rs0"