Ceph Event Based Source
Provided by: "Apache Software Foundation"
Support Level for this Kamelet is: "Preview"
Receives the bucket notifications that a Ceph Object Gateway sends to a Kafka topic, one message per event, such as a file added to or deleted from a bucket.
Use it to react as soon as a file is uploaded to or removed from a Ceph bucket, without reading the bucket over and over. Set getObject to true to receive the content of each new file. To read and remove the files of a bucket instead, use ceph-source.
Output: the body is the notification as JSON text, unchanged: a Records array that gives the event name (such as ObjectCreated:Put), the bucket name and the key of the file. The headers ceph-event-id, ceph-opaque-data, ceph-object-metadata and ceph-object-tags give the fields Ceph adds to the event; metadata and tags are a List of entries with a key and a val. Each header is also set with a ce- name, such as ce-cepheventid. With getObject true, the body of an ObjectCreated event is the content of the file instead, as a stream, with its content type in CamelAwsS3ContentType, and the other events come out as parsed JSON (a Jackson JsonNode), not as text.
Good to know: the notification and its Kafka topic are set up on the Ceph side; this Kamelet only reads the topic. A record that is not a Ceph notification (without a Records array) fails. saslAuthType has no default and must be set; NONE connects to Kafka with no login and no TLS. Without consumerGroup, each start uses a new group, so with the default autoOffsetReset of latest the events sent while it was stopped are not read.
Configuration Options
The following table summarizes the configuration options available for the ceph-event-based-source Kamelet:
| Property | Name | Description | Type | Default | Example |
|---|---|---|---|---|---|
Bootstrap Servers | Required The address of one or more Kafka brokers to connect to first, as | string | broker1:9092,broker2:9092 | ||
Authentication Type | Required How to log in to Kafka. It has no default and must be set. Enum values: * NONE * PLAIN * SCRAM_SHA_256 * SCRAM_SHA_512 * SSL * OAUTH * AWS_MSK_IAM * KERBEROS | string | |||
Topic Name | Required The Kafka topic that the Ceph Object Gateway sends the bucket notifications to, as set in the push endpoint of the Ceph topic. | string | ceph-bucket-notifications | ||
Access Key | The S3 access key of a Ceph Object Gateway user that can read the bucket. Required when | string | |||
Auto Offset Reset | Where to start when the consumer group has not read the topic before. Enum values: * latest * earliest * none | string | latest | ||
Ceph URL | The address of the Ceph Object Gateway (RGW) to read the files from, as a URL with the scheme and the port. Required when | string | http://ceph-rgw:8080 | ||
Consumer Group | The name of the Kafka consumer group. Kafka remembers how far the group has read, so a restart continues where it stopped. When it is not set, a new group is used at each start. | string | my-group-id | ||
Get Object | If true, for each | boolean | false | ||
Password | The password of | string | |||
Username | The user to log in to Kafka as. Required when | string | |||
Secret Key | The S3 secret key that goes with | string | |||
Ceph Zone Group | The name of the zone group of the Ceph Object Gateway, used as the region of the requests. Required when | string | zonegroup1 |
Dependencies
At runtime, the ceph-event-based-source Kamelet relies upon the presence of the following dependencies:
-
camel:kafka
-
camel:jsonpath
-
camel:jackson3
-
camel:kamelet
-
camel:aws2-s3
Camel JBang usage
Prerequisites
-
You’ve installed JBang.
-
You have executed the following command:
jbang app install camel@apache/camel Supposing you have a file named route.yaml with this content:
- route:
from:
uri: "kamelet:ceph-event-based-source"
parameters:
.
.
.
steps:
- to:
uri: "kamelet:log-sink" You can now run it directly through the following command
camel run route.yaml Ceph Event Based Source Kamelet Description
How Notifications Arrive
Ceph RGW pushes bucket notifications to an endpoint configured on the topic rather than being polled. This Kamelet consumes the Kafka flavour of that.
Configuring the notification and the topic on the Ceph side is done out of band, with the S3 and topic APIs, and is not something this Kamelet does.
Fetching the Object
Set getObject to true to fetch the object body from Ceph for ObjectCreated events, using the same S3 compatible path that ceph-source and ceph-sink use. With getObject left false, the notification JSON is emitted unchanged, as text. With getObject true, the other events are emitted as parsed JSON (a Jackson JsonNode) rather than as text.
When getObject is enabled, cephUrl, accessKey, secretKey and zoneGroup are required so the object can be read back from the RGW endpoint.
Ceph Extension Fields
The fields Ceph adds beyond the S3 notification specification are surfaced as headers, so a consumer does not have to re-parse the body for them. Each has a ce- prefixed CloudEvents counterpart as well.
-
ceph-event-idfromeventId- identifies the event, for spotting duplicates after a transport retransmit. RGW populates it as<timestamp>.<counter>.<etag>, unique per event. The event structure in the Ceph documentation shows it empty, but that is a placeholder in the sample rather than what RGW emits. -
ceph-opaque-datafromopaqueData- free-form information attached to the topic by the user. -
ceph-object-metadatafroms3.object.metadata- user attributes on the object, sent asx-amz-meta-headers. Ceph can also filter notifications on these. -
ceph-object-tagsfroms3.object.tags- object tags. Both this and the metadata arrive as a list of key/val entries rather than a flattened string.
s3.bucket.id is not surfaced. It is an internal Ceph identifier, useful for debugging but not for routing.
Expected Payload
The topic is expected to carry only Ceph notifications. A message without a Records array fails the exchange rather than being passed through, so that a foreign message is not mistaken for an event with empty Ceph headers.
A Ceph notification always carries the extension fields, empty when nothing is set, so the headers above are always populated. A payload missing them is not a Ceph event and fails the same way.