Pulsar Sink
Provided by: "Apache Software Foundation"
Support Level for this Kamelet is: "Stable"
Sends each message to a topic in Apache Pulsar, named by its tenant, namespace and topic name.
Use it to publish events or data for other applications to read from Pulsar, such as orders or sensor readings. Use pulsar-source to read them.
Input: the body is the payload, as text or bytes. A Map fails: turn it into JSON text first, for example with json-serialize-action. The headers of the message are not sent as Pulsar properties; to set properties, put a Map in the CamelPulsarProducerMessageProperties header. The optional key header becomes the key of the message, as the key header that pulsar-source sets; the CamelPulsarProducerMessageKey header takes precedence over it.
Output: the message is unchanged.
Good to know: it connects with no TLS and no login, unless serviceUrl starts with pulsar+ssl:// and authenticationClass is set. Over TLS, the certificate of Pulsar must be signed by an authority that the Java trust store of the integration trusts, or by one in the PEM file of tlsTrustCertsFilePath, such as a private one, and it must be issued for the host name in serviceUrl. It connects when the integration starts, and the integration fails to start when Pulsar cannot be reached, unless lazyStart is true. Headers that start with CamelPulsar, except CamelPulsarProducerMessageKey and CamelPulsarProducerMessageProperties, are removed from the message first, so they cannot change what the Kamelet does.
Configuration Options
The following table summarizes the configuration options available for the pulsar-sink Kamelet:
| Property | Name | Description | Type | Default | Example |
|---|---|---|---|---|---|
Namespace Name | Required The namespace within the tenant that holds the topic. A new Pulsar has the namespace | string | default | ||
Service URL | Required The address of the Pulsar broker, as | string | pulsar://localhost:6650 | ||
Tenant Name | Required The Pulsar tenant that the namespace belongs to. A new Pulsar has the tenant | string | public | ||
Topic Name | Required The name of the topic within the namespace to send the messages to. | string | orders | ||
Topic Type | Required Whether the topic is Enum values: * persistent * non-persistent | string | |||
Authentication Class | The Java class of the Pulsar authentication plugin to log in with, such as | string | org.apache.pulsar.client.impl.auth.AuthenticationToken | ||
Authentication Params | The settings of the plugin in | string | |||
Enable Batching | If true, messages sent close together are grouped and sent to Pulsar in one batch. A batch goes to one receiver of a | boolean | true | ||
Batching Maximum Messages | The most messages in one batch. Only used when | integer | 1000 | ||
Batch Publish Delay | How long to wait for more messages to fill a batch before sending it, in microseconds. For example, 1000 is one millisecond. Only used when | integer | 1000 | ||
Block If Queue Full | What happens when | boolean | false | ||
Compression Type | How to compress the messages on the way to Pulsar. The readers decompress them by themselves. Enum values: * NONE * LZ4 * ZLIB * ZSTD * SNAPPY | string | NONE | ||
Initial Sequence ID | The sequence ID to count from: the first message gets this number plus one. With the default, -1, the first message gets 0. Pulsar uses sequence IDs to drop duplicates when deduplication is on. | integer | -1 | ||
Lazy Start | If true, the connection to Pulsar is made when the first message is sent, not when the integration starts. The integration then starts even when Pulsar cannot be reached, and the first message takes longer. | boolean | false | ||
Maximum Pending Messages | The most messages that can wait at the same time for Pulsar to confirm them. See | integer | 1000 | ||
Message Routing Mode | How messages are spread over the partitions of a partitioned topic: Enum values: * SinglePartition * RoundRobinPartition * CustomPartition | string | RoundRobinPartition | ||
Producer Name | The name of this sender in Pulsar, shown in the topic statistics and set on each message. When it is not set, Pulsar picks a unique name. | string | |||
Send Timeout | How long Pulsar may take to confirm a message before the message fails, in milliseconds. For example, 30000 is 30 seconds. | integer | 30000 | ||
TLS Trust Certificates File | The path of a PEM file with the certificates of the authorities to trust for a | string | /etc/pulsar/ca.pem |
Dependencies
At runtime, the pulsar-sink Kamelet relies upon the presence of the following dependencies:
-
camel:pulsar
-
camel:kamelet
Camel JBang usage
Prerequisites
-
You’ve installed JBang.
-
You have executed the following command:
jbang app install camel@apache/camel Supposing you have a file named route.yaml with this content:
- route:
from:
uri: "kamelet:timer-source"
parameters:
period: 10000
message: 'test'
steps:
- to:
uri: "kamelet:pulsar-sink" You can now run it directly through the following command
camel run route.yaml Pulsar Sink Kamelet Description
Apache Pulsar Integration
This Kamelet integrates with Apache Pulsar, a distributed messaging and streaming platform built for the cloud. Pulsar provides unified messaging and streaming with high scalability and low latency.
Multi-Tenancy
Pulsar supports native multi-tenancy with namespaces and topics organized within tenants, providing isolation and resource management for different applications and teams.
Geo-Replication
Built-in geo-replication capabilities enable data replication across multiple data centers for disaster recovery and global data distribution scenarios.
Persistent and Non-Persistent Messaging
Supports both persistent messaging (with data durability) and non-persistent messaging (for low-latency scenarios) based on application requirements.
Schema Registry Integration
Pulsar includes built-in schema registry support for schema evolution and validation. This Kamelet does not use it: it sends the body as bytes, so turn structured data into JSON text first.
Cloud-Native Architecture
Designed for cloud-native deployments with:
-
Horizontal scaling capabilities
-
Kubernetes integration
-
Multi-cloud support
-
Serverless computing integration
Message Key
The optional key header becomes the key of the message. Pulsar keeps the messages with the same key in order on one partition, and a KEY_SHARED subscription hands them to the same receiver. pulsar-source sets the key header, so the key is kept when one Kamelet feeds the other. The CamelPulsarProducerMessageKey header takes precedence over key.
A batch of messages goes to one receiver of a KEY_SHARED subscription as a whole, whatever the keys in it. Set batchingEnabled to false to have the keys spread over the receivers.
Usage example
- route:
from:
uri: "kamelet:timer-source"
parameters:
message: "order placed"
steps:
- setHeader:
name: key
constant: "customer-7"
- to:
uri: "kamelet:pulsar-sink"
parameters:
serviceUrl: "pulsar://localhost:6650"
topicType: "persistent"
tenant: "public"
namespaceName: "default"
topic: "orders"
batchingEnabled: false For TLS, use a pulsar+ssl:// address in serviceUrl. The Kamelet has no trust store option: the certificate of Pulsar is checked against the Java trust store of the integration, which javax.net.ssl.trustStore can point at.