Google Pubsub Source
Provided by: "Apache Software Foundation"
Support Level for this Kamelet is: "Stable"
Receives the messages of a Google Cloud Pub/Sub subscription, one message per Pub/Sub message, and acknowledges each one once it has been processed.
Use it to react to events that other applications publish to a Pub/Sub topic, such as orders, uploads or alerts. To publish messages, use google-pubsub-sink. To react to new files in a Cloud Storage bucket through Pub/Sub notifications, use google-storage-event-based-source.
Output: the body is the data of the Pub/Sub message, as bytes. The CamelGooglePubsubMessageId and CamelGooglePubsubPublishTime headers give the ID and the publish time of the message, CamelGooglePubsubAttributes gives its attributes as a Map, and each attribute is also set as a header of its own.
Good to know: the subscription must already exist; for one that does not, it logs an error and receives nothing. A message whose processing fails is not acknowledged, so Pub/Sub delivers it again. With ackMode NONE, the route acknowledges each message itself, through the CamelGooglePubsubAcknowledge header. On a subscription with a dead-letter topic, maxDeliveryAttempts stops a message that keeps failing from being processed again. Without serviceAccountKey, it logs in with the Application Default Credentials of the environment where it runs.
Configuration Options
The following table summarizes the configuration options available for the google-pubsub-source Kamelet:
| Property | Name | Description | Type | Default | Example |
|---|---|---|---|---|---|
Project ID | Required The ID of the Google Cloud project that holds the subscription, as shown in the Google Cloud console. It is the project ID, not the project name or number. | string | my-project-123 | ||
Subscription Name | Required The name of the subscription to read from, without the | string | orders-sub | ||
Acknowledge Mode | When a message is acknowledged. Enum values: * AUTO * NONE | string | |||
Concurrent Consumers | How many messages to receive and process in parallel, each in its own stream from the subscription. | integer | 1 | ||
Max Ack Extension Period | How long, in seconds, a message that has not been acknowledged yet is kept, by extending its acknowledgement deadline. After that time, Pub/Sub delivers it again. Only used when | integer | 600 | ||
Max Delivery Attempts | A message that Pub/Sub has delivered this many times is given back (nack) without being processed. Pub/Sub delivers it again until the maximum of delivery attempts of the subscription is reached, and then moves it to the dead-letter topic. Pub/Sub only counts the deliveries on a subscription with a dead-letter topic; on another subscription, this option does nothing. When it is not set, the maximum of the dead-letter policy of the subscription is read when the Kamelet starts, which needs the permission | integer | 5 | ||
Max Messages Per Poll | The most messages to ask Pub/Sub for at once. Only used when | integer | 1 | ||
Service Account Key | The JSON key of a service account that may read the subscription, encoded in Base64: create the key in the Google Cloud console under IAM & Admin > Service Accounts > Keys, then encode the downloaded file, for example with | binary | |||
Synchronous Pull | If true, it asks Pub/Sub for messages in batches of up to | boolean | false |
Dependencies
At runtime, the google-pubsub-source Kamelet relies upon the presence of the following dependencies:
-
camel:kamelet
-
camel:google-pubsub
-
camel:jackson3
Camel JBang usage
Prerequisites
-
You’ve installed JBang.
-
You have executed the following command:
jbang app install camel@apache/camel Supposing you have a file named route.yaml with this content:
- route:
from:
uri: "kamelet:google-pubsub-source"
parameters:
.
.
.
steps:
- to:
uri: "kamelet:log-sink" You can now run it directly through the following command
camel run route.yaml Google Pubsub Source Kamelet Description
Authentication methods
This Kamelet uses Service Account Key authentication to connect to Google Cloud Pub/Sub. You need to provide:
-
A service account key encoded in base64 format
-
The Google Cloud Project ID
-
The subscription name from which to consume messages
Output format
The Kamelet consumes messages from Google Cloud Pub/Sub. The body is the data of each message, as bytes, and the attributes of the message are set as headers.
Configuration
The Kamelet requires the following parameters:
-
projectId: The Google Cloud Pub/Sub Project ID -
subscriptionName: The subscription name
Optional parameters: - serviceAccountKey: The service account key to use as credentials (must be base64 encoded) - synchronousPull: Specifies to synchronously pull batches of messages (default: false) - maxMessagesPerPoll: The maximum number of messages to receive from the server in a single API call (default: 1) - concurrentConsumers: The number of parallel streams to consume from the subscription (default: 1) - ackMode: AUTO (when not set) or NONE, see below - maxDeliveryAttempts: How many deliveries of a message to process at most, on a subscription with a dead-letter topic - maxAckExtensionPeriod: How long, in seconds, a message that is not acknowledged yet is kept (when not set: 3600)
Usage example
apiVersion: camel.apache.org/v1alpha1
kind: KameletBinding
metadata:
name: google-pubsub-source-binding
spec:
source:
ref:
kind: Kamelet
apiVersion: camel.apache.org/v1alpha1
name: google-pubsub-source
properties:
projectId: "my-gcp-project"
subscriptionName: "my-subscription"
serviceAccountKey: "{{service-account-key-base64}}"
synchronousPull: true
maxMessagesPerPoll: 10
concurrentConsumers: 2
sink:
ref:
kind: Service
apiVersion: v1
name: my-service Acknowledging Messages
With ackMode not set, or AUTO, a message is acknowledged once it has been processed. A message whose processing fails is given back to Pub/Sub, which delivers it again.
With ackMode: NONE, the Kamelet does not acknowledge the messages. The route does it, with the object in the CamelGooglePubsubAcknowledge header, for example once the message has been stored:
- route:
from:
uri: "kamelet:google-pubsub-source"
parameters:
projectId: "my-project-123"
subscriptionName: "orders-sub"
ackMode: NONE
synchronousPull: true
steps:
- to:
uri: "kamelet:log-sink"
- script:
groovy: "request.headers.CamelGooglePubsubAcknowledge.ack(exchange)" Call nack instead of ack to have Pub/Sub deliver the message again right away. A message that is neither acknowledged nor given back is delivered again: with synchronousPull, once the acknowledgement deadline of the subscription has passed; otherwise once maxAckExtensionPeriod has passed, which is one hour when it is not set.
Dead-Letter Topic
On a subscription with a dead-letter topic, Pub/Sub counts the deliveries of each message, and moves a message to the dead-letter topic once it has been delivered as many times as the dead-letter policy allows. The count is in the CamelGooglePubsubDeliveryAttempt header.
A message that has been delivered maxDeliveryAttempts times is given back without being processed, until Pub/Sub moves it. When maxDeliveryAttempts is not set, the Kamelet reads the maximum of the dead-letter policy when it starts, so the last delivery is not processed. That needs the permission pubsub.subscriptions.get; when the policy cannot be read, a warning is logged, and every delivery is processed. With maxDeliveryAttempts: 0, every delivery is processed.
- route:
from:
uri: "kamelet:google-pubsub-source"
parameters:
projectId: "my-project-123"
subscriptionName: "orders-sub"
maxDeliveryAttempts: 3
steps:
- to:
uri: "kamelet:log-sink"