Google Storage Source
Provided by: "Apache Software Foundation"
Support Level for this Kamelet is: "Stable"
Reads the files (objects) in a Google Cloud Storage bucket, one message per file, and by default deletes each file once it has been read.
Use it to process files that another system drops into a bucket, such as uploaded documents, exported reports or log files. To react to new files through Pub/Sub notifications instead of reading the bucket over and over, use google-storage-event-based-source.
Output: the body is the content of the file, as a stream. The headers give the bucket name (CamelGoogleCloudStorageBucketName), the object name, that is the path of the file in the bucket (CamelGoogleCloudStorageObjectName), the content type and the ETag of the file.
Good to know: files are deleted from the bucket after they are read, unless deleteAfterRead is set to false; then the same files are read again on every poll, about twice a second, or every delay milliseconds. With moveAfterRead, they are copied to destinationBucket first. A file whose processing fails is not deleted or moved, and is read again on the next poll. Without serviceAccountKey, it logs in with the Application Default Credentials of the environment where it runs, or with Workload Identity Federation when useWorkloadIdentityFederation is true.
Configuration Options
The following table summarizes the configuration options available for the google-storage-source Kamelet:
| Property | Name | Description | Type | Default | Example |
|---|---|---|---|---|---|
Bucket Name | Required The name of the bucket to read the files from, without | string | my-bucket | ||
Autocreate Bucket | If true, the bucket is created when it does not exist yet. | boolean | false | ||
Delay | How long to wait between two reads of the bucket, in milliseconds. For example, 60000 is one minute. When it is not set, it is 500. | integer | 60000 | ||
Auto-delete Objects | If true, each file is deleted from the bucket after it has been read. Set it to false to keep the files; they are then read again on every poll. | boolean | true | ||
Destination Bucket | The name of the bucket to move the files to, without | string | my-archive-bucket | ||
Filter | A regular expression that the whole name of a file must match for the file to be read, such as | string | .*\.csv | ||
Impersonated Service Account | The email of a service account to act as, with the credentials of Workload Identity Federation, so that the Kamelet gets the permissions of that service account. The federated identity needs the Service Account Token Creator role on it. Only used when | string | storage-reader@my-project-123.iam.gserviceaccount.com// | ||
Include Body | If true, the body is the content of the file. If false, the content is not downloaded, the body is empty, and only the headers describe the file, such as its size in | boolean | |||
Include Folders | If true, folder markers, that is empty objects whose name ends with | boolean | |||
Move Objects After Read | If true, each file is copied to | boolean | |||
Prefix | Only read the files whose name starts with this text, such as a folder followed by | string | inbox/ | ||
Service Account Key | The JSON key of a service account that may read (and delete) the files of the bucket, encoded in Base64: create the key in the Google Cloud console under IAM & Admin > Service Accounts > Keys, then encode the downloaded file, for example with | binary | |||
Use Workload Identity Federation | If true, it logs in with Workload Identity Federation instead of | boolean | |||
Workload Identity Configuration | Where to load the Workload Identity Federation credential configuration file from: the JSON file of type | string | file:/var/run/secrets/gcp/credential-config.json |
Dependencies
At runtime, the google-storage-source Kamelet relies upon the presence of the following dependencies:
-
camel:kamelet
-
camel:google-storage
-
camel:jackson3
Camel JBang usage
Prerequisites
-
You’ve installed JBang.
-
You have executed the following command:
jbang app install camel@apache/camel Supposing you have a file named route.yaml with this content:
- route:
from:
uri: "kamelet:google-storage-source"
parameters:
.
.
.
steps:
- to:
uri: "kamelet:log-sink" You can now run it directly through the following command
camel run route.yaml Google Storage Source Kamelet Description
Authentication methods
This Kamelet uses Service Account Key authentication to connect to Google Cloud Storage. You need to provide:
-
A service account key encoded in base64 format (optional, can use default credentials)
-
The name of the bucket from which to consume objects
Configuration
The Kamelet requires the following parameters:
-
bucketNameOrArn: The name of the Google Cloud Storage bucket
Optional parameters: - serviceAccountKey: The service account key to use as credentials (must be base64 encoded) - deleteAfterRead: Specifies to delete objects after consuming them (default: true) - autoCreateBucket: Specifies to automatically create the Google Cloud Storage bucket (default: false) - prefix: The prefix which is used in the BlobListOptions to only consume objects we are interested in - moveAfterRead and destinationBucket: Move each file to another bucket after it has been read - includeBody: Set it to false to read only the description of each file, not its content (when not set: true) - includeFolders: Set it to false to skip the folder markers, the empty objects whose name ends with / (when not set: true) - delay: How long to wait between two reads of the bucket, in milliseconds (when not set: 500) - useWorkloadIdentityFederation, workloadIdentityConfig and impersonatedServiceAccount: Log in with Workload Identity Federation, see below
Usage example
apiVersion: camel.apache.org/v1alpha1
kind: KameletBinding
metadata:
name: google-storage-source-binding
spec:
source:
ref:
kind: Kamelet
apiVersion: camel.apache.org/v1alpha1
name: google-storage-source
properties:
bucketNameOrArn: "my-storage-bucket"
serviceAccountKey: "{{service-account-key-base64}}"
deleteAfterRead: false
autoCreateBucket: false
prefix: "data/"
sink:
ref:
kind: Service
apiVersion: v1
name: my-service Moving Files to Another Bucket
With moveAfterRead, each file that has been processed is copied to destinationBucket, under the same name, and then deleted from the bucket it was read from. A file whose processing fails stays where it is. To read a large bucket only once a minute, and to move the files without downloading their content:
- route:
from:
uri: "kamelet:google-storage-source"
parameters:
bucketNameOrArn: "my-bucket"
moveAfterRead: true
destinationBucket: "my-archive-bucket"
includeBody: false
delay: 60000
steps:
- to:
uri: "kamelet:log-sink" With includeBody: false, the body is empty, and the headers describe the file, such as CamelGoogleCloudStorageObjectName and CamelGoogleCloudStorageContentLength. The destination bucket must exist: when it does not, a warning is logged, and the file stays in its bucket and is read again on every poll. With deleteAfterRead: false, the file is copied and kept, so it is read and copied again on every poll.
Workload Identity Federation
To run without a service account key, outside Google Cloud, the Kamelet can exchange a token of another identity provider, such as AWS, Azure or a Kubernetes service account, for Google credentials. Create the credential configuration file with gcloud iam workload-identity-pools create-cred-config, and point workloadIdentityConfig at it:
- route:
from:
uri: "kamelet:google-storage-source"
parameters:
bucketNameOrArn: "my-bucket"
useWorkloadIdentityFederation: true
workloadIdentityConfig: "file:/var/run/secrets/gcp/credential-config.json"
impersonatedServiceAccount: "storage-reader@my-project-123.iam.gserviceaccount.com"
steps:
- to:
uri: "kamelet:log-sink" With impersonatedServiceAccount, the Kamelet acts as that service account, which needs the Service Account Token Creator role granted to the federated identity. Without workloadIdentityConfig, the Application Default Credentials are used, which on GKE with Workload Identity need no setting at all.