IBM Cloud Object Storage Source
Provided by: "Apache Software Foundation"
Support Level for this Kamelet is: "Preview"
Reads the files (objects) in a bucket of IBM Cloud Object Storage, one message per file, and by default deletes each file once it has been read.
Use it to process files that another system drops into a bucket, such as uploaded documents, exported reports or log files. To upload files to a bucket, use ibm-cos-sink.
Output: the body is the content of the file, as a stream. The headers give the bucket name (CamelIBMCOSBucketName), the key, that is the path of the file in the bucket (CamelIBMCOSKey), the content type, the size and the ETag of the file.
Good to know: files are deleted from the bucket after they are read, unless deleteAfterRead is set to false; then the same files are read again on every poll. moveAfterRead moves them to another bucket instead. To read one file only, set fileName. A file whose processing fails is kept, and read again on the next poll. It logs in with an IBM Cloud API key.
Configuration Options
The following table summarizes the configuration options available for the ibm-cos-source Kamelet:
| Property | Name | Description | Type | Default | Example |
|---|---|---|---|---|---|
API Key | Required An IBM Cloud API key that may read (and delete) the files of the bucket. Create it in the IBM Cloud console under Manage > Access (IAM) > API keys, or use the | string | |||
Bucket Name | Required The name of the bucket to read the files from. | string | my-bucket | ||
Endpoint URL | Required The address of Cloud Object Storage for the region of the bucket, starting with | string | https://s3.us-south.cloud-object-storage.appdomain.cloud | ||
Service Instance ID | Required The CRN (Cloud Resource Name) of the Cloud Object Storage instance, shown in the details of the instance in the IBM Cloud console. It starts with | string | |||
Autocreate Bucket | If true, the bucket is created when it does not exist yet. | boolean | false | ||
Delay | How long to wait between two polls of the bucket, in milliseconds. For example, 60000 is one minute. | integer | 500 | 60000 | |
Auto-delete Objects | If true, each file is deleted from the bucket after it has been read. Set it to false to keep the files; they are then read again on every poll. | boolean | true | ||
Delimiter | The character that separates folders in the keys, usually | string | / | ||
Destination Bucket | The bucket to move the files to. Only used when | string | my-archive-bucket | ||
Destination Bucket Prefix | Text put before the name of each moved file in | string | processed/ | ||
Destination Bucket Suffix | Text put after the name of each moved file in | string | .done | ||
File Name | The key of one file to read, such as | string | inbox/orders.csv | ||
Include Body | If true, the body is the content of the file. If false, there is no body and only the headers describe the file. | boolean | true | ||
Include Folders | If true, folder markers (empty objects whose key ends with | boolean | true | ||
Max Messages Per Poll | The most files to read at each poll. Use 0 or a negative number for no limit. | integer | 10 | ||
Move Objects After Read | If true, each file is moved to | boolean | false | ||
Prefix | Only read the files whose key starts with this text, such as a folder followed by | string | inbox/ | ||
Region | The region of the bucket, such as | string | us-south |
Dependencies
At runtime, the ibm-cos-source Kamelet relies upon the presence of the following dependencies:
-
camel:ibm-cos
-
camel:kamelet
Camel JBang usage
Prerequisites
-
You’ve installed JBang.
-
You have executed the following command:
jbang app install camel@apache/camel Supposing you have a file named route.yaml with this content:
- route:
from:
uri: "kamelet:ibm-cos-source"
parameters:
.
.
.
steps:
- to:
uri: "kamelet:log-sink" You can now run it directly through the following command
camel run route.yaml IBM Cloud Object Storage Source Kamelet Description
Authentication methods
This Kamelet requires authentication with IBM Cloud using an API Key and Service Instance ID (CRN - Cloud Resource Name).
To obtain these credentials:
-
API Key - Create an IBM Cloud API Key through the IBM Cloud console (Manage > Access (IAM) > API keys).
-
Service Instance ID - This is the Cloud Resource Name (CRN) of your IBM COS instance, found in your IBM COS service instance details.
-
Endpoint URL - The IBM COS endpoint URL for your region (e.g., https://s3.us-south.cloud-object-storage.appdomain.cloud).
For more information, see the IBM Cloud Object Storage documentation
Usage examples
You can consume objects from an IBM COS bucket and delete them after processing:
- route:
from:
uri: "kamelet:ibm-cos-source"
parameters:
bucketName: "my-bucket"
apiKey: "{{ibm.api.key}}"
serviceInstanceId: "{{ibm.service.instance.id}}"
endpointUrl: "https://s3.us-south.cloud-object-storage.appdomain.cloud"
region: "us-south"
steps:
- to:
uri: "kamelet:log-sink" The deleteAfterRead property is true by default, ensuring each object is consumed only once.
Consuming objects without deletion
To consume objects without deleting them, set deleteAfterRead to false:
- route:
from:
uri: "kamelet:ibm-cos-source"
parameters:
bucketName: "my-bucket"
apiKey: "{{ibm.api.key}}"
serviceInstanceId: "{{ibm.service.instance.id}}"
endpointUrl: "https://s3.us-south.cloud-object-storage.appdomain.cloud"
region: "us-south"
deleteAfterRead: false
steps:
- to:
uri: "kamelet:log-sink" Moving objects after reading
Instead of deleting objects, you can move them to another bucket:
- route:
from:
uri: "kamelet:ibm-cos-source"
parameters:
bucketName: "my-bucket"
apiKey: "{{ibm.api.key}}"
serviceInstanceId: "{{ibm.service.instance.id}}"
endpointUrl: "https://s3.us-south.cloud-object-storage.appdomain.cloud"
region: "us-south"
deleteAfterRead: false
moveAfterRead: true
destinationBucket: "archive-bucket"
destinationBucketPrefix: "processed/"
steps:
- to:
uri: "kamelet:log-sink" Consuming with a prefix filter
You can filter objects using a prefix:
- route:
from:
uri: "kamelet:ibm-cos-source"
parameters:
bucketName: "my-bucket"
apiKey: "{{ibm.api.key}}"
serviceInstanceId: "{{ibm.service.instance.id}}"
endpointUrl: "https://s3.us-south.cloud-object-storage.appdomain.cloud"
region: "us-south"
prefix: "inbox/"
steps:
- to:
uri: "kamelet:log-sink" This consumes only objects with keys starting with inbox/.
Reading one file
To read one file of the bucket instead of all of them, set fileName to its key. With deleteAfterRead false, the same file is read again on every poll; with the default deleteAfterRead, it is read once and deleted, and each later poll logs a warning that the key does not exist, until a file with that key is uploaded again.
- route:
from:
uri: "kamelet:ibm-cos-source"
parameters:
bucketName: "my-bucket"
apiKey: "{{ibm.api.key}}"
serviceInstanceId: "{{ibm.service.instance.id}}"
endpointUrl: "https://s3.us-south.cloud-object-storage.appdomain.cloud"
fileName: "config/prices.csv"
deleteAfterRead: false
delay: 60000
steps:
- to:
uri: "kamelet:log-sink" Consuming only metadata
To consume only object metadata without the body, set includeBody to false:
- route:
from:
uri: "kamelet:ibm-cos-source"
parameters:
bucketName: "my-bucket"
apiKey: "{{ibm.api.key}}"
serviceInstanceId: "{{ibm.service.instance.id}}"
endpointUrl: "https://s3.us-south.cloud-object-storage.appdomain.cloud"
region: "us-south"
includeBody: false
steps:
- to:
uri: "kamelet:log-sink"