AI Observability
Since Camel 4.23
The camel-ai-observability module provides GenAI observability for Camel AI components following the OpenTelemetry semantic conventions for GenAI (stable subset aligned with Spring AI).
AI producers depend on the lightweight camel-ai-observability-api module, which records nothing by itself. GenAI observability is opt-in: add camel-ai-observability to the application, together with a tracing or metrics backend such as camel-opentelemetry2 or camel-micrometer, to enable span and metric emission. It is not part of camel-observability-services, so add it also when the application uses that module.
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-ai-observability</artifactId>
<version>x.x.x</version>
<!-- use the same version as your Camel core version -->
</dependency> When camel-opentelemetry2 and/or camel-micrometer is on the classpath, langchain4j, OpenAI, and Spring AI chat producers emit child spans and metrics per LLM call with attributes such as gen_ai.operation.name, gen_ai.system, gen_ai.request.model, gen_ai.usage.input_tokens, and gen_ai.usage.output_tokens.
OpenAI coverage includes chat-completion (including streaming), responses, embeddings, and moderation. Each maps to a gen_ai.operation.name span attribute (chat, embeddings, moderation; moderation has no token usage).
Global toggle: set camel.aiObservability.enabled=false (default is true when a tracing or metrics backend is present). Camel Main exposes the same setting via camel.aiObservability.enabled in application.properties or programmatically with main.configure().aiObservability().withEnabled(false).
Supported producers: langchain4j-chat, langchain4j-agent, langchain4j-embeddings, openai, and spring-ai-chat.
When using spring-ai-chat with only a pre-built chatClient (no chatModel on the endpoint), Camel attempts to read the backing Spring AI ChatModel via a private field on Spring AI’s internal ChatClientRequestSpec implementation. That field is not public API and may change between Spring AI releases. If extraction fails, gen_ai.system and gen_ai.request.model span attributes may report unknown. Configure chatModel explicitly on the endpoint URI or component when GenAI observability is enabled so model metadata is always populated.
New exchange headers for model identification on langchain4j components:
-
CamelLangChain4jChatRequestModel/CamelLangChain4jChatResponseModel -
CamelLangChain4jAgentRequestModel/CamelLangChain4jAgentResponseModel -
CamelLangChain4jEmbeddingsRequestModel/CamelLangChain4jEmbeddingsResponseModel
Metrics recorded (when Micrometer is available):
-
gen_ai.client.operation— operation duration timer -
gen_ai.client.token.usage— token usage counter (tags:gen_ai.token.type=input|output)
Observability levels
By default the AI producers record a small, stable set of data. The camel.aiObservability.level option records more, for backends built around LLM traces such as Langfuse. Each level includes the previous one.
| Level | Recorded |
|---|---|
| One span per LLM call with |
| The structure of agent runs: an |
| Message content: |
camel.aiObservability.level = detailed
# camel.aiObservability.contentMaxLength = 8192 Prompts, completions and tool results can hold personal or confidential data. Use full only when the tracing backend is allowed to store them.
Content and ids are never used as metric tags, and the token totals are the same at every level. At detailed the model calls of an agent run are counted and timed one by one in gen_ai.client.operation (instead of one generate_content operation for the run), and with a Micrometer ObservationRegistry the agent runs and tool calls are also Observations named camel.ai.agent and camel.ai.tool, which the Micrometer meter handler turns into timers.
The attributes added by detailed and full belong to the part of the OpenTelemetry GenAI semantic conventions that is still in development, and may change with them.
An agent that is not built on AbstractAgent (a custom Agent implementation) cannot be observed call by call: at detailed its run is recorded as one generate_content model call, as at basic.
Calls already traced by the AI framework
Some AI frameworks trace their own model calls: Spring AI does when its models are built with an ObservationRegistry that has a Micrometer Tracing handler (as Spring Boot builds them with a tracing bridge; a registry with the meter handler only records metrics), quarkus-langchain4j does when quarkus-opentelemetry is present, and a LangChain4j model does when it has the langchain4j-observation listener. When Camel uses such a model, it does not record its own span and metrics for the calls the framework records, so that each call is counted once:
-
spring-ai-chat: the model calls are left to Spring AI, and so are the Camel route tools when the tool calling manager of the model traces them (Spring AI gives some models a default manager without a registry). -
langchain4j-chatandlangchain4j-agent: the model calls are left to the framework. At leveldetailedand above, the agent run and the Camel route tools are still recorded by Camel. The framework’s spans are children of the span that is current when the endpoint is called, which is the route span: they appear next to the agent run, not under it, because Camel makes its spans current only while a processor runs. -
openai: theopenai-javainstrumentation of OpenTelemetry (in the OpenTelemetry Java agent) wraps the OpenAI client used bycamel-openaiand traces its chat completions and embeddings. When the client is wrapped, those calls are left to it, except at levelfull: the instrumentation records the messages as log events, not as span attributes, so Camel keeps its own span and logs a warning. Start the JVM with-Dotel.instrumentation.openai-java.enabled=falseto keep only the Camel spans.
Set camel.aiObservability.deferToFramework=false to record these calls with Camel as well.
Micrometer Observation
When a non-NOOP io.micrometer.observation.ObservationRegistry is in the Camel registry (for example a Spring Boot Actuator bean), each GenAI client call is recorded as a Micrometer io.micrometer.observation.Observation named gen_ai.client.operation, unless a camel-telemetry tracer other than camel-micrometer-observability is in use (for example camel-opentelemetry2): then the GenAI spans are created through that tracer, in the same tree as the route spans. At level detailed, agent runs and tool calls are Observations named camel.ai.agent and camel.ai.tool, so they stay out of the gen_ai.client.operation timer. The contextual name uses the same {operation} {model} form as GenAI telemetry spans (for example chat test-model).
That is not the same type as Camel’s org.apache.camel.component.ai.observability.GenAiObservation handle used by AI producers.
camel-micrometer-observability is not required. Bind the ObservationRegistry in the Camel registry; Spring Boot does this automatically when observation support is enabled.
When Observation is used:
-
Camel does not create the GenAI CLIENT span through
camel-telemetry/camel-opentelemetry2. -
Camel does not record the direct
gen_ai.client.operationtimer. -
Tracing is emitted only if the registry has a tracing handler. Without one, the Observation path produces no GenAI trace.
-
The operation timer is emitted only if the registry has a meter handler. Without one, the Observation path produces no operation timer.
-
Token usage counters (
gen_ai.client.token.usage) still useMeterRegistrywhen one is present.
When no ObservationRegistry is available, existing OpenTelemetry spans and Micrometer timers continue to work as before.
Low-cardinality keys: gen_ai.operation.name, gen_ai.system, gen_ai.request.model, camel.component, and error.type on failure. Token usage, the response model and the finish reason are high-cardinality key values, as in Spring AI: they reach the trace but never the metric tags. The data added by the detailed and full levels is also recorded as high-cardinality key values.
Sending AI traces to an OTLP backend
Any backend that accepts OTLP and understands the OpenTelemetry GenAI semantic conventions can show these spans as LLM calls, agent runs and tool calls. The example below sends them from Camel Main to Langfuse, with camel-opentelemetry2, camel-ai-observability, and the OpenTelemetry SDK autoconfiguration and OTLP exporter on the classpath. Import io.opentelemetry:opentelemetry-bom so that the OpenTelemetry API and SDK versions match.
camel.opentelemetry2.enabled = true
# processors such as setBody or log as spans only add noise to LLM traces
camel.opentelemetry2.disableCoreProcessors = true
camel.aiObservability.level = full export OTEL_SERVICE_NAME=support-assistant
# Langfuse accepts OTLP over HTTP only
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_ENDPOINT=https://cloud.langfuse.com/api/public/otel
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic $(printf 'pk-lf-...:sk-lf-...' | base64),x-langfuse-ingestion-version=4"
export OTEL_METRICS_EXPORTER=none
export OTEL_LOGS_EXPORTER=none
java -Dotel.java.global-autoconfigure.enabled=true -jar my-app.jar Points to check on the runtimes:
-
Spring Boot and Quarkus sample 10% of the traces by default. Set
management.tracing.sampling.probability=1.0orquarkus.otel.traces.sampler=parentbased_always_on. -
Spring AI and quarkus-langchain4j trace their own model calls; Camel leaves those calls to them (see above).
-
With the OpenTelemetry Java agent (as used with
camel-observability-services) and levelfull, set-Dotel.instrumentation.openai-java.enabled=falseso that thecamel-openaicalls are recorded once, with their messages. -
Session and user ids set as OpenTelemetry baggage (for example through
CamelBaggage_exchange properties) are sent to every downstream service that propagates baggage, including the LLM provider when its HTTP client is instrumented.
Camel TUI integration
When monitoring a running integration with camel tui and observability enabled, the AI panel usage view (Ctrl+U while the AI panel is open) combines:
-
TUI ask token usage from
camel ask/ the embedded AI prompt -
Route GenAI usage extracted from exported OpenTelemetry spans with
gen_ai.*attributes
This gives a single dashboard for developer CLI usage and production route LLM calls.