OpenTelemetry Integration

Root Signals provides comprehensive OpenTelemetry (OTel) integration. allowing you to export detailed observability data from judge executions and evaluations to your preferred observability platform.

Overview

With Root Signals' OTel integration, you can:

  • Export traces from judge and evaluation executions to any OTel-compatible backend

  • Monitor evaluation performance with detailed timing and scoring metrics

  • Correlate RS evaluations with your existing application traces

  • View detailed execution flows in your observability platform of choice

All judge runs and individual evaluator executions are automatically instrumented and exported as OTel spans, providing complete visibility into your AI evaluation pipeline.

Configuration

Organization-Level Setup

OTel configuration is managed at the organization level in Root Signals. Each organization can configure their own OTel endpoint and settings.

Required Configuration:

  • Endpoint: Your OTel traces endpoint (e.g., https://tempo.example.com:443/v1/traces)

  • Headers: Authentication headers if required by your OTel backend

  • Enabled: Toggle to enable/disable OTel export

Optional Configuration:

  • Content Capture: Include user inputs and LLM responses in spans (disabled by default for privacy)

  • Sampling Rate: Control the percentage of traces exported (default: 100%)

Popular Backend Examples

Grafana Tempo

Endpoint: https://your-tempo-instance.com/v1/traces
Headers: 
  Authorization: Bearer your-api-key

Jaeger

Endpoint: http://jaeger-collector:14268/api/traces
Headers: {}

Exported Data

Trace Structure

Each judge execution creates a parent span with child spans for individual evaluations:

└── judge.execution.JudgeName 
    ├── judge.evaluation.EvaluatorName 
    └── judge.evaluation.EvaluatorName
    └── ...

Span Attributes

Judge Span Attributes:

  • judge.name: Name of the judge being executed

  • judge.evaluator_count: Number of evaluators in the judge

  • judge.duration_seconds: Total execution time

  • judge.user_input: User input being evaluated (if content capture enabled)

  • judge.llm_response: LLM response being evaluated (if content capture enabled)

Evaluation Span Attributes:

  • evaluation.name: Evaluator name (e.g., "Accuracy", "Clarity")

  • evaluation.score: Numerical score

  • evaluation.duration_ms: Evaluation execution time in milliseconds

  • evaluation.evaluator_id: Unique identifier for the evaluator instance

  • evaluation.justification: Detailed reasoning for the score (if content capture enabled)

  • evaluation.llm_response: The response being evaluated (if content capture enabled)

Service Identification:

  • service.name: "root-signals-evaluation"

  • service.version: Root Signals API version

  • rs.component: Component type ("judge" or "evaluator")

Viewing Traces

In Root Signals

All traces are visible in the Root Signals web interface with detailed breakdowns, scores, and justifications.

In your OTel Backend

Example: Grafana/Tempo

Getting Started

  1. Configure your OTel backend (Tempo, Jaeger, etc.)

  2. Set up Root Signals OTel configuration in your organization settings

  3. Run a judge to generate your first traces

  4. View traces in both Root Signals and your OTel platform

  5. Set up dashboards and alerts based on your specific needs

Your evaluation traces will automatically appear in both the Root Signals interface and your configured OTel backend, providing comprehensive observability into your AI evaluation pipeline.

PreviousLangfuseNextVertex AI Agent Builder

Last updated