Traigent Telemetry Documentation

This document describes what telemetry data Traigent SDK collects, how it's used, retention policies, and how to opt-out.

Overview

Traigent SDK collects telemetry data to improve the optimization experience and help diagnose issues. In the open-source build, telemetry stays local unless you opt into managed services. Telemetry can be completely disabled.

What Data is Collected

Optimization Metrics

During optimization runs, Traigent collects:

Trial Lifecycle Events: - Trial suggested/intermediate/completed timestamps - Trial status (completed, failed, pruned) - Trial duration - Trial configuration. On the default portal-backed path, this includes tuned config-space key/value pairs being tested, with internal keys stripped. If you tune prompt variants or other free-text strings as configuration values, those tuned values are sent on the default path so the portal can display winning configs. - Trial metrics (accuracy, cost, latency, etc.)

Optimization Run Metadata: - Optimization run ID - Algorithm used (grid, random, bayesian, optuna) - Number of trials executed - Total optimization duration - Result source (cloud_brain, local_fallback, explicit_local, or offline) - Stop conditions triggered - Content-free tuned-variable observations can include knob names, enum/scalar values, numeric metrics, and aggregate effectuation events for backend optimization. Set TRAIGENT_TVAR_OBSERVATION=off to disable the tvar_observation_v1 metadata sub-field, or use TRAIGENT_TVAR_OBSERVATION=hashed (default) to hash free-form string values in that sub-field. Only off and hashed are supported; unsupported values fall back to hashed. This setting does not control the trial config field.

Performance Metrics: - LLM API response times - Token usage per trial - Cost per trial (when available) - Dataset size and evaluation metrics

What is NOT Collected

Outside tuned configuration values explicitly sent on the default portal-backed path, and outside observability content you explicitly opt into recording, Traigent does not collect:

• User prompts or inputs
• LLM responses or outputs
• Evaluation dataset contents
• Personal identifiable information (PII)
• API keys or credentials
• Source code or function implementations

Data Boundary and No-Egress Runs

The default portal-backed path sends tuned config-space values and numeric metrics. This is deliberate: the portal uses those values to display and compare winning configs. If a prompt variant, persona instruction, model name, or other string is part of the tuned configuration space, the default path sends that string value. It does not send dataset example inputs, expected outputs, model responses, or example metadata unless you put that content into the tuned configuration itself.

Privacy-mode redaction applies to per-trial submissions, not to session creation. The full tuned configuration space — including string values — is sent unconditionally when the session is created, before any redaction runs (see Data Boundary above). Enabling privacy mode does not withhold that payload. What it does change: subsequent per-trial submissions preserve which keys were tuned, but redact sensitive-key values and all string/free-text config values. Numeric, boolean, and None config values still pass through. This trial-level redaction is gated by the run's effective TraigentConfig.privacy_enabled setting, including configuration loaded from TRAIGENT_PRIVACY_MODE=true for compatibility. Do not use @traigent.optimize(..., privacy_enabled=True) for new code; that decorator keyword is deprecated and emits a warning. Use offline=True instead when the requirement is that tuned string values — including at session creation — never leave.

Use offline=True when your policy requires no Traigent backend egress at all:

@traigent.optimize(
    algorithm="grid",
    offline=True,
    ...
)

No-egress runs keep Traigent optimization metadata local while still allowing your own function to call LLM providers or other services.

offline=True also disables OpenTelemetry export, the langfuse client and the standalone API clients while the run is in flight, even if TRAIGENT_TRACE_ENABLED and OTEL_EXPORTER_OTLP_ENDPOINT are set. Outside an offline=True run, the observability lanes (@observe, TRAIGENT_TRACE_ENABLED) have their own content controls, described below, and carry no privacy guarantee.

Use TRAIGENT_DISABLE_TELEMETRY=true for SDK telemetry opt-out, effective TraigentConfig.privacy_enabled privacy mode to redact tuned string config values on per-trial submissions made after session creation, and offline=True for zero Traigent backend egress, including at session creation.

@observe Content Egress

The @observe decorator and ObserveContext are metadata-only by default. They can send trace names, observation names, status, timing, tags, environment, release, IDs, explicit metadata, and token/cost fields you pass, but they omit function arguments, explicit context input_data, return values, and output data unless you opt in.

To send prompt/output content for observability, opt in explicitly:

@observe("llm-call", content_mode="record")
def call_model(prompt: str) -> str:
    ...

Alternatively set TRAIGENT_OBSERVABILITY_CONTENT=record. The supported content modes are:

• metadata (default): omit input and output content
• redacted: send {"redacted": true} placeholders for input and output
• record: send captured input and output content

redact_input=True and redact_output=True force redaction placeholders for the corresponding side even when content_mode="record". The transport still applies pattern-based secret scrubbing before send, but that scrubber is a final safety net, not the content-egress policy.

Default vs. explicit, and what a later un-annotated call on the same trace does. The content_mode a trace resolves to at start_trace() (or the first @observe/ObserveContext call that creates it) becomes that trace's own base mode for its whole lifetime. A later record_observation()/end_trace()/ add_comment()/submit_feedback() call that passes no content_mode of its own continues that base — it does not fall back to the client's configured default. For example, client.start_trace("t", content_mode="redacted") followed by a plain client.record_observation(...) (no override) still emits the redacted placeholder, even if the client itself is unconfigured (client-default metadata). A per-call override may only tighten that base — never loosen it — and only when either the client's own policy was set explicitly (constructor option, TRAIGENT_OBSERVABILITY_CONTENT, or the legacy capture-content env var) or the trace's own mode was itself set explicitly at start_trace(). When neither is explicit, a per-call override is used as-is — the normal, documented way to opt one call into record/redacted capture. A trace_id this client process never locally tracked (created elsewhere, or already closed) resolves purely against the client-level policy, same as a bare start_trace().

Bound the promise: metadata mode withholds content fields (function arguments, return values, explicit input_data/output_data); arbitrary metadata you attach is sent after secret scrubbing and is NOT content-free. Anything you pass in metadata= — including a traigent_active_config/trial snapshot the SDK enriches your call with — is free-form data the caller controls, so it ships regardless of content_mode, minus pattern-based secret scrubbing and credential-shaped key redaction. Treat metadata as "no captured content", not "nothing leaves this process": if you need a value to never leave the process, do not put it in metadata.

client.add_comment() and client.submit_feedback() (trace comments and correction output) both accept an optional content_mode= keyword and otherwise obey the same resolution as @observe/record_observation (including the trace's own base mode described above): metadata mode refuses add_comment locally (there is no non-content payload for a comment to fall back to) and submit_feedback sends the rating/label fields while omitting comment/correction_output; redacted sends placeholders for both; record sends the text after the same mandatory secret scrubbing.

Dataset-conversion and content-upload endpoints (traigent.datasets.*) are a separate content-egress surface with their own contract; they are not gated by observability's content_mode and are out of scope here.

user_id is not anonymized. The SDK never generates, hashes, or maps user_id for you — whatever string you pass on start_trace(user_id=...) (or @observe(user_id=...)) is exactly what is attributed to that trace. Pattern-based scrubbing still redacts a user_id value that happens to look like an email address or another recognized secret/PII pattern in every content_mode, as a safety net — but that is a byproduct of the generic value scan, not identity protection. Pass an opaque identifier you control (an internal user/session ID, not an email address, name, or other directly-identifying value) if you need per-user attribution without exposing PII to telemetry.

@observe(observation_type=GENERATION) does not estimate LangChain or provider token usage from prompts. Pass measured input_tokens, output_tokens, total_tokens, or cost_usd from the provider response when recording generation spans; otherwise usage is reported as unknown, not zero.

Content-Free Execution Lineage

ExecutionContextDTO carries only versioned lineage identifiers. Client and environment defaults are merged into each trace; fields omitted from a per-trace DTO inherit those defaults. An explicit None clears a default and is emitted as JSON null:

from traigent.observability import ExecutionContextDTO

context = ExecutionContextDTO(toolset_id=None)
client.start_trace("agent-run", execution_context=context)

Dictionary inputs have the same explicit-null behavior. Free-form metadata and content fields are not accepted in execution context.

OpenTelemetry Tracing

OpenTelemetry tracing is opt-in and uses a separate flag from telemetry opt-out:

export TRAIGENT_TRACE_ENABLED=true

TRAIGENT_TRACE_ENABLED is the canonical tracing flag and defaults to false. It controls SDK span emission and workflow trace tracker creation. The older plural spelling TRAIGENT_TRACES_ENABLED is no longer supported as of 0.13.0 and is silently ignored. If you were using TRAIGENT_TRACES_ENABLED, migrate to TRAIGENT_TRACE_ENABLED.

Agent Workflow Spans

Traigent exposes a public helper for adding sanitized agent/node spans to the active optimization workflow trace:

from collections.abc import Mapping
from typing import Any

from traigent.observability import add_agent_span

def add_agent_span(
    node_id: str,
    *,
    span_type: str = "agent",
    input_tokens: int | None = None,
    output_tokens: int | None = None,
    cost_usd: float | None = None,
    latency_ms: float | None = None,
    model: str | None = None,
    metadata: Mapping[str, Any] | None = None,
) -> SpanResult

Returns a SpanResult receipt — accepted, reason when it was not, and dropped_metadata_keys for any metadata entries the sanitizer removed. It is falsy when the span was rejected, so if not add_agent_span(...) works. The call never raises. See Observability spans for the reason codes and the metadata rules.

The helper is safe to call from user code. If no active optimization trial or workflow trace manager exists, it logs at debug level and returns. Metadata is limited to safe numeric values, sensitive keys such as prompt/response/output are dropped, and model identifiers are validated before emission.

How Telemetry is Used

This section used to list four uses - optimization improvements, bug detection, performance analysis, usage analytics - without saying who holds the data or which runs it described. The FAQ then answered the same question the other way, calling telemetry local-only, so the file gave two answers and a reader could pick either. Both are rewritten here and below, scoped by path, because the path is what decides whether Traigent receives anything at all.

No-egress runs (offline=True): Traigent receives nothing. Everything above is written to your disk and used by your own run. There is nothing here for Traigent to use. TRAIGENT_DISABLE_TELEMETRY=true is a different lever: it silences SDK observability-telemetry emission but does not stop portal-backed session and trial egress. Use offline=True when the requirement is no Traigent backend egress at all.

Portal-backed runs: Traigent receives the data described in What Data is Collected above, minus the content listed in What is NOT Collected. That material is used to:

1. Run your optimization - the backend coordinates which configurations to try next, which is what a managed search is.
2. Record what each trial tried and achieved, so the portal can show you the winning configuration and how the others compared.
3. Tell you about your own dataset - which examples were too easy to separate configurations, which were too few to support a reliable comparison, and which look inconsistent.

Those three are scoped to your own run and reported back to you.

Beyond your run, the numeric measures and tuned configuration values may also be used in aggregate across customers: which settings tend to help which kinds of agent, and the statistics that inform how Traigent searches in future. Two things bound that, and they are the reason it stays private rather than a promise that it does. Your dataset and your model's responses are not part of it, because they are never sent. Your prompts are not part of it unless you tune them as configuration values - a prompt variant used as a tuned configuration value is sent on the default path (see Data Boundary and No-Egress Runs above) and is therefore inside this aggregate set; offline=True is the only lever that withholds it - privacy-mode redaction applies to per-trial submissions, not to the configuration space sent at session creation.

That aggregate is statistical: which settings tended to help which kinds of agent, counted over those names, values and numbers.

Two things this section deliberately does not claim. It does not say the four older bullets were wrong about Traigent's intentions - it says they were not a description of where the data is. And it makes no statement about training or fine-tuning models on this material; that answer belongs to Traigent's published policy rather than to an SDK reference, and this file will link it rather than paraphrase it.

Data Retention

No-egress runs: - All data is stored locally on your machine - Data is kept in ~/.traigent/ or your specified local_storage_path - You control retention - delete files as needed - No data is sent to the Traigent backend

Portal-backed runs: - Tuned config-space values, configuration IDs/schema, and numeric metrics can be sent to Traigent backend for optimization coordination - Retention policies depend on your managed-service agreement - You can request data deletion at any time

Opting Out of Telemetry

Complete Opt-Out

To completely disable all telemetry collection, set the environment variable:

export TRAIGENT_DISABLE_TELEMETRY=true

Or in Python before importing Traigent:

import os
os.environ["TRAIGENT_DISABLE_TELEMETRY"] = "true"

import traigent

Accepted Values

The following values are recognized as "opt-out" (case-insensitive): - "true" - "1" - "yes" - "on"

Any other value (including unset) means telemetry is enabled.

What Happens When Disabled

When TRAIGENT_DISABLE_TELEMETRY is set:

1. No telemetry events are emitted - SDK telemetry collectors return without recording
2. No metrics are sent to collectors - MetricsCollector calls are skipped
3. Optimization still works normally - Only telemetry is disabled, not functionality
4. Local results are still saved - Optimization results are still persisted locally for your use
5. Observability traces are not sent - ObservabilityClient.flush() and close() still return success=True, but items_sent is 0 and warnings says delivery was suppressed and why. Check items_sent to confirm delivery.

Telemetry Implementation

Telemetry is routed through optimization storage, observability clients, and workflow tracing. The previous SDK-local Optuna telemetry emitter has been removed with the local Optuna dependency; smart optimizer telemetry is now a backend concern.

Event Types

When telemetry is enabled, the following events may be emitted (not exhaustive):

• trial_suggested - An optimizer proposed a new trial configuration
• trial_intermediate - Intermediate metric reported (pruning signal)
• trial_completed - A trial finished successfully
• trial_failed - A trial encountered an error
• trial_pruned - A trial was pruned early
• trial_call_started / trial_call_completed - Seamless injection wrapper lifecycle

Each event includes: - event: Event type string - trial_id: Trial identifier (if applicable) - study_name: Optimization study name - timestamp: ISO 8601 timestamp - payload: Additional event-specific data

Local Storage Structure

When using offline=True, data is stored locally:

~/.traigent/
├── sessions/
│   ├── 20260101_120000_my_function.json  # Optimization session record (trials + metadata)
│   └── ...
├── cache/
│   ├── model_responses/
│   └── ...
└── .locks/

You can customize the storage location:

@traigent.optimize(
    local_storage_path="./my_optimizations",
    ...
)

Per-example content in optimization logs

By default, optimization runs also write per-trial logs under ./.traigent/optimization_logs/ (the project working directory, unless TRAIGENT_OPTIMIZATION_LOG_DIR / TRAIGENT_RESULTS_FOLDER relocate it). Each example record includes the input query, the model response, and the expected output as free text, alongside ids and metrics.

⚠️ On-disk redaction covers structured PII only (emails, Luhn-checked cards, API keys / bearer tokens, etc.). Free-text content in prompts and responses (names, addresses, proprietary data) is stored verbatim.

To keep ids and metrics while omitting that content, disable content logging:

export TRAIGENT_LOG_EXAMPLE_CONTENT=false   # also accepts 0 / no / off

or per-logger via OptimizationLogger(..., log_example_content=False). With it disabled, trial jsonl retains example_id, accuracy, cost_usd, latency_ms and error, but query / response / expected are null.

Traigent writes a .gitignore (*) into the log root so this content is not accidentally committed, but you should still keep .traigent/ out of version control and CI artifact collection in sensitive projects.

Security and Sensitive Data

Data Sanitization

Configuration data is sanitized before telemetry. Private/internal keys (starting with _) are automatically removed. On default portal-backed submissions, tuned config values are otherwise sent unchanged so the portal can display winning configs. On privacy-mode submissions, sensitive-key values and all string/free-text config values are redacted while preserving the tuned keys.

Error Handling

Telemetry failures never crash your optimization:

• If a listener raises an exception, it's logged but doesn't affect optimization
• If metrics collection fails, it's logged but optimization continues
• Telemetry is defensive and isolated from core functionality

Compliance

HIPAA and Sensitive Data

For HIPAA compliance or handling sensitive data:

1. Use offline=True for Zero Traigent Egress:

   @traigent.optimize(
       algorithm="grid",
       offline=True,
       ...
   )
   

2. Disable Telemetry:

   export TRAIGENT_DISABLE_TELEMETRY=true
   

3. Use Local Storage Only:

   @traigent.optimize(
       algorithm="grid",
       offline=True,
       local_storage_path="/secure/location",
       ...
   )
   

4. Review Evaluation Dataset: Ensure your evaluation datasets don't contain PII

Observability transport health

ObservabilityClient.get_stats() returns a process-local snapshot of buffered trace delivery. Use it to alert on local loss before a trace reaches the backend. The snapshot includes dropped_items, dropped_by_reason, queue_depth, inflight_items, oldest_inflight_age_seconds, and retry_attempts.

from traigent.observability import ObservabilityClient, ObservabilityConfig


def report_transport_health(event_type: str, payload: dict) -> None:
    # Every local drop is reported. Delivery failures are one event per batch:
    # payloads include item_count and up to 20 trace_ids alongside drop_reason,
    # cumulative dropped_items, queue_depth, and event-specific details.
    metrics_sink.emit(event_type, payload)


client = ObservabilityClient(
    ObservabilityConfig(health_callback=report_transport_health)
)
stats = client.get_stats()
if stats["dropped_items"]:
    metrics_sink.emit("observability_transport_snapshot", stats)

queue_depth counts payloads still buffered; inflight_items counts payloads already handed to a sender but not yet completed. A non-null oldest_inflight_age_seconds makes a custom sender that does not return diagnosable; custom senders should enforce their own request timeout. dropped_by_reason is bounded to the transport's in-process lifetime and distinguishes reasons such as queue_full, payload_too_large, payload_not_json_serializable, transport_closed, and batch_delivery_failed. These are local SDK metrics; they are deliberately not added to the ingest wire payload until the strict Schema and Backend contract is updated together.

health_callback is invoked only after SDK transport locks are released. It may run on a background delivery thread, and callback exceptions are swallowed. Keep it fast and safe for concurrent invocation. Batch-delivery failures emit a single batch-shaped event with item_count, up to 20 trace_ids, and trace_ids_truncated; queue, oversized-payload, serialization, and closed- transport drops each emit their own event.

flush(timeout=0) and close(timeout=0) are immediate poll-style calls: they return without starting a sender and do not emit a deadline-exceeded warning. Previously, zero was treated like the configured flush timeout. An explicit positive timeout bounds the wait; no-argument flush/close preserves synchronous delivery. The atexit close path explicitly uses the default 30-second flush_timeout, so shutdown can trade undelivered tail items for a bounded interpreter exit.

FAQ

Q: Is telemetry enabled by default?

A: Yes, for local optimization metadata. The default portal-backed path can send tuned config-space values, configuration IDs/schema, and numeric metrics; use privacy-mode redaction when backend coordination is needed without tuned string values, or offline=True for zero Traigent backend egress.

Q: How do I verify telemetry is disabled?

A: Check the environment variable:

echo $TRAIGENT_DISABLE_TELEMETRY

Or in Python:

import os
print(os.getenv("TRAIGENT_DISABLE_TELEMETRY"))

Q: Does disabling telemetry affect optimization performance?

A: No. Disabling telemetry has minimal performance impact. Optimization runs normally without telemetry.

Q: Can I enable telemetry for some optimizations but not others?

A: The TRAIGENT_DISABLE_TELEMETRY environment variable is process-wide. Use offline=True when you need to disable Traigent backend egress for a run.

Q: Where can I see what telemetry data was collected?

A: With offline=True, check the JSON files in ~/.traigent/sessions/. They contain the same trial metadata and metrics emitted to telemetry listeners.

Q: Does any of this reach Traigent?

A: Only on the portal-backed path - see What Data is Collected and What is NOT Collected above for what that path sends. A no-egress run (offline=True) sends nothing. This answer used to read "telemetry is local-only in the open-source version", which was true of a no-egress run and wrong about the default portal-backed one, and it sat one page below a section listing what Traigent does with the data it receives. Two answers to one question is worse than either.

Q: Can I contribute telemetry data to improve Traigent?

A: There is no opt-in channel for contributing telemetry beyond what a portal-backed run already sends for your own optimization. If that changes it will be opt-in and documented here.

Related Documentation

• Decorator Reference - Configuration options
• Choosing the Right Optimization Model - defaults, local search, no-egress runs, and migration guidance
• Security - Security practices
• API Reference - Full API documentation