Token and Cost Capture¶

Traigent charges each trial for the LLM calls its examples make. It reads the provider's reported usage (prompt and completion tokens), prices it, and records input_tokens, output_tokens, total_tokens, input_cost, output_cost and total_cost on every example and trial.

This page lists which calls are captured, how to price a model Traigent does not know, and what a trial reports when nothing was captured.

What is captured¶

Client	Captured when	Notes
LangChain `ChatOpenAI`, `ChatAnthropic`, Bedrock chat models: `invoke`, `ainvoke`, `abatch`, `stream`, `astream`	always (patched by the evaluator)	`ainvoke`/`abatch` are captured (#2424, #2445). `stream`/`astream` capture the last chunk, which carries usage only with `stream_usage=True`; without it the call is reported as unmeasured, not `$0`
`litellm.completion` / `litellm.acompletion`	always (patched by the evaluator)	streaming calls are not captured
Traigent's `BedrockChatClient`	always
Raw OpenAI SDK, sync and async: `chat.completions.create`, `chat.completions.parse` (also `beta.chat.completions.parse`), `completions.create`, `responses.create`, `responses.parse`, and the `with_raw_response` form of each	the OpenAI override is active (see below)	non-streaming calls that return `usage`. Responses API `input_tokens`/`output_tokens` are counted as prompt/completion tokens. Your code still receives the normal return value; a `with_raw_response` result is parsed once and the parsed object is cached, so your own `.parse()` returns it
Direct `google.genai` calls (Gemini API or Vertex AI client)	never	Traigent does not patch this client, so these calls are outside cost tracking. A trial can show a cost of 0.0, and a run with a cost cap can stop with an unmeasured-cost error. Calling Gemini through `litellm.completion` gets its usage captured.

Raw OpenAI SDK calls (including OpenAI-compatible gateways)¶

An agent that calls the OpenAI SDK directly is captured when the OpenAI framework override is active. Either turn it on for the whole process:

from traigent.integrations.llms.openai import enable_openai_optimization

enable_openai_optimization()  # patches openai.OpenAI and openai.AsyncOpenAI

or name the classes on the decorator:

client = openai.OpenAI(base_url=os.environ["LLM_BASE_URL"])  # build it once


@traigent.optimize(
    configuration_space={"model": ["gpt-4o-mini", "gpt-4o"]},
    objectives=["accuracy", "cost"],
    eval_dataset="data.jsonl",
    auto_override_frameworks=True,
    framework_targets=["openai.OpenAI", "openai.AsyncOpenAI"],
)
def answer(question: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o-mini",  # replaced by the trial's model
        messages=[{"role": "user", "content": question}],
    )
    return response.choices[0].message.content

Every call made while an example runs is charged, so an agent that calls the model several times per example is billed for all of them.

Coverage limits:

The override must be active. Without enable_openai_optimization() or framework_targets, raw OpenAI calls are not captured (issue #2441 tracks capture that needs no opt-in).
Streaming calls (stream=True) are not captured. A stream's usage arrives only after your code reads it, and only with stream_options={"include_usage": True}. The example is reported as unmeasured, not as $0.
Responses without usage (some gateways omit it) are not captured. They are reported as unmeasured.
Not captured: streaming forms (stream=True, chat.completions.stream, responses.stream), completions.create on a gateway that omits usage, and any other method (embeddings, images, audio, responses.retrieve). A client whose with_raw_response accessor was first used before the override was activated keeps the unpatched method: build the client after enabling the override, or access with_raw_response only afterwards.
The new parse and Responses paths are capture-only: Traigent records their usage but does not inject trial parameters (model, temperature, ...) into them. Only chat.completions.create and completions.create receive injected parameters.
When LangChain's ChatOpenAI.invoke or litellm.completion makes the OpenAI call for you, that wrapper records the usage and the underlying OpenAI call is not counted a second time.

Pricing a model Traigent does not know¶

Traigent prices tokens from LiteLLM's model table. A gateway alias such as acme-gateway/house-model is not in that table, so its calls are unpriced: tokens are recorded, cost is not known. What happens next depends on TRAIGENT_STRICT_COST_ACCOUNTING:

true: an unpriced model raises.
unset: a run whose objectives include cost is strict at runtime, so it stops at the first call that has neither a table price nor a provider-reported cost. A run without a cost objective records the call at $0 and the result carries an UNPRICED_MODEL_RUNTIME warning.
false: explicit opt-out. The call is recorded at $0 with the same warning. The trial is flagged cost_unpriced, and a flagged trial is left out of cost ranking (best config, Pareto front, weighted scores), so a placeholder $0 never wins as "cheapest".

Give the alias a price with TRAIGENT_CUSTOM_MODEL_PRICING_JSON (inline) or TRAIGENT_CUSTOM_MODEL_PRICING_FILE (path to the same JSON). Prices are USD per token:

export TRAIGENT_CUSTOM_MODEL_PRICING_JSON='{
  "acme-gateway/house-model": {
    "input_cost_per_token": 0.000002,
    "output_cost_per_token": 0.000005
  }
}'

The key must be the model name Traigent prices. In the custom-evaluator lane that is the trial's model setting when the configuration has one, otherwise the model the response reports.

Mock provenance in result metadata¶

A result records whether its LLM calls were simulated in result.metadata:

Key	Meaning
`mock_llm_run`	`True` if any mock response was served during the run, `False` if the run's calls were real.
`mock_llm_mixed`	Present and `True` only when both mock and real calls were observed.

Both keys are saved and restored by PersistenceManager. A result saved before these keys existed, or built by hand, has no mock_llm_run key: that means unknown, not False. Read it with result.metadata.get("mock_llm_run") and treat None as "not recorded". The results table labels a recorded mock run, and hedges an unrecorded result only while mock mode is on in the process.

When nothing was captured¶

A missing measurement is not a free call. When a custom-evaluator example has no captured usage, its result has metadata["llm_usage_measured"] = False and no token or cost keys. When no example in a trial was measured, the trial's token and cost totals are left out of its metrics (or set to None when TRAIGENT_STRICT_METRICS_NULLS=true) instead of being reported as 0. When only some examples were measured, the totals cover those examples and a warning names the coverage.

The same holds for a cost objective: a trial with no measured example has no cost (or None under strict nulls), never 0.0; cost_unmeasured is 1.0 on a trial where any example had no measured cost, and token counts estimated from text length (input from the dataset or prompt template, output from the returned string, about 4 characters per token; a system message written inside your function is not counted, so the estimate can be well below the provider's count) are labelled with tokens_estimated and are never priced. The results table shows the cost as unmeasured. Weighted best_config selection counts a missing cost as the worst cost, the Pareto front leaves out a trial that is missing a configured objective, and the batch composite score gives such a trial the failed-trial score instead of scoring it on quality alone. A constraint written as metrics.get("cost", 0) <= limit still accepts unmeasured trials (#2446, item 3); write metrics.get("cost") is not None and metrics["cost"] <= limit instead. Workflow spans for an unmeasured trial, and for a failed or pruned trial that recorded no cost, send the known spend as cost_usd (0.0 when none was reported) together with the metadata flag traigent.cost_unmeasured: true; the flag, not the number, says the cost is not a measurement. Run summaries (get_summary(), experiment_stats) withhold total_cost and report cost_unmeasured_trials and cost_lower_bound when any trial's cost is unmeasured.

A run that stops before the grid is exhausted records why in result.metadata["stop_detail"] and prints it under the results table: the SDK default max_trials=10 when none was passed, an explicit max_trials, or the unmeasured-cost safety limit. A metric_limit on a cost metric leaves unmeasured trials out of its running total rather than failing the run. Two run-level warnings say what happened:

COST_OBJECTIVE_NO_USAGE_CAPTURED: no trial captured usage, so the cost column is unmeasured. Under strict cost accounting the run fails instead.
COST_OBJECTIVE_PARTIAL_USAGE_CAPTURED: some successful trials captured usage and others did not; the cost comparison covers only the measured ones.

When a run stops because cost was not measured¶

The cost limit (cost_limit= on @traigent.optimize or .optimize(), or TRAIGENT_RUN_COST_LIMIT, default $2.00) can only bound spend it can see. When any trial reports no cost, what happens depends on whether you sized the run yourself.

You did not set max_trials or max_total_examples. Traigent is conservative: the run stops at a default safety limit of 10 trials (TRAIGENT_FALLBACK_TRIAL_LIMIT changes it) with stop_reason == "cost_limit". One unmeasured trial is enough, even if every other trial was measured. The result carries the COST_UNMEASURED_TRIAL_LIMIT_REACHED warning code and a message, which is also logged, that says how many trials were unmeasured and how many were measured.

You set max_trials or max_total_examples explicitly, on the decorator, on OptimizedFunction(...), or on .optimize()/.optimize_sync(). That is taken as consent: the safety limit does not apply, and the run goes up to the size you set. Set max_trials to run more trials. max_total_examples caps the total number of examples across trials and also counts as your explicit consent, but it does not raise max_trials, which defaults to 10, so on its own it never yields more than 10 trials. The measured part of the spend is still bounded by cost_limit. The result carries COST_UNMEASURED_TRIALS_RAN, and a cost-objective run also carries COST_OBJECTIVE_NO_USAGE_CAPTURED or COST_OBJECTIVE_PARTIAL_USAGE_CAPTURED. "Explicitly" means you passed the argument; passing the same value as the default counts.

To continue after the safety stop, either:

Get cost measured for every configuration. Call the model through a captured client (the table above). In a mixed run, find why the unmeasured configurations report no usage: a gateway that omits usage, a streaming call, or an uncaptured client. A fully measured run is bounded by its cost budget instead of the trial limit, so raise cost_limit if the budget is what stops you.
Size the run yourself. Set max_trials explicitly to run more trials, for example func.optimize_sync(max_trials=50). max_total_examples caps total examples and also counts as your explicit consent, but does not raise the default max_trials of 10. Traigent cannot tell you what the unmeasured trials cost.

A max_total_examples passed to .optimize() or .optimize_sync() is kept on the function object, so later calls on the same object keep that sample budget (and keep counting as explicitly sized) until you change it.

Build the OpenAI client once, outside the optimized function, as in the examples above. The override injects the trial's model and sampling parameters at chat.completions.create, not into the client constructor.