📊 Traigent Evaluation Guide¶

Understanding Evaluation in Traigent¶

Evaluation is how Traigent measures the success of different AI agent configurations. This guide explains what evaluation means, how it works, and how to use it effectively.

Table of Contents¶

What is Evaluation?
Understanding Accuracy
Evaluation Methods
Creating Evaluation Datasets
Custom Evaluators
Troubleshooting
Examples

What is Evaluation?¶

Evaluation in Traigent compares your AI agent's outputs against expected results to determine which configuration performs best.

@traigent.optimize(
    configuration_space={"temperature": [0.1, 0.5, 0.9]},
    objectives=["accuracy"],  # What we're measuring
    eval_dataset="data.jsonl"  # Test cases with expected outputs
)
def my_agent(input):
    # Your AI agent logic
    return output

Understanding Accuracy¶

"Accuracy" in Traigent can mean different things depending on your use case:

1. Classification Accuracy (Exact Match)¶

For tasks with discrete outputs (sentiment, categories, yes/no):

# Accuracy = % of exact matches
Input: "I love this!"  → Output: "positive" → Expected: "positive" → ✅ 100%
Input: "I hate this!"  → Output: "neutral"  → Expected: "negative" → ❌ 0%

2. Semantic Similarity (Bring Your Own Scorer)¶

For tasks with varied but equivalent outputs:

# Accuracy = semantic similarity score (from a user-supplied scoring_function)
Input: "Capital of France?"
Output: "Paris is the capital"
Expected: "Paris"
→ 95% (semantically equivalent, with your own embedding-based scorer)

Important: Traigent does not ship an embedding-based scorer. The example above only matches the paraphrase if you pass a scoring_function (or metric_functions={"accuracy": ...}) that computes semantic similarity yourself (e.g., OpenAI embeddings, sentence-transformers, or an LLM judge). The built-in default is exact / case-insensitive match — see Method 1 below.

3. Custom Metrics¶

For domain-specific evaluation:

# Accuracy = custom calculation
Input: Financial report
Output: Generated summary
Expected: Reference summary
→ ROUGE score, BLEU score, or business metrics

Evaluation Methods¶

Method 1: Default Evaluation (Exact / Case-Insensitive Match)¶

When you do not pass a scoring_function, metric_functions, or custom_evaluator, Traigent's LocalEvaluator scores accuracy by comparing the agent's output to expected_output using exact match, falling back to a case-insensitive string comparison. There is no embedding model and no LLM judge in this path.

@traigent.optimize(
    eval_dataset="qa_pairs.jsonl",  # Default: exact / case-insensitive accuracy
    objectives=["accuracy"]
)
def qa_agent(question):
    return answer

Works well for: classification, span extraction, structured outputs, and any task where the expected output is a fixed string or label.

Does not work for: Q&A, summarization, translation, or anything where paraphrases must count as correct — those need a user-supplied semantic scorer (see Method 2).

Dataset format (JSONL):

{"input": {"question": "What is AI?"}, "output": "Artificial Intelligence"}
{"input": {"question": "Who invented Python?"}, "output": "Guido van Rossum"}

Method 2: Semantic Similarity (User-Supplied Scorer)¶

For tasks where paraphrases should count as correct, supply your own semantic scorer via scoring_function (or metric_functions={"accuracy": ...}). Traigent does not ship one out of the box.

from openai import OpenAI

client = OpenAI()  # requires OPENAI_API_KEY in your own environment

def semantic_accuracy(output: str, expected: str) -> float:
    """Embedding-based cosine similarity, returned as a 0.0-1.0 score."""
    resp = client.embeddings.create(
        model="text-embedding-3-small",
        input=[str(output), str(expected)],
    )
    a, b = resp.data[0].embedding, resp.data[1].embedding
    dot = sum(x * y for x, y in zip(a, b))
    na = sum(x * x for x in a) ** 0.5
    nb = sum(x * x for x in b) ** 0.5
    return dot / (na * nb) if na and nb else 0.0

@traigent.optimize(
    eval_dataset="qa_pairs.jsonl",
    scoring_function=semantic_accuracy,  # your scorer, your dependency, your API key
    objectives=["accuracy"],
)
def qa_agent(question):
    return answer

Fail-loud behaviour: If your data ships with metadata.evaluation_type = "semantic" but you do not provide a scoring_function or custom_evaluator, LocalExecutionAdapter marks each such example as success=False, sets correct=False, attaches an error field explaining the missing capability, and emits an ERROR-level log. The aggregate metrics will report success_rate < 1.0 and accuracy_semantic = 0.0, so the problem shows up immediately instead of silently scoring paraphrases as wrong.

Method 3: Exact Match (Explicit)¶

For classification tasks where you want to be explicit:

def exact_match_score(output, expected, llm_metrics=None):
    """Returns 1.0 for exact match, 0.0 otherwise."""
    return 1.0 if str(output).strip() == str(expected).strip() else 0.0

@traigent.optimize(
    eval_dataset="classification.jsonl",
    scoring_function=exact_match_score,
    objectives=["accuracy"]
)
def classifier(text):
    return category

Method 4: Mock Mode (Development/Demo)¶

For testing without real API calls:

from traigent.testing import enable_mock_mode_for_quickstart

enable_mock_mode_for_quickstart()

Mock mode provides: - Realistic accuracy values (60-95%) - Progressive improvement simulation - No API costs - Fast iteration

Tips: - To keep all artifacts local and writable in restricted environments, set TRAIGENT_RESULTS_FOLDER to a project path: - export TRAIGENT_RESULTS_FOLDER="./.traigent_local" - The Examples Navigator pages work from file://, but some browsers block fetch() on local files. Serve via HTTP to avoid CORS: - python -m http.server 8000 → open http://localhost:8000/examples/ - On “Getting Started”, a built-in inline fallback shows demo results even on file://

Creating Evaluation Datasets¶

Format: JSONL (JSON Lines)¶

Each line is a complete JSON object:

{"input": {"text": "Great product!"}, "output": "positive"}
{"input": {"text": "Terrible service"}, "output": "negative"}
{"input": {"text": "It's okay"}, "output": "neutral"}

Best Practices¶

Representative Data: Include examples from real use cases
Balanced Classes: Equal distribution across categories
Edge Cases: Include difficult or ambiguous examples
Size: 20-100 examples for initial testing, 100+ for production

Creating Datasets Programmatically¶

import json

# Create evaluation dataset
dataset = []

# Add examples
dataset.append({
    "input": {"question": "What is 2+2?"},
    "output": "4"
})

dataset.append({
    "input": {"question": "Capital of Japan?"},
    "output": "Tokyo"
})

# Save to JSONL
with open("eval_dataset.jsonl", "w") as f:
    for item in dataset:
        f.write(json.dumps(item) + "\n")

Custom Evaluators¶

When to Use Custom Evaluators¶

Use custom evaluators when: - You have specific business metrics - Default similarity doesn't fit your use case - You need weighted scoring - You want to evaluate multiple aspects

Example: Multi-Aspect Evaluator¶

def business_score(output, expected, llm_metrics=None):
    """
    Custom evaluator for business requirements.

    Returns:
        float: Score between 0.0 and 1.0
    """
    score = 0.0

    # Check correctness (40% weight)
    if output and expected:
        if output.lower() == expected.lower():
            score += 0.4

    # Check completeness (30% weight)
    if len(output) > 10:  # Minimum length requirement
        score += 0.3

    # Check format (30% weight)
    if output.startswith(("The", "A", "An")):  # Proper sentence
        score += 0.3

    return min(1.0, score)

@traigent.optimize(
    scoring_function=business_score,
    eval_dataset="business_cases.jsonl"
)
def business_agent(query):
    return response

Example: ROUGE Score Evaluator¶

from rouge_score import rouge_scorer

def rouge_score_fn(output, expected, llm_metrics=None):
    """Evaluate using ROUGE score for summarization."""
    scorer = rouge_scorer.RougeScorer(['rouge1', 'rougeL'], use_stemmer=True)
    scores = scorer.score(expected, output)

    # Return ROUGE-L F1 score
    return scores['rougeL'].fmeasure

@traigent.optimize(
    scoring_function=rouge_score_fn,
    eval_dataset="summaries.jsonl"
)
def summarizer(text):
    return summary

Troubleshooting¶

Issue: Getting 0.0% Accuracy¶

Causes and Solutions:

Default scorer is exact match, not semantic The built-in LocalEvaluator accuracy is exact / case-insensitive string comparison. Paraphrased answers (e.g. agent returns "Paris is the capital of France", expected is "Paris") score 0.0 under the default. If you need semantic equivalence, pass your own scoring_function (see Method 2 above).

Wrong Dataset Format

# ❌ Wrong
{"question": "...", "answer": "..."}

# ✅ Correct
{"input": {"question": "..."}, "output": "..."}

Type Mismatch

# Ensure output types match
def evaluator(output, expected):
    return 1.0 if str(output) == str(expected) else 0.0

evaluation_type: "semantic" in dataset metadata without a scorer LocalExecutionAdapter does not implement embedding-based similarity. Examples tagged metadata: {"evaluation_type": "semantic"} will be marked success=False with an explicit error message until you provide a scoring_function or custom_evaluator. Check the per-example error field or look for the "Semantic evaluation requested but no semantic evaluator is configured" ERROR log.

Use Mock Mode for Testing

from traigent.testing import enable_mock_mode_for_quickstart

enable_mock_mode_for_quickstart()

Issue: Inconsistent Results¶

Solutions:

Set Random Seed

import random
import numpy as np
random.seed(42)
np.random.seed(42)

Use Larger Dataset
Minimum 50 examples for stable results
Check for Data Leakage
Ensure test data isn't in training

Examples¶

Complete Sentiment Analysis Example¶

import traigent
import json
from pathlib import Path

# Create evaluation dataset
dataset = [
    {"input": {"text": "Amazing product!"}, "output": "positive"},
    {"input": {"text": "Terrible experience"}, "output": "negative"},
    {"input": {"text": "It's acceptable"}, "output": "neutral"},
]

# Save dataset
with open("sentiment_eval.jsonl", "w") as f:
    for item in dataset:
        f.write(json.dumps(item) + "\n")

# Define simple scoring function for exact match
def sentiment_score(output, expected, llm_metrics=None):
    """Exact match for sentiment classification."""
    return 1.0 if output.lower() == expected.lower() else 0.0

# Create optimizable function
@traigent.optimize(
    configuration_space={
        "temperature": [0.0, 0.3, 0.7],
        "model": ["gpt-4.1-mini", "gpt-4.1"]
    },
    eval_dataset="sentiment_eval.jsonl",
    scoring_function=sentiment_score,
    objectives=["accuracy"],
    max_trials=10
)
def sentiment_classifier(text):
    cfg = traigent.get_config()
    temperature = cfg["temperature"]
    model = cfg["model"]
    # Your LLM logic here
    # This is simplified for example
    if "amazing" in text.lower() or "great" in text.lower():
        return "positive"
    elif "terrible" in text.lower() or "bad" in text.lower():
        return "negative"
    else:
        return "neutral"

# Run optimization
import asyncio
results = asyncio.run(sentiment_classifier.optimize())

print(f"Best config: {results.best_config}")
print(f"Best accuracy: {results.best_score:.2%}")

Q&A System with Semantic Evaluation¶

Q&A is a natural fit for semantic scoring because well-formed answers paraphrase the expected output. The example below shows the user-supplied semantic scorer (scoring_function) you need — the default scorer would mark these paraphrases as wrong.

import litellm
import traigent

def semantic_accuracy(output: str, expected: str) -> float:
    """Replace this with embeddings / an LLM judge for your domain."""
    out_tokens = {tok.lower() for tok in str(output).split()}
    exp_tokens = {tok.lower() for tok in str(expected).split()}
    if not exp_tokens:
        return 0.0
    return 1.0 if exp_tokens.issubset(out_tokens) else 0.0

@traigent.optimize(
    configuration_space={
        "temperature": [0.0, 0.5, 1.0],
        "max_tokens": [50, 100, 200],
    },
    eval_dataset="data/qa_dataset.jsonl",
    scoring_function=semantic_accuracy,  # required for paraphrase tolerance
    objectives=["accuracy", "cost"],
    max_trials=15,
)
def qa_system(question: str) -> str:
    cfg = traigent.get_config()
    response = litellm.completion(
        model="gpt-4o-mini",
        temperature=cfg["temperature"],
        max_tokens=cfg["max_tokens"],
        messages=[{"role": "user", "content": f"Answer concisely: {question}"}],
    )

    return response.choices[0].message.content

# Without a scoring_function, the same Q&A run would score paraphrased
# answers as 0.0 under the default exact / case-insensitive contract.

Summary¶

Start Simple: Use mock mode and default evaluation
Understand Your Metrics: Know what "accuracy" means for your use case
Create Good Datasets: Representative, balanced, sufficient size
Customize When Needed: Write evaluators for specific requirements
Test Thoroughly: Verify evaluation works before optimization

Next Steps¶

Custom Evaluator Patterns
Dataset creation tools are available in the repository examples.
Troubleshooting Guide