LLM Evaluation with LLM-as-Judge: How to Measure AI Quality in Production

How to evaluate LLM output quality in production using LLM-as-Judge — building automated evaluation pipelines, scoring rubrics, and golden dataset testing with Claude API. With real code examples.

Shipping an LLM feature is easy. Knowing whether it's actually working well — and catching regressions when you update your prompt or switch models — is hard.

Manual evaluation doesn't scale. You can't have a human read every LLM response in production. But unit tests don't work either — there's no single correct answer to "write a diagnosis for this Kubernetes failure."

LLM-as-Judge is the solution: use a separate, capable LLM to evaluate the output of your production LLM. It's not perfect, but it scales and it catches real regressions.

Here's how to build it properly.

The Core Idea

Instead of checking if output equals a fixed string, you ask an evaluator LLM to score the output against a rubric:

Input: [the prompt you sent to your production LLM]
Output: [what your production LLM returned]
Criteria: [what makes a good response for this use case]

Score: 1-5
Reasoning: [why this score]

The evaluator LLM (usually a more capable or different model) returns structured scores that you can track over time.

Setting Up an Evaluation Pipeline

python

import anthropic
import json
from dataclasses import dataclass
from typing import Optional
 
 
@dataclass
class EvaluationResult:
    score: int          # 1-5
    reasoning: str
    passed: bool        # True if score >= threshold
    criteria_scores: dict[str, int]  # Per-criteria breakdown
 
 
def evaluate_response(
    input_prompt: str,
    model_output: str,
    use_case: str,
    threshold: int = 3,
    evaluator_model: str = "claude-opus-4-8"  # Use a capable model for judging
) -> EvaluationResult:
    """
    Evaluate an LLM response using Claude as judge.
    
    Args:
        input_prompt: The prompt that was sent to the production LLM
        model_output: The response from the production LLM
        use_case: Context about what this response is for
        threshold: Minimum score to consider a pass (1-5 scale)
        evaluator_model: Model to use as judge
    """
    client = anthropic.Anthropic()
 
    eval_prompt = f"""You are an expert evaluator assessing the quality of an AI assistant's response.
 
## Use Case
{use_case}
 
## User Input
{input_prompt}
 
## AI Response to Evaluate
{model_output}
 
## Evaluation Criteria
 
Score each criterion from 1-5:
- **Accuracy** (1-5): Is the technical information correct? Are commands accurate?
- **Completeness** (1-5): Does it address all aspects of the question?
- **Clarity** (1-5): Is it clear and well-structured? Can a practitioner follow it?
- **Actionability** (1-5): Does it give specific, executable next steps?
- **Conciseness** (1-5): Is it appropriately detailed without unnecessary padding?
 
## Response Format
Respond ONLY with this JSON structure:
{{
  "criteria_scores": {{
    "accuracy": <1-5>,
    "completeness": <1-5>,
    "clarity": <1-5>,
    "actionability": <1-5>,
    "conciseness": <1-5>
  }},
  "overall_score": <1-5>,
  "reasoning": "<2-3 sentences explaining the score>",
  "key_issues": ["<issue 1 if any>", "<issue 2 if any>"]
}}"""
 
    response = client.messages.create(
        model=evaluator_model,
        max_tokens=500,
        messages=[{"role": "user", "content": eval_prompt}]
    )
 
    result_text = response.content[0].text
    result_data = json.loads(result_text)
 
    return EvaluationResult(
        score=result_data["overall_score"],
        reasoning=result_data["reasoning"],
        passed=result_data["overall_score"] >= threshold,
        criteria_scores=result_data["criteria_scores"]
    )

Building a Golden Dataset

A golden dataset is a collection of input/output pairs that represent good responses. You use it to detect regressions: if your score on the golden dataset drops after a prompt change, something regressed.

python

import json
from pathlib import Path
from datetime import datetime
 
 
GOLDEN_DATASET = [
    {
        "id": "k8s-crashloop-001",
        "input": "My pod is in CrashLoopBackOff. The container exits with code 1. How do I debug this?",
        "use_case": "Kubernetes troubleshooting assistant for DevOps engineers",
        "minimum_score": 4,
        "must_contain": ["kubectl logs", "previous", "--previous"],  # Response must mention these
        "must_not_contain": ["I cannot help", "I don't know"]
    },
    {
        "id": "terraform-state-001",
        "input": "Terraform is showing resources that already exist as needing to be created. What's happening?",
        "use_case": "Terraform troubleshooting assistant",
        "minimum_score": 4,
        "must_contain": ["terraform state", "import"],
        "must_not_contain": []
    },
    {
        "id": "docker-build-001",
        "input": "My Docker build is slow. Each build takes 15 minutes even for small code changes.",
        "use_case": "Docker optimization assistant",
        "minimum_score": 4,
        "must_contain": ["layer cache", "COPY", "layer"],
        "must_not_contain": []
    }
]
 
 
def run_golden_dataset_evaluation(
    production_model: str,
    evaluator_model: str = "claude-opus-4-8"
) -> dict:
    """
    Run production model against golden dataset and return evaluation results.
    Use this in CI to catch regressions before deploying prompt changes.
    """
    client = anthropic.Anthropic()
    results = []
    passed = 0
    failed = 0
 
    for test_case in GOLDEN_DATASET:
        print(f"Testing: {test_case['id']}")
 
        # Get response from production model
        response = client.messages.create(
            model=production_model,
            max_tokens=1000,
            messages=[{"role": "user", "content": test_case["input"]}]
        )
        model_output = response.content[0].text
 
        # Check must_contain / must_not_contain rules
        rule_passed = True
        rule_failures = []
 
        for required_phrase in test_case.get("must_contain", []):
            if required_phrase.lower() not in model_output.lower():
                rule_passed = False
                rule_failures.append(f"Missing required content: '{required_phrase}'")
 
        for forbidden_phrase in test_case.get("must_not_contain", []):
            if forbidden_phrase.lower() in model_output.lower():
                rule_passed = False
                rule_failures.append(f"Contains forbidden content: '{forbidden_phrase}'")
 
        # Get LLM-as-judge evaluation
        eval_result = evaluate_response(
            input_prompt=test_case["input"],
            model_output=model_output,
            use_case=test_case["use_case"],
            threshold=test_case["minimum_score"],
            evaluator_model=evaluator_model
        )
 
        overall_pass = rule_passed and eval_result.passed
 
        result = {
            "test_id": test_case["id"],
            "passed": overall_pass,
            "score": eval_result.score,
            "minimum_score": test_case["minimum_score"],
            "reasoning": eval_result.reasoning,
            "criteria_scores": eval_result.criteria_scores,
            "rule_failures": rule_failures,
        }
 
        results.append(result)
 
        if overall_pass:
            passed += 1
        else:
            failed += 1
            print(f"  FAILED: score={eval_result.score}, reasoning={eval_result.reasoning}")
            if rule_failures:
                print(f"  Rule failures: {rule_failures}")
 
    return {
        "timestamp": datetime.now().isoformat(),
        "model": production_model,
        "total": len(GOLDEN_DATASET),
        "passed": passed,
        "failed": failed,
        "pass_rate": passed / len(GOLDEN_DATASET),
        "results": results
    }
 
 
def save_evaluation_report(report: dict, output_dir: str = "eval-reports"):
    """Save evaluation results for tracking over time."""
    Path(output_dir).mkdir(exist_ok=True)
    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
    model_safe = report["model"].replace("/", "_").replace(":", "_")
    filename = f"{output_dir}/eval_{model_safe}_{timestamp}.json"
 
    with open(filename, "w") as f:
        json.dump(report, f, indent=2)
 
    print(f"Report saved: {filename}")
    return filename

Adding to CI/CD

Run golden dataset evaluations automatically when you change prompts:

yaml

# .github/workflows/llm-eval.yml
name: LLM Quality Evaluation
 
on:
  pull_request:
    paths:
      - "prompts/**"
      - "src/llm/**"
 
jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      
      - run: pip install anthropic
 
      - name: Run golden dataset evaluation
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          python -c "
          from evaluation import run_golden_dataset_evaluation, save_evaluation_report
          report = run_golden_dataset_evaluation('claude-sonnet-5')
          save_evaluation_report(report)
          
          # Fail CI if pass rate drops below 80%
          if report['pass_rate'] < 0.8:
              print(f'FAIL: Pass rate {report[\"pass_rate\"]:.0%} below threshold')
              exit(1)
          print(f'PASS: {report[\"pass_rate\"]:.0%} of golden dataset tests passed')
          "
 
      - name: Upload eval report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: eval-report
          path: eval-reports/

Tracking Quality Over Time

Log evaluation results to track model quality trends:

python

import logging
import time
 
logger = logging.getLogger(__name__)
 
 
def production_evaluate_sample(
    input_prompt: str,
    model_output: str,
    use_case: str,
    sample_rate: float = 0.05  # Evaluate 5% of production calls
):
    """
    Randomly sample production calls for ongoing quality monitoring.
    Call this in your production LLM wrapper.
    """
    import random
    if random.random() > sample_rate:
        return  # Skip this call
 
    try:
        result = evaluate_response(input_prompt, model_output, use_case)
 
        logger.info({
            "event": "llm_quality_sample",
            "score": result.score,
            "passed": result.passed,
            "use_case": use_case,
            "criteria_scores": result.criteria_scores,
            # Don't log the actual content in production for privacy
        })
    except Exception as e:
        logger.error(f"Evaluation sampling failed: {e}")
        # Don't let evaluation failures affect production

Common Pitfalls

Using the same model as judge and judged. A model judging its own outputs is biased. Use a different model or a significantly more capable version (Claude Sonnet for production, Claude Opus for evaluation).

Writing vague criteria. "Is this good?" produces inconsistent scores. "Does this include at least one executable kubectl command?" produces consistent scores.

Evaluating too frequently. LLM-as-judge calls cost money and add latency. Sample 2-10% of production calls for ongoing monitoring. Run full golden dataset evaluations only on prompt/model changes.

Forgetting calibration. Run your rubric on 20-30 human-reviewed examples before trusting it in CI. If the LLM judge disagrees with humans 40% of the time, your rubric needs work.

LLM-as-judge is not a silver bullet — an imperfect automated evaluation is still better than no evaluation at all for catching obvious regressions. Start simple, track scores over time, and refine your rubric as you see where it diverges from human judgment.

More LLMOps content? Read our LLM context window management guide and getting reliable JSON output from LLMs.

LLM Evaluation with LLM-as-Judge: How to Measure AI Quality in Production

The Core Idea

Setting Up an Evaluation Pipeline

Building a Golden Dataset

Adding to CI/CD

Tracking Quality Over Time

Common Pitfalls

Stay ahead of the curve

Related Articles

LLM Context Window Management in Production: What Nobody Tells You

LLM Observability with OpenTelemetry: Tracing LLM Calls in Production

LLM Prompt Caching: How to Cut Your Claude API Costs by 80%

Comments