What this build covers (and what an eval harness actually is)
An evaluation harness validates agent behavior across changes by combining tracing, fixed datasets for offline testing, live evaluators, and a CI pull request gate built on the Langfuse GitHub Action. This blueprint describes how those parts fit together so regressions are caught before merge and live quality stays visible.
This guide is assembled from official documentation and community reports — we did not run this build ourselves.
When you run production autonomous workflows, raw generation logging is rarely enough. An evaluation harness provides the operational guardrails needed to measure behavior systematically. Instead of guessing whether an updated system prompt, a new tool definition, or a revised model checkpoint degrades performance, you couple structured tracing with automated scoring.
As described in the Langfuse Evaluation Overview, evaluation runs in two modes: online and offline. Offline evaluation tests your existing application against a pre-defined dataset before you ship a change. Online evaluation scores live production traces to measure quality on real data and track trends over time. Tying both modes into a CI pipeline turns ad-hoc checking into a repeatable engineering baseline.
For broader tooling decisions, see the ToolBrain comparisons hub and the LLM observability breakdown. The Langfuse review covers platform fundamentals. This post stays focused on the architecture of the build.
+-------------------------------------------------------------------+
| AI Agent Execution |
| (Instrumented via Python SDK, JS/TS, LangChain, etc.) |
+---------------------------------+---------------------------------+
|
Ingests Traces & Spans via API
v
+-------------------------------------------------------------------+
| Langfuse Platform |
| (Cloud or Self-Hosted Core) |
+---------------------------------+---------------------------------+
| |
v v
+-----------------------+ +-------------------+
| Offline Evaluation | | Online Evaluation |
| - Fixed Datasets | | - Live Traces |
| - Regression Checks | | - LLM-as-Judge |
| - experiment-action | | - Human Queues |
+-----------+-----------+ +-------------------+
|
Fails Pull Request on RegressionError
v
+-----------------------+
| CI/CD Gate (GitHub) |
+-----------------------+
Stack and prerequisites
Setting up this harness requires an agent codebase in Python or JavaScript/TypeScript, access to Langfuse Cloud or a self-hosted instance, git, Docker if you self-host, and a GitHub Actions environment for CI. Langfuse documents that the instrumentation code is the same for Cloud and self-hosted deployments.
According to the official Get Started with Tracing documentation and the Python SDK documentation, the pieces are:
- Agent Runtime: Python or JavaScript/TypeScript application logic.
- Observability Layer: Langfuse Cloud, or a self-hosted instance.
- CI/CD Platform: GitHub Actions running regression jobs on pull requests.
- Container Engine: Docker and Docker Compose, only needed if you deploy the self-hosted server.
- Automation Utilities: Node, if you use the
npx skillsinstaller from the Langfuse skills repository.
You also need project API credentials (secret key and public key) created in the project settings, plus the base URL of your region or self-hosted instance. If you are new to the platform, the beginner’s guide to Langfuse walks through the dashboard.
Step 1 — Tracing: getting your agent’s runs into Langfuse
Traces enter Langfuse through the Python and JS/TS SDKs, framework integrations such as LangChain and the Vercel AI SDK, the OpenAI SDK drop-in wrappers, or OpenTelemetry. Instrumenting your application records the hierarchy of tool calls and model generations as traces and observations, on both Cloud and self-hosted instances.
The Get Started page lists the supported entry points: OpenAI SDK (Python and JS/TS), Vercel AI SDK, LangChain (Python and JS/TS), the Python SDK, the JS/TS SDK, and OpenTelemetry. It also notes that Langfuse calls spans “observations”, and that the instrumentation code is the same for Langfuse Cloud and self-hosting. Further background is in the SDK Overview and the data model documentation.
The code is open source. The Langfuse GitHub repository shows more than 10,000 commits on the main branch at the time of writing.
Below is an illustrative pattern adapted from the Python SDK example on the Get Started page. Check the current docs before copying it.
# Illustrative only: adapted from the documented Python SDK example.
# Credentials come from environment variables:
# LANGFUSE_SECRET_KEY, LANGFUSE_PUBLIC_KEY, LANGFUSE_BASE_URL
from langfuse import get_client
langfuse = get_client()
def run_agent_workflow(user_query: str) -> str:
with langfuse.start_as_current_observation(as_type="span", name="assistant-run") as span:
span.update(input=user_query)
# Nested generation for the model call
with langfuse.start_as_current_observation(
as_type="generation", name="llm-response", model="gpt-3.5-turbo"
) as generation:
response = "Simulated model output" # your real LLM call goes here
generation.update(output=response)
span.update(output=response)
# Flush events in short-lived applications such as CI jobs
langfuse.flush()
return response
The Get Started page also describes an installable agent skill that lets a coding agent add tracing for you:
npx skills add langfuse/skills --skill "langfuse"
Step 2 — Datasets and offline evaluation (the pre-ship gate)
Offline evaluation checks your application against a pre-defined dataset before a change ships, giving you a repeatable baseline. Each candidate configuration runs over the same test cases, so a drop in quality shows up before the change reaches live users.
According to the Evaluation Overview and the Datasets documentation, datasets are the way to build a reusable set of test cases, and experiments compare prompt, model, or code changes side by side. Experiments can be run via the UI, the SDK, or OpenTelemetry, as listed in the overview’s feature table. Setup details are on the Offline Evaluation Get Started page.
A typical offline loop looks like this (a general pattern, not a quoted procedure):
- Define a dataset of representative production queries and known edge cases.
- Load the dataset items into your test harness.
- Run the updated agent against each item.
- Record the outputs and attach scores to the experiment run.
# Illustrative pseudocode, NOT a real SDK API.
# load_dataset_items and record_run_result are stand-in names.
# See the Datasets and Experiments-via-SDK docs for the actual methods.
items = load_dataset_items("agent-core-benchmarks")
for item in items:
output = run_agent_workflow(item["input"])
record_run_result(item, output, run_name="pr-candidate-experiment")
Running every pull request against the same dataset makes it easier to tell a genuine prompt improvement from an unintended regression in a tool.
Step 3 — Online evaluation, annotation queues, and the evaluator toolbox
Online evaluation scores live production traces using automated LLM-as-a-judge evaluators, code evaluators, and user feedback, while annotation queues route outputs to human reviewers. The Evaluation Overview documents each of these paths, so you can combine automated and manual checks on production data and track quality trends over time.
The Evaluation Overview feature table and the Online Evaluation Get Started page point to these methods. The Core Concepts page explains how evaluators, scores, datasets, and experiments fit together.
- LLM-as-a-Judge: automatically scores live production traces. See the LLM-as-a-Judge docs.
- Code Evaluators: run deterministic checks. See the Code Evaluators docs.
- User Feedback: collects feedback from your end users. See User Feedback.
- Annotation Queues: let people review and rate traces manually, documented on the Annotation Queues page.
- Scores via API/SDK: let external scripts or pipelines send scores back to the platform. The Public API is the underlying interface.
+-------------------------------------------------------------------+
| Live Production Traces |
+---------------------------------+---------------------------------+
|
+------------------------+------------------------+
| | |
v v v
+------------------+ +------------------+ +-------------------+
| Code Evaluator | | LLM-as-Judge | | User Feedback |
| (Deterministic) | | (Automated) | | (End users) |
+--------+---------+ +--------+---------+ +---------+---------+
| | |
+------------------------+------------------------+
|
v
+-------------------------------------------------------------------+
| Scores / Manual Review |
| Optionally routed to Annotation Queues |
+-------------------------------------------------------------------+
A reasonable design is to use cheap deterministic checks for baseline sanity and reserve LLM judges for the criteria that need them. That is a design suggestion rather than a documented recommendation.
Step 4 — CI regression gating with GitHub Actions
CI regression gating adds the Langfuse GitHub Action to a pull request workflow. Your experiment script raises a RegressionError when a score violates your threshold, and the action then fails the job. This stops prompt or tool regressions from merging unnoticed.
The Evaluation Overview describes the mechanism, and the Experiments CI/CD page has the details. Two rules are documented in the Evaluation Overview:
- Add the
langfuse/experiment-actionGitHub Action to apull_requestworkflow. It works with Langfuse Cloud and self-hosted Langfuse. - Raise
RegressionErrorfrom your experiment script when a score violates your threshold. The action then fails the job.
# Illustrative skeleton only. Check the Experiments CI/CD page for the
# action's real version tag and inputs; none are shown here.
name: Agent Evaluation Gate
on:
pull_request:
branches: [main]
jobs:
evaluate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Langfuse experiment gate
uses: langfuse/experiment-action@<version>
env:
LANGFUSE_PUBLIC_KEY: ${{ secrets.LANGFUSE_PUBLIC_KEY }}
LANGFUSE_SECRET_KEY: ${{ secrets.LANGFUSE_SECRET_KEY }}
LANGFUSE_BASE_URL: ${{ secrets.LANGFUSE_BASE_URL }}
# with: <inputs per the action's documentation>
Inside the experiment script, compare scores with your baseline:
# Illustrative only. The docs name RegressionError but the sources here
# do not state its import path, so a local stub is defined.
class RegressionError(Exception):
"""Stub: use the error type the action documentation specifies."""
def verify_experiment_results(score: float, baseline: float):
if score < baseline:
raise RegressionError(
f"Quality regression detected: score {score} fell below baseline {baseline}"
)
Self-hosting the harness (Docker Compose, and when you’d outgrow it)
Running Langfuse locally or on a VM means cloning the repository, updating the secrets in docker-compose.yml, running docker compose up, and waiting about 2-3 minutes until the web container logs “Ready”. The docs call this the simplest way to try Langfuse and recommend Kubernetes for high availability.
The official Docker Compose deployment guide lists git and Docker with Docker Compose as requirements, and the steps are short:
git clone https://github.com/langfuse/langfuse.git
cd langfuse
# Update the secrets marked CHANGEME in docker-compose.yml
docker compose up
After about 2-3 minutes the langfuse-web-1 container should log “Ready”. Then open http://localhost:3000. There is no default login: click Sign up on the login page to create the first user, then create your organization and project in the UI (Docker Compose deployment).
For a VM, the guide recommends at least 4 cores and 16 GiB of memory, and enough storage for accumulating traces (Docker Compose deployment).
The guide states that the Docker Compose setup lacks high availability, scaling capabilities, and backup functionality, and that it does not support horizontal scaling without an additional load balancer. When load or uptime requirements grow, it points to the Kubernetes Helm guide.
Langfuse vs alternatives: which should you pick?
Choose Langfuse when you want tracing, online and offline evaluation, and a documented GitHub Action for CI gating in one open-source platform that runs on Cloud or self-hosted. Other tools may fit different hosting or workflow constraints, so weigh them against your own requirements.
Opik and LangWatch are the usual alternatives. Rather than duplicate feature breakdowns here, see the ToolBrain arena and the observability comparison guide.
If you prefer a managed environment, Langfuse Cloud pricing is tiered:
| Plan | Price | Units included | Overage | Data access | Users | Annotation queues |
|---|---|---|---|---|---|---|
| Hobby | Free | 50k units/month | — (not stated) | 30 days | 2 | 1 queue |
| Core | $29/month | 100k units/month | $8 per 100k extra | 90 days | Unlimited | 3 queues |
| Pro | $199/month | 100k units/month | $8 per 100k extra | 3 years | Unlimited | Unlimited |
| Teams add-on (optional on Pro) | $300/month | — | — | — | — | — |
| Enterprise | $2499/month | 100k units/month | $8 per 100k extra | 3 years | Unlimited | Unlimited |
| Self-hosted | See separate page | — | — | — | — | — |
Data source: Langfuse pricing. Overage rates are lower with volume. The Teams add-on lists Enterprise SSO, SSO enforcement, fine-grained RBAC, and a dedicated Slack/Teams support channel. Self-hosted pricing and enterprise support are on a separate pricing page.
Choose Langfuse if native GitHub Actions gating through langfuse/experiment-action and combined online and offline evaluation matter to you. If your needs are narrower, or you have hosting constraints, compare the alternatives in the guides above before committing.
Failure modes, tradeoffs, and honest caveats
An evaluation harness brings tradeoffs: judge model variance, dataset drift, the limits of a single-node Docker Compose deployment, and the cost of automated grading. Knowing these in advance helps you avoid trusting noisy quality signals or hitting infrastructure limits late.
This guide is assembled from official documentation and community reports — we did not run this build ourselves.
The first, second and fourth points below are general engineering considerations, not statements from the Langfuse documentation:
- LLM-as-a-Judge Variance: Evaluator models can be sensitive to prompt wording and may score non-deterministically. Failing builds on narrow margins can make CI flaky unless thresholds and rubrics are tuned carefully.
- Dataset Drift: A static offline dataset loses value as your agent and real-world inputs evolve. Plan to refresh it with edge cases found in production.
- Self-Hosted Infrastructure Limitations: The Docker Compose setup lacks high availability, scaling capabilities, and backup functionality (Docker Compose deployment). Running critical evaluation pipelines on it risks downtime.
- Judge Latency and Cost: Running LLM judges over every trace in a high-volume system can drive up model inference costs. Teams can combine deterministic code evaluators with sampling for the LLM judge passes.
Where to go next
To extend this harness, add coding-agent assistance for instrumentation, build custom reporting on top of the Public API, and compare Langfuse with other observability tools. Going beyond baseline regression testing lets you track quality across more of your agent’s real-world behavior.
Recommended next steps:
- Coding Agent Integration: Install the skill from the Langfuse skills repository to help automate trace instrumentation:
npx skills add langfuse/skills --skill "langfuse" - Custom Reporting Pipelines: Use the Langfuse Public API to pull data into internal dashboards or alerting.
- Deep-Dive Reading: Read the Langfuse review, the beginner’s guide to Langfuse, or browse the comparisons catalog.
📖 Related Reads
- NiteAgent — AI agent development, frameworks, and production patterns
- ToolBrain — tool reviews, LLM comparisons, and AI workflow guides
- CodeIntel Log — code quality, debugging, and software engineering benchmarks
- Hermes Tutorials — Hermes Agent setup, configuration, and advanced workflows
Cross-links automatically generated from None.
Back to all posts