llm_eval #

A test harness for LLM outputs in Dart. Write eval cases the way you write unit tests: a prompt, a list of checks, and a report you can read in CI.

llm_eval does not talk to any model provider. You hand it one function that takes a prompt and returns the model output, and it handles running cases concurrently, checking outputs, caching responses, and reporting.

Why this instead of what you already have #

Instead of plain package:test. You can already call a model inside a test and assert on the reply. What that leaves you is everything around it: the second run costs another call and can come back different. llm_eval keys each prompt to a file on disk (FileResponseCache, lib/src/file_response_cache.dart:18), so the rerun is free and byte-identical. EvalReport.toJUnitXml() (lib/src/eval_report.dart:203) writes a report your CI already renders, and BaselineCase (lib/src/baseline.dart:6) names the case that flipped since the last run you accepted.

Instead of eval. It has the wider matcher set, including RAG scoring and statistics, and it is the closest package on pub.dev to this one. It calls the provider every time: apiCallImpl posts straight to the API with nothing in front of it (lib/src/services/service.dart:169), and eval() reruns the whole test function once per numberOfRunsPerLLM (lib/src/eval_base.dart:91). Grep its 5,167 lines of lib/ for cache, junit, xml, or baseline and all four return nothing.

Reach for it when

• You have prompts in production and nothing that goes red when one of them regresses.
• The suite has to run on every pull request, so it cannot cost money or return different text each time.
• You want the result in the CI UI as a JUnit file, not in scrollback.

Skip it if you call a model in exactly one place, where a contains check in a test you already have does the same job without a cache directory to commit.

Why #

Dart and Flutter apps that call LLMs usually have no automated way to catch a prompt regression. Model output is not exact, which rules out plain string equality in a unit test. This package covers the middle ground. It gives you assertion-style checks for the properties that must hold. For the fuzzy parts, there is an optional LLM judge. A response cache keeps CI runs deterministic and free.

Quick start #

import 'package:llm_eval/llm_eval.dart';

Future<void> main() async {
  // Bind your model. Any Future<String> Function(String prompt) works:
  // an OpenAI or Anthropic SDK call, an Ollama HTTP request, a fake.
  Future<String> model(String prompt) async {
    // ... call your model client here ...
    return 'The capital of France is Paris.';
  }

  final suite = EvalSuite([
    EvalCase(
      id: 'capital-question',
      prompt: 'What is the capital of France?',
      checks: [
        Check.contains('paris'),
        Check.notContains('berlin'),
      ],
    ),
    EvalCase(
      id: 'structured-output',
      prompt: 'Return a JSON object with a "city" field.',
      checks: [
        Check.isValidJson(where: (v) => v is Map && v.containsKey('city')),
      ],
    ),
  ]);

  final report = await suite.run(model, modelId: 'my-model');
  print(report.toMarkdown());
}

Three runnable examples ship with the package:

• dart run example/llm_eval_example.dart shows the shape of a suite.
• dart run example/ci_gate.dart is the whole CI story in one file: a cached suite, a Markdown report, a JUnit file, and an exit code. Run it twice and the second run stops calling the model. It is deliberately red, because a green example teaches you nothing about what a failure looks like in your CI UI.
• dart run example/judge.dart puts a judge on three outputs and lands one of each verdict: a score above the threshold, a score below it, and a judge reply with no score in it at all.

Checks #

Custom checks implement the Check interface: a description for reports and an evaluate method that returns a CheckResult.

Fails and errors are distinct throughout. A fail means the check ran and the output did not satisfy it. An error means no verdict was possible (the judge response could not be parsed, a callback threw, the model call failed). Reports show them separately, so a broken harness is not mistaken for a failing model.

LLM as judge #

For properties that plain checks cannot express, ask another model to grade the output against a rubric:

Check.judge(
  judge: judgeModel, // often a stronger model than the one under test
  rubric: 'The answer names Paris and stays under three sentences.',
  passAt: 0.7,
)

The judge receives the rubric and the output in a fixed prompt and must answer with a SCORE: <number> line between 0.0 and 1.0. A response that cannot be parsed becomes an error result, never a silent pass or fail. So does a response with conflicting score lines, which is what a graded output that smuggles in its own SCORE: 1.0 line tends to produce. The graded output is wrapped in delimiters the judge is told to respect; this raises the bar against prompt injection without eliminating it.

example/judge.dart runs all three outcomes against a fake judge, with no key and no network: a score above passAt, a score below it, and a reply that never produces a score line.

One honest caveat: the judge is itself an LLM. Its scores are not calibrated, they drift across judge models and versions, and they can be wrong. Use judges sparingly, pin the judge model, and spot-check its verdicts against your own reading.

The judge is a nested model call that suite.run does not cache: on a warm cache the model under test is skipped but an unwrapped judge fires on every run. Wrap the judge in the same cache so it is cached too:

Check.judge(
  judge: cache.wrap(judgeModel, modelId: 'judge-v1'),
  rubric: 'The answer names Paris and stays under three sentences.',
)

cache.wrap uses the same key scheme as the suite: the judge's responses sit alongside the model responses in one cache directory. Give each judge its own modelId so pinning or changing a judge re-records only its own responses.

Caching and CI #

The core library is pure Dart and runs on every platform, including the web. FileResponseCache needs dart:io and lives in a separate library:

import 'package:llm_eval/llm_eval.dart';
import 'package:llm_eval/io.dart' show FileResponseCache;

final cache = FileResponseCache('test/llm_cache');
final report = await suite.run(model, cache: cache, modelId: 'my-model-v1');

The first run calls the model and stores each response in a file named after the SHA-256 of the model id and prompt. Later runs read the cache and never call the model. Commit the cache directory and your CI eval job is deterministic, offline, and free. Delete the directory, or change modelId, to re-record.

The cache covers the model under test. A nested call, such as the judge in a Check.judge, is not cached by suite.run, so a warm cache still calls the judge unless you wrap it with cache.wrap(judgeModel, modelId: ...) (see LLM as judge). The Markdown report's cached column likewise describes the model under test rather than any nested judge.

A regression test then looks like any other test:

test('prompt regression suite', () async {
  final report = await suite.run(
    model,
    cache: FileResponseCache('test/llm_cache'),
    modelId: 'my-model-v1',
  );
  expect(report.results, isNotEmpty);
  expect(report.errorCount, 0, reason: report.toMarkdown());
  expect(report.passRate, 1.0, reason: report.toMarkdown());
});

The same shape works as a standalone CI gate:

final report = await suite.run(model, cache: cache, modelId: 'my-model-v1');
stdout.writeln(report.toMarkdown());
if (report.results.isEmpty || report.errorCount > 0 || report.passRate < 1.0) {
  exitCode = 1;
}

Check errorCount separately from the pass rate: errors mean the harness could not produce a verdict (a judge response failed to parse, a callback threw), not that the model answered badly. Note that an empty suite has a pass rate of 1.0; guard against accidentally building zero cases, as both snippets above do.

Showing results in the CI UI #

An exit code tells the build to go red; toJUnitXml tells it which cases went red and why. Write the report where your CI looks for test results and each eval case appears as a test, failing ones expanded to the checks that failed and the model output that failed them:

File('eval-results.xml').writeAsStringSync(report.toJUnitXml());

# GitHub Actions
- run: dart run tool/eval.dart
- uses: dorny/test-reporter@v1
  if: always()
  with:
    path: eval-results.xml
    reporter: java-junit

GitLab, Jenkins, CircleCI and Buildkite read the same format. A case with a model error or an errored check becomes an <error>, any other non-passing case a <failure>, and a flaky case counts as failing with a message saying how many attempts passed. Model output is arbitrary text, so it is escaped and characters XML cannot carry are dropped; a single stray control byte would otherwise make the report unreadable to the CI system.

This repository runs that step against itself. tool/eval.dart evaluates a small local model, its cache is committed under tool/eval_cache/, and the GitHub runner has no model to call and no key to call it with: the job replays the recorded responses, writes both reports, and fails if a fixture is missing. The tool is replay-only unless you pass --record, so a cache miss goes red instead of quietly reaching for a model. See .github/workflows/ci.yaml.

Catching a regression a pass rate hides #

A threshold on the pass rate cannot see composition. Nine of ten passing before and nine of ten passing now is the same number whether nothing moved or one case broke while another was fixed. Deleting the case that was failing moves the rate the same way repairing it does.

A baseline keeps the identity of what passed, and both of those read as what they are:

tool/baseline_figure.dart draws that from a diff it computes as it runs, and it refuses to write the file if the two runs stop sharing a pass rate.

final report = await suite.run(model, modelId: 'gpt-4o-mini');

final file = File('test/eval_baseline.json');
if (!file.existsSync()) {
  file.writeAsStringSync(EvalBaseline.fromReport(report).toJsonString());
  return; // first run records, later runs compare
}

final diff = diffAgainstBaseline(
  report,
  EvalBaseline.parse(file.readAsStringSync()),
);
stdout.write(diff.toMarkdown());
if (diff.hasRegressions) exitCode = 1;

hasRegressions stops for four things: a case that stopped passing, a check whose score fell past scoreTolerance while the case still passed, a case that was steady and now disagrees between attempts, and a case that is in the baseline and missing from the run. Fixes and new cases are reported and do not stop the build.

Commit the baseline next to your tests. Re-record it in the same commit that explains why the numbers moved, which keeps the file honest about being a decision rather than a leftover.

A model swap is reported rather than refused, since a deliberate swap is exactly when the diff is worth reading:

## Baseline diff

Model changed: `gpt-4o-mini` to `gpt-4o`.

### Regressions (1)

- `refund-policy`: was passing, now failing

### Fixed (2)
...

Repeat and flakiness #

final report = await suite.run(model, repeat: 5);
print(report.flakinessRate);

Each case runs five times and the report exposes the fraction of cases whose attempts disagree. Measure flakiness without a cache: a warm cache returns the same response every time.

Reports #

EvalReport.toMarkdown() renders a summary table plus details for every non-passing case, suitable for a CI job summary. EvalReport.toJson() returns a JSON-compatible map with the full result tree (outputs, per-check verdicts, scores, latencies, cache hits) for your own tooling.

Alternatives #

Three other pub.dev packages cover nearby ground, and for some projects one of them is the better answer. The table below was filled in on 2026-08-08 by reading the published source of eval 0.0.5, vouch 0.1.0 and llm_replay_eval 0.1.0, not their descriptions.

eval is the closest neighbour: pure Dart, built on package:test, with matchers for strings, JSON, schemas, frontmatter, edit distance, judges and RAG. Because an eval there is a Dart test, you get CI reporting from package:test's own reporters and need nothing extra. It also does two things this package does not: comparing several models or prompt variants over one suite, and eval statistics with a declared winner. It has no response cache, so every rerun calls the model again.

vouch freezes a baseline of a run and diffs later runs against it. That shows what changed when you swapped the model, which llm_eval cannot do today. It is Flutter-only and layers on llm_replay_eval.

llm_replay_eval records and replays at the on-device inference boundary, for in-process Flutter models where there is no HTTP call to intercept. llm_eval's cache wraps whatever function you hand it, which covers a provider API call but is not tied to on-device inference. It is Flutter-only.

Pick llm_eval when the eval is a standalone build step in a Dart project: dart run tool/eval.dart, a committed cache, an exit code, and a JUnit file the CI UI can read, with no Flutter and one dependency. Pick eval when you would rather write matchers inside the dart test suite you already have, or you need to compare models. Pick llm_replay_eval, with vouch on top, when the model runs on the device.

Planned #

Out of scope for 1.0 and planned for later releases:

• side-by-side comparison of several models over one suite
• token and cost accounting
• dataset loaders for existing eval formats
• structured prompts: system messages and multi-turn conversations

License #

MIT