# Test the answer. Then test the action.

A convincing response is not enough. Useful evaluation checks the facts, the sources and the resulting request.

Published 2026-10-12 · Welvia

## Start with ordinary questions

Build a short set of questions that reflect real customer needs: what do you offer, where do you deliver, how much notice is required, and what happens if the customer leaves out a detail? Include a request that should succeed and one the business should decline or route to staff.

Write the expected behavior before running the test. Otherwise, it is tempting to accept an impressive answer without noticing that it missed a constraint. For a delivery request, a wrong postcode can matter more than polished wording.

## Keep the comparison fair

Use the same questions and agent settings when comparing the current information with an improved version. Preserve the source snapshots and inspect the actual answers. If the baseline already answers a question correctly, count that success. Do not make the old website artificially worse to produce a larger improvement.

Anthropic’s evaluation guidance distinguishes tasks, repeated trials, grading and actual outcomes. Its advice is especially relevant here: model behavior can vary, so a single successful attempt is not a reliability guarantee. Simple automated checks also need human review.

## Turn feedback into another test

An owner should be able to point to an answer and say which fact needs correcting. Store that feedback with the question and report. After the correction is approved, create a new information version and rerun the relevant cases. Keep the older results for comparison.

Welvia’s Juniper pilot uses controlled source snapshots and isolated inquiry preparation. That helps test the workflow, but it does not establish visibility across public search engines or all consumer assistants. Those require separate checks. Good quality reporting makes the remaining gaps visible instead of turning every green indicator into a claim about the whole internet.

## One thing to try

Review both the customer-facing answer and the saved outcome. Repeat tests and retain failures as evidence.

## Sources

- [Anthropic: Demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents)