Closed-loop AI evaluation and agentic testing

AI systems need this more than conventional software does. Outputs vary between runs, so one successful trial proves little. There is usually no one correct string, so correctness must be written as criteria. And they fail softly: a wrong answer arrives in the same confident tone as a correct one. The AI agents versus chatbots page covers the difference in expectations.

• Case generation. A generator agent writes inputs across the range the real system will meet: partial information, contradictory requests, unusual formats.
• Grading. A separate agent judges each output against explicit written criteria, so the verdict is reproducible and reviewable.
• Adversarial probing. A third role hunts for inputs that push the agent outside its scope, away from its instructions, or into confident guessing.

A grader is itself a model and can be wrong. So the criteria are written down before grading begins, and its judgments are sampled against human review.

Simulation drives the system with agents playing users: one gives requirements piecemeal, one contradicts an earlier answer, one asks for something out of scope. Context gets dropped, or an earlier constraint is forgotten. Failures that only appear on turn six are invisible to single-turn testing and obvious under simulation. Conversational systems are covered on the agentic chat assistant page.

1. 1

   Capture the whole trace

   Input, context retrieved, tools called, intermediate outputs, final answer. A final answer alone is not diagnosable.
2. 2

   Classify the cause

   Retrieval returned the wrong material, routing sent the work to the wrong step, or the instructions were ambiguous. The model may also have reasoned badly over correct inputs. Each cause has a different fix.
3. 3

   Fix the cause, not the symptom

   Patching a prompt to cover a retrieval problem just moves the failure.
4. 4

   Freeze the case as a regression test

   So that failure cannot quietly return.
5. 5

   Re-run the full suite

   A change that fixes one class of input often degrades another.

A failure can only be attributed to its cause if the handoffs are inspectable. That is why Fig AI Systems designs multi-agent orchestration around structured handoffs. Retrieval changes are measured on the vector databases and retrieval side.

A review gate pauses a workflow, shows the reviewer the output and the evidence, and records the decision. That record shows whether the written criteria match what an expert wants. It feeds straight back into the grading criteria. Over time the gates can narrow to the cases that genuinely need judgment.

Skill-enabled agents are mapped to a domain-specific corpus, so judgments are anchored to the criteria that apply. DOCX and PDF parsing with segmentation channels parallel LLM calls at high concurrency. Human-in-the-loop grading keeps inspector input where it matters.

Either way, write down what a good output looks like before building. Those sentences become the grading criteria later. More questions of this kind are answered on the Fig AI Systems FAQ.