Evaluation tests an AI system with representative tasks, expected properties and documented methods. A few demonstration prompts are not dependable evidence.
What the concept actually means
A test set contains normal, edge, failure and adversarial cases. Generative outputs are assessed using references, rubrics and expert pairwise comparison.
Why it matters in the enterprise
A baseline is recorded before changes to model, prompt, data or workflow. Regression tests then show whether local improvements create new failures.
A controlled method
The CTPM practice framework for controllable AI applications uses seven stages: understand the task, clarify context and data, apply AI deliberately, review professionally, handle deviations, approve accountably and document transfer. It is a transparent working framework, not a certification.
- Define task and impact
- Clarify data, context and permissions
- Review against domain criteria
- Control deviations, approval and evidence
CTPM practice example
CTPM practice example: Requirements extraction is tested on 60 document sections with known expected requirements. Evidence locations, omissions, invented requirements and correct uncertainty marking are measured.
Quality and test criteria
The following criteria make quality observable for this use case:
- Cases represent real frequency and impact.
- Baseline, model, prompt and data version are fixed.
- Domain reviewers are calibrated.
- Regressions block release under a clear rule.
Risks and common misconceptions
LLM-as-a-judge can scale review but inherits model bias and requires calibration. Synthetic tests do not replace difficult real cases.
Example transfer artefact
Transfer artefact: a versioned evaluation set with cases, expected criteria, scoring, failure taxonomy and release threshold.