AI Product Quality Plan the rollout

Evals for Business AI: What and How to Check

Understanding what to test in an AI system—not just “does it work,” but “does it work correctly and safely.” We break down an architectural approach to evals: from goals to checkpoints.

Discuss implementation All Academy Topics

What are “evals” in business AI?

Evals (evaluation) are systematic checks of how the AI system copes with business tasks: accuracy of decisions, safety, compliance with rules, UX. Unlike technical testing (unit/integration), evals focus on business relevance: «Does this solve the client’s problem?»

Architecture of evals: 5 levels of control

flowchart TD  
    A[Business Goal] --> B[Data: Representativeness, Balance]  
    B --> C[Roles: Who Approves, Who Reviews]  
    C --> D[API: Input/Output Validation, Fallback]  
    D --> E[Control: Metrics, Audit, Human-in-the-Loop]  

    subgraph "Quality System"  
        B & C & D & E  
    end  

    E --> F[Decision: Launch / Fix / Rollback]

Example: Customer support chat agent

Business goal: Reduce repeat contacts by 30%. Evaluations at this level check not only whether the agent responds, but also:

Does the solution repeat (if the client asks the same thing—does the agent give the same answer)?
Does the response comply with company policy (e.g., does it not promise a refund without verification)?
Does the agent offer a “human” fallback when uncertain?

The unit test will verify that the `generate_response()` method returns a string. The eval will check that the response solves the client’s problem.

Pseudocode: Simple eval check

def evaluate_response(agent_response, expected_outcome, policy_rules):
    # 1. Policy compliance check
    if not is_policy_compliant(agent_response, policy_rules):
        return {"status": "fail", "reason": "policy_violation"}

    # 2. Solution completeness check (based on query metadata)
    if not resolves_issue(agent_response, original_query):
        return {"status": "fail", "reason": "incomplete_solution"}

    # 3. Fallback behavior check
    if confidence_score(agent_response) < 0.7 and not suggests_human_review():
        return {"status": "fail", "reason": "missing_fallback"}

    return {"status": "pass", "confidence": confidence_score(agent_response)}

This eval can be run on historical queries, synthetic scenarios, and A/B tests.

Common mistakes

Accuracy metrics only: 95% accuracy—good, but if 95% is “I don’t know,” it’s a failure.
Tests on ideal data: evals on “clean” questions, but in production—Cyrillic + typos + emotional queries.
Missing fallback tests: they don’t verify whether the agent is attempting to “guess” when confidence is low.
Ignoring Roles: Eval’s is developed by the developer, but the product owner does not participate—and the result does not solve the business problem.

Checklist: What to Verify Before Launch

Level	What to check	How
Data	Representativeness, balance, drift	Scenario distribution analysis, time-based split
Roles	Who asserts, who verifies, who is responsible for regression?	RACI matrix for evaluation scenarios
API	Input validation, fallback, logging	Edge-case tests: empty query, non-JSON, toxic input
Control	Quality metrics, audit, human-in-the-loop	Dashboard for evals + manual sampling of 5% of results

Next step

Go to Roles and Responsibilities in AI Implementation — Without a clear distribution of eval functions, the quality system will collapse on the first incident. Or consider this: Human-in-the-loopto understand where and how to involve a person in the verification cycle.