A · Professional problem solving and implementation

AI evaluation and governance

Establishing evaluation, access controls, risk monitoring, human oversight, and accountability for AI systems.

Classification boundary

Include: Include AI evaluation design, release governance, risk tiers, human oversight, and accountability.

Choose a neighboring leaf when: Use AI engineering for model integration, retrieval, prompting, and tool implementation; use this leaf when governance judgment is primary. Specific controls may also involve quality, delivery, or responsibility capabilities.

This capability may appear in standards across different stages. Explore stage expectations

12 questions

Choose a group based on the evidence you need. Groups follow existing question formats, not difficulty or mandatory prerequisites. Each batch has up to eight questions.

Judgment and options12

Compare goals, constraints, options, and trade-offs.

Questions 1–8

Public benchmarks keep improving while internal task performance stalls. How would you investigate evaluation distortion?
The product wants confident AI conclusions, but evidence supports only probabilistic judgments. How would you design communication and traceability?
After AI output causes actual user harm, product, model, operations, and review teams all take part in the response. How would you divide incident handling and follow-up improvement?
After replacing a foundation model, overall metrics remain similar but some users report worse results. How would you identify what changed in model behavior and decide whether to roll back?
You need to sample production AI conversations for emerging risks, but they contain personal or commercial data. How would you design the review?
A third-party model provider changes its training, retention, or cross-border terms. When should calls pause or receive renewed approval?
Average AI quality meets the target, but a few high-risk errors can cause harm. How would you set the release gate?
An AI assistant can call internal tools, but tasks require different permissions. How would you design authorization and audit?

Questions 9–12

When the model cannot establish an answer, should the product refuse, return a low-confidence result, or escalate to a person? What distinguishes the choices?
Overall AI accuracy passes, but one language or user segment has a higher error rate. How would you assess and respond?
A new model scores much better on an offline test set, but a small production rollout shows lower task completion for one user group and more complaints. How would you determine where the evaluation set, metrics, or release decision failed?
The team needs real user conversations to reproduce model failures, but logs contain personal and sensitive business content. How would you minimize collection, redact data, authorize access, and set retention while preserving debugging value?

Back to the whole domain