A · Professional problem solving and implementation
AI engineering
Evaluation, retrieval, agents, and the engineering limits of model-backed features.
Classification boundary
Include: Include implementation of retrieval, prompting, tool use, agents, and inference services in products.
Choose a neighboring leaf when: When the core evidence is evaluation ownership, governance, risk approval, or human oversight, use the relevant quality, delivery, or responsibility capability.
This capability may appear in standards across different stages. Explore stage expectations
47 questions
Choose a group based on the evidence you need. Groups follow existing question formats, not difficulty or mandatory prerequisites. Each batch has up to eight questions.
Judgment and options44
Compare goals, constraints, options, and trade-offs.
Questions 1–8
- An AI assistant can create tickets, issue refunds, or change accounts on a user's behalf. A refund request times out, and the system cannot tell whether it completed; retrying may issue a second refund. How would you check the outcome, control retries, and ensure actions stay within the user's authorization?
- Two reviewers often disagree about the same set of AI outputs. How would you tell whether the issue is annotator disagreement, an unclear evaluation standard, or model instability?
- You built a customer-support Q&A prototype. How would you design an evaluation set to decide whether it may only assist retrieval, or can reply to customers directly?
- What does an algorithm model and the features it depends on truly cost? How do you count training, inference, feature computation and storage, labeling, and maintenance, and how do you judge whether a feature is worth keeping?
- A document assistant can submit most cases automatically, but mistakes have very different consequences. Human review has reduced errors, but the queue keeps growing. How would you decide which cases to automate or review, and keep the queue within the team’s capacity?
- A generative feature produces good answers, but peak latency and per-request cost exceed the product budget. How would you determine whether the cause is the model, context, retrieval, or call chain?
- After a foundation-model change, the average score improves, but outcomes worsen for a few important customers. How would you decide whether to ship, stage, or roll back?
- Your company wants an internal knowledge assistant, and document permissions change as people move around. How would you make it answer with evidence while keeping it from exposing what a user should not see?
No questions in this section match your target. ·
Questions 9–16
- User satisfaction with a knowledge assistant has fallen, but offline retrieval metrics are unchanged. Which parts of the system would you investigate?
- You ask an AI to review a configuration or generate a piece of code. How do you decide whether it can go live?
- Enterprise knowledge documents may change, be withdrawn, or become restricted, yet users still receive answers citing old content. How would you propagate changes from source documents through chunks, indexes, caches, and existing citations, and prove that an update or deletion took effect?
- Production answers contain incorrect citations or degrade in quality, but prompts and retrieval context may include sensitive data and cannot be logged by default. What telemetry would help distinguish retrieval, model, and tool failures while controlling data exposure?
- A new model raises overall accuracy from 91% to 94%, but errors remain concentrated in cases that affect customer rights. The team wants to ship based on the aggregate score. How would you decide?
- A RAG assistant cites an internal document, but its owner has updated the process and the new version is not indexed yet. How would you define freshness goals, propagation, and answer behavior during the gap?
- A multi-step agent reads project permissions at startup. Before step three, the user is removed from the project, but the task still holds a tool token. How would you design authorization checks and cancellation semantics?
- After changing an embedding model, old and new vectors are not directly comparable, while a full rebuild takes days. How would you design migration and query cutover?
No questions in this section match your target. ·
Questions 17–24
- A team discovers that a model may have seen some internal evaluation prompts in public corpora or fine-tuning data. How would you determine what offline scores still mean and rebuild a trustworthy evaluation?
- A team fine-tunes on user feedback, human labels, and synthetic samples, then receives a request to delete some source data. How would you trace impact and decide the scope of retraining?
- A new model improves offline evaluation, but answer quality may fall for a small set of high-value users. How would you stage a canary, observe it, and decide when to expand traffic?
- A product depends on an external model API, and the provider may restrict request frequency or change available models. How would you design switch rules and temporary shutdowns for noncritical features so quality changes after switching are visible?
- Offline evaluations keep improving while a specific user segment regresses online. Which data leakage, slice coverage, and deployment differences would you investigate?
- An AI assistant repeatedly calls search, databases, and external APIs for a small share of requests, making latency and cost far higher than average. How would you design per-task budgets, stop conditions, idempotency, and auditable execution records?
- An AI assistant retrieves documents from several business systems, where users have different permissions in each. How would you ensure retrieval, caching, summaries, and citations follow source permissions, including after access is revoked?
- A model service used by an AI feature has highly variable latency, causing user requests to exceed the end-to-end deadline. How would you allocate budgets across retrieval, model calls, and post-processing, and design fallbacks after a timeout?
No questions in this section match your target. ·
Questions 25–32
- A user submits a long AI analysis, closes the page, and later checks from another device; the task may call several tools and produce partial results. How would you design task state, cancellation, retry, and result visibility?
- An analytics assistant converts natural language into SQL against business data; a request may induce it to read sensitive columns or run an expensive query. How would you design read-only permissions, query validation, resource limits, and result redaction?
- A model begins returning structured JSON, while mobile, server, and analytics consumers upgrade at different rates; new fields may be absent or enum values may change. How would you define the output contract, handle validation failures, and preserve compatibility?
- An AI assistant reads user-uploaded documents that may tell it to ignore existing rules and call external tools. How would you distinguish data from instructions, constrain tool permissions, and verify the safeguards?
- A model has streamed partial output to a user when the connection drops or a safety check fails; the client may make the answer look complete. How would you design completion markers, withdrawal or explanatory notices, and retry behavior?
- Overall task success is unchanged after a model update, but outcomes diverge more across users by language, device, and experience. How would you define evaluation segments, manage small-sample risk, and decide whether to continue rollout?
- Two currently valid policies in the knowledge base conflict, with similar retrieval scores. The assistant must not choose whichever is more convenient. What response and escalation path would you design?
- A new assistant improves satisfaction ratings, but task completion and repeat-contact rates remain unchanged. How would you determine whether this is a real improvement?
No questions in this section match your target. ·
Questions 33–40
- Model responses may echo customer identifiers, while production debugging relies on request and response logs. How would you design collection, redaction, access, and retention?
- The team tunes prompts and model settings against the same fixed evaluation set every week. Scores rise, but new-user feedback does not improve. How would you restore evaluation trustworthiness?
- A model passes evaluations in a major market but behaves inconsistently in a new region because language, terminology, and service rules differ. How would you set rollout scope and follow-up gates?
- To reduce inference cost, the team caches answers by prompt text. User permissions, knowledge versions, and regional rules may differ. How would you define cache keys and invalidation?
- The evaluation set has grown, but many items are slight rewrites of one template, making model scores look stable. How would you audit sample quality?
- A natural-language request omits an amount limit and target account, while the agent can invoke a side-effecting tool. How would you decide whether to clarify, advise only, or require confirmation?
- A template changes only formatting and examples, but output length and tool-call frequency shift. The team wants to skip full regression. How would you tier the risk?
- A new safety policy reduces false refusals but causes sensitive disclosures in a few edge cases. Product wants to keep the satisfaction gain. How would you evaluate?
No questions in this section match your target. ·
Questions 41–44
- A hosted model upgrades without notice. Standard health checks pass, but classification boundaries and output length shift. How would you build safeguards?
- After a safety policy launch, human handoffs double while high-risk errors fall. Operations capacity is limited. How would you assess the result and tune thresholds?
- A model labels some accounts high-risk and automatically denies service, affecting customer rights. How would you set validation, appeals, and human review?
- The business changes “churn” from 60 days without login to 30 days without payment, while historical training and models use the old definition. How would you manage versions and comparisons?
No questions in this section match your target. ·
Implementation and verification2
Examine execution, verification, and correction.
Questions 1–2
- Users say answers sound plausible but omit a critical constraint present in the documents. How would you locate whether parsing, chunking, retrieval, ranking, or generation caused the failure?
- Users see response times double, but model inference time has barely changed. How would you break down the request-processing stages, locate the bottleneck, and limit impact?
No questions in this section match your target. ·
Experience and reflection1
Use past facts to examine contribution, results, and learning.
Questions 1–1
No questions in this section match your target. ·
No questions in this capability match your target. ·