TOPIC
AI engineering
These are the same questions found in the library, grouped here by the capability each one mainly examines.
76 questions
Questions 51–76 of 76 on this page.
A · Professional problem solving and implementation20
- A model begins returning structured JSON, while mobile, server, and analytics consumers upgrade at different rates; new fields may be absent or enum values may change. How would you define the output contract, handle validation failures, and preserve compatibility?
- An AI assistant reads user-uploaded documents that may tell it to ignore existing rules and call external tools. How would you distinguish data from instructions, constrain tool permissions, and verify the safeguards?
- A model has streamed partial output to a user when the connection drops or a safety check fails; the client may make the answer look complete. How would you design completion markers, withdrawal or explanatory notices, and retry behavior?
- Overall task success is unchanged after a model update, but outcomes diverge more across users by language, device, and experience. How would you define evaluation segments, manage small-sample risk, and decide whether to continue rollout?
- The team needs real user conversations to reproduce model failures, but logs contain personal and sensitive business content. How would you minimize collection, redact data, authorize access, and set retention while preserving debugging value?
- Two currently valid policies in the knowledge base conflict, with similar retrieval scores. The assistant must not choose whichever is more convenient. What response and escalation path would you design?
- A new assistant improves satisfaction ratings, but task completion and repeat-contact rates remain unchanged. How would you determine whether this is a real improvement?
- Model responses may echo customer identifiers, while production debugging relies on request and response logs. How would you design collection, redaction, access, and retention?
- Users see response times double, but model inference time has barely changed. How would you break down the request-processing stages, locate the bottleneck, and limit impact?
- The team tunes prompts and model settings against the same fixed evaluation set every week. Scores rise, but new-user feedback does not improve. How would you restore evaluation trustworthiness?
- A model passes evaluations in a major market but behaves inconsistently in a new region because language, terminology, and service rules differ. How would you set rollout scope and follow-up gates?
- To reduce inference cost, the team caches answers by prompt text. User permissions, knowledge versions, and regional rules may differ. How would you define cache keys and invalidation?
- The evaluation set has grown, but many items are slight rewrites of one template, making model scores look stable. How would you audit sample quality?
- A natural-language request omits an amount limit and target account, while the agent can invoke a side-effecting tool. How would you decide whether to clarify, advise only, or require confirmation?
- A template changes only formatting and examples, but output length and tool-call frequency shift. The team wants to skip full regression. How would you tier the risk?
- A new safety policy reduces false refusals but causes sensitive disclosures in a few edge cases. Product wants to keep the satisfaction gain. How would you evaluate?
- A hosted model upgrades without notice. Standard health checks pass, but classification boundaries and output length shift. How would you build safeguards?
- After a safety policy launch, human handoffs double while high-risk errors fall. Operations capacity is limited. How would you assess the result and tune thresholds?
- A model labels some accounts high-risk and automatically denies service, affecting customer rights. How would you set validation, appeals, and human review?
- The business changes “churn” from 60 days without login to 30 days without payment, while historical training and models use the old definition. How would you manage versions and comparisons?
No questions in this section match your target. ·
B · Business understanding and domain modeling3
- New users have little behavioral history, and the feature pipeline fills missing values with a global average, making “not observed” indistinguishable from “average behavior.” How would you represent cold-start status and evidence boundaries?
- A new model requires additional features and changes the meaning of existing ones, while online serving must still support rollback to the old model. How would you version feature contracts, deployment order, dual-read cost, and retirement criteria?
- After a feature definition is fixed, the modeling team wants a historical backfill. Training reproducibility requires the original version, while business reports want corrected values. How would you design both views?
No questions in this section match your target. ·
C · Value judgment and direction setting2
- An AI assistant increases apparent task-completion rates, but users recheck every result and some errors cause downstream rework. How would you judge whether automation actually lowers total task cost?
- The team can move all requests to a higher-performing large model, doubling average cost and increasing tail latency; a smaller model already handles most simple requests. How would you decide on tiered routing, budgets, and quality boundaries?
No questions in this section match your target. ·