TOPIC
Metrics and measurement
These are the same questions found in the library, grouped here by the capability each one mainly examines.
200 questions
Questions 1–50 of 200 on this page.
A · Professional problem solving and implementation11
- Two reviewers often disagree about the same set of AI outputs. How would you tell whether the issue is annotator disagreement, an unclear evaluation standard, or model instability?
- You built a customer-support Q&A prototype. How would you design an evaluation set to decide whether it may only assist retrieval, or can reply to customers directly?
- User satisfaction with a knowledge assistant has fallen, but offline retrieval metrics are unchanged. Which parts of the system would you investigate?
- Enterprise knowledge documents may change, be withdrawn, or become restricted, yet users still receive answers citing old content. How would you propagate changes from source documents through chunks, indexes, caches, and existing citations, and prove that an update or deletion took effect?
- A new model raises overall accuracy from 91% to 94%, but errors remain concentrated in cases that affect customer rights. The team wants to ship based on the aggregate score. How would you decide?
- A new index speeds up queries, but write latency and storage growth worsen noticeably. How would you decide whether to keep it?
- Public benchmarks keep improving while internal task performance stalls. How would you investigate evaluation distortion?
- After replacing a foundation model, overall metrics remain similar but some users report worse results. How would you identify what changed in model behavior and decide whether to roll back?
- Average AI quality meets the target, but a few high-risk errors can cause harm. How would you set the release gate?
- Overall AI accuracy passes, but one language or user segment has a higher error rate. How would you assess and respond?
- A new model improves offline evaluation, but answer quality may fall for a small set of high-value users. How would you stage a canary, observe it, and decide when to expand traffic?
No questions in this section match your target. ·
B · Business understanding and domain modeling20
- Renewal fell by six percentage points this quarter, while the share of small customers among new accounts rose. How would you segment customers, usage periods, and service cost to determine whether product value fell or the customer mix changed?
- Sales reports growth using signed contract value, but customers will not use the product for months. How would you distinguish contracting, fulfillment, revenue recognition, and customer value, and explain what the team should observe?
- A premium service tier earns more revenue but consumes more human support, and renewal has not improved. How would you determine whether it creates incremental value?
- A product is sold once but requires long-term support and upgrades. Revenue is growing while the service queue accumulates. How would you decide whether these sales still create value?
- A customer buys a bundle and then asks to refund one item. How should the original bundle discount be reallocated?
- Product, finance, and growth teams each report “active users,” but the numbers differ. What would you ask before deciding whether to standardize the definition?
- A business object moves across teams as owners change. Audit must reconstruct the owner on a past date, while daily reporting needs current ownership. How would you model both queries?
- In a behavioral event table, what exactly does one row represent? Explain how you settle its grain, and which statistics go wrong when the grain is unclear.
- A monthly close report was sent to leadership. Days later, a late refund and price correction arrive. The business wants current figures updated, while finance must explain the prior report. How would you model this?
- A business rule is corrected retroactively. Reports need corrected results, while decision-makers also need the results available at the time. How would you manage both historical views?
- A user orders through a channel and later receives a refund, chargeback, or cancellation. How would you define order value, attribution window, and channel cost so marketing analysis and finance settlement can both explain the same outcome?
- A new plan makes more users willing to try the product, but some customers who would have bought a higher-priced plan switch to the cheaper option. What commercial model and observation window would you use to judge whether the plan portfolio creates incremental value?
- The same person may encounter a product through ads, an agency, a partner, and a self-serve channel, and different parties may pay. How would you distinguish customer, buyer, user, and acquisition touchpoint in the business model?
- An enterprise customer has multiple organizations, departments, and members, and a member may act for different organizations. When designing customer activity analysis, how would you define the grain and relationships among customer, account, organization, and user?
- Support agents merge duplicate tickets and sometimes split a complex issue into several cases. When modeling service-quality data, how would you define the issue, ticket, contact, and resolution?
- Business units use an “active customer” metric, but some define it by payment, others by usage, and others by likelihood to renew. How would you decide which definitions should be standardized and which should remain distinct concepts?
- Recommendation and risk teams both need a “spend in the last 30 days” feature, but one uses calendar days and the other a rolling hour window, and refund handling differs. How would you decide whether the definition can be shared?
- A risk value is usually present in training data, but live serving often substitutes a default and model performance fluctuates. How would you find whether the cause is data collection, timing, missing-value handling, or implementation differences?
- Models, operational rules, and reports share a core feature, but the upstream business definition has changed and some consumers cannot migrate yet. How would you manage old and new definitions, ownership, and migration order?
- An upstream dependency change will increase refresh latency for a costly feature set from seconds to ten minutes. How would you determine which decisions are affected, which features must remain real time, and when a fallback value is acceptable?
No questions in this section match your target. ·
C · Value judgment and direction setting11
- Visits and registrations are growing, but match success is falling. Demand concentrates in a few regions while suppliers are scattered. How would you find the liquidity bottleneck and choose where to invest first?
- A marketplace raises commission by two points. Revenue rises short term, but some high-quality suppliers list less often; buyer conversion has not moved yet. How would you assess long-term impact?
- After a subscription price increase, monthly revenue rises 8%, but cancellations grow among small core customers. Sales wants to continue. How would you estimate net benefit and long-term risk?
- Gross margin is steady, but standard-service revenue falls while custom-service revenue grows, along with implementation and support hours. How would you determine whether growth is healthy?
- An acquisition campaign drives many registrations, but some regions cannot yet be served, reducing the share of new users who can actually receive the service. How would you define and report the campaign result?
- Brand advertising, search, and affiliate channels reach overlapping prospects, while the attribution platform assigns most conversions to the last click. How would you estimate each channel’s incremental contribution and state the limits of the conclusion?
- A checkout experiment improves payment completion, but complaints and refunds also increase. How would you judge whether the experiment succeeded, and how long would you wait before deciding?
- Product, operations, and sales reports all say “retention,” but refer to return visits, key-feature use, and renewal. How would you clarify the measures so each name maps to a user behavior and business decision?
- A membership program increases repeat-purchase frequency, but average order value and margin both decline. How would you distinguish a healthy habit change from purchases pulled forward or low-quality, promotion-driven growth?
- A new automated workflow doubles processing speed and reduces human review, but errors have a greater impact on a small group of high-value customers. How would you assess net value and decide whether to expand it?
- A major customer will pay a high price for a custom feature, but reuse by other customers is uncertain. Ongoing maintenance, permissions, and data rules add cost, and sales has promised a delivery date. How would you compare a one-off build, productizing it, a service, or declining—and decide whether to invest?
No questions in this section match your target. ·
D · Delivery and ongoing operations7
- Project requests backed by senior sponsors tend to move ahead, while compliance and infrastructure work without executive sponsorship repeatedly slips. How would you create a transparent portfolio intake and prioritization process?
- An onboarding experiment does not change overall signup-to-activation conversion, but activated users complete their first key task sooner. How would you decide whether to roll it out and revise the definition of success?
- A brand campaign has delivered its planned total impressions, but actual reach among the core target audience is below expectation, and one-third of the budget remains. How would you decide whether to continue, change the media mix, or revise the objective?
- A community event has strong registration and interaction numbers, but participants rarely continue using the product or contributing content afterward. How would you locate where the connection to product value breaks and design the next operating cycle?
- A campaign creative gets high engagement on social channels but almost no conversion in search or stores. Another version has ordinary engagement but stronger sales. How would you decide on creative iteration and channel allocation?
- A broad discount meets the overall sales target, but valuable existing customers view it as unfair, and some users delay purchases they would have made anyway. How would you adjust future audiences and offers?
- A discount briefly increases volume, competitors follow, and category prices and margins decline. How would you decide whether to keep matching, change the value proposition, or exit the promotion?
No questions in this section match your target. ·