C · Value judgment and direction setting
Metrics and causal reasoning
Metric definitions, attribution, and the limits of a conclusion.
Classification boundary
Include: Include metric definitions, comparisons, attribution, experiments, and causal evidence.
Choose a neighboring leaf when: Use business outcomes/validation when the focus is goals, baselines, guardrails, and decisions after measurement.
This capability may appear in standards across different stages. Explore stage expectations
43 questions
Choose a group based on the evidence you need. Groups follow existing question formats, not difficulty or mandatory prerequisites. Each batch has up to eight questions.
Concepts and boundaries2
Check concepts, assumptions, and conditions of use.
Questions 1–2
- How is your core metric defined? What are the numerator, denominator, time window, de-duplication, and exclusion rules, and why is it the core metric?
- How was your funnel built? Why are the steps divided the way they are, where does most of the drop-off happen, and how do you confirm the steps neither overlap nor leave gaps?
No questions in this section match your target. ·
Judgment and options40
Compare goals, constraints, options, and trade-offs.
Questions 1–8
- An acquisition campaign drives many registrations, but some regions cannot yet be served, reducing the share of new users who can actually receive the service. How would you define and report the campaign result?
- Brand advertising, search, and affiliate channels reach overlapping prospects, while the attribution platform assigns most conversions to the last click. How would you estimate each channel’s incremental contribution and state the limits of the conclusion?
- A checkout experiment improves payment completion, but complaints and refunds also increase. How would you judge whether the experiment succeeded, and how long would you wait before deciding?
- Product, operations, and sales reports all say “retention,” but refer to return visits, key-feature use, and renewal. How would you clarify the measures so each name maps to a user behavior and business decision?
- A membership program increases repeat-purchase frequency, but average order value and margin both decline. How would you distinguish a healthy habit change from purchases pulled forward or low-quality, promotion-driven growth?
- A new automated workflow doubles processing speed and reduces human review, but errors have a greater impact on a small group of high-value customers. How would you assess net value and decide whether to expand it?
- Overall conversion improves, but new users and low-connectivity regions worsen. How would you examine segmentation, selection bias, and problems hidden by the aggregate?
- After several teams optimize the same north-star metric, duplicate content and low-quality interactions appear. How would you diagnose metric distortion and adjust guardrails?
No questions in this section match your target. ·
Questions 9–16
- A campaign randomly holds out 10% of users, but they may still receive sales outreach and organic recommendations. Conversion is higher in treatment, and the channel team wants more budget. How would you assess contamination and results?
- Orders rise 30% during a discount campaign, but many come from existing customers who would have bought anyway, and product mix changes. How would you estimate incremental gross margin?
- The algorithm team expects an upgraded recommendation model to lift conversion by 3%, but operations would lose the ability to adjust key placements manually. How would you design an experiment and decision rules while accounting for value the experiment cannot measure?
- After the bot upgrade, sessions with no handoff to a human are counted as resolved, and the rate went from 72% to 91%. In the same week the handoff button moved into a submenu and the denominator changed. How do you respond today to the proposal to cancel weekend human coverage?
- What business result did your project bring? How do you confirm it came from this project and not from something else?
- Retention rose during a marketing campaign, and the team credits the channel, but a new product flow also launched and participants may already have been more active. How would you establish a credible comparison, estimate the campaign’s net lift, and state the limits of the conclusion?
- Who you compare against changes the conclusion directly. Why did you choose these comparisons, and how would your conclusion change with a different one?
- When you benchmark against a competitor's data, how do you decide what is comparable and what is not? What decision did you finally use it for?
No questions in this section match your target. ·
Questions 17–24
- The completion rate of a key task fell this week. Before explaining why, in what order do you confirm that the change is real, where it is concentrated, and what cost users bear?
- How do changes at each funnel stage ultimately act on the business metric? Improving which stage by a little moves the business metric the most, and what is that judgment based on?
- Once your metric is converted to the industry's common definition, where does it stand? What assumptions did the conversion make, and what information did it lose?
- A dashboard holds dozens of metrics and the team's attention keeps getting pulled to the most visible numbers. Pick one user task and explain how you would tier the metrics and decide which enter the core list.
- A target was set for a metric you own. How do you break it down into specific levers and actions, and judge along the way whether it can be met?
- The boss raises a business pain, such as high acquisition cost and poor retention. How do you translate it into goals the technical team can tackle, and show how technical means act on those business metrics?
- For the last new feature you launched, what were the specific before-and-after changes in the core metric, such as conversion, click-through, or retention? If the numbers fall short of expectation, along which dimensions do you usually investigate the cause?
- A feature encourages invitations, so inviters and friends may land in different experiment groups. Results are significant, but spillover cannot be fully isolated. How would you assess the conclusion?
No questions in this section match your target. ·
Questions 25–32
- After a redesign of an important flow, the expected completion rate does not improve. Support requests decline, existing users respond more positively, but new users behave worse. How would you check metric definitions and user segments, determine what effect the design had, and choose the next action?
- The team defines conversion as purchasers divided by visitors. After a redesign, visits fall while purchaser count stays roughly flat, so the rate rises. How would you decide whether the redesign improved the target user's purchase experience?
- Checkout, recommendations, and promotion teams run experiments at the same time and share traffic and revenue measures; results conflict and each team claims success. How would you identify interactions, interference, and resource conflicts?
- After workflow automation, average handling time per case falls, but overdue backlog grows and complex cases wait longer. How would you analyze queue distribution and service outcomes to judge whether the automation works?
- The team changes “active customer” from any login to completing a key task and updates the dashboard midquarter; leadership still compares year-over-year trends across the old and new definitions. How would you show the change and limits of comparability?
- A low-traffic experiment shows a large conversion lift, but only dozens of users participated and a few large transactions account for most of the gain. How would you judge stability, risk, and whether to expand the sample?
- A product change increases setup completion, but renewal and long-term use take months to observe; the team wants to invest more based on the proxy. How would you assess its link to final outcomes and the risk of deciding now?
- User activity and transaction value rise in one quarter while the team launches a feature, a competitor exits a region, and seasonal demand strengthens. How would you estimate the product's incremental contribution and state limits of the conclusion?
No questions in this section match your target. ·
Questions 33–40
- A dashboard shows a sudden drop in complaints, but upstream events are delayed; similar delays previously backfilled many complaints the next day. The on-call owner must decide whether to continue rolling out a new flow. How would you assess freshness and decision fitness?
- Overall retention rises after a redesign, but fewer new users and more established users are in the cohort; retention within each group is nearly unchanged. How would you assess the product's contribution and restructure the analysis?
- A redesign raises recommendation scores, but app-store complaints and escalations to support also increase; the sources have different users and response timing. How would you decide whether experience truly improved?
- Support quality was measured by response speed. Automation now shortens first response without reducing resolution time, and complex cases are split into multiple tickets. How would you redefine success and guard against gaming?
- During a randomized experiment, the platform changes pricing policy and affects treatment and control differently. How would you decide which conclusions the experiment still supports?
- Brand advertising expands reach, but short-term direct conversions do not change and the sales cycle may last months. How would you decide whether to wait, change creative, or stop?
- A recommendation algorithm changes content ranking and creator supply, so treatment behavior changes what the control group can see. How would you assess the experiment design?
- An onboarding test improves short-term activation, but most participants came during a promotion and long-term retention is immature. How would you judge the evidence and next step?
No questions in this section match your target. ·
Experience and reflection1
Use past facts to examine contribution, results, and learning.
Questions 1–1
No questions in this section match your target. ·
No questions in this capability match your target. ·