TOPIC
Metrics and measurement
These are the same questions found in the library, grouped here by the capability each one mainly examines.
200 questions
Extended questions 151–200 of 200 on this page.
A · Professional problem solving and implementation6
- A new assistant improves satisfaction ratings, but task completion and repeat-contact rates remain unchanged. How would you determine whether this is a real improvement?
- The team tunes prompts and model settings against the same fixed evaluation set every week. Scores rise, but new-user feedback does not improve. How would you restore evaluation trustworthiness?
- The evaluation set has grown, but many items are slight rewrites of one template, making model scores look stable. How would you audit sample quality?
- A feature targets people who have never used the product, but all usability participants are core users. Which design decisions can the current evidence support?
- A new safety policy reduces false refusals but causes sensitive disclosures in a few edge cases. Product wants to keep the satisfaction gain. How would you evaluate?
- After a safety policy launch, human handoffs double while high-risk errors fall. Operations capacity is limited. How would you assess the result and tune thresholds?
No questions in this section match your target. ·
B · Business understanding and domain modeling4
- Payment is authorized but not captured, and goods are not fulfilled. The business report counts an order, while finance records no revenue yet. How would you design cross-domain metric semantics?
- High-revenue customers frequently request urgent support, consuming team capacity and delaying critical issues for others. How would you define service prioritization?
- A missing risk feature is automatically set to zero. Latency and error rates are stable, but denial rates changed significantly. How would you judge whether the default is safe?
- A partner charges per form submission, while sales counts only qualified opportunities. Is the dispute about definitions, data feedback, or audience quality? How would you reset the agreement?
No questions in this section match your target. ·
C · Value judgment and direction setting10
- Task completion reaches 95%, but half of users rely on support or colleagues. The team plans to use completion as its launch-success metric. What would you add?
- The same customer submits changing feature requests, all pointing to delayed month-end reconciliation. How would you uncover the underlying problem and validate priority?
- The team completed its top-ranked projects, but engineers with critical skills are idle while other teams are overloaded. How would you improve allocation?
- During a randomized experiment, the platform changes pricing policy and affects treatment and control differently. How would you decide which conclusions the experiment still supports?
- A new feature reaches only half its adoption target, but many target users have not received training or access. How would you diagnose and adjust the plan?
- Brand advertising expands reach, but short-term direct conversions do not change and the sales cycle may last months. How would you decide whether to wait, change creative, or stop?
- A feature is rarely used, yet enterprise procurement and renewal interviews consistently cite it as a safeguard. How would you assess value and decide investment?
- Search returns hundreds of records, yet users repeatedly combine filters and export to spreadsheets. How would you distinguish retrieval, ranking, task understanding, and result explanation problems?
- Several teams have migrated to a shared platform and adoption meets its target, but lead time and reliability have not improved. How would you judge the investment?
- An onboarding test improves short-term activation, but most participants came during a promotion and long-term retention is immature. How would you judge the evidence and next step?
No questions in this section match your target. ·
D · Delivery and ongoing operations3
- The team believes rework is excessive, but some count bug fixes, others requirement changes, and others code rewrites. How would you define the problem and set an improvement target?
- The monthly cloud bill is under budget, but cost per transaction keeps rising; temporarily lower volume hides efficiency regression. How would you govern cost?
- The team lists release quality as a quarterly priority, but some focus on rollbacks, others on defects, and others on speed. How would you define a shared outcome?
No questions in this section match your target. ·
E · Collaboration, communication, and influence1
No questions in this section match your target. ·
F · People and organization building3
- The team hires using algorithm and communication scores, but six months later sees the largest differences in problem decomposition and cross-functional execution. How would you validate and revise the scorecard?
- Annual goals were translated into countable delivery volume at team level, while nobody owns customer outcomes. How would you inspect the cascade and reset accountability?
- Several frontline managers you oversee assess people using different evidence: delivery volume, team stability, or business impact. Their cases are hard to compare in calibration. How would you build a shared evidence framework while preserving role and team context?
No questions in this section match your target. ·
G · Learning, adaptation, and sustained growth4
- Startup time improves after your change, but caching and server changes also shipped. How would you assess your contribution?
- The team completes new security training with nearly 100% test pass rate, yet sensitive-configuration errors continue. How would you determine whether learning transferred into practice and what to change?
- You own internal data quality with no shared defect baseline or formal business target, but errors affect several downstream teams. How would you define a goal and verify progress?
- You changed code-review practices after a retrospective, but team defect rates remain flat. How would you judge effectiveness, sample size, or the mechanism?
No questions in this section match your target. ·
H · Professional responsibility and work style19
- You find a definition error in analysis you submitted, and decision-makers have already shifted resources based on it. How would you take responsibility and correct the impact?
- A deduplication change makes conversion look higher, but historical data has not been recalculated. How would you present the result?
- Retention improves after changes by product, operations, and customer success. How would you explain your contribution without overstating causality?
- You deliver an analysis and proposal with substantial AI help for research, code drafts, and documentation, while you own problem framing, validation, and final decisions. How would you explain contribution to the team?
- A quarterly target was missed. A leader proposes excluding refunds and cancellations from the main conversion rate, calling it more representative of core customers. How would you assess and communicate the change?
- At a quarterly review, you predicted a change would lift conversion. A later segmented experiment shows no overall lift and harm to one segment. How would you correct the conclusion?
- Tell me about a time you proactively corrected something you had said or a metric definition you had used. How did you find it, and how did you tell the people affected?
- A launch event you supported described a metric increase after release as an improvement “caused by the feature,” but the team had no control group and had not ruled out a concurrent campaign. How would you decide whether to correct the claim and work with marketing and business owners?
- Your quarterly report says a target was met. You later discover that the data pipeline changed the definition of “active customer” midway through the period; recalculating with the old definition puts the result below target. How would you correct the report, inform decision-makers, and repair the process?
- Describe a project or module you owned that fell short of its commitment. How did external dependencies and unvalidated assumptions affect the result? How do you explain your judgment, responsibility, and recovery actions, and what would you validate earlier next time?
- You present a successful project to a review committee. A senior leader made a key decision; you executed it and identified risks. How would you describe the team outcome and your responsibility?
- You led an architecture change that improved reliability but did not move conversion. How would you report value, failed assumptions, and your next responsibility?
- Revenue targets were met while channel budgets, product experience, and sales processes all changed. How would you separate contributions and explain causal limits?
- Your portfolio shows a shipped design that changed through research, operations, and engineering input. How would you present your original judgment, adopted work, and rejected ideas?
- You met your own commitments, but the team missed its target. A reviewer attributes the whole result to you. How would you acknowledge shared accountability and explain your impact?
- A quarterly review credits you with most of a project, but a colleague resolved the key launch risk. How would you correct the record and preserve both contribution boundaries?
- Another team provided resources at a critical moment. You did not manage them, but clarified dependencies, negotiated priority, and prepared alternatives. How would you describe your impact?
- You inherited a mature process and results improved further. How would you separate the foundation you received, your new decisions, and the natural trend?
- The packet says your work reached five teams but does not show how you influenced them or what you owned. Which decision, behavior, and outcome evidence would you add?
No questions in this section match your target. ·