Desired Outcome or Something Simpler?
A logical Analysis of two ways to measure success in Jobs-to-be-Done (JTBD)
I thought it would be interesting to let the new reasoning capabilities of the large language models perform a logical analysis of two different approaches currently used to measure success in a JTBD survey.
I crafted a fairly long prompt (including a system instruction) that I believe was objective. I’m sure there is someone who will disagree. Therefore, I’m sharing a link to the prompt so you can make that determination yourself:
The following is the response from the latest Gemini model.
The Response
<begin response>
Let's break down these two formats for evaluating Job Steps in a Jobs-to-be-Done framework and analyze their suitability for a long survey with a diverse audience.
Understanding the Core Goal:
The primary objective of these survey questions is to understand the importance and current satisfaction/performance related to specific Job Steps. This information is crucial for identifying opportunities for innovation and improvement. The chosen format needs to be clear, concise, and minimize ambiguity to ensure reliable data collection.
Analysis of Format 1: The Desired Outcome Statement
Pros:
High Precision and Actionability: Explicitly stating the direction of improvement and the metric makes the desired outcome very clear and actionable. Innovators know exactly what needs to be increased or decreased.
Reduced Ambiguity: The structured format minimizes the chance of respondents misinterpreting the question. The specific metric and object of control leave less room for subjective interpretation.
Direct Link to Measurement: The format directly ties the statement to a measurable dimension, making it easier to track progress and evaluate the impact of changes.
Clear Focus on Improvement: The explicit direction of improvement (minimize, increase) keeps the focus on the desired outcome.
Cons:
Cognitive Load: The more complex sentence structure, especially with the optional context, can increase the cognitive load for respondents, particularly those with lower educational backgrounds or those experiencing survey fatigue in a long survey.
Potential for Jargon/Unnatural Language: Phrases like "minimize the time it takes to locate a file" might sound slightly unnatural in everyday conversation, potentially leading to confusion or disengagement.
Repetitiveness: Repeating the full, structured statement for both Importance and Satisfaction questions can become monotonous and contribute to respondent fatigue in a long survey.
Abstraction: While precise, the focus on abstract metrics might be less intuitive for some respondents compared to more practical phrasing.
Analysis of Format 2: The Practical Version
Pros:
Simplicity and Clarity: The phrasing is more straightforward and uses natural language, making it easier for a wider range of respondents to understand quickly.
Lower Cognitive Load: The shorter, less structured sentences require less mental effort to process, which is beneficial in a long survey.
Intuitiveness: Phrases like "quickly and accurately locate a file" are more intuitive and align with how people naturally think about tasks.
Reduced Repetitiveness: The phrasing is more concise, potentially making the repetition across multiple questions less burdensome.
Focus on Action: The phrasing emphasizes the action itself ("locate a file"), which can be more relatable for respondents.
Cons:
Increased Ambiguity: Without explicitly stating the direction and metric, there's more room for interpretation. "Quickly" could mean different things to different people. "Accurately" is a bit more concrete but still less precise than a defined metric.
Less Actionable (Initially): While intuitive, the lack of a specific metric makes it less directly actionable for product teams. They need to infer the underlying metric (time, error rate, etc.).
Potential for Misinterpretation: Respondents might focus on different aspects of the statement. One person might prioritize "quickly," while another focuses on "accurately."
Difficulty in Measurement: Measuring "quickness" or "effort" can be more subjective and harder to standardize compared to measuring "time" or "likelihood."
Analysis of the Lead-ins:
Desired Outcome Statement Lead-in: The lead-in clearly defines the dimensions being measured (Importance and Satisfaction) and directly uses the full Desired Outcome Statement. This reinforces the precise nature of the evaluation.
Practical Version Lead-in: This lead-in uses "Effort" instead of "Satisfaction," which is a significant difference. "Effort" is often a more concrete and less emotionally charged metric than "Satisfaction," potentially leading to more objective responses. The inclusion of "Frequency" for consumption chain jobs is a valuable addition for understanding the context of the job.
Determining the Better Format:
Considering the need for a comprehensive and long survey with a wide range of educational backgrounds, Format 2 (The Practical Version) is likely the better choice.
Reasoning:
Reduced Cognitive Load and Improved Comprehension: In a long survey, minimizing cognitive load is crucial to maintain respondent engagement and data quality. The simpler language of Format 2 is easier for a diverse audience to understand quickly, reducing the risk of confusion and fatigue.
Natural Language and Intuitiveness: Format 2 uses more natural language, which resonates better with respondents and makes the questions feel less like a technical exercise. This can lead to more thoughtful and accurate responses.
Focus on Effort: The use of "Effort" as a dimension in Format 2 is a strength. Effort is often a more tangible and less subjective measure than "Satisfaction." Respondents can more easily assess the effort required for a task. While satisfaction is important, it can be influenced by various factors beyond the specific Job Step.
Mitigating Ambiguity (with careful wording): While Format 2 has the potential for ambiguity, this can be mitigated by carefully choosing the wording of the statements. Focusing on clear and commonly understood terms is key. For example, instead of just "quickly," you could use "locate the file without significant delays."
Frequency as Context: The optional "Frequency" dimension in Format 2 provides valuable context, especially for understanding the importance of frequently performed Job Steps.
Why Format 1 is Less Ideal for This Scenario:
Risk of Respondent Fatigue and Disengagement: The more complex and repetitive nature of Format 1 can lead to respondent fatigue, especially in a long survey. This can result in rushed or less thoughtful answers.
Potential for Confusion: The structured phrasing, while precise, might be confusing or off-putting to respondents with lower educational backgrounds.
Abstraction: The focus on abstract metrics might be less intuitive and relatable for some respondents.
Recommendation and Refinements:
While Format 2 is generally better, here are some recommendations for its implementation:
Careful Wording: Pay close attention to the wording of the Practical Version statements to minimize ambiguity. Use clear, concise, and commonly understood language.
Consider Adding a "Clarity" Check: In initial pilot testing, consider adding a question about how clear each statement is to respondents.
Provide Examples: Offer clear examples of what "effort" means in the context of the survey.
Potentially Incorporate "Frequency" into Format 1: If the precision of Format 1 is desired for certain critical Job Steps, consider adapting it to include a "Frequency" dimension as well.
Final Verdict:
For a long survey with a wide range of educational backgrounds, Format 2 (The Practical Version), with careful attention to wording, is the more effective choice. It balances clarity, ease of understanding, and the collection of actionable data while minimizing respondent burden. The focus on "Effort" as a key dimension is also a significant advantage.
</end response>
While you may not agree, this reconciles very well with my experience. I don’t need to do the same thing over and over again for 30 years to make that determination. In fact, that’s probably what leads to bias. I decided to use a more practical version based on the reactions I was seeing from stakeholders once the realization sunk in that consumers would have to answer these questions for 40-50 minutes.
It appears that practicality is the winner. However, it was close.
Are you practical?
Please share your thoughts in the comments.
Mike Boysen - www.pjtbd.com
Why fail fast when you can succeed the first time?
Book an appointment: https://pjtbd.com/book-mike
Grab my JTBD Masterclass: https://mc.zeropivot.us/s/mc-1
Get the whole customer management thing done on a single platform:
https://pjtbd.com/tech-stack