We Love Speed: Understanding Cognitive Biases in Performance Measurement

Transcript

1. Understanding Cognitive Biases
   in Performance Measurement
   Finding the factors that lead to abandonment
   https://speakerdeck.com/bluesmoon/we-love-speed-understanding-cognitive-biases-in-performance-measurement
   

   View Slide

2. Philip Tellis
   Principal RUM Distiller @ Akamai
   ● Analyses real user performance data from mPulse
   ● Author of the OpenSource boomerang RUM library
   twitter:@bluesmoon ⦿ github:@bluesmoon
   speakerdeck:@bluesmoon
   

   View Slide

3. BIAS is an
   Expectation
   Our Journey Today...
   ★ Understanding Cognitive
   Biases
   ★ Signs of cognitive biases in
   browsing data
   ★ What can we do?
   

   View Slide

4. Understanding Bias
   Good, Bad, Normal?
   

   View Slide

5. Similarity Zero-Risk
   False Memory Expedience
   Experience Proximity
   Survivorship Negativity
   Safety Loss Aversion
   If you have a brain, you have bias.
   

   View Slide

6. Bias stems from experience – It’s Normal
   ● Helps us learn
   Perceptual/Sensory Dissonance
   ● Keeps us safe
   Safety Bias, Loss Aversion, Negativity Bias
   ● Find our people
   Similarity Bias, Proximity Bias
   Boston Shipyard Artist’s Community
   

   View Slide

7. https://upload.wikimedia.org/wikipedia/commons/6/65/Cognitive_bias_codex_en.svg
   

   View Slide

8. Cognitive Biases
   ● Similarity Bias
   ● Expedience Bias
   ● Experience Bias
   ● Proximity Bias
   ● Safety Bias
   ● Serial-position effect
   ● False memory
   ● Duration neglect
   ● Peak–end rule
   ● Negativity bias
   ● Escalation of commitment
   ● Loss aversion
   ● Zero-risk bias
   ● Next-in-line effect
   ● Misattribution of memory
   ● Sunk cost
   ● Levels-of-processing
   ● Spacing effect
   

   View Slide

9. Cognitive Biases - Related to Performance on the Web
   ● Similarity Bias
   ● Expedience Bias
   ● Experience Bias
   ● Proximity Bias
   ● Safety Bias
   ● Serial-position effect
   ● False memory
   ● Duration neglect
   ● Peak–end rule
   ● Negativity bias
   ● Escalation of commitment
   ● Loss aversion
   ● Zero-risk bias
   ● Next-in-line effect
   ● Misattribution of memory
   ● Sunk cost
   ● Levels-of-processing
   ● Spacing effect
   

   View Slide

10. Cognitive Biases - This Talk
    ● Similarity Bias
    ● Expedience Bias
    ● Experience Bias
    ● Proximity Bias
    ● Safety Bias
    ● Serial-position effect
    ● False memory
    ● Duration neglect
    ● Peak–end rule
    ● Negativity bias
    ● Escalation of commitment
    ● Loss aversion
    ● Zero-risk bias
    ● Next-in-line effect
    ● Misattribution of memory
    ● Sunk cost
    ● Levels-of-processing
    ● Spacing effect
    

    View Slide

11. Pause
    Statistique
    A 500ms connection speed delay resulted in
    up to a 26% increase in peak frustration
    and up to an 8% decrease in engagement.
    Tammy Everts – The impact of network speed on emotional engagement
    

    View Slide

12. some definitions
    Bounce Rate: Percentage of users on the site who leave after viewing one page.
    Retention Rate: Percentage of users on a particular page who remain on the
    site for at least one more page view.
    Conversion Rate: Percentage of users on the site who complete a goal or
    particular task.
    Goal: A task like a conversion, purchase, visiting a particular page, or viewing
    a certain number of pages.
    Frustration Index: A metric derived from multiple timers on a page that
    correlates with user frustration during page load.
    

    View Slide

13. Serial-Position Effect
    …is the tendency of a person to recall the first and last items
    in a series best, and the middle items worst.
    Ebbinghaus, Hermann (1913). On memory: A contribution to experimental psychology
    

    View Slide

14. Serial-Position Effect
    …is the tendency of a person to recall the first and last items
    in a series best, and the middle items worst.
    ● Retention rate might be a function of the first and latest pages
    ● The recency effect suggests that the latest page has a higher weight
    Ebbinghaus, Hermann (1913). On memory: A contribution to experimental psychology
    

    View Slide

15. Peak-End Rule
    People judge an experience largely based on how they felt at
    its peak & at its end, rather than the sum or average of
    every moment of the experience.
    Kahneman, Daniel (2000). "Evaluation by moments, past and future"
    

    View Slide

16. People judge an experience largely based on how they felt at
    its peak & at its end, rather than the sum or average of
    every moment of the experience.
    ● Retention rate depends on the best/worst and latest performance
    ● Conversion rate depends on the best/worst performance and that of the page
    just before the conversion
    Peak-End Rule
    Kahneman, Daniel (2000). "Evaluation by moments, past and future"
    

    View Slide

17. Negativity Bias
    Even when of equal intensity, things of a more negative
    nature have a greater effect on one's psychological state and
    processes than neutral or positive things.
    Baumeister, Roy F
    .; Finkenauer, Catrin; Vohs, Kathleen D. (2001). "Bad is stronger than good"
    

    View Slide

18. Negativity Bias
    Even when of equal intensity, things of a more negative
    nature have a greater effect on one's psychological state and
    processes than neutral or positive things.
    ● Conversion rate should correlate with the ratio, or average of worst
    experience to best experience.
    ● Active Listening can confound the results
    Baumeister, Roy F
    .; Finkenauer, Catrin; Vohs, Kathleen D. (2001). "Bad is stronger than good"
    

    View Slide

19. Escalation of Commitment / Sunk Cost
    An individual or group facing increasingly negative
    outcomes continue the behavior instead of altering course.
    A greater tendency to continue an endeavor once an
    investment in money, effort, or time has been made.
    Staw, Barry M. (1997). "The escalation of commitment: An update and appraisal"
    Arkes, Hal R.; Ayton, Peter (1999). "The sunk cost and Concorde effects"
    

    View Slide

20. Escalation of Commitment / Sunk Cost
    An individual or group facing increasingly negative
    outcomes continue the behavior instead of altering course.
    A greater tendency to continue an endeavor once an
    investment in money, effort, or time has been made.
    ● High session length for really bad performing sessions
    ● Retention/conversion rate increases as session length increases
    Staw, Barry M. (1997). "The escalation of commitment: An update and appraisal"
    Arkes, Hal R.; Ayton, Peter (1999). "The sunk cost and Concorde effects"
    

    View Slide

21. Hypotheses…
    ● The most recent experience is
    very impactful.
    ● The best and/or worst
    experiences are impactful.
    ● The first experience may be
    impactful.
    ● The amount of time someone
    stays on the site is impactful.
    Pacific Islander Navigation Map, Museum of Fine Arts, Boston
    https://www.flickr.com/photos/bluesmoon/1266590108/
    

    View Slide

22. Pause
    Statistique
    Wikipedia found that a 4% temporary
    improvement to page load time resulted in an
    equally temporary 1% increase in user
    satisfaction.
    Wiki Research: Analyzing Wikipedia Users’ Perceived Quality Of Experience
    

    View Slide

23. Detecting Bias
    Identifying Cognitive Biases in Browsing Data
    

    View Slide

24. ● Collection: Real user performance data collected with boomerang
    ● Sessions: Anonymous session ID attached to continuous sessions;
    discarded after 30 minutes of inactivity. Limited to sessions of 30
    pages or fewer.
    ● Samples: Analysis was done across multiple websites with millions
    of data points each.
    ● Timers: We looked at Page Load Time (PLT), Time to Interactive
    (TTI), Largest Contentful Paint (LCP) and Frustration Index for Full
    Page as well as Single Page Apps.
    Notes about the Data
    

    View Slide

25. First, Last, Fastest, Slowest
    ● There is a strong negative correlation between
    conversion rate and the performance of the first page.
    3.5% @ 1.8s
    0.8% @ 18s
    1.6% @ 9s
    

    View Slide

26. First, Last, Fastest, Slowest
    ● There is a strong negative correlation between
    conversion rate and the performance of the first page
    – this is related to bounce rate.
    40% @ 1.1s
    60% @ 18s
    50% @ 9s
    

    View Slide

27. First, Last, Fastest, Slowest
    ● There is a strong negative correlation between
    conversion rate and the performance of the first page
    – this is related to bounce rate.
    ● The last page distribution has strong drop after an
    initial peak. The two peaks are for XHR & Full Page.
    13.5% @ 300ms
    0.4% @ 18s
    1% @ 5.5s
    

    View Slide

28. First, Last, Fastest, Slowest
    ● There is a strong negative correlation between
    conversion rate and the performance of the first page
    – this is related to bounce rate..
    ● The last page distribution has strong drop after an
    initial peak. The two peaks are for XHR & Full Page.
    ● The fastest page has to be really fast.
    10.5% @ 500ms
    0.4% @ 9s
    1% @ 5s
    

    View Slide

29. Conversions x First, Last, Fastest, Slowest
    ● There is a strong negative correlation between
    conversion rate and the performance of the first page
    – this is related to bounce rate..
    ● The last page distribution has strong drop after an
    initial peak. The two peaks are for XHR & Full Page.
    ● The fastest page has to be really fast. Too slow, and
    users bounce.
    ● Correlation with the slowest page is a little weird…
    

    View Slide

30. 0.5% @ 1s
    3.4% @ 19s
    3% @ 5s
    ● It seems that conversions increase as performance gets worse
    ● It turns out that a slow experience is part of the conversion
    flow.
    ● The low conversion rate on the left is a result of bounces.
    Very fast pages are typically caused by JavaScript errors
    resulting in a mostly blank page.
    (we see the same when the fastest page is under 100ms)
    Is Slower Better?
    

    View Slide

31. looking at the 2nd Slowest Instead…
    0.5% @ 1s
    1% @ 19s
    4.8% @ 2s
    1.9% @ 6s
    1.3% @ 12s
    

    View Slide

32. Conversions x First, Last, Fastest, Slowest
    ● There is a strong negative correlation between
    conversion rate and the performance of the first page.
    ● The last page distribution has strong drop after an
    initial peak. The two peaks are for XHR & Full Page.
    ● The fastest page has to be really fast. Too slow, and
    users bounce.
    ● The slowest page doesn’t matter, but you cannot have
    too many slow pages.
    

    View Slide

33. Retention Rate x First, Last, Fastest, Slowest
    ● Retention Rate of a page varies based on the page.
    ● For Homepages and other Landing pages, the performance of the first page
    appears to be the biggest indicator of retention.
    ● For Product Detail, Category, and Search Results Pages, it’s a combination of
    the fastest & latest, and sometimes the first page.
    ● The worst and second worst performing pages do not have an impact on
    retention.
    

    View Slide

34. Negativity Bias
    ● To determine if negativity bias is in play, we
    look at combinations of the best and 2nd
    worst performing pages.
    ● The ratio (worst/best) has a strong negative
    correlation with conversions.
    ● The geometric mean has a high, narrow peak.
    ● A heatmap shows low tolerance for deviations
    in the fastest load time, and inverse
    dependence between the fastest and slowest
    times.
    Ratio of Slowest to Fastest
    Geometric MEAN of Slowest & Fastest
    Fastest →
    ← Slowest
    1
    10
    20
    30
    40
    50
    60
    0 10 20 30 40 50
    0 0.4 0.7 1.0 1.3 1.6 1.9 2.2 2.5
    

    View Slide

35. Uhh… What does all of that
    mean?
    

    View Slide

36. Negativity Bias
    1
    10
    20
    30
    40
    50
    60
    0 0.4 0.7 1.0 1.3 1.6 1.9 2.2 2.5
    ● We have a practical lower bound on the fastest page
    ● We have a tolerable upper bound on the fastest page
    ● Slow pages are tolerated only when paired with a fast
    page that’s at least 15x faster.
    ● This results in an upper bound on the slowest page.
    Fastest →
    0 10 20 30 40 50
    ← Slowest
    

    View Slide

37. A greater tendency to continue an
    endeavor once an investment in money,
    effort, or time has been made.
    Escalation of Commitment / Sunk Cost
    

    View Slide

38. A greater tendency to continue an
    endeavor once an investment in money,
    effort, or time has been made.
    29% after 30
    0.6% after 5 pages
    7.1% after 10
    21% after 20
    Pages ->
    Conversion Rate ->
    Escalation of Commitment / Sunk Cost
    

    View Slide

39. Looking across Load Times…
    1
    10
    20
    30
    40
    50
    60
    70
    80
    90
    100
    110
    115
    0 10 20 30 40 50
    0.1s 2s 4s 6s 8s 10s 15s 20s 25s 30s
    Load Time ->
    <- Number 0f Pages
    Conversion Rate
    

    View Slide

40. Pause
    Statistique
    The average rise in mobile users' heart rates caused
    by delayed web pages — equivalent to the anxiety
    of watching a horror movie alone.
    Ericsson ConsumerLab neuro research 2015
    38%
    

    View Slide

41. Accounting for Bias
    What do we do with this knowledge?
    

    View Slide

42. Focus performance improvements on
    a few key pages.
    

    View Slide

43. The performance of the first page
    affects bounces.
    

    View Slide

44. The performance of the fastest page
    and last page affects retention.
    

    View Slide

45. The slowest page in a session should
    be no more than 15x the latency of
    the fastest page.
    

    View Slide

46. Acknowledging when you didn’t meet the
    user’s expectations can alleviate negative
    perceptions.
    Practice Active Listening
    https://affect.media.mit.edu/pdfs/02.klein-moon-picard.pdf
    https://uxdesign.cc/the-fastest-way-to-pinpoint-frustrating-user-experiences-1f8b95bc94aa
    https://doi.org/10.1016/j.ijhcs.2004.01.002
    https://www.sciencedirect.com/science/article/abs/pii/S1071581904000060?via%3Dihub
    

    View Slide

47. A fast page increase pages per session
    which in turn increase the likelihood
    of a conversion.
    

    View Slide

48. Pause
    Statistique
    Users are most patient when using the
    web from the office and least patient
    when using their phones.
    Median Lethal Frustration Index study in mPulse data
    

    View Slide

49. Developer Bias
    Biases when studying the data
    

    View Slide

50. https://upload.wikimedia.org/wikipedia/commons/6/65/Cognitive_bias_codex_en.svg
    

    View Slide

51. Cognitive Biases – Developer Edition
    ● Amdahl's Law
    Assuming every millisecond is the same.
    ● Outcome Bias
    Choosing data that confirms past outcomes.
    ● Survivorship Bias
    Assuming what we’ve measured is all there is.
    ● Selection Bias
    Choosing dimensions based on our instincts.
    ● Pareidolia
    Preferring data that renders interesting shapes.
    ● Insensitivity to Sample Size
    Forgetting that smaller samples have larger variance.
    ● Clustering Illusion
    Seeing patterns in small samples where none exist.
    ● Confirmation Bias
    Choosing data that confirms our pre-existing beliefs.
    

    View Slide

52. Ignoring Amdahl’s Law
    You may have read reports that say something like:
    “every 100ms decrease in homepage load
    time worked out to a 1% increase in
    conversion”
    Citation redacted to protect the innocent
    

    View Slide

53. Survivorship Bias
    ● In 2012, Youtube made their site lighter but aggregate
    performance got worse.
    ● It turns out that new users who previously could not access the site
    were now coming in at the long tail.
    ● The site appeared slower in aggregate, but the number of users
    who could use it had gone up.
    Chris Zacharias: Page Weight Matters.
    

    View Slide

54. Insensitivity To Sample Size
    We often get questions like:
    “Why is performance on tablets worse than
    performance on mobile devices?”
    It turns out that mobile generally has 50x the amount of traffic than tablets.
    That results in far less variance in the data.
    A customer recently asked me this question.
    

    View Slide

55. Anscombe’s Quartet
    Anscombe's Quartet Frank Anscombe
    Plot of Anscombe's Quartet by Schutz & Avenue
    ● 4 data sets with the same summary statistics:
    ○ 𝜇
    x
    = 9, 𝜇
    y
    = 7.5
    ○ s
    x
    2 = 11, s
    y
    2 = 4.125
    ○ 𝜌
    x,y
    = 0.816
    ○ Linear Regression Line: y=3
    ○ ℝ2 = 0.67
    ● Anscombe’s Quartet shows us why it’s
    important to visualize data and not just look
    at summary stats
    

    View Slide

56. fin!
    Vous êtes magnifique!
    Thank you!
    

    View Slide

57. References
    ● Ebbinghaus, Hermann (1913). On memory: A contribution
    to experimental psychology
    ● Kahneman, Daniel (2000). "Evaluation by moments, past
    and future"
    ● Baumeister, Roy F
    .; Finkenauer, Catrin; Vohs, Kathleen D.
    (2001). "Bad is stronger than good"
    ● Staw, Barry M. (1997). "The escalation of commitment: An
    update and appraisal"
    ● Arkes, Hal R.; Ayton, Peter (1999). "The sunk cost and
    Concorde effects"
    ● The impact of network speed on emotional engagement
    ● Ericsson ConsumerLab neuro research 2015
    ● Wikipedia Paper on User Satisfaction v/s Performance
    ● Toward a more civilized design: studying the effects of
    computers that apologize
    ● The fastest way to pinpoint frustrating user experiences
    ● Serial-position effect
    ● Peak–end rule
    ● Negativity bias
    ● Escalation of commitment / Sunk cost
    ● Levels-of-processing
    ● Amdahl's Law
    ● Outcome Bias
    ● Survivorship Bias
    ● Selection Bias
    ● Pareidolia
    ● Insensitivity to Sample Size
    ● Clustering Illusion
    ● Confirmation Bias
    ● Time Saving Bias
    

    View Slide