Presentation metrics - Dasdan€¦ · Implicit presentation metrics • Statistical metrics –...

26
© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009. 150 150 Presentation metrics Implicit Explicit • Small sample set • Slow data collection Statistical metrics Log analysis of real traffic Online user studies • Large sample set • Fast data collection User studies Editorial

Transcript of Presentation metrics - Dasdan€¦ · Implicit presentation metrics • Statistical metrics –...

Page 1: Presentation metrics - Dasdan€¦ · Implicit presentation metrics • Statistical metrics – Precision/recall based on editorial data – Example: • Title from various sources

© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009. 150 150

Presentation metrics

Implicit Explicit

•  Small sample set •  Slow data collection

•  Statistical metrics •  Log analysis of real traffic •  Online user studies

•  Large sample set •  Fast data collection

•  User studies •  Editorial

Page 2: Presentation metrics - Dasdan€¦ · Implicit presentation metrics • Statistical metrics – Precision/recall based on editorial data – Example: • Title from various sources

© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009. 151

Explicit presentation metrics

•  User studies –  In-home

•  Early ideation –  Generative studies

•  Participatory, paper printouts –  Usability

•  Prototypes •  Eye tracking studies

–  Mental models –  Focus groups

•  Editorial –  Comparative

•  Preferential or judgment values between contender configurations –  Perceived vs. actual

•  How well does presentation convey the content of the landing page?

Page 3: Presentation metrics - Dasdan€¦ · Implicit presentation metrics • Statistical metrics – Precision/recall based on editorial data – Example: • Title from various sources

© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009. 152

Implicit presentation metrics

•  Statistical metrics –  Precision/recall based on editorial data –  Example:

•  Title from various sources (directories, web page, dynamic) •  Editors rate titles

•  Log analysis of real traffic –  User engagement

•  CTR, +/- clicks, query reformulations –  Session analysis –  User independent data (speed)

•  Online user studies –  Online surveys

Page 4: Presentation metrics - Dasdan€¦ · Implicit presentation metrics • Statistical metrics – Precision/recall based on editorial data – Example: • Title from various sources

© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009. 153

Online user studies

•  Goal: –  To measure product experience

•  Various types –  General surveys of a product –  Task-specific exercises –  Commercial products: userzoom.com, keynote.com

•  Two dimensions of product experience: 1.  Measured user experience

•  Example: –  User given a set of tasks. –  What is the task completion success rate?

2.  Perceived user experience •  Many sub-dimensions

–  Easy or difficult to use –  Performance (e.g. response times) –  User-friendliness (e.g. fun, not user-engaging)

•  Example: –  User given a set of tasks. –  How easily (in her opinion) did user complete tasks?

[A’09]

Page 5: Presentation metrics - Dasdan€¦ · Implicit presentation metrics • Statistical metrics – Precision/recall based on editorial data – Example: • Title from various sources

© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009. 154

Online user studies: The impact of a new release*

Product Experience Over Time

65

70

75

80

85

90

Time 1 Time 2 Time 3

PEM

Yahoo Competitor 1 Competitor 2

* To protect proprietary data and information this chart does not represent data from an actual study

Page 6: Presentation metrics - Dasdan€¦ · Implicit presentation metrics • Statistical metrics – Precision/recall based on editorial data – Example: • Title from various sources

© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009. 155

Online user studies: Use-based product survey

Page 7: Presentation metrics - Dasdan€¦ · Implicit presentation metrics • Statistical metrics – Precision/recall based on editorial data – Example: • Title from various sources

© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009. 156

Online user studies: Use-based product survey

Page 8: Presentation metrics - Dasdan€¦ · Implicit presentation metrics • Statistical metrics – Precision/recall based on editorial data – Example: • Title from various sources

© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009. 157

Online user studies: Use-based product survey

Performance

Satisfaction

Learnability

Page 9: Presentation metrics - Dasdan€¦ · Implicit presentation metrics • Statistical metrics – Precision/recall based on editorial data – Example: • Title from various sources

158

High resolution image seen by the Fovea

Reduced visual acuity experienced by the parafovea

Progressively reducing visual acuity from the periphery of the retina

© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009

Page 10: Presentation metrics - Dasdan€¦ · Implicit presentation metrics • Statistical metrics – Precision/recall based on editorial data – Example: • Title from various sources

159

Users use parafoveal preview to identify the parts most likely to have relevant information based on the location of boldfaced terms

© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009

Page 11: Presentation metrics - Dasdan€¦ · Implicit presentation metrics • Statistical metrics – Precision/recall based on editorial data – Example: • Title from various sources

160

Familiar summary patterns draw user attention and clicks

Users are relatively blind to unfamiliar summary patterns.

© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009

Page 12: Presentation metrics - Dasdan€¦ · Implicit presentation metrics • Statistical metrics – Precision/recall based on editorial data – Example: • Title from various sources

© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009.

Eye tracking studies

161

Reading (Yahoo! Finance) Scanning (Yahoo! Finance)

Page 13: Presentation metrics - Dasdan€¦ · Implicit presentation metrics • Statistical metrics – Precision/recall based on editorial data – Example: • Title from various sources

© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009. 162 162

Heat maps

Highest density of clicks concentrated

in hottest zone.

[E’05]

Page 14: Presentation metrics - Dasdan€¦ · Implicit presentation metrics • Statistical metrics – Precision/recall based on editorial data – Example: • Title from various sources

© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009. 163 163

Golden triangle

Golden triangle is habituated.

•  Result #1 is always more trusted and is considered more relevant by default.

•  Scan path gets narrower, and the user spends less time reading lower down the page.

Page 15: Presentation metrics - Dasdan€¦ · Implicit presentation metrics • Statistical metrics – Precision/recall based on editorial data – Example: • Title from various sources

© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009. 164 164

Bolding in scan path

•  Users use bolding in titles to rapidly scan the SRP.

•  Bolding in scan path is critical to making users notice a result.

•  If a result is not bolded here, it is not noticed, and hence cannot be judged as relevant.

Page 16: Presentation metrics - Dasdan€¦ · Implicit presentation metrics • Statistical metrics – Precision/recall based on editorial data – Example: • Title from various sources

© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009. 165 165

Design changes for Shortcuts

Page 17: Presentation metrics - Dasdan€¦ · Implicit presentation metrics • Statistical metrics – Precision/recall based on editorial data – Example: • Title from various sources

© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009. 166 166

Design changes for Shortcuts

Before Bolding all over

Conversational title style

After Bolding in scan line

To the point title (query term – property)

[ROK’07]

Page 18: Presentation metrics - Dasdan€¦ · Implicit presentation metrics • Statistical metrics – Precision/recall based on editorial data – Example: • Title from various sources

© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009. 167

1.

2.

3.

4.

167

SearchMonkey templates

Page 19: Presentation metrics - Dasdan€¦ · Implicit presentation metrics • Statistical metrics – Precision/recall based on editorial data – Example: • Title from various sources

© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009. 168 168

How design templates evolve Case study: SearchMonkey

Users didn’t notice pictures in most cases

Images are consistently ignored except in some specific intents.

Deep links & structured data increased perception of clutter

The perception of ads Increased significantly

1st Round of SM Templates tested

Page 20: Presentation metrics - Dasdan€¦ · Implicit presentation metrics • Statistical metrics – Precision/recall based on editorial data – Example: • Title from various sources

© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009. 169 169

How design templates evolve Case study: SearchMonkey

Moving the image out of the critical scan path helped users selectively discover it, without disrupting scanability.

Deep links were more discoverable when presented separately from the image

2nd Round of SM Templates tested

Page 21: Presentation metrics - Dasdan€¦ · Implicit presentation metrics • Statistical metrics – Precision/recall based on editorial data – Example: • Title from various sources

© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009.

Key problems

•  What is the best way to manage user attention? – Seeing vs. noticing – When do we cross from being helpful to

being overwhelming (volume)? – How many different types of formats can

coexist (diversity)? •  How can presentation support user

intent? •  How can presentation be used to

communicate genre and topic?

170

Page 22: Presentation metrics - Dasdan€¦ · Implicit presentation metrics • Statistical metrics – Precision/recall based on editorial data – Example: • Title from various sources

© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009. 171 171

References

•  [ABD’06] E. Agichtein, E. Brill, and S.T. Dumais (2006), Improving web search ranking by incorporating user behavior, SIGIR’06.

•  [ABDR’06] E. Agichtein, E. Brill, S.T. Dumais, and R. Ragno (2006). Learning user interaction models for predicting web search preferences,SIGIR’06.

•  [A’09] W. Albert (2009), Unmoderated usability testing: experience from the field, Usability Professionals Association Conference Panel.

•  [CKP’08] D. Chakrabarti, R. Kumar, and K. Punera (2008), Generating Succinct Titles for Web URLs, KDD’08.

•  [CKP’09] D. Chakrabarti, R. Kumar, and K. Punera (2009), Quicklink Selection for Navigational Query Results, WWW’09.

•  [CADW’07] C. Clarke, E. Agichtein, S. Dumais, and R. White (2007), The influence of caption features on clickthrough patterns in web search, SIGIR’07.

•  [E’05] Enquiro Eye Tracking Reports I & II, http://www.enquiroresearch.com/, June, 2005.

•  [HLZF’06] E. Hovy, C. Lin, L. Zhou, and J. Fukumoto (2006), Automated Summarization Evaluation with Basic Elements. LREC’06.

•  [KO’09] T. Kanungo and D. Orr (2009), Predicting Readability of Short Web Summaries, WSDM’09.

•  [MK’08] D. Metzler and T. Kanungo (2008), Machine Learned Sentence Selection Strategies for Query-Biased Summarization, SIGIR’08.

•  [ROK’07] D.E. Rose, D. Orr, and R.G.P. Kantamneni, Summary attributes and perceived search quality, WWW’07.

Page 23: Presentation metrics - Dasdan€¦ · Implicit presentation metrics • Statistical metrics – Precision/recall based on editorial data – Example: • Title from various sources

© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009. 172 172

Contributions and acknowledgments

•  Prasad Kantamneni – [email protected] –  Customer insights –  General consultation, eye-tracking study slides, and open problems

•  Rob Aseron – [email protected] –  Search & advertising metrics & analysis team –  Online user studies

•  Youssef Billawala – [email protected] –  Search result presentation –  Search result presentation issues, implicit metrics

•  Diane Yip – [email protected] –  Interaction designer –  Slides on SearchMonkey prototypes

Page 24: Presentation metrics - Dasdan€¦ · Implicit presentation metrics • Statistical metrics – Precision/recall based on editorial data – Example: • Title from various sources

173

Conclusions

Page 25: Presentation metrics - Dasdan€¦ · Implicit presentation metrics • Statistical metrics – Precision/recall based on editorial data – Example: • Title from various sources

© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009. 174

Conclusions

•  Continuous improvement needs metrics and analysis.

•  We have discussed measuring user satisfaction from many key angles. –  Optimizing for these metrics (the synthesis angle) is a

topic of its own. •  We listed many open problems for metrics.

–  There are lots of open problems for the synthesis part too.

•  We reviewed a set of pointers to the technical literature.

174

Page 26: Presentation metrics - Dasdan€¦ · Implicit presentation metrics • Statistical metrics – Precision/recall based on editorial data – Example: • Title from various sources

175

Q&A Thank you

Please send us your feedback {dasdan, kostas, emrev}@yahoo-inc.com