Presentation metrics - Dasdan€¦ · Implicit presentation metrics • Statistical metrics –...
Transcript of Presentation metrics - Dasdan€¦ · Implicit presentation metrics • Statistical metrics –...
© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009. 150 150
Presentation metrics
Implicit Explicit
• Small sample set • Slow data collection
• Statistical metrics • Log analysis of real traffic • Online user studies
• Large sample set • Fast data collection
• User studies • Editorial
© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009. 151
Explicit presentation metrics
• User studies – In-home
• Early ideation – Generative studies
• Participatory, paper printouts – Usability
• Prototypes • Eye tracking studies
– Mental models – Focus groups
• Editorial – Comparative
• Preferential or judgment values between contender configurations – Perceived vs. actual
• How well does presentation convey the content of the landing page?
© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009. 152
Implicit presentation metrics
• Statistical metrics – Precision/recall based on editorial data – Example:
• Title from various sources (directories, web page, dynamic) • Editors rate titles
• Log analysis of real traffic – User engagement
• CTR, +/- clicks, query reformulations – Session analysis – User independent data (speed)
• Online user studies – Online surveys
© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009. 153
Online user studies
• Goal: – To measure product experience
• Various types – General surveys of a product – Task-specific exercises – Commercial products: userzoom.com, keynote.com
• Two dimensions of product experience: 1. Measured user experience
• Example: – User given a set of tasks. – What is the task completion success rate?
2. Perceived user experience • Many sub-dimensions
– Easy or difficult to use – Performance (e.g. response times) – User-friendliness (e.g. fun, not user-engaging)
• Example: – User given a set of tasks. – How easily (in her opinion) did user complete tasks?
[A’09]
© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009. 154
Online user studies: The impact of a new release*
Product Experience Over Time
65
70
75
80
85
90
Time 1 Time 2 Time 3
PEM
Yahoo Competitor 1 Competitor 2
* To protect proprietary data and information this chart does not represent data from an actual study
© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009. 155
Online user studies: Use-based product survey
© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009. 156
Online user studies: Use-based product survey
© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009. 157
Online user studies: Use-based product survey
Performance
Satisfaction
Learnability
158
High resolution image seen by the Fovea
Reduced visual acuity experienced by the parafovea
Progressively reducing visual acuity from the periphery of the retina
© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009
159
Users use parafoveal preview to identify the parts most likely to have relevant information based on the location of boldfaced terms
© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009
160
Familiar summary patterns draw user attention and clicks
Users are relatively blind to unfamiliar summary patterns.
© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009
© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009.
Eye tracking studies
161
Reading (Yahoo! Finance) Scanning (Yahoo! Finance)
© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009. 162 162
Heat maps
Highest density of clicks concentrated
in hottest zone.
[E’05]
© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009. 163 163
Golden triangle
Golden triangle is habituated.
• Result #1 is always more trusted and is considered more relevant by default.
• Scan path gets narrower, and the user spends less time reading lower down the page.
© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009. 164 164
Bolding in scan path
• Users use bolding in titles to rapidly scan the SRP.
• Bolding in scan path is critical to making users notice a result.
• If a result is not bolded here, it is not noticed, and hence cannot be judged as relevant.
© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009. 165 165
Design changes for Shortcuts
© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009. 166 166
Design changes for Shortcuts
Before Bolding all over
Conversational title style
After Bolding in scan line
To the point title (query term – property)
[ROK’07]
© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009. 167
1.
2.
3.
4.
167
SearchMonkey templates
© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009. 168 168
How design templates evolve Case study: SearchMonkey
Users didn’t notice pictures in most cases
Images are consistently ignored except in some specific intents.
Deep links & structured data increased perception of clutter
The perception of ads Increased significantly
1st Round of SM Templates tested
© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009. 169 169
How design templates evolve Case study: SearchMonkey
Moving the image out of the critical scan path helped users selectively discover it, without disrupting scanability.
Deep links were more discoverable when presented separately from the image
2nd Round of SM Templates tested
© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009.
Key problems
• What is the best way to manage user attention? – Seeing vs. noticing – When do we cross from being helpful to
being overwhelming (volume)? – How many different types of formats can
coexist (diversity)? • How can presentation support user
intent? • How can presentation be used to
communicate genre and topic?
170
© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009. 171 171
References
• [ABD’06] E. Agichtein, E. Brill, and S.T. Dumais (2006), Improving web search ranking by incorporating user behavior, SIGIR’06.
• [ABDR’06] E. Agichtein, E. Brill, S.T. Dumais, and R. Ragno (2006). Learning user interaction models for predicting web search preferences,SIGIR’06.
• [A’09] W. Albert (2009), Unmoderated usability testing: experience from the field, Usability Professionals Association Conference Panel.
• [CKP’08] D. Chakrabarti, R. Kumar, and K. Punera (2008), Generating Succinct Titles for Web URLs, KDD’08.
• [CKP’09] D. Chakrabarti, R. Kumar, and K. Punera (2009), Quicklink Selection for Navigational Query Results, WWW’09.
• [CADW’07] C. Clarke, E. Agichtein, S. Dumais, and R. White (2007), The influence of caption features on clickthrough patterns in web search, SIGIR’07.
• [E’05] Enquiro Eye Tracking Reports I & II, http://www.enquiroresearch.com/, June, 2005.
• [HLZF’06] E. Hovy, C. Lin, L. Zhou, and J. Fukumoto (2006), Automated Summarization Evaluation with Basic Elements. LREC’06.
• [KO’09] T. Kanungo and D. Orr (2009), Predicting Readability of Short Web Summaries, WSDM’09.
• [MK’08] D. Metzler and T. Kanungo (2008), Machine Learned Sentence Selection Strategies for Query-Biased Summarization, SIGIR’08.
• [ROK’07] D.E. Rose, D. Orr, and R.G.P. Kantamneni, Summary attributes and perceived search quality, WWW’07.
© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009. 172 172
Contributions and acknowledgments
• Prasad Kantamneni – [email protected] – Customer insights – General consultation, eye-tracking study slides, and open problems
• Rob Aseron – [email protected] – Search & advertising metrics & analysis team – Online user studies
• Youssef Billawala – [email protected] – Search result presentation – Search result presentation issues, implicit metrics
• Diane Yip – [email protected] – Interaction designer – Slides on SearchMonkey prototypes
173
Conclusions
© Dasdan, Tsioutsiouliklis, Velipasaoglu, 2009. 174
Conclusions
• Continuous improvement needs metrics and analysis.
• We have discussed measuring user satisfaction from many key angles. – Optimizing for these metrics (the synthesis angle) is a
topic of its own. • We listed many open problems for metrics.
– There are lots of open problems for the synthesis part too.
• We reviewed a set of pointers to the technical literature.
174
175
Q&A Thank you
Please send us your feedback {dasdan, kostas, emrev}@yahoo-inc.com