Presentation metrics - Dasdan€¦ · Implicit presentation metrics • Statistical metrics –...

Presentation metrics

Implicit Explicit

•  Small sample set •  Slow data collection

•  Statistical metrics •  Log analysis of real traffic •  Online user studies

•  Large sample set •  Fast data collection

•  User studies •  Editorial

Explicit presentation metrics

•  User studies –  In-home

•  Early ideation –  Generative studies

•  Participatory, paper printouts –  Usability

•  Prototypes •  Eye tracking studies

–  Mental models –  Focus groups

•  Editorial –  Comparative

•  Preferential or judgment values between contender configurations –  Perceived vs. actual

•  How well does presentation convey the content of the landing page?

Implicit presentation metrics

•  Statistical metrics –  Precision/recall based on editorial data –  Example:

•  Title from various sources (directories, web page, dynamic) •  Editors rate titles

•  Log analysis of real traffic –  User engagement

•  CTR, +/- clicks, query reformulations –  Session analysis –  User independent data (speed)

•  Online user studies –  Online surveys

Online user studies

•  Goal: –  To measure product experience

•  Various types –  General surveys of a product –  Task-specific exercises –  Commercial products: userzoom.com, keynote.com

•  Two dimensions of product experience: 1.  Measured user experience

•  Example: –  User given a set of tasks. –  What is the task completion success rate?

2.  Perceived user experience •  Many sub-dimensions

–  Easy or difficult to use –  Performance (e.g. response times) –  User-friendliness (e.g. fun, not user-engaging)

•  Example: –  User given a set of tasks. –  How easily (in her opinion) did user complete tasks?

[A’09]

Online user studies: The impact of a new release*

Product Experience Over Time

Time 1 Time 2 Time 3

Yahoo Competitor 1 Competitor 2

* To protect proprietary data and information this chart does not represent data from an actual study

Online user studies: Use-based product survey

Performance

Satisfaction

Learnability

High resolution image seen by the Fovea

Reduced visual acuity experienced by the parafovea

Progressively reducing visual acuity from the periphery of the retina

Users use parafoveal preview to identify the parts most likely to have relevant information based on the location of boldfaced terms

Familiar summary patterns draw user attention and clicks

Users are relatively blind to unfamiliar summary patterns.

Eye tracking studies

Reading (Yahoo! Finance) Scanning (Yahoo! Finance)

Heat maps

Highest density of clicks concentrated

in hottest zone.

[E’05]

Golden triangle

Golden triangle is habituated.

•  Result #1 is always more trusted and is considered more relevant by default.

•  Scan path gets narrower, and the user spends less time reading lower down the page.

Bolding in scan path

•  Users use bolding in titles to rapidly scan the SRP.

•  Bolding in scan path is critical to making users notice a result.

•  If a result is not bolded here, it is not noticed, and hence cannot be judged as relevant.

Design changes for Shortcuts

Before Bolding all over

Conversational title style

After Bolding in scan line

To the point title (query term – property)

[ROK’07]

SearchMonkey templates

How design templates evolve Case study: SearchMonkey

Users didn’t notice pictures in most cases

Images are consistently ignored except in some specific intents.

Deep links & structured data increased perception of clutter

The perception of ads Increased significantly

1st Round of SM Templates tested

How design templates evolve Case study: SearchMonkey

Moving the image out of the critical scan path helped users selectively discover it, without disrupting scanability.

Deep links were more discoverable when presented separately from the image

2nd Round of SM Templates tested

Key problems

•  What is the best way to manage user attention? – Seeing vs. noticing – When do we cross from being helpful to

being overwhelming (volume)? – How many different types of formats can

coexist (diversity)? •  How can presentation support user

intent? •  How can presentation be used to

communicate genre and topic?

References

•  [ABD’06] E. Agichtein, E. Brill, and S.T. Dumais (2006), Improving web search ranking by incorporating user behavior, SIGIR’06.

•  [ABDR’06] E. Agichtein, E. Brill, S.T. Dumais, and R. Ragno (2006). Learning user interaction models for predicting web search preferences,SIGIR’06.

•  [A’09] W. Albert (2009), Unmoderated usability testing: experience from the field, Usability Professionals Association Conference Panel.

•  [CKP’08] D. Chakrabarti, R. Kumar, and K. Punera (2008), Generating Succinct Titles for Web URLs, KDD’08.

•  [CKP’09] D. Chakrabarti, R. Kumar, and K. Punera (2009), Quicklink Selection for Navigational Query Results, WWW’09.

•  [CADW’07] C. Clarke, E. Agichtein, S. Dumais, and R. White (2007), The influence of caption features on clickthrough patterns in web search, SIGIR’07.

•  [E’05] Enquiro Eye Tracking Reports I & II, http://www.enquiroresearch.com/, June, 2005.

•  [HLZF’06] E. Hovy, C. Lin, L. Zhou, and J. Fukumoto (2006), Automated Summarization Evaluation with Basic Elements. LREC’06.

•  [KO’09] T. Kanungo and D. Orr (2009), Predicting Readability of Short Web Summaries, WSDM’09.

•  [MK’08] D. Metzler and T. Kanungo (2008), Machine Learned Sentence Selection Strategies for Query-Biased Summarization, SIGIR’08.

•  [ROK’07] D.E. Rose, D. Orr, and R.G.P. Kantamneni, Summary attributes and perceived search quality, WWW’07.

Contributions and acknowledgments

•  Prasad Kantamneni – kprasad@yahoo-inc.com –  Customer insights –  General consultation, eye-tracking study slides, and open problems

•  Rob Aseron – raseron@yahoo-inc.com –  Search & advertising metrics & analysis team –  Online user studies

•  Youssef Billawala – billawal@yahoo-inc.com –  Search result presentation –  Search result presentation issues, implicit metrics

•  Diane Yip – dyip@yahoo-inc.com –  Interaction designer –  Slides on SearchMonkey prototypes

Conclusions

•  Continuous improvement needs metrics and analysis.

•  We have discussed measuring user satisfaction from many key angles. –  Optimizing for these metrics (the synthesis angle) is a

topic of its own. •  We listed many open problems for metrics.

–  There are lots of open problems for the synthesis part too.

•  We reviewed a set of pointers to the technical literature.

Q&A Thank you

Please send us your feedback {dasdan, kostas, emrev}@yahoo-inc.com

Presentation metrics - Dasdan€¦ · Implicit presentation metrics • Statistical metrics –...

Documents

Transcript of Presentation metrics - Dasdan€¦ · Implicit presentation metrics • Statistical metrics –...

Transmedia metrics: Towards a new Transmedia Engagement Metrics model

The metrics

Social Metrics

Wallapop Metrics

Credit Risk Metrics

Metrics presentation

Implicit Implicit Scala

Ch4 Performance metrics

Software Metrics - University of Colorado Boulderkena/classes/5828/s12/presentation... · What are software metrics? ! Tools for anyone involved in software engineering to understand

SIKAP IMPLICIT MAHASISWA TERHADAP PEDULI ATAU … Implicit Mahasiswa... · SIKAP IMPLICIT MAHASISWA TERHADAP PEDULI ATAU ... mencerminkan reaksi otomatis yang bersifat segera dan

I metrics nov2011

Memes, Metrics + Milestones

Sabr metrics

Ensenyament explicit implicit

Mind The Metrics

Implicit Explicit Scala

Fleet Performance Metrics - Amazon Web Services - Fl… · o Presentation on Fleet Performance Metrics ... If you can’t measure something, ... one must be able to measure downtime.

Twitter Metrics

Mastering metrics - Poznań

Scor 10 Metrics