1. Data-driven Design Making Software & Websites Better Through Use of Statistics Patrick McKenzie April 15th, 2010 Creative Commons Licensed CC-BY.Please feel free to share.

2. Outline

Introduction

Art & Science

Rationale

Instrumentation

Funnel Analysis

A/B Testing

Conclusion

A/B

3. D:Downloads2264763977_fbeb2e34ba.jpg The Plan 4. What Happened D:DropboxMy DropboxPhotosSakura Season 2010IMG_0348.JPG 5. D:Downloads3452417503_88c32d1397.jpg What Happened Next 6. Started Business 7. Art into Science Supplement the vision and craft of designers with the logic, regimentation, and process of science. 8. Rationale

Decrease Costs

Decrease Uncertainty

Increase Satisfaction

Increase Success

Increase Profits

9. Designing With Data

Get Facts to Answer The Questions:

• What are users doing?

• Which users are doing things?

• Where are they doing things?

• (Maybe) Why are they doing it?

10. Types of Instrumentation

Analytics software

Heatmapping

Surveys

Ad hoc analysis (Excel, etc)

Custom software

11. CrazyEgg 12. Landing Page

Photo of problem solved goes here

13. Landing Page (Fixed) 14. Paving The Trails

Picture of paving a path

15. Instrumenting Software 16. Instrumenting Software #2 17. Funnel Analysis

Many applications have several steps followed by a goal.

Only a percentage of users clear each step.

Funnel analysis finds where folks are falling out, to concentrate efforts on improvement

Viewed Landing Page Signed Up For Trial Did Work $$ 18. Software

Google Analytics

• Free but limited

• Good for starter

Mixpanel

• Cheap

• API

• Harder to use

19. Define Funnel

Signs In

Opens Word List

Saves Word List

Customizes Cards

Creates Cards

Downloads Card

20. Collect Data

Signs In: 260

Opens Word List: 218 (83%)

Saves Word List: 191 (87%)

Customizes Cards 161 (84%)

Creates Cards: 131 (81%)

Downloads Card 126 (95%)

Total Conversion Rate 126 / 260 = ~48.4% 21. Identify Areas for Improvement

Signs In: 260

Opens Word List: 218 (83%)

Saves Word List: 191 (87%)

Customizes Cards 161 (84%)

Creates Cards: 131 (81%)

Downloads Card 126 (95%)

Total Conversion Rate 126 / 260 = ~48.4% 22. Implement Improvements

Open Word List needlessly complicated: makeeasier.

Save Button was unclear: tweak microcopy, button design.

Customize Cards needlessly complicated: show less options.

23. Before 24. After 25. Collect Metrics

Signs In: 194

Opens Word List: 155 (79%)

Saves Word List: 151 (97%)

Customizes Cards: 127 (84%)

Creates Cards: 112 (88%)

Downloads Card 126 (98%)

Up from 83% Up from 81% Total Conversion Rate goes from 48.5% to 56.7% 16.6% lift in customer success / revenue for 1 hour of work . 26. http://farm1.static.flickr.com/21/33346213_e5d5842024.jpg Pretend You Own A Bar Salty Peanut by EuroMagic. Half of Customers Get... Half of Customers Get... 27. A/B Testing

Divide similar population into two groups randomly.

Treat the groups differently.

Measure difference in outcomes.

Scientific Method applied to design .Eliminates bias, randomness, poor test design.

28. Actual A/B Test Half of Users See... Half of Users See... 29. Software Tools

Google Website Optimizer

• Terrible.Do not use.

Visual Website Optimizer

• visualwebsiteoptimizer.com

• WYSIWYG alternative creation.Verynice.

• Free for now.

A/Bingo or similar

• Rails A/B testing library.

• Programmer friendly, powerful, easy to use.

30. Core A/B Testing Loop

Identify conversion and instrument.

Create alternatives.

Assign users randomly to alternatives

Watch behavior.

Identify winning alternative.

Iterate, iterate, iterate...

31. Landing Pages experianvasm experianvbsm Source: whichtestwon.com 32. Signup Forms http://www.bingocardcreator.com/blog-images/signup-ab-tests/old-boring-signup.png 33. Signup Forms http://www.bingocardcreator.com/blog-images/signup-ab-tests/new-hotness-signup.png 34. Signup Forms Vast contact dealer form 35. New Features http://farm3.static.flickr.com/2422/3981364314_d4b30cb739.jpg

• 60 ~ 80% of features do not change user behavior

• Eric Ries

36. You can quickly create bingo cards using our included wizards! Old Version 37. C:UserspatrickPictures ew-version.png Spot The New Feature 38. Core A/B Testing Discipline

Accept that you do not know what is right.( Hardest lesson. )

Test frequently, as a matter of routine.(Helps to have one-line testing.)

Test changes in isolation.

Use evidence from tests to inform your marketing and development plans.

39. Conclusion

What we learned today:

• Importance of tracking user interaction.

• Funnel analysis measuring user success and goal completion.

• A/B tests improving marketing message and product design.

40.

Thanks for listening.

[email_address] My email.

http://www.kalzumeus.com My blog.

http://www.abingo.org A/Bingo software (OSS)

http://www.startuplessonslearned.com Eric Ries, very useful.

41. Microsoft logo: http://www.flickr.com/photos/scobleizer/2264763977/ Sleeping Salaryman: http://www.flickr.com/photos/kansai/3452417503/ Edamame (Green Soybeans): http://www.flickr.com/photos/kankan/33346213/ Salty Peanuts: http://www.flickr.com/photos/euromagic/2351628831/ Facepalm Statue: http://www.flickr.com/photos/cesarastudillo/3981364314/ Photo Credits