1

The Networked Economy (17) :Information Management, Strategy, and Innovation: , , Innovation Culture people & data | www.weigend.comAndreas S. Weigend, Ph.D. # people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 Three ingredients for innovation DataComputer ScienceDatabase ResearchDomain expertiseBehavioral EconomicsMarketingFinanceMethodologyStatistics Data AnalysisMachine Learningplus: Culture

# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 Was better in previTo take advantage of this

Relevant research

Data/TechnologyAmazon.com 50M Customers15M Products

Multiple touchpoints

Methodology/ PROCESSWhy Stats: QUANTIFYING UNCERTAINTY-- might want to insert MBA sheet HIGH NOISE(there is alwas EXTERNAL stuff as src of noise)Decision always made in uncertain environmentHIGH DIMENSIONALITYCharacterizationPrediction and DiagnosisE.g. Probabilistic Models

DomainE.g., Psychology, Behavioral EconomicsWhat actions can be taken?Pricing, promotions

Need for COLLABORATION

2Q: Understanding cross-channel shopping behaviorDrive online shoppers to traditional channels through trackable means E.g., giving them codes to redeemDrive offline shoppers to websiteE.g., Banana Republic giving offers with offline purchase to first online purchase Banana RepublicSurvey shoppersSurvey employeesMerge online and off-line transaction databases Collect additional dataVideoRFIDs# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 Doubleclick

3Approaches towards understanding customersStated preferencesSurveysOnlineOffline

Revealed preferencesAnalyze clicksMethodologies Server data Observe the storeBrowser data (e.g., toolbar)(Follow customer across sites

Gain insights by combining surveys with click analysis

# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 4Analyze clicksDefinitionsVisitVisit begins with first click on the siteTo avoid counting redirects, require minimum of 5 seconds until next click 5Visit ends after inactivity on the site of a specified time (e.g.,30 minutes)1530SessionWithin a site, session and visit used interchangeably Across sites (i.e., browser - or proxy data), session denotes a continuous string of visits Includes competitor sites and othersPurchaseA visit that contains a order confirmation page from the site# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 5Browser collected dataAdvantagesAdvantage of browser-side data Behavior across sitesOf course, also behavior within site (for a smaller subset of server log)Caching of web pages no problem (compared to site data and proxy data)DisadvantagesUser needs to download softwarePrivacy issuesMediaMetrix provides URLs onlyMediaMetrixURLNo purchase dataNo site characteristics# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 6An early study of online customer behavior Johnson et al. (1999): The Data MediaMetrix Panel from July 1997 through Feb. 99. (June 98 for CD, Books) MediaMetrix199771999219986CDTotal of 19,465 panelists, capture every URL.19465URLThere was still little shopping June 97-Feb. 99 976-992

# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 7Research Questions Characterize paths through website Modeling Online Browsing and Path Analysis Using Clickstream Databy Alan L. Montgomery, Shibo Li, Kannan Srinivasan, and John C. Liechty.Marketing Science (2004). Predict intention and modality of the visit Seize the Occasion by Horacio D. Rozanski, Gerry Bollman, and Martin Lipman. Strategy and Business (2001).Understand and influence conversion A Model of Web Site Browsing Behavior Estimated on Clickstream Data by Randolph E. Bucklin and Catarina Sismeiro, J of Marketing Research 40 (2003).Dynamic Conversion Behavior at E-Commerce Sites by Wendy W. Moe and Peter S. Fader. Management Science (2004).Compute and apply customer network value Mining the Network Value of Customers by Pedro Domingos and Matt Richardson, KDD-2001. ACM Press.# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800

# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 9

16.01

Also want to make recommendations!!# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 Revealed preferences vs stated preferencesLeverage users, not manual laborVery clean data!

Not manual editing!!!Belief: automatic

10Also want to make recommendations!!

Substitutes (buy instead of)Customers who shopped for X also shopped for Y (based on clicks)ZXComplements (buy in addition to)Customers who bought X also bought YXY

# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 Revealed preferences vs stated preferencesLeverage users, not manual laborVery clean data!

Not manual editing!!!Belief: automatic

11

# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 KEEP THIS AS FOCUSING STORY: WHAT DOES xxx do for us?

12

# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 Which one is better? No such thing without agreeing on objective

13Result: Right vs LeftMetricsConversion rate: Percentage of visits placing an order Order size: Number of additional (from the second page) items put into cart ResultYour Shopping Cart on right is about 1% better than on left Your Shopping Cart1%Some details

All customersExisting customersCart-adds from 2nd page: +0.6%Cart-adds from 2nd page:+0.8%Wishlist-adds: +1.4%DVD Cart-adds: DVD+0.8%DVD (USD):+1.1%DVD (USD):+1.0%# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 Total revenues: GMS(in treatment over control)

BUT: Interaction

14Amazon Shopping Cart RecommendationsSituation and ideaAdd an item to your shopping cart at a websiteMost sites show the cartAt Amazon, Greg Linden had the idea of showing recommendations based on cart itemsEvaluationPro: cross-sell more itemsCon: distract people from checking out VP asked to stop work on this idea- VP As with many new things, hard to decideA/B test was runA/B Feature is live on the sitefrom Greg Lindens Blog:http://glinden.blogspot.com/2006/04/early-amazon-shoppingcart.html

# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 ASW TO INNOVATIO15DiscussionStrength and weaknesses of such experiments??ExamplesSearch Inside the BookSponsored LinksPricing, promotions, Short-term hit vs sustainable long-term effectThe science behind itGeneralize from individual experiments to create scientific insightsDesign experiments appropriately

# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 Consequence: meaning = use

16Controlled ExperimentsMany names for same conceptA/B testsA/B Treatment and control groupsControlled experimentsRandomized Experimental Design

# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 Controlled ExperimentsMain ideaRandomly split traffic between two versionsTreatment: New idea(s)Control: Current live versionCollect metrics of interestAnalyzeStatistical testsData mining

# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 How old is this methodology?

First known controlled experiment in the 1700s18British captain noticed lack of scurvy in Mediterranean shipsCHad half the sailors eat limes (treatment), half did not (control)Experiment was so successful, British sailors are still called limeyslimeysNote: Success despite no understanding of vitamin C deficiencyC # people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 Advantages of Controlled ExperimentsControlled experiments test for causal relationships, not simply correlationsThey insulate external factors

They are a standard requirement for drug approval by US Federal Drug Administration (FDA )

But like most great things, there are problems and its important to recognize them

# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 Issues with Controlled ExperimentsIf you dont know where you are going, any road will take you thereLewis Carroll

Organization has to agree on key metric(s) to improveWhile it may seem obvious that we need to know if were improving, its not easy to get clear agreementIf nothing else, bringing this question to the surface is a great benefit to the org!

# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 Issues with Controlled ExperimentsQuantitative metrics, not always explanations of whyFor example, we may know that lemons work against scurvy, but not why; it may take a while to understand vitamin C deficiencyCData Mining may help identify segments where difference is large, leading to better understandingUsability studies also useful at explaining # people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 Issues with Controlled ExperimentsPrimacy effectChanging navigation in a website may degrade the customer experience (temporarily), even if the new navigation is betterEvaluation may need to focus on new users, or run for a long periodMultiple experimentsEven though the methodology shields an experiment from other changes, statistical variance increases making it harder to get significant resultsIt is useful to avoid multiple changes to the same area. QA also becomes harder when tests interact# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 23Good example of primacy: Office 2007 vs. Office 2003Issues with Controlled ExperimentsConsistency/contaminationOn the web, assignment is usually cookie-based, but people may use multiple computers, erase cookies, etc. Typically a small issuecookiecookieLaunch events / media announcements sometimes preclude controlled experiments/The journalists need to be shown the new version# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 24Good example of primacy: Office 2007 vs. Office 2003Issues with Controlled ExperimentsShort-term vs. Long-termHard to assess long term effects, such as customer abandonmentExample: if you optimize for ads for click-through revenues, you might plaster the site with ads. Long-term concerns should be part of metric (e.g., revenue per pixels of real estate on the window)# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800

# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 MIGHT WANT TO ADD POST a9 pictire

Points to make0) a9 launched mid-April 20041) after having used m/c learning, we can then get to the essence directly2) purpose of goldbox (+ the 10 hits)3) book

26Long-term effects Displaying Googles sponsored links within Amazon.coms site? Google Initial effect positive but how to model whether this helps or hurts the company in the long run?

Offer free shipping Competitors quickly adopt, how to model the overall game?

Pricing, Promotions

# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 Title: What can go wrong?

Add: Competitive changes (nonstationarity), drift of norms

27Lesson : A/B Experiments A/B(1) Formulate relevant questions (2)Ask your customers:Decide on set of metricsCreate a set of actionsRun A/B testA/BLearn from the experimentAct

# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 28Action: Invest USD 10M to improve customer satisfaction1000Base decision on analysis of behavioral dataQuantifyModelActConsider Increase selection? Increase availability? Reduce clutter on web site? Improve product search algorithms?

# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 DM to drive Strategic decisionHypothesis discovery

29Survey . Please let us know what you plan to do at Amazon.com todayFree responses, hand-coded into non-exclusive categoriesMultiple assignments possibleAverage: 1.2 categories per response1.2Number of responses: 10231023Response rate: 3.1%3.1%Date: February 11-12, 2003200311-12

# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 Say: self-selectionTurns out that respondents are more likely to be good customers / they care

REDO THISDrop THANK YOU, instead translation

Point of entryNote: Bias towards shopping

30Why do people visit? 35% Research: Response indicates having a target or topic for which customer wants to gather information in depth 31% Browse: Looks at items casually at a more general level than Research/16% Buy: Response indicates intent to buy in this visit (buy a book)10%Complain: Complains about some feature of the site9%Post-buy: Mentions checking order status, other account activities7% Community: Mentions usage of some feature of community, like reviews7% Price: Mentions getting information about price for specific items (does not include looking for deals Browse)# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 Part of EXPLICITLY COLLECTEDNote: A survey is a qualitative instrument primarily used for exploration. When extracting percentages, large systematic biases are possible due to the self-selection of respondents.

Unaided CHECK RECOGNIZED VS UNRECOGNIZED

31Why do people visit?2.9% Goldbox: Mentions Goldbox activity2.2% Gift: Indicates looking for a gift for someone else2.0% Sell: Mentions a selling activity1.4% Personalization: Mentions personalization feature (e.g., recommendations)0.8% Used: Mentions finding or transacting a used item# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 32Stated vs revealed preferences Obtain insights by combining individual survey response with click analysis:

Look at those who ended up buying something:Only about one-half of those making a purchase indicated that they wanted to buy something in this visit1/2

Look at those who said they wanted to buy something:Only about one-third of those indicating intent to buy ended up making a purchase in that visit1/3

# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 Starting with a purchase and looking back in time, when did the customer first look at the item Of those who did not indicate intend to buy, but actually bought:48% indicated browse48%42% indicated research42%

Influence / help the customers finding things whether or not they are .

Impact on satisfaction

Are they impulse buyers?How clear are respondents about their intentions?

IS EXAMPLE OF THE THREE CIRCLES

Q: How many buy without initially stating an intent to buy?

Q: How many who stated they want to buy actually buy?

33Customer satisfactionRelate satisfaction rating to:Visit intent (from survey)Actions (from clickstream)Intermediate goalPredict satisfactionUltimate goalIncrease satisfactionNumber of responses: 61176117Dates: March 1524, 20032003315-24

# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 34Outcome of the analysisCreated a9.com for web search and product searcha9.com# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800

How long ago did a customer first look at the detail page of an item eventually purchased? (i.e., conditioning on purchase)# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 20% of items bought today were looked at beforeAnalyze behavior of purchased items

How long does it take a customer to buy an item after they first looked at it?Distribution of time between ordering an item and the first look at its detail page

36(6)-37How long do customers take between adding an item to cart and checking out?

5% of items are purchased after having been in the cart for more than one week5%90.5% of ordered items were in cart for less than 24 hours before purchase90.5%24# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 37What makes great groups Great Groups? Source: Warren Bennis, USC, 1997http://www.leadertoleader.org/knowledgecenter/L2L/winter97/bennis.html# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 A shared dream At the heart of every Great Group is a shared dream.

All Great Groups believe that they are on a mission from God, that they could change the world, make a dent in the universe. They are obsessed with their work. It becomes not a job but a fervent quest. That belief is what brings the necessary cohesion and energy to their work.

E.g., Google wants to make the worlds information accessible Google

Source: Warren Bennis, USC, 1997http://www.leadertoleader.org/knowledgecenter/L2L/winter97/bennis.html

# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 There is a price to be part of the groupMembers pay a personal price.

Membership in a Great Group isnt a day job; it is a night and day job. Divorces, affairs, and other severe emotional fallout are typical, especially when a project ends. So groups strike a Faustian bargain for the intensity and energy that they generate.

# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 Curators, not creatorsGreat Groups make strong leaders.

On one hand, Great Groups all nonhierarchical, open, and very egalitarian. Yet they all have strong leaders. Thats the paradox of group leadership. You cannot have a great leader without a Great Group -- and vice versa. In an important way, these groups made the leaders great. The leaders were connoisseurs of talent, more like curators than creators.

# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 Abandon egos for the dream

Great groups manage conflict by abandoning individual egos to the pursuit of the dream.

At a critical point in the Manhattan Project a great chemist who later served as Dwight Eisenhowers chief scientific advisor threatened to quit because he couldnt get along with a colleague. Project leader Robert Oppenheimer simply said, George, how can you leave this project? The free world hangs in the balance. So conflict, even with these diverse people, is resolved by reminding people of the mission.

Robert Oppenheimer

E.g., Amazon, Google et al focus on measurement.

# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 RecruitingGreat Groups are the product of meticulous recruiting.

Cherry-picking the right talent for a group means knowing what you need and being able to spot it in others. It also means understanding the chemistry of a group. Candidates are often grilled, almost hazed, by other members of the group and its leader. You see the same thing in great coaches. They can place the right people in the right role. And get the right constellations and configurations within the group.

Hiring practice at Amazon, Microsoft et al. And firing practice (top grading).# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 AgeGreat Groups are usually young.

The average age of the physicists at Los Alamos was about 25. Oppenheimer -- the old man -- was in his 30s. But Great Groups are also young in their spirit, ethos, and culture. Most important, because theyre young and naive, group members dont know whats supposed to be impossible, which gives them the ability to do the impossible. Great Groups dont lack the experience of possibilities.

Los Alamos 25

Average age of employees at Google in 2007: 28 years.200728# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 DeliverReal artists ship.

Steve Jobs constantly reminded his band of Apple renegades that their work meant nothing unless they brought a great product to market. In the end, Great Groups have to produce a tangible outcome external to themselves. Most dissolve after the product is delivered; but without something to show for their efforts, the most talented assemblage becomes little more than a social club or a therapy group.

Yes!

# people & data | www.weigend.com | +1 650 906-5906 | +49 174 906-5906 | +86 138 1818 3800 Books_SampleDataHHIDSEXBIRTHTIMEACTIVEFDOMAINURL2118F29-May-3005OCT97:16:33:406www.amazon.com/exec/obidos/subst/home/home.html/0280/8610963/4075582118F29-May-3005OCT97:16:33:4610www.amazon.com/exec/obidos/ats/query/page/0280/8610963/4075582118F29-May-3005OCT97:16:33:5627www.amazon.com/exec/obidos/ats/query/0280/8610963/4075582118F29-May-3005OCT97:16:34:232www.amazon.com/exec/obidos/quicksearch/query/0280/8610963/4075582118F29-May-3005OCT97:16:34:3915www.amazon.com/exec/obidos/ats/query/0280/8610963/4075582118F29-May-3005OCT97:16:34:5437www.amazon.com/exec/obidos/quicksearch/query/0280/8610963/4075582118F29-May-3005OCT97:16:35:311www.amazon.com/exec/obidos/ats/query/0280/8610963/4075582118F29-May-3005OCT97:16:35:320www.amazon.com/exec/obidos/ats/query/page/0280/8610963/4075582118F29-May-3005OCT97:16:35:321www.amazon.com/exec/obidos/subst/home/home.html/0280/8610963/4075582118F29-May-3005OCT97:16:35:3910www.barnesandnoble.com/index.asp?userid=0b970iog7e2118F29-May-3005OCT97:16:35:4919shop.barnesandnoble.com/booksearch/search.asp?userid=0b970iog7e2118F29-May-3005OCT97:16:36:085shop.barnesandnoble.com/booksearch/results.asp?userid=0b970iog7e&mscssid=r3k3qpxn7csh22118F29-May-3005OCT97:16:36:1317shop.barnesandnoble.com/booksearch/search.asp?userid=0b970iog7e2118F29-May-3005OCT97:16:36:3016shop.barnesandnoble.com/booksearch/results.asp?userid=0b970iog7e&mscssid=r3k3qpxn7csh22118F29-May-3005OCT97:16:36:460shop.barnesandnoble.com/booksearch/search.asp?userid=0b970iog7e2118F29-May-3005OCT97:16:36:461www.barnesandnoble.com/index.asp?userid=0b970iog7e2118F29-May-3005OCT97:16:38:253www.books.com/scripts/news.exe2118F29-May-3005OCT97:16:38:341www.books.com/scripts/search1.exe?sid~gnsl5wduel0htne2118F29-May-3005OCT97:16:38:4714www.books.com/scripts/search.exe?sid~gnsl5wduel0htne