8/13/2019 An Cova 2000
1/81
Analytic Insights Group Slide 1 ancova2000.ppt (08/00)
Analyzing Test Results (ANCOVA)
Using this presentation: This is an internal trainingpresentation created for use at Testing Boot Camp.Select slides may be extracted for use with clients.Filename: ANCOVA2000.PPTPrepared by: Valerie Skala and Bob Bakshis based on
charts created by Bruce Richardson and Andy FisherLast Revised: August 8, 2000Other resources: Steve LowtherKey Words: Analysis of Covariance, ANCOVA, test
analysis, covariate
8/13/2019 An Cova 2000
2/81
Analytic Insights Group Slide 2 ancova2000.ppt (08/00)
Analyzing Test Results
using Analysis of Covariance
(ANCOVA)
8/13/2019 An Cova 2000
3/81
Analytic Insights Group Slide 3 ancova2000.ppt (08/00)
Signposts
Part 1 -- For Everyone Intro to Analysis of Covariance The process of test analysis
How ANCOVA works Explanation of covariates Exercise: Non-Standard Designs
Part 2 -- For Statisticians More details on the ANCOVA model
8/13/2019 An Cova 2000
4/81
Analytic Insights Group Slide 4 ancova2000.ppt (08/00)
Case StudyPepsi is interested in determining the effect of a $40million ad campaign on sales of Aquafina bottledwater.
They need to see at least a +15% boost in brand
sales to make the ad campaign profitable.They run the ad campaign in Dallas.
They compare their sales in the test market pre-to-
post, using Houston as the control, and see a net+10% boost in sales.
Should they roll the ad campaign nationally?
8/13/2019 An Cova 2000
5/81
Analytic Insights Group Slide 5 ancova2000.ppt (08/00)
The Answer: It Depends!
Raw numbers do NOT tell the whole story!
What if one of these scenarios took place?
Test Cell Control Cell Net
Comp Dealing flat Comp Dealing UP
Aquafina Sales UP Aquafina Sales flat Advertis ing effect may
really be zero.Comp Dealing UP Comp Dealing flat
Aquafina Sales UP Aquafina Sales flat Advertis ing effect may bemuch greater.
8/13/2019 An Cova 2000
6/81
Analytic Insights Group Slide 6 ancova2000.ppt (08/00)
We need to make sure were not justseeing changes in sales caused by:
Pre existing differences between the test andcontrol cell -- no match is perfect E.g., category development
Differences that arose during the test period
between test & control cells -- the marketplaceis always changing E.g., price, feature, display -- of the test brand or
a key competitor
Unknown variables /normal random variation-- aka error variance
What we cant control by design,we attempt to control with statistics.
8/13/2019 An Cova 2000
7/81Analytic Insights Group Slide 7 ancova2000.ppt (08/00)
ANCOVA is an ex post facto statisticaltechnique that
Levels the playing field so we can isolate the true test effect Identify imbalances in non-treatment factors, pre vs. post and test vs.
control Use weekly data to estimate the effect of these imbalances on sales Adjust the 4 sales means for these imbalances Report a covariate-adjusted net test effect
Provides a statistical test of significance How confident are we that the reported test effect could not be due
strictly to chance? What is the likely error range around our estimate of the effect?
Analysis of Covariance (ANCOVA) answers the question:What would the test effect be if the test and control markets were held equal
in all respects except for the test treatment?
8/13/2019 An Cova 2000
8/81Analytic Insights Group Slide 8 ancova2000.ppt (08/00)
We have weekly data on sales, price, & merchandisingfor the test brand, competitors, and total category
5/10 6/7 7/5 8/2 8/30 9/27 10/25 11/22 12/20 1/17 2/14 3/14 4/11 5/9 6/6 7/4 8/1 8/29 9/26
4 Weeks Ending
0
200
400
600
800
1000 Volume Sales Price
$2.00
$3.00
$4.00
$5.00
$6.00
Pre PostPrice
8/13/2019 An Cova 2000
9/81Analytic Insights Group Slide 9 ancova2000.ppt (08/00)
We can summarize the weekly data into 4 means
Dummy variables are added to each store-week record toindicate which Group (test or control) and which Period(pre or post) the record belongs in.
CumPre-TestWeeks
CumPost-Test
WeeksChange
AggregatedTest Group A B (B - A)
AggregatedControl Group C D (D - C)
Diff of Diffs
% Effect
Measure (sales, price, merchandising)
D %
8/13/2019 An Cova 2000
10/81Analytic Insights Group Slide 10 ancova2000.ppt (08/00)
But ANCOVA is only one piece of the process
A good analysis starts with a gooddesign
ANCOVA cannot
fix a bad design.
Keep your brain engagedand your team talking
There are many pieces to ananalysis, not just the ANCOVA
8/13/2019 An Cova 2000
11/81Analytic Insights Group Slide 11 ancova2000.ppt (08/00)
Signposts
Part 1 -- For Everyone Intro to Analysis of Covariance The process of test analysis How ANCOVA works Explanation of covariates Exercise: Non-Standard Designs
Part 2 -- For Statisticians More details on the ANCOVA model
8/13/2019 An Cova 2000
12/81Analytic Insights Group Slide 12 ancova2000.ppt (08/00)
What to discuss at project kick- off before its too late!
Review proposal, including test objectives anddesign . Any known issues or concerns with sample size,
sensitivity, unusual factors we may (or may not) beable to control for?
What client expectations have been set or shouldbe set?
Do match criteria line up with what we plan toanalyze?
8/13/2019 An Cova 2000
13/81Analytic Insights Group Slide 13 ancova2000.ppt (08/00)
What to discuss at project kick- off before its too late!(continued)
Expand everyones knowledge base. Review marketing plans -- do we have full info? Review category and brand dynamics -- slow
moving? Heavily merchandised or couponed?
8/13/2019 An Cova 2000
14/81Analytic Insights Group Slide 14 ancova2000.ppt (08/00)
What to discuss at project kick- off before its too late!(continued)
If were doing toplines, what do we need to doto ensure that final analysis data will line up? Calculate volume sales per $MM ACV adjusted
for distribution or not? Adjust and project trial and repeat figures?
8/13/2019 An Cova 2000
15/81Analytic Insights Group Slide 15 ancova2000.ppt (08/00)
1. From ANALYZER get unadjusted data and calculate rawtest effect.
+0.03 - -0.66 = +0.69 / 2.83 = +24.4%
3.49 3.493.52 2.83
Test Markets Control Markets0.00
1.00
2.00
3.00
4.00
5.00
Volum e Sales per $MM ACV Pre Post
8/13/2019 An Cova 2000
16/81Analytic Insights Group Slide 16 ancova2000.ppt (08/00)
2. Review unadjusted results. Any obvious covariates jump out: number UPCs moving, price, merchandising-- for test brand and key competitors?
5/10 6/7 7/5 8/2 8/30 9/2710/2511/2212/201/17 2/14 3/14 4/11 5/9 6/6 7/4 8/1 8/29 9/26
4 Weeks Ending
0
1
2
3
4
5Test Brand Volume Sales per $MM ACV Test Control
Pre-test match Test
8/13/2019 An Cova 2000
17/81Analytic Insights Group Slide 17 ancova2000.ppt (08/00)
3. Determine the dependent variable and independentvariables to include in the analysis.
- Should we assume 100% distribution on test brand or not?- Is brand share large, such that we should adjust for All Other (category
minus test brand) development instead of Total Category development?
Adjustments Made to level the Playing Field
Causal Factors(covariates)
Diff of Diffs(Post - Pre,Test - Cntrl)
Impact on TestEffect
Test Brand % BVol on Disp Only _+4_
Test Brand % BVol on Feat Only _+5_
Test Brand Average Price _-3_
CompA Average Price _-2_
8/13/2019 An Cova 2000
18/81Analytic Insights Group Slide 18 ancova2000.ppt (08/00)
4. Run ANCOVA with just one test brand covariate (e.g.,NUPC1), to make sure raw means match Analyzer #s.*
* This QC check helps to ensure that the right stores and weeks havebeen pulled in, the right projection system and scan rate adjustmentshave been used, and the stub is brand flagged correctly. ANCOVA #sshould match InfoView numbers within 1%. If were losing too manyobservations due to BIDs with no data; may need to re-BID or changethe Assume Distribution parameter to YES
3.49 3.493.522.83
Test Control0.00
1.00
2.00
3.00
4.00
5.00Volum e Sales per $MM ACV
Pre Post
3.49 3.493.522.83
Test Control0.001.00
2.00
3.00
4.00
5.00Volum e Sales per $MM ACV
Pre Post
ANCOVA Analyzer
8/13/2019 An Cova 2000
19/81Analytic Insights Group Slide 19 ancova2000.ppt (08/00)
Test BrandRaw vs. Adjusted Results
+ 24.4%
+ 12.0%
Raw Test
Effect
Adjusted Test
Effect
How is each covariate impacting the test effect?
Causal Factors (covariates)
Incrs inCovariateCauses ?
Chg in Sales
Test Brand %BVol Disp Only + Test Brand %BVol Feat Only + Test Brand %BVol TPR Only + Test Brand %BVol Feat+Disp +
CompA %BVol Any Merch - Category Vol/$MM ACV +
Test Brand Price - Competitor Price +
Adjustments to Level the Playing Field
5. Run kitchen sink ANCOVA with all covariates
8/13/2019 An Cova 2000
20/81Analytic Insights Group Slide 20 ancova2000.ppt (08/00)
Test BrandRaw vs. Adjusted Results
+ 24.4%
+ 12.0%
Raw Test
Effect
Adjusted Test
Effect
6. On the third and consecutive ANCOVA runs,add/remove/modify covariates to improve the model.
Causal Factors (covariates*)
Diff ofDiffs
Impact onTest Effect
Test Brand % BVol on Disp Only _+4_ _-6.4_
Test Brand % BVol on Feat Only _+5_ _-6.0_
Test Brand Average Price _-3_ _-2_
CompA Average Price _-2_ _+2_
Adjustments to Level the Playing Field
Focus on the covariates that represent 75% of theadjustment.
8/13/2019 An Cova 2000
21/81Analytic Insights Group Slide 21 ancova2000.ppt (08/00)
7. Report adjusted test effect with confidence level and interval.
+12%
+20%
Raw Test
Effect
Covariate Adjusted
Test Effect*
+12%
+20%
+28%
Confidence Interval
Low End*
Best Estimate
of Test Effect
Confidence Interval
High End*
* 99% confidence level * 80% confidence interval
8/13/2019 An Cova 2000
22/81
Analytic Insights Group Slide 22 ancova2000.ppt (08/00)
Each outlet is analyzed individually, but only theCombined Outlet result is reported to the client.
Outlet(% of "Analyzed Product"
Volume by Outlet)
Unadjusted % Test Effect
Covariate Adjusted
% Test Effect
Confidence Level
Comb ined Out lets (100%) 18.6 8.0 98%
Drug (40%) 20.8 9.0 99%
Food (30%) 24.4 12.0 98%
Mass Merch (30%) 9.9 2.5 81%
8. After analyzing each outlet individually,calculate the combined-outlet effect.
8/13/2019 An Cova 2000
23/81
Analytic Insights Group Slide 23 ancova2000.ppt (08/00)
About three-fourths oftests had a statistically
significant effect
Of those that worked,almost one-fourth had aneffect greater than 20%.
73% 27% 22 8
Up to 5%
5.1 to 10%
10.1 to 20%
> 20% effect
32%
23%
23%
23%
n = 30
9. Compare the results of this testvs. benchmarks
8/13/2019 An Cova 2000
24/81
Analytic Insights Group Slide 24 ancova2000.ppt (08/00)
Are we confident that our figures are accurate? Do all of the data tie together?
Are we drawing the correct conclusions from the data? Are any of the findings going to be politically sensitive? What actionable recommendations can we make?
10. Analyst explains the model and the final results indetail. Client presentation is prepared.
8/13/2019 An Cova 2000
25/81
Analytic Insights Group Slide 25 ancova2000.ppt (08/00)
11. Project test market results to the roll-out universe
Translate test market results to Total U.S.Consider post period differences between testmarket and national in terms of:
- sales development and trends- distribution of key items- competitive environment- merchandising
Compare to action standard and calculate payout
8/13/2019 An Cova 2000
26/81
Analytic Insights Group Slide 26 ancova2000.ppt (08/00)
Expected national unit sales 95,000,000 in implementation year*
x % test effect x 0.12 ---------------
Incremental national unit sales expected from new TV advertising 11,400,000
11A. Translate test market results to national
* Brand sales have been declining from the 100 million units sold in the pre-test year.
+
8/13/2019 An Cova 2000
27/81
Analytic Insights Group Slide 27 ancova2000.ppt (08/00)
11B. Compare test effect to action standard.Calculate pay-out.
This program should pay for itself in approx. 3 years.
Incremental unit sales in 1 year 11,400,000x Profit per unit x $0.50Incremental profit in 1 year $ 5,700,000
+ 50%* of year 1 effect in year 2 $ 2,850,000+ 25%* of year 1 effect in year 3 $ 1,425,000Total incremental profit in 3 yrs $ 9,975,000Investment in TV advertising $10,000,000*
* These assumptions are loosely based on the 1989 IRI "How Advertising Worksstudy. Of 44 ad weight tests on established brands that showed a significantpositive effect in the test year, the average effects were +22% in year 1, +14%
in year 2, and +7% in year 3.
8/13/2019 An Cova 2000
28/81
Analytic Insights Group Slide 28 ancova2000.ppt (08/00)
Take-Aways from this section
8/13/2019 An Cova 2000
29/81
Analytic Insights Group Slide 29 ancova2000.ppt (08/00)
Signposts
Part 1 -- For Everyone Intro to Analysis of Covariance The process of test analysis How ANCOVA works Explanation of covariates Exercise: Non-Standard Designs
Part 2 -- For Statisticians More details on the ANCOVA model
8/13/2019 An Cova 2000
30/81
Analytic Insights Group Slide 30 ancova2000.ppt (08/00)
,
,,
,
0 1 2 3
X
02
4
6
8
10Equation for this line: y = 3 + 2x
Remember the linear regression equation? y = b + 1X
Y
y = Total Sales, b = Base Sales, X = Level of Display,1 = strength of display's influence on sales
Actual obs ervations
ANCOVA starts with linear regression, to measure theimpact of each marketing mix variable on sales
8/13/2019 An Cova 2000
31/81
Analytic Insights Group Slide 31 ancova2000.ppt (08/00)
In multiple linear regression, we fit a plane to the datapoints.
* **
***
*
_
_
_ _ _
_ _
1.8
2.0
2.2
2.4
2.6
2.8
3.0
S a
l e s
R a
t e
Group0.5 0
68
10 12
14
8/13/2019 An Cova 2000
32/81
Analytic Insights Group Slide 32 ancova2000.ppt (08/00)
In multiple linear regression, we fit a plane to the datapoints. This plane can be defined with an equation.
A simplified example is shown below.
.y = b + 1X1 + 2X2
Sales Rate = Base Sales Rate + (Effect of Variable1 * Value of Variable1)+ (Effect of Variable2 * Value of Variable2)
Sales Rate = 2.0043 + (0.1783 * 1) + (0.0174 * 11)
2.0043 = Sales rate when there is no display0.1783 = Effect of being in the test group
1 = 1 for test group, 0 for control group0.0174 = Increase in sales when display increases 1%11 = % of base volume on display
8/13/2019 An Cova 2000
33/81
Analytic Insights Group Slide 33 ancova2000.ppt (08/00)
ANCOVA combines two statistical techniques
Multiple Linear Regression Using all store-week observations (regardless of group or period), we
measure the degree to which the level of a non-treatment variable, orcovariate (X) influences sales.
The coefficient ( 1) represents the % difference in sales we wouldexpect from a 1% imbalance in the level of that variable.
The program partitions sales variability into components, representingsales attributable to:
Each covariate, as defined by the analyst The Group-by-Period term, or actual test effect Remaining unexplained variance, or error variance
When we have an equation that seems to predict sales well, the fourraw sales means are adjusted to represent what sales would havebeen, if each covariate were set to the average level across all groupsand periods
8/13/2019 An Cova 2000
34/81
Analytic Insights Group Slide 34 ancova2000.ppt (08/00)
ANCOVA combines two statistical techniques
continued
Analysis of Variance
Compares the 4 adjusted means to determine whetherthe difference is greater than the products normal sales
variation.
Tests for statistical significance: how sure are we thatthere is a difference?
Considers the degree of unexplained variability in thesales data to estimate a likely error range around theestimated test effect.
ANCOVA dj h 4 l h h
8/13/2019 An Cova 2000
35/81
Analytic Insights Group Slide 35 ancova2000.ppt (08/00)
ANCOVA adjusts the 4 sales means to what theywould be if display were at the average level
Pre-Test Post-Test Diff % Effect
RAW SALES
Ad Group 100 120 20 9.1%
No Ad Group 100 110 10DISPLAY
Ad Group 5.0 7.0 Avg = 6.125
No Ad Group 6.0 6.5
ADJ SALES
Ad Group 105.0 116.1 11.1 3.1%
No Ad Group 100.6 108.3 7.8
8/13/2019 An Cova 2000
36/81
Analytic Insights Group Slide 36 ancova2000.ppt (08/00)
Core Principles of Testing
Its better to be vaguely rightthan precisely wrong .
Draw conclusions only from properlyadjusted data.
We cant adjust for everything, butthe more we DO adjust for, thecloser we can get to the rightanswer.
2 TPRs1 Display
2 Features
2 TPRs1 Feature
Comparing Test vs.
Control is like using anold-fashioned scale:It needs a skillful touchto balance.
8/13/2019 An Cova 2000
37/81
Analytic Insights Group Slide 37 ancova2000.ppt (08/00)
Core Principles of Testing
Gut-check and reality-check . The covariate adjustments and the final
adjusted test effect must make sense whencompared to the unadjusted sales &merchandising data.
A red light should go off if we see a largeadjustment to volume for the price ofcompetitive brand A, but the raw data shows
only a small difference in brand As price. Test your hypotheses in more than one way
and look for converging evidence
8/13/2019 An Cova 2000
38/81
Analytic Insights Group Slide 38 ancova2000.ppt (08/00)
Core Principles of Testing
A test is successful when we learnsomething new and actionable which has adirect impact on our profitability.
Many tests do NOT return a positive result --after all, if a proposition was as sure thing, itwouldnt need to be tested.
Still every test IS a success when it yields sometype of actionable new information or gets usthinking in new directions
8/13/2019 An Cova 2000
39/81
Analytic Insights Group Slide 39 ancova2000.ppt (08/00)
How is Testings ANCOVA different from MM Models?
Testing analyses have a more narrow goal:
We are only interested in adjusting thegroup/period sales means in order to isolate thetest effect.
We are not interested in the ability of the equationto predict the sales impact of changes in a specificvariable.
Testing removes outlier* HHs for panel analyses.Testing also drops outlier store-weeks when
appropriate.* HHs whose category volume is >= 3 standard deviations from the mean
by market (test+control HHs combined), by period (pre and post)
8/13/2019 An Cova 2000
40/81
Analytic Insights Group Slide 40 ancova2000.ppt (08/00)
How is Testings ANCOVA different from MM Models?(continued)
Modeling analyses generally have a broader goal:
To create a marketing mix model that can correctlyestimate the sales impact of each variableindividually
Modeling drops imputed store-weeks; Testing keeps them.
Modeling mean-centers all records; Testing does not.
8/13/2019 An Cova 2000
41/81
Analytic Insights Group Slide 41 ancova2000.ppt (08/00)ancova00.pr4
Signposts
Part 1 -- For Everyone Intro to Analysis of Covariance The process of test analysis How ANCOVA works
Explanation of covariates Exercise: Non-Standard Designs
Part 2 -- For Statisticians
More details on the ANCOVA model
8/13/2019 An Cova 2000
42/81
Analytic Insights Group Slide 42 ancova2000.ppt (08/00)
Typical Panel CovariatesFMOV - corrects for imbalances in the store environment
experienced by panelists. Brand share in store is assumed toreflect all in-store conditions . For each store-week-cell, VFMOV = Brand Share (store) times a Panelist
Store Trips. Alternatively, try SFMOV = Brand Share (STORE) times PanelCategory Volume. One benefit of this measure is that it should capture theeffect of Frequent Shopper Program discounts. If some element of the
store environment is part of the test treatment, use individual merch.covariates instead.
ALOYAL - corrects for imbalances in test vs. control panelistspre-test loyalty to the brand . For each week-cell ALOYAL = the average pre-test %loyalty to the brand,
among those panelists who bought the category this week. Dont use incats with long purchase cycle or where a HH may have dual loyalty (onesoap for women and one for men).
A variation on ALOYAL is ADJ_EV, which multiplies the %loyalty* categoryvolume purchased. Dont use with high share brands.
% of brand volume purchased with a coupon
S A V E $ 1
8/13/2019 An Cova 2000
43/81
Analytic Insights Group Slide 43 ancova2000.ppt (08/00)
Even in BehaviorScan, the cleanest test design, ANCOVAmakes an important difference about 40% of the time.
60 %
24 %16 %
Less than 5% 5 - 10% More than 10%
Difference Between Raw and Adjusted Test Effect
% of Tests
* 5% is usually the minimum "significant" effect seen.
8/13/2019 An Cova 2000
44/81
Analytic Insights Group Slide 44 ancova2000.ppt (08/00)
Typical Covariates in CST & MMT Analyses
Merchandising Covariates % of base volume on feature only, display only, feature+display,and/or price reduction only.
Price Covariates Average price per volume (or unit)
Or a combination of Base Price per volume (or unit) and AveragePercent Off base price, when on deal.
Other Covariates ACV Wtd average number of UPCs moving (variety) Category volume sales / $MM ACV (CDI)
If test brand has a large market share, use All Other Category (minustest brand).
Assumes brand volume is correlated with category volume, which maynot always be true.
Dummy variables to flag presence of an unusual event or out-ofdesign weeks
2 5 % O f f
$ 1 . 9 9
e v e r y d a y
V a r i e t y
8/13/2019 An Cova 2000
45/81
Analytic Insights Group Slide 45 ancova2000.ppt (08/00)
Merchandising Covariates
For the test brand, we usually break in-storemerchandising into 7 mutually exclusive conditionsFor competitive brands, we typically use a single AnyMerchandising covariate.
Feature Only
Feature andDisplay
Price Reductiononly
Display only
Feature and Display
Price Reduction &
Display
Feature, Display, & PriceReduction
8/13/2019 An Cova 2000
46/81
Analytic Insights Group Slide 46 ancova2000.ppt (08/00)
Price CovariatesFor the Test brand , price usually is segmented into components: Base (non-promoted)price -- although base price rately
changes Average % discount off base price, for UPCs or TRP (% of
base volume sold on any price reduction completes thepicture)
For competitors (and sometimes for the test brand, if thisapproach seems to work better), we use a single price covariate:
Average Price per Volume
WATCH OUT: IRI does not yet capture average price paid afterFrequent Shopper discountsWATCH OUT: Price is generally correlated with the promotioncovariates.
ANCOVA makes an important difference in
8/13/2019 An Cova 2000
47/81
Analytic Insights Group Slide 47 ancova2000.ppt (08/00)
ANCOVA makes an important difference inabout half of all Controlled Store Tests.
48 %
31 %20 %
Less than 3% 3 - 8% More than 8%
Difference Between Raw and Adjus ted Test Effect
% of Tests (sample of 194 Controlled Store Tests)
* 3% is usually the minimum "signif icant" effect seen.
ANCOVA is most important in Matched Market Tests
8/13/2019 An Cova 2000
48/81
Analytic Insights Group Slide 48 ancova2000.ppt (08/00)
ANCOVA is most important in Matched Market Tests,making an important difference in two-thirds of cases.
34 % 34 % 32 %
Less than 3% 3 - 8% More than 8%
Difference Between Raw and Adjus ted Test Effect
% of Tests (sample of 133 Matched Market Tests)
* 3% is typically the smallest "significant" eff ect seen.
8/13/2019 An Cova 2000
49/81
Analytic Insights Group Slide 49 ancova2000.ppt (08/00)
How do we decide which covariates to start with?
Test design (treatment)
Suggestions from client and category experts.
IRI analysts experience with similar tests.
Prior category studies such as cross-purchasing,brand switching, price elasticity, or marketing mix.
Each covariate must logically have an influence on sales,or act as a surrogate for other influencing factors.
8/13/2019 An Cova 2000
50/81
8/13/2019 An Cova 2000
51/81
Analytic Insights Group Slide 51 ancova2000.ppt (08/00)
Which covariates do we keep?
Covariates that are significantly correlated with thetest brands volume (p 80%) and have the logical sign . . .even if they arent making a large adjustment to thetest effect.
Covariates that are not significantly correlated withthe test brands volume, but do have the logical signand arent making a large impact on the test effect --doesnt hurt to leave them in.
8/13/2019 An Cova 2000
52/81
Analytic Insights Group Slide 52 ancova2000.ppt (08/00)
Signals that the model can still be improved
You have a large but insignificant test effect may be due to 1
unusual week (unusual promotion with big sales bump) in either baseor test period, causing unusual variability.One covariate represents more than 30% of the total adjustment, whenthe adjustment is significant.
A covariate is causing a large adjustment that does not seemreasonable given the raw data e.g., test cell experienced a $0.01advantage on average price, reducing raw effect of +10% to adjustedeffect of -2%.
A covariate has a counter- intuitive sign e.g., an increase incompetitor As merchandising having a positive effect on the testbrands volume.
A covariate is not significant, but is having a large impact on the testeffect.The test effect is significant, but the confidence interval is very wide Consider additional covariates to reduce the error variation.
8/13/2019 An Cova 2000
53/81
Analytic Insights Group Slide 53 ancova2000.ppt (08/00)
Signals that the model can still be improved continued
Common fixes to consider:Look for outlier observations in the data with unusually high levels offeature, display and/or TPR.If no unusual causal conditions are apparent, consider the possibility of anunadvertised Frequent Shopper discount. Solution to both of the above: if problem is in just 1-3 weeks, drop the
outlier week(s). If the problem seems to affect more weeks, considerusing a dummy variable.
If the problem is with the pricing variables on the test brand, reduce thenumber of covariates involving price.Eliminate Number of UPCs Moving, especially if there was no significantchange in the means for this covariate. If you know of a significant newproduct entry (e.g., test brand introduced a new flavor), see if it rolled intotest and control markets on different timing; you might create a dummyvariable to indicate weeks where the new item is present.Roll competitive products into larger aggregates.Make sure youve included all relevant competitors in the category
8/13/2019 An Cova 2000
54/81
Analytic Insights Group Slide 54 ancova2000.ppt (08/00)
Signals that the model can still be improved continued
Common fixes to consider continued Check for colinearity, or correlation (r close to |1|), betweencovariates; drop duplicative covariates.Look for category-driven explanations such as seasonality thattends to make all brands promote on the same timing and createcolinearity.Check if covariate is correlated with the test treatment, so thatby adjusting for it, you are washing out the treatment effect.Check for homogeneity of slopes (possibility that the variablemay affect sales differently across groups).Make sure the error term (the remaining variance in sales notexplained by the covariates) is both small and randomlydistributed.
Bottom line: Can you defend the design and output of your model?
8/13/2019 An Cova 2000
55/81
8/13/2019 An Cova 2000
56/81
Analytic Insights Group Slide 56 ancova2000.ppt (08/00)
Examples: when to use a custom covariate
In a Matched market test, you learn that the test marketreceived a coupon drop while the control did not. Create a covariate for couponing
Or a juice/iced tea MMT, the test markets experienced anunusually hot period that the control markets did not.
Create a covariate to adjust for weekly temperature.In a Controlled Store price test, you had an out-of designcondition in one or two weeks (I.e., price not at thedesignated level)
Add a field to flag the out of design weeks.In a CST of a complex display unit, you audited test storesto make sure the condition of display was up to spec. Create a covariate for quality of display
8/13/2019 An Cova 2000
57/81
Analytic Insights Group Slide 57 ancova2000.ppt (08/00)
Non-Standard Designs
We usually analyze VOLUME sales per $MM ACVWhere Available. In what situation (I.e., when testing what kind of variable)
would it be better to analyze UNIT sales rate ?
Dollar sales rate? Volume sales rate assuming 100% distribution?
Share?
Name two situations in which we would NOT useDISPLAY as a covariate
8/13/2019 An Cova 2000
58/81
Analytic Insights Group Slide 58 ancova2000.ppt (08/00)ancova00.pr4
Signposts
Part 1 -- For Everyone Intro to Analysis of Covariance The process of test analysis How ANCOVA works
Explanation of covariates Exercise: Non-Standard Designs
Part 2 -- For Statisticians
More details on the ANCOVA model
W h e n y o u ' r
e r e a d y t o
t a k e t h e n e
x t s t e p u p t
h e
l e a r n i n g c u
r v e.
8/13/2019 An Cova 2000
59/81
Analytic Insights Group Slide 59 ancova2000.ppt (08/00)
There are many methods of test evaluation
t-tests (comparing 2 means) and F-tests (comparing 4means)
Analysis of Variance (ANOVA)
Baseline Burps (regressions) Analysis of Covariance (ANCOVA)
Marketing Mix (regressions)
Share / Attraction models (logits)
8/13/2019 An Cova 2000
60/81
Analytic Insights Group Slide 60 ancova2000.ppt (08/00)
ANCOVA is IRIs Method of Choice
ANCOVA is a statistical method specifically designedto evaluate the response to a treatment whileeliminating the influences of extraneous (non-test)factors.
ANCOVA provides: adjustments for non-test factors, also called nuisance
factors or noise
an estimate of the effect of the test treatment confidence intervals for that estimate
8/13/2019 An Cova 2000
61/81
Analytic Insights Group Slide 61 ancova2000.ppt (08/00)
Concepts of ANCOVAThe ANCOVA model uses a least squaresmethodology, like regression. We use Type 3 sum of squares, because it does not require
equal #s of store -week observations.
Statistical tests are performed by comparing theparameter estimates to unexplained variability.
In IRIs model, the Group * Period term representsthe increase in volume sales due to the test
treatment.
Our confidence level comes from testig whether theG*P term is zero.
8/13/2019 An Cova 2000
62/81
Analytic Insights Group Slide 62 ancova2000.ppt (08/00)
A typical model for test analysisY = + 1G + 2P + 3G*P + 4Pr + M + ... + e
where Y = Sales rate (the dependent variable we're trying to predict)
0 = The Y intercept, generally set to 0 in Tes tingx = The regres sion coefficient for covariate x (1, 2, 3, etc.)
G = 1 for test, 0 for control -- indicates the group affiliation of eachdata record
P = 1 for post, 0 for pre -- indicates the period affiliation of the dataG*P = 1 for post-period, test cell; 0 for all other observations. The group*period term = the effect of the test treatment
Pr = A price variable, like avg $/vol for each brandflag-store-weekM = A merchandising variable, like % of base vol on feat+disp... = Other covariatese = Error term -- the residual unexplained variance in sales
8/13/2019 An Cova 2000
63/81
Analytic Insights Group Slide 63 ancova2000.ppt (08/00)
Calculating the coefficient for each covariate
represents the strength of the relationshipbetween the covariate, x, and sales, y.
For each covariate x : (y = sales, n = # weeks) x = SPxy
SSx
Sum of the cross products of X and Y:SPxy = sum(X*Y) - sum(X) * sum(Y)
n
Sum of squares for X:SSx = sum (X^2) - sum(X)^2
n
Example of data records: for a test with 3 pre & 3 post
8/13/2019 An Cova 2000
64/81
Analytic Insights Group Slide 64 ancova2000.ppt (08/00)
Example of data records: for a test with 3 pre & 3 postweeks, and 3 stores per group
Average Price for the Test Brand is the only covariate
Vol/$MM ACV Group Period Store G*P Price
1.2 1 1 1 0 1.991.5 1 1 1 0 1.890.8 1 1 1 0 1.991.6 1 2 1 1 1.891.4 1 2 1 1 1.991.4 1 2 1 1 1.99
2.4 2 1 2 0 1.992.3 2 1 2 0 1.993.1 2 1 2 0 1.892.6 2 2 2 0 1.992.5 2 2 2 0 1.992.5 2 2 2 0 1.99
8/13/2019 An Cova 2000
65/81
Analytic Insights Group Slide 65 ancova2000.ppt (08/00)
The group, period, and group*period terms
The regression coefficient has to be interpreteddifferently for these terms than for a continuousvariable like price of merchandising.
Specifically, the coefficient is equal to the SALESmean for this group/period, when all otherindependent variables (like price or merchandising)are equal to zero.
8/13/2019 An Cova 2000
66/81
Analytic Insights Group Slide 66 ancova2000.ppt (08/00)
Basis for covariate adjustments
The size of the adjustment to sales depends on: the regression coefficient for each covariate the imbalance in the mean value of each covariate
(e.g., average price) pre-post, test-control.
The raw sales means are adjusted by setting eachcovariate to its overall average level across all groupsand periods.
l l h d d l
8/13/2019 An Cova 2000
67/81
Analytic Insights Group Slide 67 ancova2000.ppt (08/00)
Calculating the covariate-adjusted sales means
The adjusted sales means are the sum of: the average sales level as predicted by the sum ofcovariates at their overall average level plus
the test treatments estimated impact on sales
AS_11 = sum( c * c) + 3G*P
AS_11 = adjusted sales for brandflag 11, the test brand.c = the c th covariate, including the Group, period, and G*P terms
8/13/2019 An Cova 2000
68/81
8/13/2019 An Cova 2000
69/81
Analytic Insights Group Slide 69 ancova2000.ppt (08/00)
ANOVA -- testing for statistical significance
A statistical F-test is performed to determine if thereis a significant difference between groups, betweenperiods, and a group-by-period (G*P) interactionterm.
The value of F is calculated as follows:F = SS effect / degrees of freedom
SS error / degrees of freedom
The smaller the error term, the larger the F value,and the more likely that the effect will be significant.
8/13/2019 An Cova 2000
70/81
Analytic Insights Group Slide 70 ancova2000.ppt (08/00)
Assumptions in our use of ANCOVAThe independent variables are measured without error. pretty safe
The relationship between sales and each covariate is linear. We know this isnt true. For example, the relationship betweensales and price is not linear, but curvilinear. But we use linearregression because the outputs are more easily interpreted forevaluation of tests.
The variables are distributed normally.
We know this isnt true: weekly sales per store, for example, aredeal-driven. But we do make sure the residual error is randomlydistributed.
The model has been appropriately specified, including all covariatesthat are relevant and no incorrect covariates.The independent variables are not highly correlated with one another
These are both up to the analyst to ensure in the building of themodel .
The variance of errors at all values of X is constant (homoscedasticity) We can check for this by . . .
8/13/2019 An Cova 2000
71/81
Analytic Insights Group Slide 71 ancova2000.ppt (08/00)
Do we ever use ANOVA for test analysis? ANOVA is just ANCOVA without any covariates.There are almost always some covariates that we willwant to adjust for.In the rare instance where there are no covariates toadjust for, we would use ANOVA to determine whether the
pre-post, test-control difference in the raw sales means isstatistically significant. Example 1: A No Couponing test vs. Couponing control,
where the coupon $s in the test cell are being reallocated totrade spending.
Example 2: A category shelf reset test in a national retailerwhich has a high degree of control and standardization inpricing and merchandising by store, such that there wouldntbe an differences of that sort to adjust for.
8/13/2019 An Cova 2000
72/81
Analytic Insights Group Slide 72 ancova2000.ppt (08/00)
Example: Output of the final model -- Panel
Period Conf LevModel Name Group Name Pre Test Diff %Effect 1T 2TADJ PURVOL1 Test Cell 3.11 3.55 0.44
Ctrl Cell 3.62 3.12 -0.50 30% 93% 86% +0.94
80% CL 2T** 90% CL 2T Adjusted Lower Upper Lower Upper
Mean* Bound Bound Bound Bound ADJ PURVOL1 0.94 0.04 0.56 -0.03 0.63
read as 4% - 56%, -3% to 63%
* This is really the Diff of Diffs from the previous section.** The +/- 26 point confidence interval around the 30% es timate of the test effect sugges ts you might have been able to improve the model with more/different covariates. Also, try a 1T test and a lower CL.
(including all important covariates)
Means
8/13/2019 An Cova 2000
73/81
Analytic Insights Group Slide 73 ancova2000.ppt (08/00)
Example: Impact of Covariates -- PanelDependent Impact
Variable Covariate on Effect Coefficient P-ValueVOLA1 ALOYAL1 - 0.05 + 5.66 .02
FMOV1 +0.13 + 0.17 .02 PCTCOU1 +0.03 +11.39 .02
DOD T,Pre T,Post C,Pre C,Post ALOYAL1 +0.009 0.116 0.105 0.220 0.201 FMOV1 - 0.734 20.47 18.93 19.15 18.33 PCTCOU1 - 0.002 0.042 0.017 0.048 0.026 (numb ers have been rounded)
Interpretation: One minus the P-value = the significance of this covariate. Each of these covariates is
significant at the 92% CL. The coeff icient of each covariate is correctly signed. The test celhad an "unfair" advantage on ALOYAL (+0.009 Diff of Diffs); ALOYA L has a positivecoefficient, indicating that an increase in ALOYAL w ill increase sales; so the impact w as adow nw ard adjustment on the test effect.
l f h f l d l k
8/13/2019 An Cova 2000
74/81
Analytic Insights Group Slide 74 ancova2000.ppt (08/00)
Example: Output of the final model -- Store/Market
RAW MEANDependent Grp 1-Test Grp 2-Ctrl DiffVariable Base Test Base Test EffectVOLA1 0.84 0.79 1 .03 0.83 18
____ ____ _____
ADJUSTED MEAN Conf. LevelDependent Grp 1-Test Grp 2-Ctrl DiffVariable Base Test Base Test Effect* 1T 2TVOLA1 0.75 0.71 0.98 0.81 16 99 99
____ ____ _____
Dependent Adjusted 80% C.I. (2T)Variable Mean* Lower UpperVOLA1 0.16 0.10 0.22 (+/- 4 to 6 points)
* These are the same -- the adjus ted % tes t effect.
Summary of Raw and Adjusted Effects(including all important covariates)
8/13/2019 An Cova 2000
75/81
Analytic Insights Group Slide 75 ancova2000.ppt (08/00)
Example: Impact of Covariates -- Store/Market
Dependent Relative EstimateVariable Covariate Impact Impact CoefficientVOLA1 PBFE1 -1.05 -0.0048 0.0024
PBDI1 -0.09 -0.0004 0.0044 PBFD1 -0.26 -0.001 0.0126
G1,P1 G1,P2 G2,P1 G2,P2 DOD T,Pre T,Post C,Pre C,Post P-valuePBFE1 2.0 8.3 9.0 11.1 9.8 .0001
PBDI1 0.1 0.9 0.7 2.0 1.7 .0001 PBFD1 0.1 0.3 0.3 0.6 0.5 .0003
Inte r pr eta t ion : PBFE1 = Percent of Base volume on FEature only, Test Brand. PBDI1 - Disp lay only,
PBFD1 - Feature+Disp lay. An increase in Feature has a positive effect on sales (coefficient+0.0024). The test cell had an advantage on Feature, and thus there is a negative IMPACT.Relative impact = impact of this covariate as a % of the sum of all impacts.
8/13/2019 An Cova 2000
76/81
Analytic Insights Group Slide 76 ancova2000.ppt (08/00)
Example: Output of the vanilla model for an MMT
(including only one covariate)
Output of General Linear Models (PROCGLM) Procedure
Dependent Variable: VOLA1 - Brand volume per $MM ACVWeight: STRMKTWT (store market weight)
Source DF Type III SS Mean Square F Value Pr > FGroup *Period 1 167.5 167.5 2.90 0.0887
Pr > F = probability that the unadjusted test effect is significant (greaterthan the F). One minus Pr = the confidence level: 1 - .0887 = 91% CL
8/13/2019 An Cova 2000
77/81
Analytic Insights Group Slide 77 ancova2000.ppt (08/00)
QUESTIONS?
8/13/2019 An Cova 2000
78/81
Analytic Insights Group Slide 78 ancova2000.ppt (08/00)
The Bobs Bologna Case Bobs Bologna was considering doubling their adweight.Bob selected two representative markets in which totest his heavy-up media plan.
A pair of matching markets were selected to serve ascontrol
After twenty-four weeks, no difference between cellswas seen.Further investigation, in the form of an ANCOVA, wasnecessary
8/13/2019 An Cova 2000
79/81
Analytic Insights Group Slide 79 ancova2000.ppt (08/00)
Bobs Bologna Case -- continued
Pre-TestMean Post-TestMean Change Diff ofDiffs % TestEffect Conf.Level
Raw Test 18.7 21.2 2.5 0.1 0.5% 54%
Raw Ctrl 19.8 22.2 2.4 Adj Test 19.3 23.1 3.8 0.6 2.7 % 82%
Adj Ctrl 20.1 23.3 3.2
8/13/2019 An Cova 2000
80/81
8/13/2019 An Cova 2000
81/81
Bobs Bologna Case -- continued
In the raw data, the treatment was: advantaged by the imbalance in features disadvantaged by the imbalance in displays
Adjustments were made accordingly: a small negative impact for features a large positive impact for displays
Conclusion: After adjusting for promotionalimbalances, the additional advertising was shown tostatistically significantly provide incremental sales.
Top Related