Post on 08-Jan-2016
description
1841f06detprob3
MM Stroustrup Ch26
Comments?
Agree or disagree with his testing approach?
2841f06detprob3
MM Stroustrup ch26
Additional topics?
3841f06detprob3
L2asg C++
Good job – very interesting programs Common mistakes
– No failure set or not well labelled– Easy fault = large failure set
4841f06detprob3
Failure Set
-15
-10
-5
0
5
10
15
-15 -10 -5 0 5 10 15
Series1
5841f06detprob3
Failure Sets
What was the shape of your failure sets?
How could test case selection criterion improve the probability of finding these faults?
6841f06detprob3
From last year’s 1-minute paper
How do we measure test adequacy? Or what is test adequacy?
7841f06detprob3
Test terms
Fault
Failure
Error
8841f06detprob3
Test terms
Test selection criterion – a methodology for selecting test cases
Test adequacy criterion – a methodology for deciding when to stop testing
9841f06detprob3
Testing Basics
Detection probability
10841f06detprob3
Detection Probability
The probability of detecting a specific fault using a specified testing strategy
OrThe probability that a randomly-generated test set
that satisfies a specific criterion will detect a specified fault
11841f06detprob3
Operational Profile
The set of inputs from which operational data is chosen is called the operational profile– It usually includes a description of the
probability of which points will be used Is 7.99 more likely to be encountered at a
WalMart POS terminal than at an exclusive store that only sells fur coats?
12841f06detprob3
Methodology to evaluate testing criterion
Pick sample program, sample faults, sample operational profile, two criteria.
For each criterion, randomly generate test suites that satisfy the criterion.
Calculate the percentage of the suites that detect each of the faults.
The criterion that has the higher percentage is better.
13841f06detprob3
Triangle Example
cin >> a >> b >> c ;type = “scalene”;if (a == b || a == c || b == c) type= “isosceles”;if (a == b && a == c) type = “equilateral”;if (a >= b+c || b >= a+c || c >= a+b) type=“not a triangle”;if (a <= 0 || b <= 0 || c <= 0) type=“bad input”;cout<< type;
14841f06detprob3
Control Flow Graph
I: cout
B: type = isoscles
D: type = equilateral
F: type = not a triangle
H: type = bad input
A: type = scalene
C: if
E: if
G: if
Operational profile
3,3,3 abcdegi equi3,3,4 abcegi isos3,3,5 abcegi isos3,3,6 abcefgi not3,4,3 abcegi isos3,4,4 abcegi isos3,4,5 acegi scal3,4,6 acegi scal
All inputs are equally likely
15841f06detprob3
What are the failure probability for each color (separately)?
cin >> a >> b >> c ;type = “scalene”;if (a == b || a == c && b == c) type= “isosceles”;if (a == b || a == c) type = “equilateral”;if (a >= b+c || b >= a+c || c > a+b) type=“not a triangle”;if (a <= 0 || b <= 0 || c <= 0) type=“bad input”;cout<< type;
Blue Green Red
16841f06detprob3
TTYP – probability of detection
What is the probability of detection with one randomly chosen test case?
What is the probability of detection with two randomly chosen test cases?
17841f06detprob3
TTYP – per path
What is the probability of detection with one randomly chosen test case per path?
What is the probability of detection with an equal number of randomly chosen test cases?
18841f06detprob3
TTYP – smaller subdomains
What might be better smaller subdomains?
19841f06detprob3
TTYP - subdomains
Are paths the best subdomains? Would a functional decomposition be
better? Should we re-define the term subdomain?
20841f06detprob3
fault 1 fault 2 fault 3 fault 4
criterion 1 0.08 0.03 0.04 0.85
criterion 2 0.07 0.1 0.05 0.91
criterion 3 0.06 0.12 0.08 0.88
criterion 4 0.05 0.4 0.06 0.89
Comparing Criteria
21841f06detprob3
Solving this choice
What are the assumptions?
Analogy with dice?
22841f06detprob3
TTYP – smaller subdomains
What might be better smaller subdomains? Would MCC (multiple condition coverage)
be better subdomains
23841f06detprob3
TTYP2 – C0 and C1 coverage
How do we deal with C0 and C1 coverage since they are not subdomain testing methodologies?
24841f06detprob3
TTYP3
How could you estimate the det prob of C0 or C1 testing?
25841f06detprob3
Marble Problem
Assume that there is a bag of marbles from which marbles are drawn with replacement.
What is the maximum likelihood estimate of p (the probability of drawing a purple) marble if you draw exactly n purple marbles in a row?
? P such that (p)n(1-p) is max or ? P such that (p)n = .5
26841f06detprob3
Evaluating Testing Methods by Delivered Reliability
Frankl, Hamlet, Littlewood, Strigini
IEEE TOSE Aug98
For Thursday, Aug 30 – study through section 2.3