Achieving Test Case Efficiency By Combining Use Case Scenarios And

Orthogonal Arrays Randall Rice,

Randall Rice Consulting Group,USA

Th15

Achieving Test Case Efficiency by Combining Use Case Scenarios and Orthogonal Arrays

Randall W. Rice, CTFLRice Consulting Services, Inc.

www.riceconsulting.com

© 2006, Rice Consulting Services, Inc.

3

Agenda

Objectives and Benefits Use Cases Test Scenarios Test Conditions Test Case Matrix Orthogonal Arrays Caveats Final Thoughts

4

Objectives and Benefits

n Provide a consistent process to create test cases from use cases

n Reduce the number of test cases via optimization techniques

n Leverage traditional test design techniques, such as boundary value analysis & equivalence classes

5

Tasks in the Process1. Identify the Use Cases

2. Identify the Test Scenarios

3. Develop a Test Scenario Matrix

4. Identify the Driving Test Conditions

5. F ind the “B est F it” O rthogonal A rray

6. Map the Conditions to the Array

7. Document the Test Cases

6

Use Casesn Describe a sequence of interactions

between systems and usersn Comprised of following components:

n Brief Descriptionn Participating Actorsn Use Case Diagramn Main Course (MC)n Alternate Courses (AC)n Rules

7

Use Case to Test Case Process Flow

Project

Capability

1..n

1

1..n

1

Main Course Alternate Course

Use Case

1..n

1

1..n

1

1

1

1

1

1..n1..n

1

Test Case

Test Scenario11 11

1..n

1

1..n

1

1..n

1

1..n

1

8

Use Case Diagram – ATM1

ATM Withdrawal

^Customer

^CustomerDB

^ATM System

^DDASystem

^SavingsSystem

9

Test Scenarios

End Use Case

Begin Use Case

Main Course

Alternate Course 1

Alternate Course 3

End Use Case

Alternate Course 4

End Use Case

Alternate Course 2

10

Coursesn Main – Successful Withdrawal (MC)n Alternate

n Fast Cash (AC1)n Multiple Transactions (AC2)n Cancelled Transaction (AC3)

n Exceptionaln Login Failure (EX1)n Over Limit (EX2)n Insufficient Balance (EX3)n Invalid Account Status (EX4)n Card Reported Stolen (EX5)

11

Concerns About Use Cases as a Basis for Test Cases

n Use Cases may contain gaps and inaccuracies.n Reviews are needed prior to test scenario design.

n Use Cases often convey conditions to be perform ed, but seldom convey the “edges” and negatives.n Use boundary-value analysis and other test case

design techniques to complete the picture.

12

Test Scenario Matrix

13

Test Conditionsn System response to user based on

procedural rules.n Field validation and business rules

n e.g. Withdrawal Amount is required and is restricted to values in multiples of 20.

n e.g. Receipt Required is not required and can be indicated by pressing either the “Yes” or “N o” key on the A T M . T he default is “N o”.

n e.g. Daily Limit is a pre-defined value of $500 per day per customer.

14

Test Conditionsn Bank Customer – Y or N (2)n Daily Limit Reached – Y or N (2)n NSF Balance – Y or N (2)n Login Limit Reached – Y or N (2)n Receipt Required – Y or N (2)n Account Types (6)

n Student, Senior, Regular, Premier, Free, Ultimate

n Invalid Amount – Multiple of 20 or not (2)

15

Total Number of Test CasesConditions (Factors) Values (Levels)

Bank Customer 2

Daily Limit Reached 2

NSF Balance 2

Login Limit Reached 2

Receipt Required 2

Account Type 6

Invalid Amount 2

16

Test Scenario Matrix with Conditions Example (Optional)

Test Conditions Added

The Test Conditions Drive the Use Case Scenarios.

18

So, What is a Workable Test Approach?

n Test everything?n Take a sample?n Perform exploratory testing?n Automate?

19

W hat’s the N eed?n One of the biggest problems for testers is

how to deal with software complexity and multiple interactions.

n It would be great to be able to test all com binations of variables, but w e just don’t have the resources, even with automated test tools.

n Orthogonal arrays give us the next best thing – the ability to test pairs of parameters.

20

Orthogonality

n “A n orthogonal array is a tw o -dimensional array of numbers that has this interesting property – choose any two columns in the array, all the combinations will occur in every colum n pair.”n Lee Copeland, A Practitioner’s

Guide to Software Test Design

21

The Fault Model Behind Orthogonal Array Testingn Integration and interaction between

software units is a major source of defects.

22

Double-mode Defects

Condition A Condition CCondition B

Each of these conditions may work correctly in isolation.

But when combinedwith another valuemay be more likelyto reveal a defect.

23

What Research Indicates

n There have been three studies done where defects were researched after the fact.

n Research has shown that the highest likelihood of an integration defect relates to interaction between pairs of items or parameters.n www.pairwise.org

24

Example of Research Findings

n [Evaluating FDA recall class failures in medical devices we established that] [...] out of the 109 reports that [were] detailed [enough], 98% showed that the problem could have been detected by testing the device with all pairs of parameter settings. n D.R. Wallace and D.R. Kuhn, 2001n http://csrc.ncsl.nist.gov/staff/kuhn/final-rqse.pdf

25

The Likelihood of Triple-mode or Higher Defects

Condition B Condition DCondition CCondition A Condition E

Double-mode defects are m ore likely…

than Triple-modeor higher defects.

26

What Common Sense Indicatesn It’s im possible to test all com binations

in applications of any level of complexity.

n Random performance of conditions is insufficient and gives no level of confidence in the application.

27

The Value of Designing Tests with Orthogonal Arrays n You know for certain all pairwise conditions

have been identified.n You get the greatest test coverage with a

minimal number of test conditions.n You get an even distribution of pairwise

combinations.n The arrays already exist. All you have to do is

find the right one and plug in the values.

28

Exact Fit Orthogonal Array

n MA.XX.6.1.2.6n M A stands for “M ixed A rray”n XX Runs (Test Cases)n 1 Factor has 6 Levelsn 6 Factors have 2 levels

n However, in actual practice, we seldom get an exact fit.

29

Likely Candidates

n If w e go to N eil Sloan’s w eb siten http://www.research.att.com/~njas/oadir/

n We find some arrays that are close:n OA.49.8.7.2n MA.36.6.1.3.6.2.3n MA.18.3.6.6.1

n We want the one with the fewest rows.

30

Best Fit Orthogonal Array

n MA.18.3.6.6.1n 18 Runs (Test Cases)n 1 Factor (condition) has 6 Levels (values)n 6 Factors have 3 levels

n This is larger than what we actually need.

n But, we have reduced 384 test cases to 18.

31

The Actual Array Looks Like This

32

Array MappingAccount Type

0 = Student1 = Senior Citizen2 = Regular3 = Premier4 = Free5 = Ultimate

Bank Customer0 = No1 = Yes

Daily Limit Reached0 = No1 = Yes

NSF Balance0 = No1 = Yes

Login Limit Reached0 = No1 = Yes

Receipt Required0 = No1 = Yes

Amount0 = Not Multiple of 201 = Multiple of 20

33

Orthogonal Array with Assignments

“G aps” can befilled with anyvalue.

34

T hen… W e N eed to A ssociate Each Row With a Test Scenario

35

Finally, We Fill The Gaps

Each row represents a test case with a combination of conditions.

36

AllPairs Output

384 cases reduced to 12.

37

Caveats About Pairwise Testingn Pairwise testing is simply a technique built on

findings from initial research that double-mode failures are much more likely than triple-mode (or higher) failures.

n However, there is no rule or guarantee that your applications will not experience a triple-mode failure.

n Therefore, if you have good cause to test more than pairwise cases, by all means do so.

38

Caveats About Pairwise Testing (2)

n The conditions placed in the array must be logically possible to perform with each other.n Mutually exclusive conditions will not work

in creating test cases unless it is a test to prevent such a condition.

39

Caveats About Pairwise Testing (3)

n When dealing with test scenarios that span applications with diverse types of input, you will need to limit the array to contain related types of input.

A

D

B

C

E

F

G

40

Final Thoughtsn This process gives:

n A way to view pairwise testing at a user scenario level

n Reduction of test cases via optimization techniquesn This can provide a minimal number of test cases for

regression testing.n It is a starting point, not the complete set of cases you

will need for high confidence.n The ability to incorporate techniques such as

boundary value analysis and equivalence classes

41

Resources for Orthogonal Arrays and Pairwise Testing

n A Practitioner’s G uide to Softw are Test Design by Lee Copelandn This is an excellent resource for many

types of test planning as well.n Introducing Software Testing by

Louise Tamresn Also discusses orthogonal arrays

n www.pairwise.orgn Contains many resources for pairwise

testing and orthogonal arrays.

42

Resources for Orthogonal Arrays and Pairwise Testing (2)

n Sources of pre-identified orthogonal arraysn Quality Engineering Using Robust Design by

Madhav S. Phadken N eil Sloan’s collection at

www.research.att.com/~njas/oadir/index.htmln SAS page on orthogonal arrays managed by

Warren F. Kuhfeldn http://support.sas.com/techsup/technote/ts723.html

n Orthogonal Array Testing Strategy (OATS) Technique.n http://www.seilevel.com/OATS.html

43

Resources for Orthogonal Arrays and Pairwise Testing (3)

n All Pairs Tooln James Bach has written a tool called

ALLPAIRS, a command-line PERL script that will create all-pairs tables.

n James' site is http://www.satisfice.com/.

44

Finally…

n Thanks to Ron Rissel and Mike Stevens at Vanguard Mutual Funds for their research and trials of this approach.

n To hear a replay of this session:n http://www.riceconsulting.com/pres/index.htm

45

Bio - Randall W. Ricen Over 30 years experience in building and testing

information systems in a variety of industries and technical environments

n Certified Software Quality Analystn Certified Software Testern ASTQB Certified Tester – Foundation leveln Director on the American Software Testing

Qualification Board (ASTQB)n Chairperson, 1995 - 2000 Q A I’s annual softw are

testing conferencen Co-author with William E.Perry, Surviving the Top

Ten Challenges of Software Testing and Testing Dirty Systems

n Principal Consultant and Trainer, Rice Consulting Services, Inc.

46

Contact Information

Randall W. Rice, CTFLRice Consulting Services, Inc.P.O. Box 6127Moore, OK 73153Ph: 405-691-8075Fax: 405-691-1441Web site: www.riceconsulting.come-mail: rrice@riceconsulting.com