Jonathan Misurda (jmisurda@cs.pitt.edu) James A. Clause, Juliya L. Reed, Bruce R. Childers

Department of Computer Science University of Pittsburgh Pittsburgh, Pennsylvania 15260 USA

Mary Lou Soffa Department of Computer Science University of Virginia Charlottesville, Virginia 22904 USA

Demand-Driven Structural Testing with Dynamic Instrumentation

Software Testing

n  What is Software Testing? q  Gather information about the behavior of the software being

developed or modified.

n  Why test software? q  Gain confidence and insights into software correctness q  Create quality, robust programs

n  Why is testing hard? q  Testing is expensive

n  50-60% of the total cost of software development q  Adds complexity to development process

n  One testing technique is not enough

Structural Software Testing

n  Structural Testing q  The process of discovering run-time properties of a

program specifically pertaining to control-flow

q  Demonstrate adequacy of the test cases

q  Different granularities of structures: n  Node coverage records statements (basic blocks) n  Branch coverage records control-flow edges n  Def-Use coverage records pairs of variable definitions

and uses

q  Over multiple test cases until coverage criteria

Current Testing Tools

n  E.g.: JCover, Clover, IBM Rational PurifyPlus

n  Static instrumentation with test probes q  Probes are injected instrumentation to gather coverage

information

n  Limitations: q  Expensive in both time and space q  Only one type of test q  Limited code regions and scopes q  Cannot define their own ways to test

Our Approach

n  Demand-driven structural testing q  Specification driven: User written tests q  Test plans: Recipe of how & where to test q  Path specific: Instrument only what is needed q  Dynamic: Insert & remove instrumentation

n  Jazz: A scalable & flexible framework for testing q  Automatically apply multiple test strategies q  Handle large programs q  Allows user customization of tests

Jazz Overview

Application

IDE & Test GUI

Test Specification

Test Planner

Test Plan

Test Analyzer and Result Reporter

Test Dynamic Instrumenter

Test Results

Select the regions and test types Allows user customization

Automatically generate where and how to instrument

Run the program with demand-driven instrumentation

Collect the results

Test Specifier

1a

1b

3

2

Specify the regions to test 1a. Highlight line regions 1b. Select classes and methods

List of desired tests

Click to create a test

Test Planner

n  Test plan: Where and how to test program n  Test storage, probe location table,

instrumentation payload n  Can combine payloads to create custom strategies

Application

Test Planner

Test Specification

Test Instrumentation Payload

Test Plan

Global Storage & Probe Location Table

Test Planner

n  Challenges: q  When to insert test probes q  Where to test/instrument q  What to do at each probe

n  A planner determines three things: q  Seed Probe Set – probes which are initially inserted in a

test region. q  Coverage Probe Set – probes that can be inserted and

removed on demand along a path. q  Probe Lifetime – whether a probe must be re-inserted

after being hit by its “control flow successor” basic blocks.

Branch Coverage Example Plan public class simpleDemo {

public static void main(String[] args) { int evenSum = 0; int oddSum = 0;

for(int i = 0; i < 100; i++) { if(i%2==0) { evenSum += i; }

else { oddSum += i; } } System.out.println("The sum of the even numbers from 0 to 100 is: " + evenSum);

System.out.println("The sum of the odd numbers from 0 to 100 is: " + oddSum); }

}

DEFINITIONS { NAME: d_method, REGION_D, LOCATION: FILE simpleDemo.java {

CLASS simpleDemo, METHOD main }

} BODY {

DO BRANCH_TEST ON REGION d_method UNTIL: 85% }

simpleDemo.java

simpleDemo.testspec

Branch Coverage Planner

n  Record which edges are executed q  Determine (source, sink) edge pairs hit at run-time q  Source is a branch q  Sink can be taken & not-taken target of branch

Probe Set Members Seed Probe First block in testing region Coverage Probe Sinks of currently instrumented block Probe Lifetime Until all incoming edges have been

covered

Branch Coverage Example

Probe Loc Tab Storage Block Next Hit 1 2,3 2 4 3 4 4 1 Test Payload Mark edge hit Insert at next point Remove instrumentation

1

2 3

4 Coverage probe Hit

N, N

N

N

N

Y, Y

Y

Y

Y

Seed probe

Demo

n  Branch coverage of music player q  Test region is whole program (JOrbis) q  Test input is a song

n  Compared q  Traditional approach with static instrumentation q  Demand-driven approach with dynamic instrumentation

With Dynamic Instrumentation

With Static Instrumentation

Demo

n  Branch coverage of music player q  Test region is whole program (JOrbis) q  Test input is a song

n  Compared q  Traditional approach with static instrumentation q  Demand-driven approach with dynamic instrumentation

With Dynamic Instrumentation

With Static Instrumentation

Demo

n  Branch coverage of music player q  Test region is whole program (JOrbis) q  Test input is a song

n  Compared q  Traditional approach with static instrumentation q  Demand-driven approach with dynamic instrumentation

With Dynamic Instrumentation

With Static Instrumentation

Other Planners

n  Node Coverage Planner q  Record which statements are executed Probe Set Members Seed Probe First statement in the testing region Coverage Probe Next reachable statement Probe Lifetime Removed as soon as statement is executed

n  Def-Use Coverage Planner q  Record which variable definition reaches an executed use

Probe Set Members Seed Probe All variable definitions Coverage Probe Uses associated with all definitions Probe Lifetime Defs removed after all reachable uses are

covered. Uses are removed as executed

Dynamic Instrumentation

n  Test plan targets an instrumentation API

n  FIST instrumentation engine [WOSS’04] q  Retargetable & reconfigurable q  Dynamic insertion & removal of instrumentation q  Binary level instrumentation (post JIT)

n  Uses fast breakpoints [Kessler]: Replace existing instruction with a jump to instrumentation

Experimental Methodology

n  Test Specification q  Implemented as an Eclipse 3.0 plugin

n  Test Planner & Dynamic Instrumenter q  Implemented using the Jikes RVM version 2.1.1 q  On the x86 platform

n  SPECjvm98 benchmarks test input q  unloaded 2.4 Ghz Pentium IV, 1GB of memory q  RedHat Linux 7.3

n  Traditional vs. Jazz (in same implementation) q  Compared coverage, run-time, memory

Coverage Results

n  Coverage – same reported by both tools

Branch Node Def-Use compress 58% 90.6% 89.8%

jess 46.8% 80.3% 71.8% db 44.4% 76.9% 75.0%

javac 38.9% 75.0% 66.9% mpegaudio 60.9% 88.7% 90.5%

mtrt 50.6% 90.3% 87.3% jack 55.6% 82.2% 73.4%

Run-time Performance

Static Branch Dynamic Branch

Static Node Static Def-Use Dynamic Node Dynamic Def-Use

Memory Requirements

Branch Node Def-Use Demand Static Demand Static Demand Static

compress 7.9 7.5 4.4 3.7 8.8 5.6 jess 50.2 60.3 34.1 29.7 70.3 44.8

db 9.7 8.9 5.0 4.1 9.8 5.1 javac 178.9 186 107.3 85.3 202.0 125.3

mpegaudio 24.7 29.5 15.0 14.7 34.8 22.3 mtrt 22.4 23 12.5 11.0 26.1 16.1 jack 73.4 78 46.5 37.6 89.0 46.8

• In Kilobytes

Summary

n  New demand-driven approach q  A tool (Jazz) for structural testing q  Dynamic instrumentation guided by a program’s

execution q  Minimally intrusive q  User configurable and flexible testing

n  Very low overhead q  E.g., branch coverage tool is 3-4x faster than

traditional approaches

For Further Information…

jmisurda@cs.pitt.edu

http://www.cs.pitt.edu/coco

Test Results 1 Click to run test suite

2 Results

Static Instrumentation

n  Shortcomings of static instrumentation: q  Not scalable: Instrumentation left in place causes

unnecessary increase in overhead q  Inflexible: Only certain tests, languages, and

platforms q  Cumbersome: Requires recompilations for

dedicated testing q  Intrusive: Addition of testing code may alter or

mask defects

Run-time Performance

-10%

30%

70%

110%

150%

190%

230%

270%

compress jess db javac mpeg mtrt jack avg

% s

low

dow

n ov

er b

asel

ine

Static Branch Dynamic Branch Static NodeDynamic Node Static Def-Use Dynamic Def-Use

Branch Testing

0102030405060708090

100

Tim

e (s

econ

ds)

compress jess db javac mpeg mtrt jack avg

Dynamic Static None

Node Testing

0102030405060708090

100

Tim

e (s

econ

ds)

compress jess db javac mpeg mtrt jack avg

Dynamic Static None

Def-Use Testing

0102030405060708090

100

Tim

e (s

econ

ds)

compress jess db javac mpeg mtrt jack avg

Dynamic Static None