32 Automated Unit Test Generation for Classes with Environment Dependencies Andrea Arcuri, Gordon...
-
Upload
camilla-bates -
Category
Documents
-
view
223 -
download
0
Transcript of 32 Automated Unit Test Generation for Classes with Environment Dependencies Andrea Arcuri, Gordon...
/ 32
Automated Unit Test Generation for Classes with Environment Dependencies
Andrea Arcuri, Gordon Fraser, Juan Pablo Galeotti
ASE 2014
Presented by Yongbae Park
/ 322
Overview• Motivation: low coverage & non-deterministic test results in
automated test generation for a class interacting with environment1. Branches depending on the environment that may not be covered by a
sequence of method calls of a class under test
2. Passing test cases generated by a tool can fail as the environment changes• The environment is inputs from outside of a class that a class cannot control
such as network, file system, system time, and user inputs
• Approach: use a mock library interacting with the virtual environment to increase coverage and produce deterministic test result• Users of a test generation tool do not have to build their own mock library• The mock library provides deterministic inputs from the virtual environment
separated from the real environment
• Evaluation: the mock library increases branch coverage and decreases the number of test cases making non-deterministic results• EvoSuite with the mock library produces test cases whose branch coverage is
increased by 183% and the number of tests with non-deterministic test results is reduced by 99.2%
/ 323
Contents• Background
• How EvoSuite generates test suites using a genetic algorithm
• Motivation• The limitation of EvoSuite in a class with environment dependency
• Approach• Mocking Java standard API to control the environment
• Empirical study• Related Works• Conclusion
/ 324
Background – EvoSuite• Given class under test (CUT), EvoSuite automatically generates a
test suite using a genetic algorithm 1. Create 2 initial test suites by adding method calls randomly and insert the
test suites into current generation
2. Select two test suites from current generation
3. Create 2 new test suites by crossover (exchange test cases of the suites)
4. Modify two test suites (test drivers) from step 3 with mutation operators (remove, insert, change operators)
5. Insert the new two test suites from step 4 to next generation if coverage of the new two test suites are higher than that of parents
6. Repeat 2~5 until next generation has sufficient number of test suites
7. Repeat 2~6 until time limit is reached or all branches are covered
8. Select a test suit with the highest branch coverage and insert assertions by observing differences in execution traces between the original CUT and a mutated CUT
/ 325
Example for EvoSuite• Message class contains 2 members: created meaning
message creation date and text meaning contents of a message• The message creation date is set in the constructor of Message
public class Message { private Date created; private String text; public Message(String text) { this.created=new Date(); this.text=(text==null?””:text); } public String toString() { return created+”,”+text; } public String getText() { return text;} }
12345678910111213
/ 326
Initial Test Suite Generation• EvoSuite randomly inserts a new statement in empty test cases of two
initial test suites until the length of test cases reaches the maximum length• EvoSuite add a method call whose callee is a method of a class under test or a
method call of an object that is available at the end• A parameter of the created method call is selected from available values, null, or a
random value
• The maximum length for each test case is set by random number
public class Message { private Date created; private String text; public Message(String text) { this.created=new Date(); this.text=(text==null?””:text); } public String toString() { return created+”,”+text; } public String getText() { return text;} }
123456789
10111213
public class TestSuite1 { public void test0() { //length 3
} public void test1() { //length 2
} }
Message v0 = new Message(“e”);
Message v0 = new Message(null);
Message v1 = new Message(“c”);String v2 = v1.toString();
String v1 = v0.getText();
/ 327
Crossover• EvoSuite selects test suites and from current population, and
moves last test cases of to and first test cases of to where is a random number
public class TestSuite1 { public void test0() { Message v0 = new Message(“e”); Message v1 = new Message(“c”); String v2 = v1.toString();} public void test1() { Message v0 = new Message(null); String v1 = v0.toString();} }
public class TestSuite2 { public void test0() { Message v0 = new Message(“a”); String v1 = v0.toString(); String v2 = v1.trim();} public void test1() { Message v0 = new Message(null); String v1 = v0.getText(); String v2 = v0.toString();} }
public class TestSuite3 { public void test0() { Message v0 = new Message(“e”); Message v1 = new Message(“c”); String v2 = v1.toString();} public void test1() { Message v0 = new Message(“a”); String v1 = v0.toString(); String v2 = v1.trim();} }
public class TestSuite4 { public void test0() { Message v0 = new Message(null); String v1 = v0.toString();} public void test1() { Message v0 = new Message(null); String v1 = v0.getText(); String v2 = v0.toString();} }
/ 328
Mutation – Insert Operator• Insert operator adds a new statement at a random position by
using one of the two ways1. Add a method call statement whose callee is a method of a class
under test
2. Add a method call of an object that is available at the random position• A parameter of the created method call is selected from available
values, null, or a random value
public class TestSuite4 { public void test0() { Message v0 = new Message(null); String v1 = v0.toString();} }
public class TestSuite4 { public void test0() { Message v0 = new Message(null); String v1 = v0.toString(); Message v2 = new Message(“b”);} }
public class TestSuite4 { public void test0() { Message v0 = new Message(“a”); String v1 = v0.toString(); boolean v2 = v1.equals(null);} }
Adding a method call of a class under test
Adding a method call of an available object
/ 329
Mutation – Remove & Change Operator• Remove operator randomly selects a statement in a test case
and remove the statement• Change operator randomly changes a callee or a parameter of a
method invocation statement in a test case• Change operator selects a new callee method that whose return type is
same as the original method• Change operator changes a parameter of a method call into a value which
is available from the previous statements or a random value public class TestSuite4 { public void test0() { Message v0 = new Message(null); String v1 = v0.toString();} }
public class TestSuite4 { public void test0() { Message v0 = new Message(null); String v1 = v0.getText();} }
public class TestSuite4 { public void test0() { Message v0 = new Message(“a”); String v1 = v0.toString();} }
callee method change parameter change (random value)
/ 3210
Inserting Assertions• EvoSuite adds assertions by observing differences
between an execution trace of the original class under test and that of a mutated version of a class under test• An assertion is added if values of a variable (primitive or String
type) are different in the two execution traces• EvoSuite mutates the original class under test using the insert,
remove, change operators in test generation
/ 3211
Inserting Assertions - Example
public class Message { //original private Date created; private String text; public Message(String text) { this.created=new Date(); this.text=(text==null?””:text); } public String toString() { return created+”,”+text;}…}
123456789
10
public class Message { //mutated private Date created; private String text; public Message(String text) { this.created=new Date();
} public String toString() { return created+”,”+text;}…}
123456789
10
public class TestSuite1 { public void test0() { Message v0 = new Message(null); String v1 = v0.toString();} }
v1 = “2014.10.02,” v1 = “2014.10.02,null”
Execution result of Message Execution result of mutated Message
assertEquals(v1,”2014.10.02,”)
/ 3212
Generated Test Case• EvoSuite creates a test case with an assertion containing a
string of the created date of the test case• test0() uses a string “2014.10.2,”
• However, the test case will fail when the test case is executed on a different day
public class Message { private Date created; private String text; public Message(String text) { this.created=new Date(); this.text=(text==null?””:text); } public String toString() { return created+”,”+text;} }
123456789
10
public class TestSuite1 { public void test0() { Message v0 = new Message(null); String v1 = v0.toString(); assertEquals(v1,”2014.10.02,”);} }
/ 3213
Motivation• The test result of a class interacting with the environment is non-
deterministic (not reproducible)• The author defines the environment as inputs from outside of a class
• E.g. a state of virtual machine (free & total memory space), a state of operating system (system time, file system, user inputs)
• A test case is unstable if the test case passed when test case is generated but may fail in the next execution with different environment
Class under test
Virtual Machine
Operating System
File system
Time & Date
Memory state User inputs
/ 3214
Environment Mocking• “Mocking” is the replacement real classes with modified
classes that behave deterministically• In the following example, Date is replaced by MockDate that
always returns a fixed value
public class Message { private Date created; private String text; public Message(String text) { this.created=new MockDate(); this.text=(text==null?””:contents); } public String toString() { return created+”,”+text;} }public class MockDate() { public String toString() { return “2014.10.02”;} }
123456789
1011121314
public class TestSuite1 { public void test0() { Message v0 = new Message(null); String v1 = v0.toString(); assertEquals(v1,”2014.10.02,”);} }
/ 3215
Overview of Environment Mocking• A test generation tool with a generic mock library create a
stable test suite with higher branch coverage without user’s (tester) efforts• A mock library of the paper mocks console inputs, file I/O, general
API class of Java standard API• The mock library consist of a custom InputStream class for console
inputs, 11 classes of file I/O (e.g. File), 12 general API classes (e.g. System, Runtime)
• The mock library have helper methods that set the environment in a test case
• The technique replaces standard library with the mock library using Java bytecode instrumentation
• Insert operator of EvoSuite inserts the helper methods in test cases to generate test suites that controls the initial environment
/ 3216
Console Inputs• A customized InputStream called SystemInUtil has
a helper method addInputLine(String) so that a test case can write contents of console inputs• The console contents of SystemInUtil is reset before every test
execution• In instrumentation, System.io in a class under test is changed
into SystemInUtil.io
/ 3217
File I/O• The authors mocked 11 JVM file I/O API classes
• Mock classes are children classes of the mocked classes • The authors overridden the methods of the mocked classes to access
virtual file system instead of real file system of operating system• For example, the authors overridden 37 methods among 52 methods of File class
• Test cases become independent from each other and there is no negative side-effect such as file system corruption
• The authors created helper methods such as appendLineToFile() to control the initial state of the virtual file systemjava.io.File java.io.PrintStream
java.io.FileInputStream java.io.PrintWriter
java.io.FileOutputStream java.util.logging.FileHandler
java.io.RandomAccessFile javax.swing.JFileChooser
java.io.FileReader java.io.FileWriter
javax.swing.filechooser.FileSystemView
/ 3218
General JVM Calls• The authors mocked 12 JVM general API classes to
control the environment such as time and random numberClass name Environment
java.lang.Exception
Stack trace messagejava.lang.Throwable
java.util.logging.LogRecord
java.lang.Thread
java.lang.Runtime Memory usage & the number of processors
java.lang.System
Current system time & datejava.util.Date
java.util.Calendar
java.util.GregorianCalendar
java.lang.Class Reflection (the order of Method objects)
java.lang.MathRandom number
java.util.Random
/ 3219
Stack Trace (1/2)• StackTest has a method getStackTrace() which returns a
stack trace string of Throwable object • The message contains the name of a thread and the name of class that executes
the test case• The test case fails if the name of a thread or test runner is changed
public class StackTest { public static String getStackTrace(Throwable t) { PureStringWriter sw = new PureStringWriter(); PrintWriter pw = new PrintWriter(); t.printStackTrace(pw); return sw.toString();} }public class TestSuite { public void test0() { String result = StackTest.getStackTrace(new Throwable()); assertEqual("Exception in thread \“Thread-0\" java.lang.Throwable\n \tat TestRunner.run(TestRunner.java:120)\n \tat TestSuite.test0(Test.java:10)" , result);} }
123456789101112131415
/ 3220
Stack Trace (2/2)• Stack trace message depends on the string representation
of Thread object and JUnit runner• Stack trace message contains the string representation with thread
id depending on the number of created thread• JUnit runner is a class that executes each test case and bottom of
stack trace message contains the class name of JUnit runner class• A tester can use custom JUnit runner to modify test execution process
• Mocked Thread, Exception, Throwable, and LogRecord create stack trace message deterministically• The mock class of Thread print the string representation
deterministically• The mock class of Thread, Exception, Throwable, and LogRecord does not prints custom JUnit runner class in stack trace
/ 3221
Memory & Processor (1/2)• Performance executes a certain task and measure
execution time and memory usage• getMemoryUsage() returns a string containing the amount of
memory used by the task• The test case fails if free memory space is changed by execution of
a test runner or other test cases
public class Performance { public static String getMemoryUsage() { performTask(); return "Memory: "+(Runtime.totalMemory()-Runtime.freeMemory());} }public class TestSuite { public void test0() { String result = Performance.getMemoryUsage(); assertEqual(“Memory: 341”, result);} }
12345678910
/ 3222
Memory & Processor (2/2)• Runtime provides system specific values such as the
number of processors and memory usage• availableProcessors() returns the number of available
processors• freeMemory(), totalMemory(), and maxMemory() return
memory usage
• availableProcessors(), freeMemory(), totalMemory(), and maxMemory() of a mock class of Runtime return constant values
/ 3223
System Time (1/2)• Message object contains the creation date of a message
• The generated test case contains a string of the day when the test case was created
• The test case fails if the test case is executed on 7next day
public class Message { private Date created; private String msg; public Message(String msg) { created = new Date(); this.msg = msg; } public Date getCreated() { return created;} }public class TestSuite { public void test0() { Message msg = new Message(); assertEqual(“2014.10.02”, msg.getCreated().toString());} }
123456789101112131415
/ 3224
System Time (2/2)• Mock methods and classes of currentTimeMillis(), nanoTime(), Date, Calendar, and GeregorianCalendar returns default time• The mock library has helper methods to set initial system time • Each call of the system time methods increases the time by 1
/ 3225
Reflection (1/2)• getAvailableMethods() returns a string of a list of public
methods• If test0() is executed before test1(), the execution result of test0() is
pass• Otherwise, the execution result of test0() is fail
public class LoadLibrary { public static String getAvailableMethods(Class c) { String result = “”; for (Method each : c.getMethods() ) { result += each.getName() + “\n”; } return result;} }public class TestSuite { public void test0() { String methods = getAvailableMethods(TestSuite.class); assertEqual(“test0\ntest1”, methods); public void test1() { …} }
123456789101112131415
/ 3226
Reflection (2/2)• The order of an array returned by reflection methods is
non-deterministic• The order of an array that is returned by reflection methods
(getClasses(), getMethods()) depends on method invocation order or class loader implementation
• The reflection methods of a mock class of Class return sorted lists of available methods, classes, annotations, etc.
/ 3227
Random Number• random() returns position data in 2D space randomly
• The content of the returned object can be different from the previous execution
• Mock classes of Math and Random return the next random number deterministically
public class Position { public int x, y; public static Position random() { Position result = new Position(); result.x = (int)Math.random(); result.y = (int)Math.random(); return result;} }public class TestSuite { public void test0() { Position p = Position.random(); assertEqual(2, p.x);} }
12345678910111213
/ 3228
Static Field Initialization• A class under test that modifies static field may change
behavior of test cases if the execution order of the test cases is changed• EvoSuite creates unstable test because EvoSuite does not reset
static field nor restart Java virtual machine
• The extended EvoSuite calls static initialization methods of classes whose static fields are modified to reset static field with low overhead• Every writing to a static field is instrumented with a method call that
records the modification• A test suite generated by EvoSuite calls the static initialization
methods before the execution of each test case
/ 3229
Empirical Study• RQ1: does controlling the environment successfully
increase coverage on known cases of environmental interaction?
• RQ2: does controlling the environment resolve known issues of unstable tests?
• RQ3: How do results generalize to the SF100 corpus?
• The authors applied extended EvoSuite to SF100 corpus• SF100 corpus is 100 Java projects randomly selected from
SourceForge• SF100 corpus contains 11,219 Java classes
• EvoSuite created test cases up to 3 minutes for each class
/ 3230
RQ1: Coverage – Setup• The authors manually selected 30 classes from SF100
corpus that interact with the environment• The classes that uses environment-related API are selected
• The authors compared coverage of 6 different configurations• Base: the default EvoSuite• Console: EvoSuite with console input mocking• VFS: EvoSuite with virtual file system• JVM: EvoSuite with general API call mocking• Static: EvoSuite with static field initialization• All: EvoSuite with console input, virtual file system, general API call
mocking and static field initialization
/ 3231
RQ1: Coverage – Experiment Results• The technique increases branch coverage by 183%(=(0.82-0.29)/0.29) in average
• The blue numbers are statistically significant differences at
No. BaseExtended
Console VFS JVM Static All
1 0.10 0.10 0.76 0.10 0.10 0.78
2 0.75 0.75 0.75 0.75 1.00 1.00
3 0.09 0.08 0.53 0.08 0.41 0.80
4 0.26 0.74 0.27 0.25 0.26 0.73
5 0.40 0.40 0.89 0.40 0.40 0.99
6 0.60 0.60 0.60 0.61 0.99 0.99
7 0.20 0.20 0.77 0.20 0.20 0.77
8 0.29 0.29 0.40 0.28 0.29 0.40
9 0.94 0.94 0.94 0.95 1.00 1.00
10 0.83 0.83 0.83 0.95 0.83 0.94
11 0.12 0.12 0.71 0.12 0.12 0.70
12 0.08 0.68 0.09 0.09 0.11 0.64
13 0.24 0.24 0.24 0.24 0.79 0.86
14 0.12 0.12 0.12 0.12 0.12 0.93
15 0.33 0.33 0.80 0.33 0.33 0.80
No. BaseExtended
Console VFS JVM Static All
16 0.22 0.22 0.74 0.22 0.22 0.73
17 0.00 0.00 0.87 0.00 0.00 0.87
18 0.00 0.00 1.00 0.00 0.00 1.00
19 0.30 0.33 0.77 0.30 0.69 0.79
20 0.00 0.00 1.00 0.00 0.00 1.00
21 0.79 0.79 0.77 0.67 0.88 0.85
22 0.67 0.68 0.69 0.61 0.99 0.96
23 0.08 0.08 0.99 0.08 0.08 0.99
24 0.25 0.25 0.87 0.25 0.25 0.99
25 0.07 0.07 0.56 0.07 0.07 0.56
26 0.22 0.66 0.22 0.22 0.22 0.63
27 0.28 0.62 0.30 0.27 0.25 0.64
28 0.03 0.03 0.56 0.03 0.03 0.54
29 0.34 0.34 0.79 0.34 0.34 0.80
30 0.20 0.20 0.79 0.20 0.20 0.79
Avg. 0.29 0.35 0.65 0.29 0.37 0.82
/ 3232
RQ2: Unstable Tests – Setup• The authors manually selected 30 classes from SF100
corpus from which the default EvoSuite creates unstable tests • There are 2 classes that are used in both RQ1 and RQ2
• The authors compared coverage of 6 different configurations same as the experiments in RQ1
/ 3233
RQ2: Unstable Tests – Experiment Results• The technique reduces the number of unstable tests per class by 99.2%(=(2.43-
0.02)/2.43)• The blue numbers are statistically significant differences at
No. BaseExtended
Console VFS JVM Static All
1 2.72 2.52 2.52 2.49 0.00 0.00
2 1.14 1.16 1.20 2.06 0.00 0.00
3 2.99 3.04 3.17 3.21 0.14 0.13
4 1.27 1.28 1.35 1.74 0.00 0.00
5 2.37 2.36 2.33 2.19 0.00 0.07
6 4.20 4.19 4.24 4.15 1.55 0.01
7 1.33 1.40 1.39 0.00 1.40 0.00
8 0.64 0.75 0.79 0.82 0.06 0.08
9 0.84 0.98 0.75 0.00 0.77 0.00
10 7.16 7.09 6.93 0.03 6.94 0.00
11 0.61 0.62 0.61 0.00 0.62 0.00
12 2.50 2.52 2.52 2.38 0.00 0.00
13 5.53 5.57 5.47 5.87 0.00 0.00
14 3.73 3.66 3.41 0.02 1.73 0.00
15 6.86 6.79 7.50 4.41 7.05 0.30
No. BaseExtended
Console VFS JVM Static All
16 1.75 1.73 1.47 1.57 0.00 0.00
17 1.19 1.23 1.03 0.00 1.17 0.00
18 2.59 2.87 2.55 0.09 2.27 0.11
19 2.49 2.39 2.93 0.02 1.62 0.00
20 1.91 1.90 1.87 0.00 1.77 0.00
21 3.96 3.87 2.40 3.81 0.00 0.00
22 0.93 0.93 0.78 0.00 0.80 0.03
23 1.82 1.59 1.28 0.00 1.55 0.00
24 3.26 3.67 3.66 0.00 3.38 0.00
25 1.83 1.76 1.79 0.12 1.60 0.16
26 2.13 2.12 2.21 0.00 2.19 0.00
27 1.91 1.86 2.18 1.79 0.01 0.00
28 1.63 2.01 1.87 0.86 1.22 0.00
29 1.22 1.22 1.22 0.00 1.19 0.00
30 0.57 0.54 0.57 0.67 0.00 0.00
Avg. 2.43 2.45 2.40 1.27 1.30 0.02
/ 3234
RQ3: Generalization• The authors compared the default EvoSuite (base) and the
extended EvoSuite (all) by applying both of them to all 11,219 classes of SF100 corpus
• The branch coverage increased by 1.8% (=(77.9%-76.5)/76.5)• The default EvoSuite achieved 76.5% of branch coverage and the
extended EvoSuite achieved 77.9%• The extended EvoSuite achieved 2,803 additional branches
• The number of classes from which EvoSuite creates unstable tests is reduced by 53.8% (=(1,452-671)/1,452)• The default EvoSuite created unstable tests from 1,452 classes while
the extended EvoSuite created unstable tests from 671 classes
/ 3235
Related Works• Mocking frameworks help developers write mock classes
of user classes, not class of the standard library• The mocking frameworks such as Moles (C#), Jmock (Java)
provide mechanisms to create mock class and replace the original classes with the mock classes
• The developers should change mock classes as the original class changes
• The paper aims mocking Java standard API which does not change frequently• Once mock library is built, the mock library can be applied to any
class under test
/ 3236
Conclusion• Mocking library interacting with the virtual environment
increases coverage and reduces unstable tests in automated unit test generation• The modified EvoSuite creates test cases that controls initial
environment to increase branch coverage• The generated test cases are stable because interactions with the
virtual environment is deterministic
• The authors shown the empirical results that using mocking library increase branch coverage and reduces unstable tests from EvoSuite for 100 Java projects