Test driven software development

Timo Suomela

September 13, 2006

Seminar paper

Timo Suomela

Test driven software development

September 13, 2006 11 pages

testing-driven, mock objects, continuous integration

This document is a short introduction to the process of test-driven software devel-

opment.

D.2.5 [Testing and debugging]

Tiedekunta/Osasto � Fakultet/Sektion � Faculty Laitos � Institution � Department

Tekijä � Författare � Author

Työn nimi � Arbetets titel � Title

Oppiaine � Läroämne � Subject

Työn laji � Arbetets art � Level Aika � Datum � Month and year Sivumäärä � Sidoantal � Number of pages

Tiivistelmä � Referat � Abstract

Avainsanat � Nyckelord � Keywords

Säilytyspaikka � Förvaringsställe � Where deposited

Muita tietoja � övriga uppgifter � Additional information

HELSINGIN YLIOPISTO � HELSINGFORS UNIVERSITET � UNIVERSITY OF HELSINKI

ii

Contents

1 Introduction 1

2 The test-driven development cycle 2

2.1 Test list . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2.2 Red/green/refactor . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2.3 Bene�ts of TDD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

3 Unit test case design 5

3.1 Mock objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

3.2 Dependency injection . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

4 Continuous integration 7

4.1 Single source repository . . . . . . . . . . . . . . . . . . . . . . . . . 7

4.2 Automated build . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

4.3 Self-testing code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

4.4 The master build . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

5 Summary 10

References 11

1

1 Introduction

Test-driven software development (often abbreviated TDD) has been popularized by

Kent Beck in association with eXtreme Programming [Bec04], one of several agile

software development processes. Although TDD is one of the cornerstones of XP,

it can easily be adopted to be used in any iterative software development process

without adopting any of the other practices of XP.

Test-driven software development (often abbreviated TDD) is far more than a simple

testing technique. TDD is primarily a programming technique that has the nice side

e�ect of ensuring that the code being developed is thoroughly unit tested. TDD only

covers structural testing (whitebox testing), traditional testing techniques are still

required to cover such aspects as functional testing and user acceptance testing.

The starting premise with TDD is the same as with any other testing technique.

A test is succesfull only if it helps to uncover one or more defects in the system

being developed. When a test fails, progress has been made since a problem has

been identi�ed and that problem can now be �xed. More importantly, these tests

provide a clear measure of progress once they no longer fail. As long as the test cases

are carefully selected, TDD increases con�dence in that the system being developed

meets the requirements de�ned for it.

Properly implemented, TDD achives 100 percent statement coverage, something that

traditional testing techniques do not guarantee. Although full statement coverage is

not the best indicator for the quality of the testing process, it increases con�dence

in the code base, since each line of code is exercised at least once during a test run.

2

2 The test-driven development cycle

The complete paradigm of Test-driven software development has been summarized

by Kent Beck in two rules [Bec04]:

1. New business code is written only when an automated test has failed.

2. Any duplicate code is eliminated by refactoring.

These are two simple rules, but they generate complex individual and group be-

haviour with technical implications such as the following:

1. The design process is organic, design decisions are made based on feedback

provided by running code.

2. Each developer must write his or her own test cases because it is impractical

to wait 20 times a day for a dedicated tester to write the tests.

3. The development environment must provide repid response to small changes.

4. The design of the software system being developed must consist of many highly

cohesive, loosely coupled components to make testing easier.

2.1 Test list

Before starting to code a new feature or modifying an existing one, a list of tests

must be compiled. Since it is practically impossible to design a test suite that will

�nd every single �aw in a non-trivial piece of software, the selection process must be

planned carefully. The selection of tests is beyond the scope of this paper but follows

the same rules as in traditional testing. According to Newkirk and Vorontsov, this

process should take about 15-20 minutes for a feature that is estimated to take about

4 hours to implement [NeV04].

After the list has been compiled, each test is implemented and then crossed of the

list. Once each test has been implemented and succesfully run, the new feature or

modi�cation has been completed.

3

2.2 Red/green/refactor

The process of implementing each test in the test list is de�ned by red, green, refactor.

The goal of this process is to work in small veri�able steps tha provide immedia

feedback. The steps are descibed by Newkirk and Vorontsov as follows [NeV04]:

1. Adding a new test.

In TDD the implementation of each new feature begins with writing a test

case, usually using a framework from the XUnit-famliy of frameworks. The

developer selects a test from the test list discussed above.

2. Running all tests and seeing the new one fail.

This step may seem unnescessary at �rst. The purpose of running all tests

at this point is to validate the new test. If the code passes the new test case

without requiring any changes, the test case is obviously �awed and requires

revising. The new test must also fail for the expected reason. Testing frame-

works usually employ visual cues when reporting success or failure of a test.

At this point the new test is colored red.

3. Implementing the new feature.

In this step the new feature is implemented by writing code that is 'good

enough'. The aim is not to write perfect code, but pass the test. The code

will be improved in a later step. This is to ensure that no untested code is

written.

4. Running all tests and see them succeed.

If all test cases pass, the developer has a positive con�rmation that the new

feature is implemented correctly and all previously implemented tested features

still work. The testing framework will show a green �ag for each passed test.

If a test fails, the developer must go back to the previous step and add or

change some code and then run the tests again.

5. Refactoring the code to remove duplication.

The last step involves cleaning up the code. By frequently running all test

cases the developer can be con�dent that the code meets the same requirements

before and after refactoring. The cycle is repeated until all test cases on the

original test list are implemented and passed.

4

The process described above forces the developer to work in very smal steps. The

steps are so small they may even seem ridiculous to the uninitiated. However, the

smaller the steps are, the faster the developer gets feedback about a mistake he or

she has made in the form of a failed test. The alternative to developing in small

steps is to make a lot of changes to the code and then run test. If the test fails,

backtracking to �nd out which changes caused the failure will take a great amount

of time. Working in smaller steps will also decrease the need for a debugger since

the developer gets immediate feedback after each change and therefore will know

exactly where to look in case of a failure. A nice side e�ect of this is increased

con�dence in the quality of the code.

2.3 Bene�ts of TDD

In a TDD-environment a developer him or herself is reponsible for writing tests that

verify the quality of his or her code. Traditionally the roles of developer and tester

have been separated to increase the chances of a test suite to �nd faults in the system

under test. A kind of 'blindness' may prevent the developer from writing tests for

the edge cases he or she did not consider when coding a new feature or modifying

an existing one.

Fowler says this is true [FoF99]. It's easy for a developer to overlook errors in his or

her own code, but Fowler considers the value of the fast turn-around of the TDD-

approach to be grater than the value of having separate testers. TDD does not cover

the whole range of testing during the software development cycle, system tests and

acceptance test must still be designed, written and executed independently of the

coding e�ort.

In a controlled expirement conducted to evaluating the e�ectiveness of TDD, Erdog-

mus et. al. divided the a group of participating undergraduate students in two. The

experiment group used TDD to develop a piece of software, the control group used a

more conventional development technique, writing tests after the actual implemen-

tation. Both groups followed an incremental process, adding new features one at

a time and regression testing them. According to the study, students in the TDD

group tendet to write more tests on the average. Students who wrote more tests

also tendet to be more productive. Additionaly, the quality of the code increased

linearly with the number of written tests, independent of the development strategy

employed.

5

3 Unit test case design

The idea of TDD in itself is language agnostic. The principles can be applied to

any iterative software development process but the main focus is on modern object-

oriented languages like Java, C and Python.

In a TDD-environment, all unit test cases must be repeatable and must not depend

on or interfere with each other. It must be possible to change the order in which

the test cases are executed. It must be possible to repeatedly execute a single test

case without need for a manual setup before the start of the test case or cleaning

up after the test has �nished. According to Beck, good unit tests must run fast, in

isolation and use real data (e.g. a copy of production data) [?].

3.1 Mock objects

In a modular software system, each module provides one or more services. Some

of these services are consumed by other modules in the same system to provide

other, more complex services. The dependency on other modules makes isolating

the module under test a non-trivial task, especially if one of the referred modules is a

complex system in itself. An example of such a complex auxilary system is a database

or a network resource like a web service. Sometimes the actual implementation of a

referred module is not available at coding time and testing must be done against a

set of interfaces. In many cases the task of setting up the modules needed to execute

a single test case may include lots of time-consuming, fragile e�ort, such as running

services or placing hardware in a known state.

Mock objects provide a way to deal elegantly with this type of situation [Ham04] in

a object-oriented language. A mock object is a simulation of a real object. Mock

objects implement the interface of the real object and can be set up to behave in

a predictable way. Mock objects also validate that the code using the objects does

so in the expected order. The unit under test must call the methods of a mock

object in the expected order, passing in the expected parameters. This veri�cation

capability is what separates mock objects from traditional stubs used in top-down

integration testing [Bin99].

6

3.2 Dependency injection

Dependency injection (sometimes referred to as 'inversion of control') is a fairly

modern design principle that is humbling in its sheer simplicity. Rather than letting

the components (instances of classes) of a software system declare explicit dependen-

cys, e.g. by instantiating collaborating objects or looking them up using the locator

pattern, they refer to them via interface only. The concrete implementations are

'injected' into the components by an assembly mechanism (sometimes called an

inversion of control -container) using setter-methods.

Paired with mock objects, the principle of dependency injection makes isolating a

unit under test a trivial proceeding.

7

4 Continuous integration

In the classical v-model of software development, unit testing (often referred to as

module testing) commences once the coding phase has been completed (�gure 1).

The focus of unit testing is to eliminate errors in the functionality a single module

provides. The tested modules are then �tted together in a process called integration

testing to eliminate errors in the collaboration of one or more modules.

Figure 1: The v-model of classic software testing

Barbey, in his masters thesis, has pointed out, that unit testing can be considered

a special form of integration testing, the integrated units being the methods or

functions of the module under test. From this point of view, integration testing

completely covers the structural testing e�ort. Since integration is not an single

event but a process,

Continuous integration, as Fowler points out [FoF99], is a set of practices that has

been around in one form or the other for a long time. Although it is a fundamental

part of the eXtreme Programming -paradigm, it can be (and often is) adopted

whithout even considering the other practices of XP.

4.1 Single source repository

The �rst requirement for succesfull continuous integration is a well known (well

known to the development team that is) repository for the source code. In addition

to the source code, the repository must contain everything that is required to compile

the source code and run the unit tests. This includes build �les, third party libraries,

test scripts, database schemas, etc. Only a minimal amount of tools should be

required to check out the source code from the source code repositor and build it

8

on a virgin machine. Typical examples of things no present in the repository are

large and stable and/or di�cult to install like a compiler, a database manager or

a web server. The repository should not contain any artifacts that are products of

the build process, i.e. executables.

The source code repository should contain a single mainline (or main branch) of

the project currently being developed. Each developer should work on the mainline,

other branches should be used only to �x bugs in older, allready released versions

of the project.

4.2 Automated build

Building a project from its sources and getting it to run on a development machine

can be a very complicated task, e.g. a typical n-tiered web-application requires a

database for the backend, a web server for the frontend and often even a middleware

server.

Like many tasks in the software development process, the build process can - and

should be - automated. The more commands a developer has to remember to build

and deploy a piece of software, the more likely he or she is to make a mistake.

Automated build systems are nothing new, the *nix world has pro�ted from make

tool for decades. Modern examples of automated environments are Ant and Maven

for Java and Nant and MSBuild for the .NET framework. Each of these tools can

be con�gured to compile the source code, install a database schema, run all tests

and deploy the system, all with issuing a single command.

The complete build of a system can take an enormous amount of time, so it is cruical

that the selected build tool is capable of analyzing the source code after a set of

changes and only re-build the nescessary artifacts. A good build script will let the

developer skip selected subgoals.

4.3 Self-testing code

An important part of each build is the execution of the test cases written by the

developers. Fowler calls these tests build veri�cation tests (BVT). If all test succeed,

the build is considered succesfull. If one or more tests fail, the build is considered

broken. Of course self-testing code is no silver bullet, since even successfull tests

don't prove the absence of bugs.

9

4.4 The master build

The purposes of the master build is to �nd integration problems in a multi-developer

environment as early as possible. The build daemon periodically checks the source

code repository and determines whether any new changes have been commited since

the last build. If there's new code in the repository, then it starts building, using

the following steps:

1. Makes a full check out from the repository.

2. Invokes a build script that compiles the source code.

3. Runs all unit tests.

4. If no tests fail, the build is considered successfull and the source code is labeled

with a running build number in the repository.

5. Informs the developers of the status of the build, e.g. via email.

The time needed to compile the source code may become an issue if the frequency

of the master builds is high. This issue is resolved by either reducing the frequency

or employing an incremental compilation strategy, compiling only the source code

that actually has changed. Even if using an incremental compiler, a complete build

should be build at least once a day.

10

5 Summary

Test-driven software development is test-�rst approach to software development. No

business code is written before a repeatable unit test case has been written to assert

the validity of the feature. The moment all tests go from red to green, coding stops.

After each fail-code-pass iteration the maintainability of the code is increased by

systematically factoring out duplicate code.

Combined with continuous integration, test-driven software development provides

fast feedback for the programmer and allows the development process to act faster

on changing requirements.

11

References

Bar97 Barbey, S., Test selection for speci�cation-based unit testing of object-

oriented software based on formal speci�cations. Master's thesis, Ecole

Polytechnique Federale de Lausanne, Lausanne, EPDFL, 1997.

Bec02 Beck, K., Test-Driven Development By Example. Addison Wesley Pro-

fessional, 2002.

Bec04 Beck, K., Extreme Programming Explained: Embrace Change, Second

Edition. Addison Wesley Professional, 2004.

EMT05 Erdogmus, H., Morisio, M. and Torchiano, M., On the e�ectiveness of

the test-�rst approach to programming. IEEE Transactions on Software

Engineering, 31,3(2005), pages 226�237.

FoF99 Fowler, M. and Foemmel, M., Continuous integration, 1999. http:

//martinfowler.com/articles/originalContinuousIntegration.

html. [10.9.2006]

Fow04 Fowler, M., Dependency injection, 2004. http://martinfowler.com/

articles/injection.html. [10.9.2006]

Ham04 Hamill, P., Unit Test Frameworks. O'Reilly, 2004.

NeV04 Newkirk, J. W. and Vorontsov, A. A., Test-Driven Development in

Microsoft .NET. Microsoft Press, 2004.

Bin99 v. Binder, R., Testing Object-Oriented Systems. Addison-Wesley, 1999.