Workshop TDD4ES

16/11/2010

EP 1

iwt-090191 TDD4ES

Workshop

Building blocks for

Test-Driven Development of Embedded Software

16/11/2010

dr. ing. Jeroen Boydens

ing. Piet Cordemans

ing. Sille Van Landschoot

ing. Wim Catteeuw

@ep_research http://ep.khbo.be

16/11/2010 TDD4ES

EP research group

2

http://ep.khbo.be

Founded: 2006 – 2007

Firstname.Surname@khbo.be

dr. ing. Jeroen Boydens + founder + professor + project leader

ing. Sille Van Landschoot + TDD4ES + lab assistant

ing. Piet Cordemans + TDD4ES + PhD student + lab assistant

ing. Wim Catteeuw + APS4ES

ing. Olivier Rosseel + academic staff + ICTO coach

Workshop TDD4ES

16/11/2010

EP 2

16/11/2010 TDD4ES 3

EP research group

Threefold mission:

Education Research Social services

• Software courses Prof. Bachelor EO/ICT Ac. Bachelor EO/ICT Master ICT

• With K.U.Leuven Concurrent software Software testing Transactional behavior

• Tetra/PWO/theses Embedded development

methodologies Gaming frameworks

• Seminars Software testing State-of-the-art

technologies

• LED eg. Medical

Reminder System

16/11/2010 TDD4ES 4

IWT - Tetra 090191: TDD4ES

Test on target Test on host Remote testing

Case studies

3 strategies

Literature study

Manual

• State of the art • Best practices

CMS

http://ep.khbo.be/TDD4ES

Workshop TDD4ES

16/11/2010

EP 3

16/11/2010 TDD4ES

User committee TDD4ES IWT/090191

5

www.edna.eu www.tesco.be

www.qstar.be

www.marelec.com

www.unitron.be

www.summa.be

www.sirris.be

www.televic.com

www.newtec.eu

www.vandewiele.com

www.dspvalley.com www.kdg.be

www.kuleuven.be www.fmtc.be

www.iwt.be

16/11/2010 TDD4ES

Content

Session 1 (Tuesday 16/11/2010)

1. Introduction

2. TDD in practice

3. TDD for embedded software

Session 2 (Thursday 18/11/2010)

1. Test on target

2. Test on host

3. Remote testing 6

Workshop TDD4ES

16/11/2010

EP 4

16/11/2010 TDD4ES

Content: session 1

1. Introduction

1. Testing embedded software

2. Test-Driven Development

3. Unit test framework

4. TDD strategy

5. TDD pitfalls

2. TDD in practice

1. Do it yourself

2. Temperature library

3. UnitTest++

4. Exercise

7

16/11/2010 TDD4ES

Content: session 1 cont’d

3. TDD for embedded software

1. Embedded constraints

2. TDD migration strategies

3. Test on target in practice

4. Do it yourself

5. EmbUnit

6. Exercise

7. Testable design

8

Workshop TDD4ES

16/11/2010

EP 5

16/11/2010 TDD4ES

1 Introduction

9

1. Testing embedded software

2. Test-Driven Development

3. Unit test framework

4. TDD strategy

5. TDD pitfalls

16/11/2010 TDD4ES

1.1 Testing embedded software

10

Quality assurance in embedded software development is mostly limited to debugging and final testing, only

focusing on the current issue…

Embedded software testing gap

Importance versus state of testing

Lifecycle

Co

st

of

Re

pa

ir

Specification Programming ProductionDesign

1000

0

100

10

© B.W. Boehm, 1981

Software Engineering Economics

Workshop TDD4ES

16/11/2010

EP 6

16/11/2010 TDD4ES

Lifecycle

Co

st

of

Re

pa

ir

Specification Programming ProductionDesign

1000

0

100

10

© B.W. Boehm, 1981

Software Engineering Economics

1. Find bugs early

2. Measurable quality

3. Increasing confidence

4. Detecting regression

5. Encapsulating third party code

11

Timely detection of bugs by automated testing

Running test suite frequently during development Incrementally expanding the test suite Support from unit testing framework

1.1 Testing embedded software

16/11/2010 TDD4ES

1.2 Test-Driven Development

TDD cycle

1. Write failing test (red)

• New behavior

• Minimal skeleton to get through compilation

2. Write code to pass test (green)

• Minimal implementation

3. Refactor

• No new behavior

• Clean code

• Keep tests passing

12

Workshop TDD4ES

16/11/2010

EP 7

16/11/2010 TDD4ES 13

1.2 Test-Driven Development

Advantages

1. Code is tested while it’s written

2. Fast feedback cycle

3. Extensive & safe refactoring

4. Focus on current functionality

5. Tests become living documentation

Disadvantages

1. 2x code

increase development time

16/11/2010 TDD4ES

1.3 Unit test framework

Why?

• Organization

• Execution

• Reuse

• Reporting

Many exist for C/C++ (>40) minUnit, CppUnitLite, UnitTest++, GoogleTest, CxxTest,

EmbUnit, …

Or: write your own (TDD-style)

14

Workshop TDD4ES

16/11/2010

EP 8

16/11/2010 TDD4ES

1.3 Unit test framework

xUnit model

15

16/11/2010 TDD4ES

1.4 TDD strategy

Black box tests vs. White box tests

a.k.a. glass box tests

16

• Internal perspective • Implementation • Focus on developer • 20% coverage • Test Afterwards

• External perspective • Specification • Focus on end-user • 80% coverage • Test First

Workshop TDD4ES

16/11/2010

EP 9

16/11/2010 TDD4ES

Write test first

• Inspiration

• Specifications

• Use cases

• Notes

• …

• Procedure

1. Define interface

2. Select values for test

« By writing a test before implementing the item under test, attention is focussed on the item’s interface and observable behavior » Beck (2003)

17

1.4 TDD strategy

16/11/2010 TDD4ES

1.5 TDD pitfalls

18

• Broken window syndrome

– One broken test, which remains unresolved, leads quickly to many broken tests.

• Test cancer [Fowler]

– Removing broken tests is a bad habit.

Maintaining tests is necessary

Workshop TDD4ES

16/11/2010

EP 10

16/11/2010 TDD4ES

2 TDD in practice

19

1. Do it yourself

2. Temperature library

3. UnitTest++

4. Exercise

16/11/2010 TDD4ES

2.1 Do it yourself

Software

• Virtual machine

Admin rights

Slow

• IDE: Eclipse CDT

Open source

Customizable

Popular

20

Workshop TDD4ES

16/11/2010

EP 11

16/11/2010 TDD4ES

2.1 Do it yourself

Procedure

21

1. Explanation

2. Demonstration

3. Exercise similar to demo

4. Solution

16/11/2010 TDD4ES

2.2 Temperature library

01_TDDonHost_TempLibrary

Requirements • Stores internally a temperature value in Kelvin

• Offers an interface to get and set values in Kelvin & Celsius

Goals • TDD without hardware involvement

• Getting acquainted with TDD

22

Workshop TDD4ES

16/11/2010

EP 12

16/11/2010 TDD4ES

2.3 UnitTest++

23

Hello unit test framework: UnitTest++

RunTests.cpp

#include "UnitTestppHeaders\UnitTest++.h"

TEST(alwaysFail)

{

CHECK(false);

}

int main()

{

return UnitTest::RunAllTests();

}

16/11/2010 TDD4ES

CHECK(true); //succeeds

CHECK_EQUAL(1, 2); //fails

CHECK_EQUAL("EP", "ep"); //fails

CHECK_CLOSE(3.14, 3.1415, 0.01); //succeeds

24

• Grouping

• Setup & Teardown

• Assertions

2.3 UnitTest++

SUITE(ThermometerTests)

{ TEST(TestName){}

TEST(AnotherTestName){}

}

struct TestFixture

{ TestFixture(){/*setup*/}

~TestFixture(){/*teardown*/}

};

TEST_FIXTURE(TestFixture, TestName) {}

Workshop TDD4ES

16/11/2010

EP 13

16/11/2010 TDD4ES

2.4 Exercise

25

Some inspiration…

Kelvin = Celsius + 273,15

KelvinTest

CelsiusTest

Celsius2KelvinTest

Kelvin2CelsiusTest

Celsius2Kelvin2CelsiusTest

…

16/11/2010 TDD4ES

3 TDD for embedded software

26

1. Embedded constraints

2. TDD migration strategies

3. Test on target in practice

4. Do it yourself

5. EmbUnit

6. Exercise

7. Testable design

Workshop TDD4ES

16/11/2010

EP 14

16/11/2010 TDD4ES

3.1 Embedded constraints

27

TDD 1. Tests + Program +

Unit test framework

Embedded 1. Limited memory

footprint

2. Automated testing

3. Fast programming cycle

2. Hardware dependencies

3. Slow programming cycle

16/11/2010 TDD4ES

3.1 Embedded constraints

28

Embedded TDD challenges

1. Limit the memory footprint needed for TDD on

target

2. Tests for

1. Hardware independent code 2. Hardware aware code 3. Hardware specific code

3. Maintain a fast programming cycle

Workshop TDD4ES

16/11/2010

EP 15

16/11/2010 TDD4ES

3.2 Migration strategies

29

Test on target

Test on host

Remote testing

1. Limited number of tests

2. Minimalistic framework

3. Program code

1. Unlimited number of tests

2. Full option framework

3. Program code with hardware virtualization

1. Unlimited number of tests

2. Full option framework with remote testing functionality

3. Program code with remote testing functionality

16/11/2010 TDD4ES

3.3 Test on target in practice

30

Tests

Workshop TDD4ES

16/11/2010

EP 16

16/11/2010 TDD4ES

3.4 Do it yourself

• EasyARM development board

– Mikroelektronika

– Temperature sensor (DS1820)

– 2x16 LCD

– RS232

– Drivers library (Keil compiler)

• Two versions (1.01 & 1.51)

– Hardware changes (switch on/off, …)

– Two different programming tools

31

Hardware

16/11/2010 TDD4ES

3.4 Do it yourself

• 2 versions

– 2.01

– 2.51

• Load hex file

• Write to flash of target

32

Workshop TDD4ES

16/11/2010

EP 17

16/11/2010 TDD4ES

3.5 EmbUnit

33

• Written in C

• No library includes needed

– But flexible inclusion of stdio possible

• Small memory footprint

– 8KB on it’s own

– 18KB with stdio library

– 34KB with UART driver on LPC2148

• Adaptable & extensible

• Verbose

• Not scalable

– (+/- 2 KB per test)

• Limited functionality

16/11/2010 TDD4ES

3.5 EmbUnit

34

Hello unit test framework: EmbUnit

main.cpp

#include “EmbUnit_lib/embUnit.h”

#include “EmbUnitTests.h”

int main()

{ extern TestRef FirstTest(void);

TestRunner_start();

TestRunner_runTest(FirstTest());

TestRunner_end();

return 0;

}

Workshop TDD4ES

16/11/2010

EP 18

16/11/2010 TDD4ES

3.5 EmbUnit

35

Hello unit test framework: EmbUnit EmbUnitTests.cpp

#include “EmbUnit_lib/embUnit.h”

static void setUp(void){}

static void tearDown(void){}

static void HelloTest(void){TEST_ASSERT(true)}

TestRef FirstTest(void)

{ EMB_UNIT_TESTFIXTURES(fixtures)

{

new_TestFixture(“HelloTest”, HelloTest)

};

EMB_UNIT_TESTCALLER(FirstTestset,

“FirstTestset”,setUp,tearDown,fixtures);

return (TestRef) &FirstTestset; }

16/11/2010 TDD4ES 36

• Assertions

3.5 EmbUnit

• No floating point support

• Advanced output possible – XML outputter library

• Scripts available to deal with verbosity

TEST_ASSERT(true); //succeeds

TEST_ASSERT_MESSAGE(false, “message”) //fails

TEST_ASSERT_EQUAL_INT(1,2) //fails

TEST_ASSERT_EQUAL_STRING(“EP”, “ep”) //fails

TEST_ASSERT_NULL(pointer) //NULL = succeeds

Workshop TDD4ES

16/11/2010

EP 19

16/11/2010 TDD4ES

3.6 Exercise

37

02_TDDonTarget_TempLibrary Requirements

• Port and extend the temperature library to the EasyARM development board

Goals

• TDD without hardware involvement on target • Getting acquainted with TDD on target

16/11/2010 TDD4ES

3.6 Exercise

38

Some inspiration…

Kelvin = Celsius + 273,15

Celsius = (Fahrenheit - 32) * 5/9

KelvinTest

CelsiusTest

FahrenheitTest

Celsius2KelvinTest

Kelvin2CelsiusTest

Fahrenheit2KelvinTest

Kelvin2FahrenheitTest

Fahrenheit2CelsiusTest …

Workshop TDD4ES

16/11/2010

EP 20

16/11/2010 TDD4ES

3.7 Testable design

39

3-Tier design

Hardware aware

Hardware specific

Hardware independent

• Reuse

• Change implementation

• Data hiding

• Testable design

16/11/2010 TDD4ES

3.7 Testable design

40

Example problem:

Generic definition of pin & port number of a hardware component, without duplication all over the code.

Typical C solution:

/* hardware.h */

#define MEANINGFUL_NAME extern const int pinNumber;

/* hardware.c */

const int pinNumber = 1 << N;

/* main.c */

init_hardware(MEANINGFUL_NAME);

Workshop TDD4ES

16/11/2010

EP 21

16/11/2010 TDD4ES

3.7 Testable design

41

(simplified adapter pattern)

TempSensor - port - pin # getPin() # getPort()

DS1820

production

TempSensorAdapter + getPin() + getPort()

testing

unable to call from test

C++ solution

notation UML 2.0

16/11/2010 TDD4ES

Next session

Session 2 (Thursday 18/11/2010)

1. Test on target (continued)

2. Test on host

3. Remote testing

42

«Program testing can be a very effective way to show the presence of bugs, but is hopelessly inadequate for showing their absence. » Dijkstra (1972)

Workshop TDD4ES

16/11/2010

EP 22

16/11/2010 TDD4ES

Literature

• K. Beck. Test-Driven Development: By Example. Addison-Wesley, 2003.

• K. Beck and C. Andres. Extreme Programming Explained: Embrace Change (2nd ed.). Addison-Wesley, 2004.

• B.W. Boehm. Software Engineering Economics (Prentice-Hall Advances in Computing Science & Technology Series). Prentice Hall PTR, October 1981.

• J. Boydens, P. Cordemans, and E. Steegmans. Test-Driven Development of Embedded Software. In European Conference on the Use of Modern Information and Communication Technologies, 2010.

• E. Dijkstra. The humble programmer. Commun. ACM, 15(10):859866, 1972.

• M. Feathers. Working Effectively with Legacy Code. Prentice Hall PTR, 2005.

• M. Fowler. UML Distilled (3rd ed.). Addison-Wesley, 2004.

• M. Fowler. Refactoring: Improving the design of existing code. Addison-Wesley, 1999.

• E. Gamma, R. Helm, R. Johnson, and J. Vlissides. Design patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley, 1995.

• J. Grenning. Progress before hardware. Agile Alliance Newsletter, 4:74_79, 2004.

43

16/11/2010 TDD4ES

Literature continued

• M. Karlesky, W. Bereza, and C. Erickson. Effective test driven development for embedded software. In IEEE 2006 Electro/Information Technology Conference, 2006.

• R. Koss, and J. Langr. Test-Driven Development in C. In C/C++ Users Journal, 2002.

• M. Müller and F. Padberg. About the return on investment of test-driven development. In International Workshop on Economics-Driven Software Engineering Research EDSER-4, 2003.

• R. Osherove. The Art of Unit Testing. Manning, 2009.

44