Gerlof Hoekstra, Atos

OMG, What Have We Done !?

www.eurostarconferences.com

@esconfs#esconfs

Gerlof Hoekstra (NL) Started as software engineer Early 90’s: Testing!

Sr Test Consultant/-Manager at Atos

From software testing to large scale integration testing◦ Complex business processes◦ Many stakeholders◦ Many (software) suppliers

I had a dream…

Implemented standard test procedures

10 years later: everything still the same

Dream came true: but does this make me happy?

Assembling my E2E test team

Certified test professionals or ….?

What has happened?

Somewhere, something must have gone wrong….

The Test Plan Independent Testing Test Reporting & The Go / No Go Advice

Entry/exit criteria

Defect databases

Defect handling workflows

Test case specifications

Defect severity definitions

…

…

Raise a

defectnew Open Assigned

Accept

defectanalyse Y fixed Deployedfix defect

in progress

Deliver fix

to test

Retest

defectRetest OK Y Validated Closed

PostponedOn hold, postpone defect N

RejectedN Accept

rejectionY

N

The creation of the test plan involves the following activities:1. Establishing the assignment2. Understanding the assignment3. Determining the test basis4. Analysing the product risks5. Determining the test strategy6. Estimating the effort7. Determining the planning8. Allocating test units and test techniques9. Defining the test products10.Defining the organisation11.Defining the infrastructure12.Organising the management13.Determining the test project risks and countermeasures14.Feedback and consolidation of the plan

The creation of the test plan involves the following activities:1. Establishing the assignment2. Understanding the assignment3. Determining the test basis4. Analysing the product risks5. Determining the test strategy6. Estimating the effort7. Determining the planning8. Allocating test units and test techniques9. Defining the test products10.Defining the organisation11.Defining the infrastructure12.Organising the management13.Determining the test project risks and countermeasures14.Feedbackand consolidation of the plan

Now look at this !?!

Shall we strip the Test Plan to the bone and never write those big, boring and useless documents anymore?

Shall we strip the test planning process to the bone and neverdo it alone at the office but always in heavy interaction with our stakeholders?

“Testing requires skills that developers generally don’t have”

“Testing should be done by independent professionals, specialists”

“You can’t let a butcher test his own meat”

Really??

Testing is part of our life …

“We strictly separate development and testing…”

“We pay the developing party a fixed price for the product they deliver…”

“We pay the testing provider a fixed price for each defect they discover…”

“But we subtract that from the fee for the developing party!”

Nail down entry/exit & acceptance criteria

Hire independent test supplier for acceptance testing, preferably fixed price based on:◦ function points◦ number of test cases◦ number of requirements◦ ---

Test supplier executes acceptance tests

The customer accepts

Is this the way to get a product accepted by its stakeholders?

Don’t we reduce testing to just a series of contractual checks?

Does this customer realize what he is (not) getting?

If we need to ‘dig somewhat deeper’: how about the contract?

Shouldn’t we (the test providers) at least warn our customer?

Numbers/Graphs/KPI’s

# test cases executed

# test cases passed

$ test cases failed

# defects found/fixed/open

Coverage %

Defect density

Defect Detection %

Lists of open defects

Traffic lights

GO/NO GO ADVICE

Does this mean:

we have a quality product now?

the product complies to the design?

we have a usable product?

We have reached 95% multiple condition

coverage

What on earth does that mean??

Must be good, 95% is almost everything

Multiple condition coverage

If my customer asks me (the test manager) for a go/no go advice, I consider that as a personal failure. I did not properly do my job!

If I did, what would my advice be worth anyway?

We covered 95% of the area

We found 60 mines

We disarmed 58 mines

The 2 mines we did not disarm are here

These are MSM MKII mines containing 4Kg of Composition B explosives

IS THIS USEFUL INFORMATION ?

WOULD YOU APPRECIATE MY GO/NO GO ADVISE?

Do you consider your ‘mature’ test process as a bless or a curse?

Shouldn’t I complain and just be glad with what we have accomplished?

Consider you own test plan template. If you would strip it to the bone:◦ What parts would you immediately throw out?◦ What should definitely be in there?

Consider your own test planning process. How could you make it◦ Lighter?◦ More effective?◦ More interactive?◦ Less boring?

To test suppliers◦ (Ethical question): Do you join the rat race of test commoditization/

should we warn customers for the consequences of excessive separation of testing? How can we?

To customers◦ Does your purchase department really know what they are doing when

they define all those fancy KPI’s.

◦ How can you build positive incentives into test SLA’s to stimulate testers to use their brains in stead of just following procedures and recipes?

Suggestions for alternative ways of reporting?

Can you imagine why I never use the traffic lights symbols in test reporting?

What would be a good alternative?