10 (PES) Testiranje Elektronskih Sistema - Deo 1

8/18/2019 10 (PES) Testiranje Elektronskih Sistema - Deo 1

1/58

6/17/2011

1

1

Testiranje

elektronskihsistema (1)


2/58

6/17/2011

2

2


3/58

6/17/2011

3

3


4/58

6/17/2011

4

System Design at a Crossroad• Productivity Crisis

– 21% Productivity Increase / Year vs.

– 58% Complexity Increase / Year

Technology

Productivity

Crisis

Productivity

IP-0,18 µ

Technology

Productivity

Crisis

Productivity

Crisis

Productivity

IP-IP-0,18 µ

Productivity

Crisis

Productivity

IP-0,18 µ

Productivity

Crisis

Productivity

Crisis

Productivity

IP-IP-0,18 µ

4

Validation

Techniques

Design

Methods

1988 1992 1996 2000 Time

G a t e s

RTL

oc s

0,8 µ

0,6 µ

0,5 µ

0,35 µ HW/SWCo-Design

(HLS)

Validation

Techniques

Validation

Techniques

Design

Methods

Design

Methods

Design

Methods

1988 1992 1996 2000 Time

G a t e s

RTL

oc soc s

0,8 µ

0,6 µ

0,5 µ

0,35 µ HW/SWCo-Design

HW/SWCo-Design

(HLS)

Validation

Techniques

Design

Methods

1988 1992 1996 2000 Time

G a t e s

RTL

Blocks

0,8 µ

0,6 µ

0,5 µ

0,35 µ HW/SW

Co-Design

(HLS)

Validation

Techniques

Validation

Techniques

Design

Methods

Design

Methods

Design

Methods

1988 1992 1996 2000 Time

G a t e s

RTL

BlocksBlocks

0,8 µ

0,6 µ

0,5 µ

0,35 µ HW/SW

Co-Design

HW/SW

Co-Design

(HLS)


5/58

6/17/2011

5

Electronic Design Automation (EDA)• System on a Chip development:

Yesterday’s chip is today’s functional block!

New design methodologies are neededNew design methodologies are needed

2.5 million gates

New Design Paradigm

2.5 million gates

New Design Paradigm

5

3.0µ 1.0µ 0.5µ 0.2µ

20K gatesSchematics

& simulation

50K gatesSchematics& Synthesis

500k gates

Simulation,Emulation,Synthesis,

Formal equivalence

Source: ICE3.0µ 1.0µ 0.5µ 0.2µ

20K gatesSchematics

& simulation

50K gatesSchematics& Synthesis

500k gates

Simulation,Emulation,Synthesis,

Formal equivalence

Source: ICE


6/58

6/17/2011

6

Testing and Quality

How much

to test?

Cost oftesting

Cost of quality

Cost

Cost oftesting

Cost of quality

Cost

7

e pro emof testingcan only becontained

not solved”

T.Williams Quality

Cost ofthe fault

100%0%Optimum

test / quality

Quality

Cost ofthe fault

100%0%Optimum

test / quality

6/17/2011


7/58

6/17/2011

7

The Role of Test

Dependability

Security Safety

There is no security

on the earth,there is onlyopportunity

DouglasMcArthur (General)

10

Fault-Tolerance

Fault DiagnosisTest

BIST

Test Diagnosis

Design for testability:

Reliability

System


8/58

6/17/2011


9/58

6/17/2011

9

How Much to Test?

Paradox:

264 input patterns (!)for 32-bit accumulator

will be not enough.

A short will change the circuit

Time can be your best friendor your worst enemy(Ray Charles)

Y = F x x x

12

,

and you will need because of that

265 input patterns

Paradox: Mathematicians counted that Intel 8080

needed for exhaustive testing 37 (!) yearsManufacturer did it by 10 secondsMajority of functions will never activated

during the lifetime of the system

&&

x1

x2

x3

yState q

Y = F(x1, x2, x3,q)

*

1

1

Bridging fault0

6/17/2011


10/58

6/17/2011

10

The Problem is Money?

Cost oftesting

Cost of qualityCost

How to succeed? Try too hard!

How to fail? Try too hard!

(From American Wisdom)

13

Quality

Cost ofthe fault

100%0%Optimum

test / quality

Conclusion:

“The problem of testing

can only be contained

not solved”

T.Williams

F a u l t C o v e r a g e

Testcoverage

function

Time

6/17/2011


11/58

6/17/2011

11

Hierarchy

Paradox:

To generate a test

for a block in a system,

The best place to start is

with a good title.Then builda song around it.(Wisdom of country music)

14

needed2 days and 2 nights

An engineerdid it by hand

with 15 minutes

So, why

computers?

System

16 bit

counter

&

1Sequence

of 216 bits

Sea of gates

6/17/2011


12/58

6/17/2011

12

20

6/17/2011


13/58

13

System testing

21

6/17/2011


14/58

14

System testing

• Testing the system as a whole to validatethat it meets its specification and the

22

o ect ves o ts users.

6/17/2011


15/58

15

Development testing

• Hardware and software components should be tested

as they are developed and as sub-systems are created.These testing activities include:

– Unit testing.

23

–

– Sub-system testing

• However, these tests cannot cover:

– Interactions between components or sub-systems

where the interaction causes the system to behavein an unexpected way

– The emergent properties of the system

6/17/2011


16/58

16

System testing• Intended to test the system as a whole rather than individual

system components

– Integration testing• As the system is integrated, it is tested by the system

developer for specification compliance

–

24

• The behaviour of the system is tested underconditions of load

– Acceptance testing

• The system is tested by the customer to check if itconforms to the terms of the development contract

• Inevitably, system testing reveals errors which wereundiscovered during component testing

6/17/2011


17/58

17

Integration testing

25

6/17/2011


18/58

18

Integration testing

• Concerned with testing the system as it is integrated

from its components• Integration testing is normally the most expensive

activity in the systems integration process

26

– Interface testing where the interactions between

sub-systems and components are tested

– Property testing where system properties such asreliability, performance and usability are tested

6/17/2011


19/58

19

What is integration testing

• Testing in which software components, hardware

components, or both together are combined andtested to evaluate interactions between them.

27

• ntegrat on test ng usua y go t roug severarealword business scenarios to see whether thesystem can successfully complete workflow tasks

• Integration plan specifies the order of combiningthe modules into partial systems

6/17/2011


20/58

20

Integration test planning

• Integration testing is complex and time-consuming

and planning of the process is essential. The largerthe system, the earlier this planning must start andthe more extensive it must be.

28

• Integration test planning may be the responsibilityof a separate V & V (verification and validation)team. It is good to be undertaken by a group whichis separate from the development team.

6/17/2011


21/58

21

Test planning activities

• Identify possible system tests using the requirements

document• Prepare test cases and test scenarios to run these

system tests

29

, ,

simulators to support system testing

• Prepare, if necessary, operational profiles for thesystem

• Schedule the testing activities and estimate testingcosts

6/17/2011


22/58

22

Testing Activities

Tested

Subsystem

RequirementsAnalysis

Document

SystemDesign

Document

Unit

UnitTest

User

Manual

RequirementsAnalysis

Document

Subsystem

SubsystemCode

30

Subsystem

Code

FunctionalIntegration

Unit

Tested

Subsystem

Tested Subsystem

Test Test

Test

TestCode

Functioning

SystemIntegrated

Subsystems

6/17/2011


23/58

23

Component / Module testing• A unit is the smallest testable piece. In software, a

unit is defined as a database trigger, stored

procedure, function, report, form, batch loadprogram or PL/SQL program.

• Unit testing is the testing that is performed to

31

va a e a e un oes sa s y s unc onaspecification and/or that its implementationstructure does match the intended designstructure.

• Module /Component consists of units integratedinto a module that performs a specific businessfunction

6/17/2011


24/58

24

Drivers and Stubs

• Driver: A program that calls the interface procedures of

the module being tested and reports the results.

– A driver simulates a module that calls the module

32

• Stub: A program that has the same interface proceduresas a module that is being called by the module beingtested but is simpler.

– A stub simulates a module called by the module

currently being tested

6/17/2011


25/58

25

Drivers and Stubs

Driver Module

Stub

33

call call

6/17/2011


26/58

26

How to test this system ??

Analysis

Embedded Controller

34

Heart support device

Control

Software

Com Port

Patients

Database.

6/17/2011


27/58

27

Artifacts and Roles for Integration Testing

Testengineer

Componentengineer

Integrationtester

Systemtester

35

Use-casemodel

Test case

Testprocedure

Test

evaluationTestplan Test

component

r

. . . . . . . . . . . . . . . . . . responsible for . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Defectmanagement

6/17/2011


28/58

28

What we have here…

• Developers do everything they can to make the

software work.• Testers do everything they can to make the software

fail.

36

6/17/2011


29/58

29

Integration testing approaches

• Different approaches to integration testing

37

– Bottom-up – Top-down

– Big-bang

– Sandwich

6/17/2011


30/58

30

Bottom-up Integration

A

D

B

38

C

E F G

H

I

6/17/2011


31/58

31

Bottom-Up Integration• Only terminal modules are tested in isolation

• Modules at lower levels are tested using the

previously tested higher level modules• This is done repeatedly until all subsystems are

included in the testing

•

39

the test case input to the interface of the modulebeing tested.

• However, stubs are not needed since we are startingwith the terminal modules and use already tested

modules when testing modules in the lower levels• Disadvantage:

Tests the most important subsystem last!

6/17/2011


32/58

32

Top-down Integration

A

D

B

40

C

E F G

H

I

6/17/2011


33/58

33

Top-down Integration• Only modules tested in isolation are the modules

which are at the highest level

• After a module is tested, the modules directly calledby that module are merged with the already testedmodule and the combination is tested

41

•

the test

• Requires stub modules to simulate the functions ofthe missing modules that may be called.

• However, drivers are not needed since we arestarting with the modules which is not used by anyother module and use already tested modules whentesting modules in the higher levels

6/17/2011


34/58

34

Sandwich Testing Strategy• Combines top-down strategy with bottom-up

strategy

• The system is view as having three layers

– A target layer in the middle

– A layer above the target

– A la er below the tar et

42

– Testing converges at the target layer• How do you select the target layer if there are more

than 3 layers?

– Heuristic: Try to minimize the number of stubsand drivers

• Top and Bottom Layer Tests can be done in parallel

• Does not test the individual subsystemsthoroughly before integration

6/17/2011


35/58

35

Other Approaches to Integration

• Big Bang Integration

– Every module is unit tested in isolation

43

– er a o e mo u es are es e ey are aintegrated together at once and tested

6/17/2011


36/58

36

Implementation Approaches Advantages and Disadvantages:

• Bottom-upadvantages

– Separately

• Bottom-updisadvantages

– Drivers must be written

44

e ugge mo u es

– System test by

integrating

previously debuggedmodules

– Testing upper-level

modules is easier

– Upper-level, criticalmodules are built last

– Drivers are more

difficult to write than

stubs – User interfaces are top-

level modules

6/17/2011


37/58

37


• Top-down

advantages – Separately

debugged modules

• Top-down

disadvantages – Stubs must be written

– Low-level, critical

45

– System test by

integrating

previously debuggedmodules

– Stubs are easier tocode than drivers

– User interfaces are

top-level modules

modules built last – Testing upper-level

modules is difficult

6/17/2011


38/58

38


• Big bangadvantages

– None

• Big bangdisadvantages

– Difficult to debug

46

– Much throwaway code

– Critical and peripheralmodules not

distinguished

– User does not seeproduct until very late inthe development cycle

6/17/2011


39/58

39

Summary of testing

• Verify operation at normal parameter values

(a black box test based on the unit’s requirements)• Verify operation at limit parameter values

(black box)

47

• Verify operation outside parameter values

(black box)

• Ensure that all instructions execute(statement coverage)

• Check all paths, including both sides of allbranches

(decision coverage)

6/17/2011


40/58

40

Summary of testing

• Check the use of all called objects• Verify the handling of all data structures

48

• er y t e an ng o a es

• Check normal termination of all loops

(part of a correctness proof)

• Check abnormal termination of all loops

6/17/2011


41/58

41

Summary of testing

• Check normal termination of all recursions

• Check abnormal termination of all recursions

49

• Verify the handling of all error conditions

• Check timing and synchronization

• Verify all hardware dependencies

6/17/2011


42/58

42

Interface testing

Test

cases

50

BA

C

6/17/2011


43/58

43

What is an interface?• An agreed mechanism for communication between

different parts of the system

• System interface classes

– Hardware interfaces• Involving communicating hardware units

– Hardware/software interfaces

•

51

software – Software interfaces• Involving communicating software components or

sub-systems – Human/computer interfaces

• Involving the interaction of people and the system – Human interfaces

• Involving the interactions between people in theprocess

6/17/2011


44/58

44

Hardware interfaces• Physical-level interfaces

– Concerned with the physical connection of different

parts of the system e.g. plug/socket compatibility,physical space utilisation, wiring correctness, etc.

• Electrical-level interfaces

52

– oncerne w e e ec r ca e ec ron c

compatibility of hardware units i.e. can a signalproduced by one unit be processed by another unit

• Protocol-level interfaces

– Concerned with the format of the signals

communicated between hardware units.

6/17/2011


45/58

45

Software interfaces• Parameter interfaces

– Software units communicate by setting pre-definedparameters

• Shared memory interfaces – Software units communicate through a shared area

of memory

53

–

type• Procedural interfaces – Software units communicate by calling pre-defined

procedures• Message passing interfaces

– Software units communicate by passing messagesto each other

6/17/2011


46/58

46

Parameter interfaces

54

SS1SS2

Parameter list

6/17/2011


47/58

47

Shared memory interfaces

SS1 SS2 SS3

55

Shared memory area

6/17/2011


48/58

48

Procedural interfaces

56

SS1SS2

Defined procedures(API)

6/17/2011


49/58

49

Message passing interfaces

57

SS1SS2

Exchangedmessages

6/17/2011


50/58

50

Interface errors

• Interface misuse

– A calling component calls another component andmakes an error in its use of its interface e.g.parameters in the wrong order

58

– A calling component embeds assumptions aboutthe behaviour of the called component which areincorrect

• Timing errors

– The calling and called component operate at

different speeds and out-of-date information is

accessed

6/17/2011


51/58

51

Stress testing

59

6/17/2011


52/58

52

Stress testing• Exercises the system beyond its maximum design load

– The argument for stress testing is that system

failures are most likely to show themselves at theextremes of the system’s behaviour

– Tests failure behaviour

60

– When a system is overloaded, it should degrade

gracefully rather than fail catastrophically

• Particularly relevant to distributed systems

– As the load on the system increases, so too does

the network traffic. At some stage, the network islikely to become swamped and no useful work can

be done

6/17/2011


53/58

53

Acceptance testing

61

6/17/2011


54/58

54

Acceptance testing

• The process of demonstrating to the customer that the

system is acceptable• Based on real data drawn from customer sources. The

system must process this data as required by the

62

• Generally carried out by customer and systemdeveloper together

• May be carried out before or after a system has been

installed

6/17/2011


55/58

55

Performance testing

• Concerned with checking that the system meets its

performance requirements – Number of transactions processed per second

– Response time to user interaction

63

– me o comp e e spec e opera ons

• Generally requires some logging software to beassociated with the system to measure its performance

• May be carried out in conjunction with stress testingusing simulators developed for stress testing

6/17/2011


56/58

56

Reliability testing

• The system is presented with a large number of ‘typical’

inputs and its response to these inputs is observed• The reliability of the system is based on the number of

incorrect outputs which are generated in response to

64

• The profile of the inputs (the operational profile) mustmatch the real input probabilities if the reliability

estimate is to be valid

6/17/2011


57/58

57

Security testing

• Security testing is concerned with checking that the

system and its data are protected from accidental ormalicious damage

• Unlike other types of testing, this cannot really be tested

65

.

against unanticipated as well as anticipated attacks• Security testing may be carried out by inviting people to

try to penetrate the system through security loopholes

6/17/2011


58/58

58

Key points• System testing involves testing the system as a

whole. It is a critical activity in systems integration

• Systems testing includes interface testing, stresstesting and acceptance testing

66

• Advance planning of integration testing is essentialand may be the responsibility of a separate group

• Interface testing is the principal type of defecttesting carried out at this stage

10 (PES) Testiranje Elektronskih Sistema - Deo 1

Documents

Transcript of 10 (PES) Testiranje Elektronskih Sistema - Deo 1