Slides of Test and SQA
-
Upload
hanhmonbeo -
Category
Documents
-
view
249 -
download
1
Transcript of Slides of Test and SQA
Kiểm thử và Đảm bảo Chất lượng Phần mềm
Software Testing and SQA
Chapter 9
Quản lý chất lượng PhầnMềm
9.2Kiểm thử và Đảm bảo Chất lượng Phần mềm
Mục tiêu
Giới thiệu quá trình quản lý chất lượng vàcác hoạt động QLCL chínhGiải thích vai trò của chuẩn trong QLCLGiải thích khái niệm về độ đo phần mềm, đọđo dự báo và độ đo điều khiểnGiải thích độ đo có thể sử dụng ntn để đánhgiá chất lượng phần mềm và hạn chế củacách đo phần mềm
9.3Kiểm thử và Đảm bảo Chất lượng Phần mềm
Topics covered
Process and product qualityQuality assurance and standardsQuality planningQuality control
9.4Kiểm thử và Đảm bảo Chất lượng Phần mềm
Software quality management
Concerned with ensuring that the required level of quality is achieved in a software product.Involves defining appropriate quality standards and procedures and ensuring that these are followed.Should aim to develop a ‘quality culture’where quality is seen as everyone’s responsibility.
9.5Kiểm thử và Đảm bảo Chất lượng Phần mềm
What is quality?
Quality, simplistically, means that a product should meet its specification.This is problematical for software systems
There is a tension between customer quality requirements (efficiency, reliability, etc.) and developer quality requirements (maintainability, reusability, etc.);Some quality requirements are difficult to specify in an unambiguous way;Software specifications are usually incomplete and often inconsistent.
9.6Kiểm thử và Đảm bảo Chất lượng Phần mềm
The quality compromise
We cannot wait for specifications to improve before paying attention to quality management.We must put quality management procedures into place to improve quality in spite ofimperfect specification.
9.7Kiểm thử và Đảm bảo Chất lượng Phần mềm
Scope of quality management
Quality management is particularly important for large, complex systems. The quality documentation is a record of progress and supports continuity of development as the development team changes.For smaller systems, quality management needs less documentation and should focus on establishing a quality culture.
9.8Kiểm thử và Đảm bảo Chất lượng Phần mềm
Quality management activities
Quality assuranceEstablish organisational procedures and standards for quality.
Quality planningSelect applicable procedures and standards for a particular project and modify these as required.
Quality controlEnsure that procedures and standards are followed by the software development team.
Quality management should be separate from project management to ensure independence.
9.9Kiểm thử và Đảm bảo Chất lượng Phần mềm
Quality management and software development
9.10Kiểm thử và Đảm bảo Chất lượng Phần mềm
The quality of a developed product is influenced by the quality of the production process.This is important in software development as some product quality attributes are hard to assess.However, there is a very complex and poorly understood relationship between software processes and product quality.
Process and product quality
9.11Kiểm thử và Đảm bảo Chất lượng Phần mềm
Chất lượng căn cứ vào tiến trình (1)
There is a straightforward link between process and product in manufactured goods.More complex for software because:
The application of individual skills and experience is particularly important in software development;External factors such as the novelty of an application or the need for an accelerated development schedule may impair product quality.
Care must be taken not to impose inappropriate process standards - these could reduce rather than improve the product quality.
9.12Kiểm thử và Đảm bảo Chất lượng Phần mềm
Chất lượng căn cứ vào tiến trình (2)
9.13Kiểm thử và Đảm bảo Chất lượng Phần mềm
Define process standards such as how reviews should be conducted, configuration management, etc.Monitor the development process to ensure that standards are being followed.Report on the process to project management and software procurer.Don’t use inappropriate practices simply because standards have been established.
Practical process quality
9.14Kiểm thử và Đảm bảo Chất lượng Phần mềm
Standards are the key to effective quality management.They may be international, national, organizational or project standards.Product standards define characteristics that all components should exhibit e.g. a common programming style.Process standards define how the software process should be enacted.
Đảm bảo chất lượng và chuẩn chất lượng
9.15Kiểm thử và Đảm bảo Chất lượng Phần mềm
Encapsulation of best practice- avoids repetition of past mistakes.They are a framework for quality assurance processes - they involve checking compliance to standards.They provide continuity - new staff can understand the organisation by understanding the standards that are used.
Tầm quan trọng về chuẩn
9.16Kiểm thử và Đảm bảo Chất lượng Phần mềm
Chuẩn sản phẩm và chuẩn tiến trình
Product standards Process standards
Design review form Design review conduct
Requirements document structure Submission of documents to CM
Method header format Version release process
Java programming style Project plan approval process
Project plan format Change control process
Change request form Test recording process
9.17Kiểm thử và Đảm bảo Chất lượng Phần mềm
Các vấn đề về chuẩn
They may not be seen as relevant and up-to-date by software engineers.They often involve too much bureaucratic form filling.If they are unsupported by software tools, tedious manual work is often involved to maintain the documentation associated with the standards.
9.18Kiểm thử và Đảm bảo Chất lượng Phần mềm
Involve practitioners in development. Engineers should understand the rationale underlying a standard.Review standards and their usage regularly. Standards can quickly become outdated and this reduces their credibility amongst practitioners.Detailed standards should have associated tool support. Excessive clerical work is the most significant complaint against standards.
Standards development
9.19Kiểm thử và Đảm bảo Chất lượng Phần mềm
ISO 9000
An international set of standards for quality management.Applicable to a range of organisations from manufacturing to service industries.ISO 9001 applicable to organisations which design, develop and maintain products.ISO 9001 is a generic model of the quality process that must be instantiated for each organisation using the standard.
9.20Kiểm thử và Đảm bảo Chất lượng Phần mềm
ISO 9001
Management responsibility Quality system
Control of non-conforming products Design control
Handling, storage, packaging anddelivery
Purchasing
Purchaser-supplied products Product identification and traceability
Process control Inspection and testing
Inspection and test equipment Inspection and test status
Contract review Corrective action
Document control Quality records
Internal quality audits Training
Servicing Statistical techniques
9.21Kiểm thử và Đảm bảo Chất lượng Phần mềm
ISO 9000 certification
Quality standards and procedures should be documented in an organisational quality manual.An external body may certify that an organisation’s quality manual conforms to ISO 9000 standards.Some customers require suppliers to be ISO 9000 certified although the need for flexibility here is increasingly recognised.
9.22Kiểm thử và Đảm bảo Chất lượng Phần mềm
ISO 9000 and quality management
9.23Kiểm thử và Đảm bảo Chất lượng Phần mềm
Documentation standards
Particularly important - documents are the tangible manifestation of the software.Documentation process standards
Concerned with how documents should be developed, validated and maintained.
Document standardsConcerned with document contents, structure, and appearance.
Document interchange standardsConcerned with the compatibility of electronic documents.
9.24Kiểm thử và Đảm bảo Chất lượng Phần mềm
Documentation process
9.25Kiểm thử và Đảm bảo Chất lượng Phần mềm
Document standards
Document identification standardsHow documents are uniquely identified.
Document structure standardsStandard structure for project documents.
Document presentation standardsDefine fonts and styles, use of logos, etc.
Document update standardsDefine how changes from previous versions are reflected in a document.
9.26Kiểm thử và Đảm bảo Chất lượng Phần mềm
Document interchange standards
Interchange standards allow electronic documents to be exchanged, mailed, etc. Documents are produced using different systems and on different computers. Even when standard tools are used, standards are needed to define conventions for their use e.g. use of style sheets and macros.Need for archiving. The lifetime of word processing systems may be much less than the lifetime of the software being documented. An archiving standard may be defined to ensure that the document can be accessed in future.
9.27Kiểm thử và Đảm bảo Chất lượng Phần mềm
Quality planning
A quality plan sets out the desired product qualities and how these are assessed and defines the most significant quality attributes.The quality plan should define the quality assessment process.It should set out which organisational standards should be applied and, where necessary, define new standards to be used.
9.28Kiểm thử và Đảm bảo Chất lượng Phần mềm
Quality plans
Quality plan structureProduct introduction;Product plans;Process descriptions;Quality goals;Risks and risk management.
Quality plans should be short, succinct documents
If they are too long, no-one will read them.
9.29Kiểm thử và Đảm bảo Chất lượng Phần mềm
Software quality attributes
Safety Understandability Portability
Security Testability Usability
Reliability Adaptability Reusability
Resilience Modularity Efficiency
Robustness Complexity Learnability
9.30Kiểm thử và Đảm bảo Chất lượng Phần mềm
Quality control
This involves checking the software development process to ensure that procedures and standards are being followed.There are two approaches to quality control
Quality reviews;Automated software assessment and software measurement.
9.31Kiểm thử và Đảm bảo Chất lượng Phần mềm
Quality reviews
This is the principal method of validating the quality of a process or of a product.A group examines part or all of a process or system and its documentation to find potential problems.There are different types of review with different objectives
Inspections for defect removal (product);Reviews for progress assessment (product and process);Quality reviews (product and standards).
9.32Kiểm thử và Đảm bảo Chất lượng Phần mềm
Types of review
Review type Principal purpose
Design or programinspections
To detect detailed errors in the requirements, design or code. A checklist ofpossible errors should drive the review.
Progress reviews To provide information for management about the overall progress of theproject. This is b oth a process and a product review and is concerned withcosts, plans and schedules.
Quality reviews To carry out a t echnical analysis of product components or documentation tofind mismatches between the specification and the component design, code ordocumentation and to ensure that defined quality standards have been followed.
9.33Kiểm thử và Đảm bảo Chất lượng Phần mềm
A group of people carefully examine part or all of a software system and its associated documentation.Code, designs, specifications, test plans, standards, etc. can all be reviewed.Software or documents may be 'signed off' at a review which signifies that progress to the next development stage has been approved by management.
Quality reviews
9.34Kiểm thử và Đảm bảo Chất lượng Phần mềm
Review functions
Quality function - they are part of the general quality management process.Project management function - they provide information for project managers.Training and communication function -product knowledge is passed between development team members.
9.35Kiểm thử và Đảm bảo Chất lượng Phần mềm
Quality reviews
The objective is the discovery of system defects and inconsistencies.Any documents produced in the process may be reviewed.Review teams should be relatively small and reviews should be fairly short.Records should always be maintained of quality reviews.
9.36Kiểm thử và Đảm bảo Chất lượng Phần mềm
Comments made during the review should be classified
No action. No change to the software or documentation is required;Refer for repair. Designer or programmer should correct an identified fault;Reconsider overall design. The problem identified in the review impacts other parts of the design. Some overall judgement must be made about the most cost-effective way of solving the problem;
Requirements and specification errors may have to be referred to the client.
Review results
9.37Kiểm thử và Đảm bảo Chất lượng Phần mềm
Software measurement and metrics
Software measurement is concerned with deriving a numeric value for an attribute of a software product or process.This allows for objective comparisons between techniques and processes.Although some companies have introduced measurement programmes, most organisations still don’t make systematic use of software measurement.There are few established standards in this area.
9.38Kiểm thử và Đảm bảo Chất lượng Phần mềm
Any type of measurement which relates to a software system, process or related documentation
Lines of code in a program, the Fog index, number of person-days required to develop a component.
Allow the software and the software process to be quantified.May be used to predict product attributes or to control the software process.Product metrics can be used for general predictions or to identify anomalous components.
Software metric
9.39Kiểm thử và Đảm bảo Chất lượng Phần mềm
Predictor and control metrics
9.40Kiểm thử và Đảm bảo Chất lượng Phần mềm
A software property can be measured.The relationship exists between what we can measure and what we want to know. We can only measure internal attributes but are often more interested in external software attributes.This relationship has been formalised and validated.It may be difficult to relate what can be measured to desirable external quality attributes.
Metrics assumptions
9.41Kiểm thử và Đảm bảo Chất lượng Phần mềm
Internal and external attributes
9.42Kiểm thử và Đảm bảo Chất lượng Phần mềm
The measurement process
A software measurement process may be part of a quality control process.Data collected during this process should be maintained as an organisational resource.Once a measurement database has been established, comparisons across projects become possible.
9.43Kiểm thử và Đảm bảo Chất lượng Phần mềm
Product measurement process
9.44Kiểm thử và Đảm bảo Chất lượng Phần mềm
Data collection
A metrics programme should be based on a set of product and process data.Data should be collected immediately (not in retrospect) and, if possible, automatically.Three types of automatic data collection
Static product analysis;Dynamic product analysis;Process data collation.
9.45Kiểm thử và Đảm bảo Chất lượng Phần mềm
Data accuracy
Don’t collect unnecessary data The questions to be answered should be decided in advance and the required data identified.
Tell people why the data is being collected. It should not be part of personnel evaluation.
Don’t rely on memory Collect data when it is generated not after a project has finished.
9.46Kiểm thử và Đảm bảo Chất lượng Phần mềm
A quality metric should be a predictor of product quality.Classes of product metric
Dynamic metrics which are collected by measurements made of a program in execution;Static metrics which are collected by measurements made of the system representations;Dynamic metrics help assess efficiency and reliability; static metrics help assess complexity, understandability and maintainability.
Product metrics
9.47Kiểm thử và Đảm bảo Chất lượng Phần mềm
Dynamic and static metrics
Dynamic metrics are closely related to software quality attributes
It is relatively easy to measure the response time of a system (performance attribute) or the number of failures (reliability attribute).
Static metrics have an indirect relationship with quality attributes
You need to try and derive a relationship between these metrics and properties such as complexity, understandability and maintainability.
9.48Kiểm thử và Đảm bảo Chất lượng Phần mềm
Software product metricsSoftware metric Description
Fan in/Fan-out Fan-in is a measure of the number of functions or methods that call some other functionor method (say X). Fan-out is the number of functions that are called by function X. Ahigh value for fan-in means that X i s tightly coupled to the rest of the design andchanges to X will have extensive knock-on effects. A high value for fan-out suggeststhat the overall complexity of X m ay be high because of the complexity of the controllogic needed to coordinate the called components.
Length of code This is a measure of the size of a program. Generally, the larger the size of the code of acomponent, the more complex and error-prone that component is likely to be. Length ofcode has been shown to be one of the most reliable metrics for predicting error-proneness in components.
Cyclomatic complexity This is a measure of the control complexity of a p rogram. This control complexity maybe related to program understandability. I discuss how to compute cyclomaticcomplexity in Chapter 22.
Length of identifiers This is a measure of the average length of distinct identifiers in a p rogram. The longerthe identifiers, the more likely they are to be m eaningful and hence the moreunderstandable the program.
Depth of conditionalnesting
This is a measure of the depth of nesting of if-statements in a program. Deeply nested ifstatements are hard to understand and are potentially error-prone.
Fog index This is a measure of the average length of words and sentences in documents. The higherthe value for the Fog index, the more difficult the document is to understand.
9.49Kiểm thử và Đảm bảo Chất lượng Phần mềm
Object-oriented metricsObject-orientedmetric
Description
Depth of inheritancetree
This represents the number of discrete levels in the inheritance tree where sub-classes inherit attributes and operations (methods) from super-classes. Thedeeper the inheritance tree, the more complex the design. Many different objectclasses may have to be understood to understand the object classes at the leavesof the tree.
Method fan-in/fan-out
This is directly related to fan-in and fan-out as described above and meansessentially the same thing. However, it may be appropriate to make adistinction between calls from other methods within the object and calls fromexternal methods.
Weighted methodsper class
This is the number of methods that are included in a class weighted by thecomplexity of each method. Therefore, a simple method may have a complexityof 1 and a large and complex method a much higher value. The larger the valuefor this metric, the more complex the object class. Complex objects are morelikely to be more difficult to understand. They may not be logically cohesive socannot be reused effectively as super-classes in an inheritance tree.
Number ofoverridingoperations
This is the number of operations in a super-class that are over-ridden in a sub-class. A high value for this metric indicates that the super-class used may not bean appropriate parent for the sub-class.
9.50Kiểm thử và Đảm bảo Chất lượng Phần mềm
Measurement analysis
It is not always obvious what data means Analysing collected data is very difficult.
Professional statisticians should be consulted if available.Data analysis must take local circumstances into account.
9.51Kiểm thử và Đảm bảo Chất lượng Phần mềm
Measurement surprises
Reducing the number of faults in a program leads to an increased number of help desk calls
The program is now thought of as more reliable and so has a wider more diverse market. The percentage of users who call the help desk may have decreased but the total may increase;A more reliable system is used in a different way from a system where users work around the faults. This leads to more help desk calls.
9.52Kiểm thử và Đảm bảo Chất lượng Phần mềm
Key points
Software quality management is concerned with ensuring that software meets its required standards.Quality assurance procedures should be documented in an organisational quality manual.Software standards are an encapsulation of best practice.Reviews are the most widely used approach for assessing software quality.
9.53Kiểm thử và Đảm bảo Chất lượng Phần mềm
Key points
Software measurement gathers information about both the software process and the software product.Product quality metrics should be used to identify potentially problematical components.There are no standardised and universally applicable software metrics.
Danh sách các test tools (nc opensource) • jUnit (test unit cho Java) • csUnit (test unit cho .NET) • Nunit (.Net framework) • jTestcase • Jfunc (test function) • Ant/Nant
Cách thực hiện:
• Nghiên cứu các công cụ kiểm thử • Viết một phần mềm đơn giản bằng java hoặc .NET • Sử dụng các công cụ kiểm thử để test cho phần mềm trên.
Các loại kiểm thử • Unit test • Regrestion test • Function test • Integrated test • Test UML test • DB test
Kiểm thử và Đảm bảo Chất lượng Phần mềm
Kiểm thử và Đảm bảo Chất lượng Phần mềm
Software Testing and SQA
GVC Thạc Bình Cường, [email protected]@mail.hut.edu.vn
Mob: 0913226660Dept. of SE, Faculty of IT, HUT
Acknowledgments: Many thanks to Prof. Ian Sommerville and Prof. Roger S. Pressman
for providing the materials for this course
1.2Kiểm thử và Đảm bảo Chất lượng Phần mềm
Vài nét về môn học KT&BĐCL PM
Tổng quan về môn họcMục tiêu Đối tượngTổ chức môn họcYêu cầu kỹ năng và hiểu biếtSách giáo trìnhTự học
Giới thiệu về đối tượng
1.3Kiểm thử và Đảm bảo Chất lượng Phần mềm
Course goalsA. Knowledge of testing process, classification
scheme, terminologyB. Ability to classify test terminologyC. Knowledge of/skill with several test generation
methodsD. Understanding of possibilities, limitations, required
effort, mutual relations of test generation methodsE. Ability to determine given a situation which test
generation on method is suitableF. Verification and validationG. Software cost estimationH. SQA Management
1.4Kiểm thử và Đảm bảo Chất lượng Phần mềm
Course overview
1 Introduction: Concepts 2 V&V 3 Testing4 Data-based black box testing 5 Structure-based white box testing6 Structure-based black box testing7 Integration testing8 Software reliability engineering9 Testing in industry10 SW cost estimation11 SW quality management12 Configuration management13 Additional: Real-time & Embedded systems design & Test
1.5Kiểm thử và Đảm bảo Chất lượng Phần mềm
Required knowledge, skills
Language (speak/understand/write) [Vietnamese/English/Japanese/…]
IT Fundamentals (Computer science):Skill in programming (making+correcting errors)SE Fundamentals, software development processBehavioural specification languages: process algebra, state machines, petri nets
Mathematics:Logic, sets, reasoningStatistics
1.6Kiểm thử và Đảm bảo Chất lượng Phần mềm
Textbooks and References
1. Ian Sommerville: “Software Engineering”, 7th Ed., 2004.2. Roger S. Pressman: “Software Engineering: A Practitioner's Approach”, 6th
Ed., McGraw-Hill, 2004.3. John Musa: “Software Reliability Engineering”, McGraw-Hill, 1998. 4. Barry W. Boehm et al.: “Software Cost Estimation with COCOMO II”,
Prentice Hall PTR, 2000.5. Websites on SW Testing and SQA6. Materials, Handouts7. David E. Simon: “An Embedded Software Primer”, Addison-Wesley, 1999.8. Frank Vahid and Tony Givargis: “Embedded System Design. A Unified
Hardware/Software Introduction”, John Wiley & Sons, Inc.2002.9. Phillip A. Laplante: “Real-time Systems Design and Analysis. An
Engineer’s Handbook”, 2nd Ed., IEEE Press, 1997.10. A.W. Leigh: “Real-time Software for Small Systems”, Sigma Press, 1987.
1.7Kiểm thử và Đảm bảo Chất lượng Phần mềm
Outline
Course overviewChapter 1: Introduction to the subject
What is... error/bug/fault/failure/testing?Overview of the testing processClassifying aspects of testing
1.8Kiểm thử và Đảm bảo Chất lượng Phần mềm
Chapter 1Introduction: Concepts
What is... error/bug/fault/failure/testing?Overview of the testing processClassifying aspects of testing
1.9Kiểm thử và Đảm bảo Chất lượng Phần mềm
What is...
error: something wrong in software itself
fault: manifestation of an error(observable in software behaviour)
failure: something wrong in software behaviour
(deviates from requirements)requirements:for input i,give output 2*(i^3)
(so 6 yields 432)
software:i=input(STDIN);i=double(i);i=power(i,3);output(STDOUT,i);
output (verbose):input: 6doubling input..computing power..output: 1728
bug
error fault
fault + failure
1.10Kiểm thử và Đảm bảo Chất lượng Phần mềm
What is...
Testing: by experiment,
find errors in software (Myers, 1979)establish quality of software (Hetzel, 1988)
a successful test: finds at least one errorgives quality judgment with maximum confidence with minimum effort
1.11Kiểm thử và Đảm bảo Chất lượng Phần mềm
What’s been said?
Dijkstra:Testing can show the presence of bugs, but not the absence
Beizer:1st law: Every method you use to prevent or find bugs leaves a residue
of subtler bugs, for which other methods are needed2nd law: Software complexity grows to the limits of our ability to manage
it
Beizer:Testers are not better at test design than programmers are at code
design
...Let’s start with a global look at the testing process
1.12Kiểm thử và Đảm bảo Chất lượng Phần mềm
Concept map of the testing process
1.13Kiểm thử và Đảm bảo Chất lượng Phần mềm
Dimensions of software testing
1. What is tested?a. Software characteristics (design/embedded?/language/...)
b. Requirements (what property/quality do we want from the software)
c. Purpose (what development phase, what kind of errors, what next step)
2. How to test?a. Method (adequacy criterion: how to generate how many tests)
b. Assumptions (limitations, simplifications, heuristics)
c. Organization (documentation, implementation, execution:platform, interactive, batch)
d. Who tests? (programmer, user, third party)
3. How are the results evaluated?
1.14Kiểm thử và Đảm bảo Chất lượng Phần mềm
Dimensions + concept map
1a
1b
1c2a2b
2c
2d
3
1.15Kiểm thử và Đảm bảo Chất lượng Phần mềm
1b. Requirements
functional: the behaviour of the system is correctprecise
non-functional:performance, reliability, compatibility, robustness (stress/volume/recovery), usability, ...possibly quantifiable, always vague
1.16Kiểm thử và Đảm bảo Chất lượng Phần mềm
1c. Test purpose
Software development phases (V-model):
implementationcode
detaileddesign
specification
requirements acceptancetest
systemtest
integrationtest
unittest
1.17Kiểm thử và Đảm bảo Chất lượng Phần mềm
1c. Test purpose
What errors to find?typical typesfunctional mistakesunimplemented required features‘software does not do all it should do’
implemented non-required features‘software does things it should not do’
1.18Kiểm thử và Đảm bảo Chất lượng Phần mềm
2a. Test method dimensions
data-based
structure-based
black box white boxerror seedingtypical errors
efficiency...
1.19Kiểm thử và Đảm bảo Chất lượng Phần mềm
2b. Assumptions, limitations
Single/multiple fault:clustering/dependency of errors
Testability: can software be tested?
time, resources, accessibility,
Perfect repair...
1.20Kiểm thử và Đảm bảo Chất lượng Phần mềm
2c. Test organization
Documentationfor reuse!
Implementationplatformbatch?inputs, coordination, ...
Executionduration, timing, interruptionsmanual/interactive or automatedin parallel on several systems
1.21Kiểm thử và Đảm bảo Chất lượng Phần mềm
Dimensions + concept map classify aspects of testing
smoke testingrequirements: only major functionspurpose: often system testingmethod: error guessingassumption: (regression testing) a bit of testing suffices
capture/replay tooltest implementation & execution: the tool records test input
behaviour and saves it for automatic reusetraceability matrix
requirements, test set: the matrix shows relationship between these, gives some coverage confidence
2.1Kiểm thử và Đảm bảo Chất lượng Phần mềm
Chapter 2Verification and Validation
Thẩm tra và Phê Chuẩn
Software Testing and SQA
GVC Thạc Bình CườngDept. of SE, Faculty of IT, HUT
2.2Kiểm thử và Đảm bảo Chất lượng Phần mềm
Objectives
To introduce software verification and validation and to discuss the distinction between them.To describe the program inspection process and its role in V & V.To explain static analysis as a verification technique.To describe the Cleanroom software development process.
2.3Kiểm thử và Đảm bảo Chất lượng Phần mềm
Topics covered
Verification and validation planningSoftware inspectionsAutomated static analysisCleanroom software development
2.4Kiểm thử và Đảm bảo Chất lượng Phần mềm
Verification: "Are we building the product right?”.
The software should conform to its specification.
Validation:"Are we building the right product?”.
The software should do what the user really requires.
Verification vs validation
2.5Kiểm thử và Đảm bảo Chất lượng Phần mềm
Is a whole life-cycle process - V & V must be applied at each stage in the software process.Has two principal objectives
The discovery of defects in a system;The assessment of whether or not the system is useful and useable in an operational situation.
The V & V process
2.6Kiểm thử và Đảm bảo Chất lượng Phần mềm
V& V goals
Verification and validation should establish confidence that the software is fit for purpose.This does NOT mean completely free of defects.Rather, it must be good enough for its intended use and the type of use will determine the degree of confidence that is needed.
2.7Kiểm thử và Đảm bảo Chất lượng Phần mềm
V & V confidence
Depends on system’s purpose, user expectations and marketing environment
Software functionThe level of confidence depends on how critical the software is to an organisation.
User expectationsUsers may have low expectations of certain kinds of software.
Marketing environmentGetting a product to market early may be more important than finding defects in the program.
2.8Kiểm thử và Đảm bảo Chất lượng Phần mềm
Software inspections. Concerned with analysis of the static system representation to discover problems (static verification)
May be supplement by tool-based document and code analysis
Software testing. Concerned with exercising and observing product behaviour (dynamic verification)
The system is executed with test data and its operational behaviour is observed
Static and dynamic verification
2.9Kiểm thử và Đảm bảo Chất lượng Phần mềm
Static and dynamic V&V
2.10Kiểm thử và Đảm bảo Chất lượng Phần mềm
Can reveal the presence of errors NOT their absence.The only validation technique for non-functional requirements as the software has to be executed to see how it behaves.Should be used in conjunction with static verification to provide full V&V coverage.
Program testing
2.11Kiểm thử và Đảm bảo Chất lượng Phần mềm
Defect testingTests designed to discover system defects.A successful defect test is one which reveals the presence of defects in a system.
Validation testingIntended to show that the software meets its requirements.A successful test is one that shows that a requirements has been properly implemented.
Types of testing
2.12Kiểm thử và Đảm bảo Chất lượng Phần mềm
Defect testing and debugging are distinct processes.Verification and validation is concerned with establishing the existence of defects in a program.Debugging is concerned with locating and repairing these errors.Debugging involves formulating a hypothesis about program behaviour then testing these hypotheses to find the system error.
Testing and debugging
2.13Kiểm thử và Đảm bảo Chất lượng Phần mềm
The debugging process
2.14Kiểm thử và Đảm bảo Chất lượng Phần mềm
Careful planning is required to get the most out of testing and inspection processes.Planning should start early in the development process.The plan should identify the balance between static verification and testing.Test planning is about defining standards for the testing process rather than describing product tests.
V & V planning
2.15Kiểm thử và Đảm bảo Chất lượng Phần mềm
The V-model of development
2.16Kiểm thử và Đảm bảo Chất lượng Phần mềm
The structure of a software test plan
The testing process.Requirements traceability.Tested items.Testing schedule.Test recording procedures.Hardware and software requirements.Constraints.
2.17Kiểm thử và Đảm bảo Chất lượng Phần mềm
Lập kế hoạch kiểm thử
Quá trình kiểm thửMô tả các giai đoạn chính của quá trình kiểm thử .
Lần theo dấu vếtNgười sử dụng hầu như chỉ quan tâm đến hệ thống có đáp ứng các yêu cầu và kiểm thử cần lập kế hoạch sao cho các yêu cầu được kiểm thửmọt cách độc lập .
Tested itemsThe products of the software process that are to be tested should be specified.
Testing scheduleAn overall testing schedule and resource allocation for this schedule. This, obviously, is linked to the more general project development schedule.
2.18Kiểm thử và Đảm bảo Chất lượng Phần mềm
Lập kế hoạch kiểm thử(cont.)
Thủ tục ghi chép lại kiểm thử. It is not enough simply to run tests. The results of the tests must be systematically recorded. It must be possible to audit the testing process to check that it been carried out correctly.
Các yêu cầu về phần cứng và phần mềmThis section should set out software tools required and estimated hardware utilisation.
Ràng buộcConstraints affecting the testing process such as staff shortages should be anticipated in this section.
2.19Kiểm thử và Đảm bảo Chất lượng Phần mềm
Software inspections
These involve people examining the source representation with the aim of discovering anomalies and defects.Inspections not require execution of a system so may be used before implementation.They may be applied to any representation of the system (requirements, design,configuration data, test data, etc.).They have been shown to be an effective technique for discovering program errors.
2.20Kiểm thử và Đảm bảo Chất lượng Phần mềm
Inspection success
Many different defects may be discovered in a single inspection. In testing, one defect ,may mask another so several executions are required.The reuse domain and programming knowledge so reviewers are likely to have seen the types of error that commonly arise.
2.21Kiểm thử và Đảm bảo Chất lượng Phần mềm
Inspections and testing
Inspections and testing are complementary and not opposing verification techniques.Both should be used during the V & V process.Inspections can check conformance with a specification but not conformance with the customer’s real requirements.Inspections cannot check non-functional characteristics such as performance, usability, etc.
2.22Kiểm thử và Đảm bảo Chất lượng Phần mềm
Program inspections
Formalised approach to document reviewsIntended explicitly for defect detection (not correction).Defects may be logical errors, anomalies in the code that might indicate an erroneous condition (e.g. an uninitialised variable) or non-compliance with standards.
2.23Kiểm thử và Đảm bảo Chất lượng Phần mềm
Inspection pre-conditionsA precise specification must be available.Team members must be familiar with the organisation standards.Syntactically correct code or other system representations must be available. An error checklist should be prepared.Management must accept that inspection will increase costs early in the software process.Management should not use inspections for staff appraisal ie finding out who makes mistakes.
2.24Kiểm thử và Đảm bảo Chất lượng Phần mềm
The inspection process
2.25Kiểm thử và Đảm bảo Chất lượng Phần mềm
Inspection procedure
System overview presented to inspection team.Code and associated documents are distributed to inspection team in advance.Inspection takes place and discovered errors are noted.Modifications are made to repair discovered errors.Re-inspection may or may not be required.
2.26Kiểm thử và Đảm bảo Chất lượng Phần mềm
Inspection rolesAuthor or ownerThe programmer or designer responsible for producing the program or document. Responsible for fixing defects discovered during the inspection process.
InspectorFinds errors, omissions and inconsistencies in programs and documents. May also identify broader issues that are outside the scope of the inspection team.
ReaderPresents the code or document at an inspection meeting.
2.27Kiểm thử và Đảm bảo Chất lượng Phần mềm
Inspection roles (cont.)
Scribe (Secretary)Records the results of the inspection meeting.
Chairman or moderator Manages the process and facilitates the inspection. Reports process results to the Chief moderator.
Chief moderatorResponsible for inspection process improvements, checklist updating, standards development etc.
2.28Kiểm thử và Đảm bảo Chất lượng Phần mềm
Inspection checklists
Checklist of common errors should be used to drive the inspection.Error checklists are programming language dependent and reflect the characteristic errors that are likely to arise in the language.In general, the 'weaker' the type checking, the larger the checklist.Examples: Initialisation, Constant naming, loop termination, array bounds, etc.
2.29Kiểm thử và Đảm bảo Chất lượng Phần mềm
Inspection checks 1a
•Data faults• Are all program variables initialised before
their values are used?• Have all constants been named?• Should the upper bound of arrays be
equal to the size of the array or Size -1?• If character strings are used, is a
delimiter explicitly assigned?• Is there any possibility of buffer overflow?
2.30Kiểm thử và Đảm bảo Chất lượng Phần mềm
Inspection checks 1b•Control faults
• For each conditional statement, is the condition correct?
• Is each loop certain to terminate?• Are compound statements correctly
bracketed?• In case statements, are all possible cases
accounted for?• If a break is required after each case in
case statements, has it been included?
2.31Kiểm thử và Đảm bảo Chất lượng Phần mềm
Inspection checks 1c
•Input/output faults•Are all input variables used?•Are all output variables assigned a value before they are output?
•Can unexpected inputs cause corruption?
2.32Kiểm thử và Đảm bảo Chất lượng Phần mềm
Inspection checks 2a•Interface faults
• Do all function and method calls have the correct number of parameters?
• Do formal and actual parameter types match?
• Are the parameters in the right order? • If components access shared memory, do
they have the same model of the shared memory structure?
2.33Kiểm thử và Đảm bảo Chất lượng Phần mềm
Inspection checks 2b•Storage management faults
• If a linked structure is modified, have all links been correctly reassigned?
• If dynamic storage is used, has space been allocated correctly?
• Is space explicitly de-allocated after it is no longer required?
•Exception management faults• Have all possible error conditions been
taken into account?
2.34Kiểm thử và Đảm bảo Chất lượng Phần mềm
Inspection rate
500 statements/hour during overview.125 source statement/hour during individual preparation.90-125 statements/hour can be inspected.Inspection is therefore an expensive process.Inspecting 500 lines costs about 40 man/hours effort - about £2800 at UK rates.
2.35Kiểm thử và Đảm bảo Chất lượng Phần mềm
Automated static analysis
Static analysers are software tools for source text processing.They parse the program text and try to discover potentially erroneous conditions and bring these to the attention of the V & V team.They are very effective as an aid to inspections - they are a supplement to but not a replacement for inspections.
2.36Kiểm thử và Đảm bảo Chất lượng Phần mềm
Static analysis checks 1Fault class and Static analysis checkData faults
• Variables used before initialisation• Variables declared but never used• Variables assigned twice but never used
between assignments• Possible array bound violations • Undeclared variables
Control faults• Unreachable code• Unconditional branches into loops
2.37Kiểm thử và Đảm bảo Chất lượng Phần mềm
Static analysis checks 2
Input/output faults• Variables output twice with no intervening
assignment• Interface faults• Parameter type mismatches• Parameter number mismatches• Non-usage of the results of functions• Uncalled functions and procedures
Storage management faults• Unassigned pointers• Pointer arithmetic
2.38Kiểm thử và Đảm bảo Chất lượng Phần mềm
Stages of static analysis
Control flow analysis. Checks for loops with multiple exit or entry points, finds unreachable code, etc.Data use analysis. Detects uninitialised variables, variables written twice without an intervening assignment, variables which are declared but never used, etc.Interface analysis. Checks the consistency of routine and procedure declarations and their use
2.39Kiểm thử và Đảm bảo Chất lượng Phần mềm
Stages of static analysisInformation flow analysis. Identifies the dependencies of output variables. Does not detect anomalies itself but highlights information for code inspection or reviewPath analysis. Identifies paths through the program and sets out the statements executed in that path. Again, potentially useful in the review processBoth these stages generate vast amounts of information. They must be used with care.
2.40Kiểm thử và Đảm bảo Chất lượng Phần mềm
LINT static analysis138% more lint_ex.c#include <stdio.h>printarray (Anarray) int Anarray;{ printf(“%d”,Anarray); }
main (){ int Anarray[5]; int i; char c; printarray (Anarray, i, c); printarray (Anarray) ;}
139% cc lint_ex.c140% lint lint_ex.c
lint_ex.c(10): warning: c may be used before setlint_ex.c(10): warning: i may be used before setprintarray: variable # of args. lint_ex.c(4) :: lint_ex.c(10)printarray, arg. 1 used inconsistently lint_ex.c(4) :: lint_ex.c(10)printarray, arg. 1 used inconsistently lint_ex.c(4) :: lint_ex.c(11)printf returns value which is always ignored
2.41Kiểm thử và Đảm bảo Chất lượng Phần mềm
Use of static analysis
Particularly valuable when a language such as C is used which has weak typing and hence many errors are undetected by the compiler,Less cost-effective for languages like Java that have strong type checking and can therefore detect many errors during compilation.
2.42Kiểm thử và Đảm bảo Chất lượng Phần mềm
Verification and formal methods
Formal methods can be used when a mathematical specification of the system is produced.They are the ultimate static verification technique.They involve detailed mathematical analysis of the specification and may develop formal arguments that a program conforms to its mathematical specification.
2.43Kiểm thử và Đảm bảo Chất lượng Phần mềm
Arguments for formal methods
Producing a mathematical specification requires a detailed analysis of the requirements and this is likely to uncover errors.They can detect implementation errors before testing when the program is analyzed alongside the specification.
2.44Kiểm thử và Đảm bảo Chất lượng Phần mềm
Arguments against formal methods
Require specialized notations that cannot be understood by domain experts.Very expensive to develop a specification and even more expensive to show that a program meets that specification.It may be possible to reach the same level of confidence in a program more cheaply using other V & V techniques.
2.45Kiểm thử và Đảm bảo Chất lượng Phần mềm
The name is derived from the 'Cleanroom' process in semiconductor fabrication. The philosophy is defect avoidance rather than defect removal.This software development process is based on:
Incremental development;Formal specification;Static verification using correctness arguments;Statistical testing to determine program reliability.
Cleanroom software development
2.46Kiểm thử và Đảm bảo Chất lượng Phần mềm
The Cleanroom process
2.47Kiểm thử và Đảm bảo Chất lượng Phần mềm
Cleanroom process characteristics
Formal specification using a state transition model.Incremental development where the customer prioritises increments.Structured programming - limited control and abstraction constructs are used in the program.Static verification using rigorous inspections.Statistical testing of the system.
2.48Kiểm thử và Đảm bảo Chất lượng Phần mềm
Formal specification and inspections
The state based model is a system specification and the inspection process checks the program against this model.The programming approach is defined so that the correspondence between the model and the system is clear.Mathematical arguments (not proofs) are used to increase confidence in the inspection process.
2.49Kiểm thử và Đảm bảo Chất lượng Phần mềm
Specification team. Responsible for developing and maintaining the system specification.Development team. Responsible for developing and verifying the software. The software is NOT executed or even compiled during this process.Certification team. Responsible for developing a set of statistical tests to exercise the software after development. Reliability growth models used to determine when reliability is acceptable.
Cleanroom process teams
2.50Kiểm thử và Đảm bảo Chất lượng Phần mềm
The results of using the Cleanroom process have been very impressive with few discovered faults in delivered systems.Independent assessment shows that the process is no more expensive than other approaches.There were fewer errors than in a 'traditional' development process.However, the process is not widely used. It is not clear how this approach can be transferred to an environment with less skilled or less motivated software engineers.
Cleanroom process evaluation
2.51Kiểm thử và Đảm bảo Chất lượng Phần mềm
Key points
Verification and validation are not the same thing. Verification shows conformance with specification; validation shows that the program meets the customer’s needs.Test plans should be drawn up to guide the testing process.Static verification techniques involve examination and analysis of the program for error detection.
2.52Kiểm thử và Đảm bảo Chất lượng Phần mềm
Key points
Program inspections are very effective in discovering errors.Program code in inspections is systematically checked by a small team to locate software faults.Static analysis tools can discover program anomalies which may be an indication of faults in the code.The Cleanroom development process depends on incremental development, static verification and statistical testing.
Kiểm thử và Đảm bảo Chất lượng Phần mềm
Software Testing and SQA
Chương 3Kiểm thử phần mềm
3.2Kiểm thử và Đảm bảo Chất lượng Phần mềm
Mục tiêu
Thảo luận sự Phân biệt giữa kiểm thửchấp nhận và kiểm thử phát hiện lỗi Mô tả các nguyên lý của kiểm thử hệthống và kiểm thử thành phần Mô tả chiến lược tạo ra các trường hợp kiểm thử hệ thống Hiểu các đặc trưng cần thiết của công cụ sử dụng cho tự động hoá kiểm thử
3.3Kiểm thử và Đảm bảo Chất lượng Phần mềm
Các chủ đề
Kiểm thử Hệ thống Kiểm thử thành phần Thiết kế trường hợp kiểm thửTự động hoá kiểm thử
3.4Kiểm thử và Đảm bảo Chất lượng Phần mềm
The testing process
Component testing Testing of individual program components;Usually the responsibility of the component developer (except sometimes for critical systems);Tests are derived from the developer’s experience.
System testingTesting of groups of components integrated to create a system or sub-system;The responsibility of an independent testing team;Tests are based on a system specification.
3.5Kiểm thử và Đảm bảo Chất lượng Phần mềm
Testing phases
3.6Kiểm thử và Đảm bảo Chất lượng Phần mềm
Defect testing
The goal of defect testing is to discover defects in programsA successful defect test is a test which causes a program to behave in an anomalous wayTests show the presence not the absence of defects
3.7Kiểm thử và Đảm bảo Chất lượng Phần mềm
Testing process goals
Validation testingTo demonstrate to the developer and the system customer that the software meets its requirements;A successful test shows that the system operates as intended.
Defect testingTo discover faults or defects in the software where its behaviour is incorrect or not in conformance with its specification;A successful test is a test that makes the system perform incorrectly and so exposes a defect in the system.
3.8Kiểm thử và Đảm bảo Chất lượng Phần mềm
The software testing process
3.9Kiểm thử và Đảm bảo Chất lượng Phần mềm
Only exhaustive testing can show a program is free from defects. However, exhaustive testing is impossible,Testing policies define the approach to be used in selecting system tests:
All functions accessed through menus should be tested;Combinations of functions accessed through the same menu should be tested;Where user input is required, all functions must be tested with correct and incorrect input.
Testing policies
3.10Kiểm thử và Đảm bảo Chất lượng Phần mềm
System testing
Involves integrating components to create a system or sub-system.May involve testing an increment to be delivered to the customer.Two phases:
Integration testing - the test team have access to the system source code. The system is tested as components are integrated.Release testing - the test team test the complete system to be delivered as a black-box.
3.11Kiểm thử và Đảm bảo Chất lượng Phần mềm
Integration testing
Involves building a system from its components and testing it for problems that arise from component interactions.Top-down integration
Develop the skeleton of the system and populate it with components.
Bottom-up integrationIntegrate infrastructure components then add functional components.
To simplify error localisation, systems should be incrementally integrated.
3.12Kiểm thử và Đảm bảo Chất lượng Phần mềm
Incremental integration testing
3.13Kiểm thử và Đảm bảo Chất lượng Phần mềm
Testing approaches
Architectural validationTop-down integration testing is better at discovering errors in the system architecture.
System demonstrationTop-down integration testing allows a limited demonstration at an early stage in the development.
Test implementationOften easier with bottom-up integration testing.
Test observationProblems with both approaches. Extra code may be required to observe tests.
3.14Kiểm thử và Đảm bảo Chất lượng Phần mềm
Release testing
The process of testing a release of a system that will be distributed to customers.Primary goal is to increase the supplier’s confidence that the system meets its requirements.Release testing is usually black-box or functional testing
Based on the system specification only;Testers do not have knowledge of the system implementation.
3.15Kiểm thử và Đảm bảo Chất lượng Phần mềm
Black-box testing
IeDu lieu kiem thu
OeOutput test results
SHe thong
Inputs causinganomalousbehaviour
Outputs which revealthe presence ofdefects
3.16Kiểm thử và Đảm bảo Chất lượng Phần mềm
Testing guidelines
Testing guidelines are hints for the testing team to help them choose tests that will reveal defects in the system
Choose inputs that force the system to generate all error messages;Design inputs that cause buffers to overflow;Repeat the same input or input series several times;Force invalid outputs to be generated;Force computation results to be too large or too small.
3.17Kiểm thử và Đảm bảo Chất lượng Phần mềm
Testing scenario
A student is studying Vietnam History and has been asked to write a paper on ‘Vietnamese people's anti-American war during 1954-1975". To do this, she needs to find sources from a range of libraries, including the US ones. She logs on to the LIBSYS system and uses the search facility to discover if she can access original documents from that time. She discovers sources in various US university libraries and downloads copies of some of these. However, for one document, she needs to have confirmation from her university that she is a genuine student and that use is for non-commercial purposes. The student then uses the facility in LIBSYS that can request such permission and registers her request. If granted, the document will be downloaded to the registered library’s server (e.g. HUT's e-Library) and printed for her. She receives a message from LIBSYS telling her that she will receive an e-mail message from the Department of Information-Documentation Services when the printed document is available for collection.
3.18Kiểm thử và Đảm bảo Chất lượng Phần mềm
System tests
1. Test the login mechanism using correct and incorrect logins to checkthat valid users are accepted and invalid users are rejected.
2. Test the search facility using different queries against known sources tocheck that the search mechanism is actually finding documents.
3. Test the system presentation facility to check that information aboutdocuments is displayed properly.
4. Test the mechanism to request permission for downloading.
5. Test the e-mail response indicating that the downloaded document isavailable.
3.19Kiểm thử và Đảm bảo Chất lượng Phần mềm
Use cases
Use cases can be a basis for deriving the tests for a system. They help identify operations to be tested and help design the required test cases.From an associated sequence diagram, the inputs and outputs to be created for the tests can be identified.
3.20Kiểm thử và Đảm bảo Chất lượng Phần mềm
Collect weather data sequence chart
3.21Kiểm thử và Đảm bảo Chất lượng Phần mềm
Performance testing
Part of release testing may involve testing the emergent properties of a system, such as performance and reliability.Performance tests usually involve planning a series of tests where the load is steadily increased until the system performance becomes unacceptable.
3.22Kiểm thử và Đảm bảo Chất lượng Phần mềm
Stress testing
Exercises the system beyond its maximum design load. Stressing the system often causes defects to come to light.Stressing the system test failure behaviour.. Systems should not fail catastrophically. Stress testing checks for unacceptable loss of service or data.Stress testing is particularly relevant to distributed systems that can exhibit severe degradation as a network becomes overloaded.
3.23Kiểm thử và Đảm bảo Chất lượng Phần mềm
Component testing
Component or unit testing is the process of testing individual components in isolation.It is a defect testing process.Components may be:
Individual functions or methods within an object;Object classes with several attributes and methods;Composite components with defined interfaces used to access their functionality.
3.24Kiểm thử và Đảm bảo Chất lượng Phần mềm
Object class testing
Complete test coverage of a class involvesTesting all operations associated with an object;Setting and interrogating all object attributes;Exercising the object in all possible states.
Inheritance makes it more difficult to design object class tests as the information to be tested is not localised.
3.25Kiểm thử và Đảm bảo Chất lượng Phần mềm
Weather station object interface
3.26Kiểm thử và Đảm bảo Chất lượng Phần mềm
Weather station testing
Need to define test cases for reportWeather, calibrate, test, startup and shutdown.Using a state model, identify sequences of state transitions to be tested and the event sequences to cause these transitionsFor example:
Waiting -> Calibrating -> Testing -> Transmitting -> Waiting
3.27Kiểm thử và Đảm bảo Chất lượng Phần mềm
Objectives are to detect faults due to interface errors or invalid assumptions about interfaces.Particularly important for object-oriented development as objects are defined by their interfaces.
Interface testing
3.28Kiểm thử và Đảm bảo Chất lượng Phần mềm
Interface testing
3.29Kiểm thử và Đảm bảo Chất lượng Phần mềm
Interface types
Parameter interfacesData passed from one procedure to another.
Shared memory interfacesBlock of memory is shared between procedures or functions.
Procedural interfacesSub-system encapsulates a set of procedures to be called by other sub-systems.
Message passing interfacesSub-systems request services from other sub-systems.
3.30Kiểm thử và Đảm bảo Chất lượng Phần mềm
Interface errorsInterface misuse
A calling component calls another component and makes an error in its use of its interface e.g. parameters in the wrong order.
Interface misunderstandingA calling component embeds assumptions about the behaviour of the called component which are incorrect.
Timing errorsThe called and the calling component operate at different speeds and out-of-date information is accessed.
3.31Kiểm thử và Đảm bảo Chất lượng Phần mềm
Interface testing guidelines
Design tests so that parameters to a called procedure are at the extreme ends of their ranges.Always test pointer parameters with null pointers.Design tests which cause the component to fail.Use stress testing in message passing systems.In shared memory systems, vary the order in which components are activated.
3.32Kiểm thử và Đảm bảo Chất lượng Phần mềm
Test case design
Involves designing the test cases (inputs and outputs) used to test the system.The goal of test case design is to create a set of tests that are effective in validation and defect testing.Design approaches:
Requirements-based testing;Partition testing;Structural testing.
3.33Kiểm thử và Đảm bảo Chất lượng Phần mềm
Requirements based testing
A general principle of requirements engineering is that requirements should be testable.Requirements-based testing is a validation testing technique where you consider each requirement and derive a set of tests for that requirement.
3.34Kiểm thử và Đảm bảo Chất lượng Phần mềm
LIBSYS requirements
• The user shall be able to search either all of the initial set of databases or select a subset from it.
• The system shall provide appropriate viewers for the user to read documents in the document store.
• Every order shall be allocated a unique identifier (ORDER_ID) that the user shall be able to copy to the account’s permanent storage area.
3.35Kiểm thử và Đảm bảo Chất lượng Phần mềm
LIBSYS tests• Initiate user search for searches for items that are known to
be present and known not to be present, where the set ofdatabases includes 1 database.
• Initiate user searches for items that are known to be presentand known not to be present, where the set of databasesincludes 2 databases
• Initiate user searches for items that are known to be presentand known not to be present where the set of databasesincludes more than 2 databases.
• Select one database from the set of databases and initiateuser searches for items that are known to be present andknown not to be present.
• Select more than one database from the set of databasesand initiate searches for items that are known to be presentand known not to be present.
3.36Kiểm thử và Đảm bảo Chất lượng Phần mềm
Partition testing
Input data and output results often fall into different classes where all members of a class are related.Each of these classes is an equivalence partition or domain where the program behaves in an equivalent way for each class member.Test cases should be chosen from each partition.
3.37Kiểm thử và Đảm bảo Chất lượng Phần mềm
Equivalence partitioning
3.38Kiểm thử và Đảm bảo Chất lượng Phần mềm
Equivalence partitions
3.39Kiểm thử và Đảm bảo Chất lượng Phần mềm
Search routine specificationprocedure Search (Key : ELEM ; T: SEQ of ELEM;
Found : in out BOOLEAN; L: in out ELEM_INDEX) ;
Pre-condition-- the sequence has at least one elementT’FIRST <= T’LAST
Post-condition-- the element is found and is referenced by L( Found and T (L) = Key)
or-- the element is not in the array( not Found andnot (exists i, T’FIRST >= i <= T’LAST, T (i) = Key ))
3.40Kiểm thử và Đảm bảo Chất lượng Phần mềm
Inputs which conform to the pre-conditions.Inputs where a pre-condition does not hold.Inputs where the key element is a member of the array.Inputs where the key element is not a member of the array.
Search routine - input partitions
3.41Kiểm thử và Đảm bảo Chất lượng Phần mềm
Testing guidelines (sequences)
Test software with sequences which have only a single value.Use sequences of different sizes in different tests.Derive tests so that the first, middle and last elements of the sequence are accessed.Test with sequences of zero length.
3.42Kiểm thử và Đảm bảo Chất lượng Phần mềm
Search routine - input partitions
Sequence ElementSingle value In sequenceSingle value Not in sequenceMore than 1 value First element in sequenceMore than 1 value Last element in sequenceMore than 1 value Middle element in sequenceMore than 1 value Not in sequence
Input sequence (T) Key (Key) Output (Found, L)17 17 true, 117 0 false, ??17, 29, 21, 23 17 true, 141, 18, 9, 31, 30, 16, 45 45 true, 717, 18, 21, 23, 29, 41, 38 23 true, 421, 23, 29, 33, 38 25 false, ??
3.43Kiểm thử và Đảm bảo Chất lượng Phần mềm
Sometime called white-box testing.Derivation of test cases according to program structure. Knowledge of the program is used to identify additional test cases.Objective is to exercise all program statements (not all path combinations).
Structural testing
3.44Kiểm thử và Đảm bảo Chất lượng Phần mềm
Structural testing
3.45Kiểm thử và Đảm bảo Chất lượng Phần mềm
Pre-conditions satisfied, key element in array.Pre-conditions satisfied, key element not in array.Pre-conditions unsatisfied, key element in array.Pre-conditions unsatisfied, key element not in array.Input array has a single value.Input array has an even number of values.Input array has an odd number of values.
Binary search - equiv. partitions
3.46Kiểm thử và Đảm bảo Chất lượng Phần mềm
Binary search equiv. partitions
3.47Kiểm thử và Đảm bảo Chất lượng Phần mềm
Binary search - test cases
Input array (T) Key (Key) Output (Found, L)17 17 true, 117 0 false, ??17, 21, 23, 29 17 true, 19, 16, 18, 30, 31, 41, 45 45 true, 717, 18, 21, 23, 29, 38, 41 23 true, 417, 18, 21, 23, 29, 33, 38 21 true, 312, 18, 21, 23, 32 23 true, 421, 23, 29, 33, 38 25 false, ??
3.48Kiểm thử và Đảm bảo Chất lượng Phần mềm
Path testing
The objective of path testing is to ensure that the set of test cases is such that each path through the program is executed at least once.The starting point for path testing is a program flow graph that shows nodes representing program decisions and arcs representing the flow of control.Statements with conditions are therefore nodes in the flow graph.
3.49Kiểm thử và Đảm bảo Chất lượng Phần mềm
Binary search flow graph
3.50Kiểm thử và Đảm bảo Chất lượng Phần mềm
1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 141, 2, 3, 4, 5, 141, 2, 3, 4, 5, 6, 7, 11, 12, 5, …1, 2, 3, 4, 6, 7, 2, 11, 13, 5, …Test cases should be derived so that all of these paths are executedA dynamic program analyser may be used to check that paths have been executed
Independent paths
3.51Kiểm thử và Đảm bảo Chất lượng Phần mềm
Test automation
Testing is an expensive process phase. Testing workbenches provide a range of tools to reduce the time required and total testing costs.Systems such as Junit support the automatic execution of tests.Most testing workbenches are open systems because testing needs are organisation-specific.They are sometimes difficult to integrate with closed design and analysis workbenches.
3.52Kiểm thử và Đảm bảo Chất lượng Phần mềm
A testing workbench
3.53Kiểm thử và Đảm bảo Chất lượng Phần mềm
Testing workbench adaptation
Scripts may be developed for user interface simulators and patterns for test data generators.Test outputs may have to be prepared manually for comparison.Special-purpose file comparators may be developed.
3.54Kiểm thử và Đảm bảo Chất lượng Phần mềm
Key points
Testing can show the presence of faults in a system; it cannot prove there are no remaining faults.Component developers are responsible for component testing; system testing is the responsibility of a separate team.Integration testing is testing increments of the system; release testing involves testing a system to be released to a customer.Use experience and guidelines to design test cases in defect testing.
3.55Kiểm thử và Đảm bảo Chất lượng Phần mềm
Key pointsInterface testing is designed to discover defects in the interfaces of composite components.Equivalence partitioning is a way of discovering test cases - all cases in a partition should behave in the same way.Structural analysis relies on analysing a program and deriving tests from this analysis.Test automation reduces testing costs by supporting the test process with a range of software tools.
1Kiểm thử và Đảm bảo Chất lượng Phần mềm
Software Testing and SQA
Chương 4
Phương pháp kiểm thử
4.2Kiểm thử và Đảm bảo Chất lượng Phần mềm
Nội dung chính
Data-based black box testingclassificationmethods
boundary-value testingequivalence partitioningdecision tableserror guessing, random testing
ExampleStructure-based white box testingData/structure-based white box testingStructure-based black box testing
4.3Kiểm thử và Đảm bảo Chất lượng Phần mềm
Dimensions of software testing
1. What is tested?a. Software characteristics (design/embedded?/language/...)
b. Requirements (what property/quality do we want from the software)
c. Purpose (what development phase, what kind of errors, what next step)
2. How to test?a. Method (adequacy criterion: how to generate how many tests)
b. Assumptions (limitations, simplifications, heuristics)
c. Organization (documentation, implementation, execution:platform, interactive, batch)
d. Who tests? (programmer, user, third party)
3. How are the results evaluated?
4.4Kiểm thử và Đảm bảo Chất lượng Phần mềm
Dimensions + concept map
1a
1b
1c2a2b
2c
2d
3
4.5Kiểm thử và Đảm bảo Chất lượng Phần mềm
Test purpose
Software development phase:
implementationcode
detaileddesign
specification
requirements acceptancetest
systemtest
integrationtest
unittest
4.6Kiểm thử và Đảm bảo Chất lượng Phần mềm
Test method dimensions
data-based
structure-based
black box white boxerror seedingtypical errors
efficiency...
4.7Kiểm thử và Đảm bảo Chất lượng Phần mềm
Classifying data-based black box testing
allrequirements: some logical/mathematical input/output relationpurpose: unit testing, functional mistakesmethod: black box, data-basedassumption: both single-/multiple-fault assumption
boundary value testingmethod: typical errors (boundaries of domains)
equivalence partitioning/decision tablesmethod: efficiency
4.8Kiểm thử và Đảm bảo Chất lượng Phần mềm
Running example: mortgage
input: gender [m,f], age [18-55], monthly salary [0-10000]output: mortgage total for one personrequirements:
mortgage= salary*factor
program: (41-50 years) 35(46-55 years) 30old(31-40 years) 50(36-45 years) 55middle(18-30 years) 70(18-35 years) 75young
femalemalecategory
Mortgage (male:Boolean, age:Integer, salary:Integer): Integerbeginif (male) thenreturn ((18<=age<35)?(75*salary):(31<=age<40)?(55*salary):(30*salary))
else /* female */return ((18<=age<30)?(75*salary):(31<=age<40)?(50*salary):(35*salary))
fi;end
4.9Kiểm thử và Đảm bảo Chất lượng Phần mềm
Testing the mortgage program
each test case = input combination + expected output valuecomplete testing requires all input combinations: infeasibleselect some combinationsassumptions:
single/multiple faultinput type: quantity, enumerable, boundedtypical errorsefficiency: redundant input combinations
4.10Kiểm thử và Đảm bảo Chất lượng Phần mềm
Boundary value testing
detect errors to do with input domain boundsfor integer input x with domain [a,b], test input values around a and b# tests = for n inputs, 4n+1 input combinationsassumes:
independent quantity inputssingle-fault assumption
d
c
a b
y
x
4.11Kiểm thử và Đảm bảo Chất lượng Phần mềm
Robustness boundary value
also test values outside the domain# tests = for n input variables, 6n+1 input combinations
a test: <x_nom, y_max+1, expected output>
d
c
a b
y
x
maximal value
minimal value
nominal value
4.12Kiểm thử và Đảm bảo Chất lượng Phần mềm
Worst-case boundary value
multiple-fault assumption# tests = for n input variables, 5n input combinations
d
c
a b
y
x
4.13Kiểm thử và Đảm bảo Chất lượng Phần mềm
Worst-case robustness bv
multiple-fault assumption, tests also outside the domain# tests =for n input variables, 7n input combinations
d
c
a b
y
x
4.14Kiểm thử và Đảm bảo Chất lượng Phần mềm
Subsume relations
Method A subsumes method B if A tests at least what B tests (possibly more)Which subsume relations for boundary value variants?(Assume one fixed nominal value for each input)
d
c
a b
y
x
boundary value
A B
worst-case robustness
worst-case robustness
4.15Kiểm thử và Đảm bảo Chất lượng Phần mềm
Boundary value testing Mortgage
ObservationsInput gender is not a quantityBoundaries are not fixed:Inputs gender and age are not independent!
Variant ‘boundary value’ may not detect error(if nominal value for gender is ‘male’)
Many more errors in the program than we can expect any boundary value variant to detect
4.16Kiểm thử và Đảm bảo Chất lượng Phần mềm
Boundary value testing summary
Coverage: not good#tests: moderate to very manyUsage: straightforward, easy to implementWhen to use:
independent inputsenumerable quantities, e.g. age(obviously) when suspecting boundary errors
See literature:ZhuJorgensen
4.17Kiểm thử và Đảm bảo Chất lượng Phần mềm
Equivalence partitioning
detect errors to do with computational mistakesfor integer input x with domain [a,b] partitioned in s_x subdomains, test input values from each subdomainassumes:
independent variablesredundancy in subdomain
f
e
a b
y
x
g
c d
weak normal variant:• assumes:
– independent variables– single-fault assumption
• #tests = max s_x for any input x
4.18Kiểm thử và Đảm bảo Chất lượng Phần mềm
Equivalence partitioning
strong normal variant:assumes:
multiple-fault assumption#tests = Π s_x for each input x
f
e
a b
y
x
g
c d
4.19Kiểm thử và Đảm bảo Chất lượng Phần mềm
Equivalence partitioning
weak robust variant:tests outside domainassumes:
independent variablessingle-fault assumption
#tests =max s_x for any input x+ 2*(#inputs)
f
e
a b
y
x
g
c d
4.20Kiểm thử và Đảm bảo Chất lượng Phần mềm
Equivalence partitioning
strong robust variant:tests outside domainassumes:
multiple-fault assumption#tests = Π (s_x+2) for each
input xf
e
a b
y
x
g
c d
4.21Kiểm thử và Đảm bảo Chất lượng Phần mềm
Subsume relations
Which subsume relations for equivalence partition variants?(Assume same value selected for each input, for
each subdomain)
x
weak normal
strong robust
strong normal weak robustf
e
a b
y
x
g
c d
4.22Kiểm thử và Đảm bảo Chất lượng Phần mềm
Equivalence partitioning summary
Coverage: moderate#tests: small to moderateUsage: some study of requirements neededWhen to use:
independent inputsenumerable quantitieswhen suspecting computational errorswhen redundancy can be assumedmay easily be combined with boundary value
4.23Kiểm thử và Đảm bảo Chất lượng Phần mềm
Decision table testing
also known as cause-effect graphingdetect errors to do with computational mistakesinput are partitioned through conditionsassumes: dependent variables#tests = depends on granularity of conditions/actions
xa3: faulty inputsx
xxa: correct answer type 1a2: correct answer type 2
ynn_c3: z even?-n-yc2: x<=y?-ynyc1: x>=0?
rule 5-8rule 4rule 2, 3rule 1conditions/actions - : don’t care_ : fixed value
4.24Kiểm thử và Đảm bảo Chất lượng Phần mềm
Decision table testing Mortgage
a5: 50*agea4: 55*age
a2: 75*agea3: 70*age
c5: < young?
c2: young?c3: middle?c4: old?
a6: 30*agea7: 35*age
a1: wrong inputc7: 0 <= salary <= 10000?1c6: > old?
c1: male?conditions/actions rule 1 rule 2
4.25Kiểm thử và Đảm bảo Chất lượng Phần mềm
Decision table testing summary
Coverage: good#tests: moderate to largeUsage: much study of requirement neededWhen to use:
dependent inputswhen suspecting computational errorsmay be combined with boundary testing (tricky)
See literature:ZhuJorgensen
4.26Kiểm thử và Đảm bảo Chất lượng Phần mềm
Error guessing & Random Testing
Error guessingNo method involved, just experiment and see if something goes wrongSome people have a knack for exposing failures (e.g. young children!)
Random testingSelect input combinations randomly
Both can be very effective, but should be used in addition to methodic testing
4.27Kiểm thử và Đảm bảo Chất lượng Phần mềm
Guidelines data-based black box testing
Inputs dependent on each other?decision table
Suspect computational errors?decision table or equivalence partition (see 1st bullet)
Suspect boundary errors?boundary value
Single-fault assumption?strong/worst-case variant
Suspect errors outside the domain?robust variant
Want to know what a non-average user does?error guessing, random testing
Combinations can and should be constructed
sensibly!
4.28Kiểm thử và Đảm bảo Chất lượng Phần mềm
Outline
Data-based black box testingStructure-based white box testing
classificationexampleadequacy criteriagenerate tests?guidelines
Data/structure-based white box testingStructure-based black box testing
4.29Kiểm thử và Đảm bảo Chất lượng Phần mềm
Dimensions + concept map
1a
1b
1c2a2b
2c
2d
3
4.30Kiểm thử và Đảm bảo Chất lượng Phần mềm
Test purposeSoftware development phase:
implementationcode
detaileddesign
specification
requirements acceptancetest
systemtest
integrationtest
unittest
4.31Kiểm thử và Đảm bảo Chất lượng Phần mềm
Test method dimensions
data-based
structure-based
black box white boxerror seedingtypical errors
efficiencycoverage
4.32Kiểm thử và Đảm bảo Chất lượng Phần mềm
Classifying structure-based white box testing
all criteriarequirements: some logical/mathematical
input/output relationsoftware characteristics: available, imperative
programming language purpose: unit testing, functional mistakesmethod: white box, structure-based, coverage,
efficiencyassumption: both single and multiple-fault
assumption
4.33Kiểm thử và Đảm bảo Chất lượng Phần mềm
Running example: triangle
input: a,b,c: [1-..]output: equilateral/isosceles/
scalene/not a trianglerequirements: mathematically
correcttesting: input combination,
expected outputstructure: each test is a pathcoverage: when ...
paths/branches/... executed
1. Triangle (a, b, c: Integer): String2. IsATriangle: Boolean3. begin4. if (a<=b+c or b<=a+c or c<=b+c)5. then IsATriangle := true6. else IsATriangle := false fi;7. if (IsATriangle)8. then if (a=b and b=c)9. then return “Equilateral”10. else if (a≠b and a≠c and b≠c)11. then return “Scalene”12. else return “Equilateral” fi; fi;13. else return “Not a triangle” fi;14. end
5 (6?) errors!
isosceles equilateral scalene
4.34Kiểm thử và Đảm bảo Chất lượng Phần mềm
Testing the triangle program
5. 6.
7.
8.
9.
13.
10.
11. 12.
14.
4. 1. Triangle (a, b, c: Integer): String2. IsATriangle: Boolean3. begin4. if (a<=b+c or b<=a+c or c<=b+c)5. then IsATriangle := true6. else IsATriangle := false fi;7. if (IsATriangle)8. then if (a=b and b=c)9. then return “Equilateral”10. else if (a≠b and a≠c and b≠c)11. then return “Scalene”12. else return “Equilateral” fi; fi;13. else return “Not a triangle” fi;14. end
programgraph
4.35Kiểm thử và Đảm bảo Chất lượng Phần mềm
Testing the triangle program
each test represents a path: a=b=c=1
5. IsATriangle := true 6. IsATriangle := false
7. IsATriangle?
8. (a=b and b=c)?
9. “Equilateral”
13. “Not a triangle”
10. (a≠b and a≠c and b≠c)?
11. “Scalene” 12. “Equilateral”
14. end
4. (a<=b+c or b<=a+c or c<=b+c)?yes
no
yes
yes
yes
no
no
no
4.36Kiểm thử và Đảm bảo Chất lượng Phần mềm
Testing the triangle program
each test represents a path: a=b=c=1: “Equilateral”coverage:
nodesedges...
all paths feasible? no! (this is in general
undecidable)
5. IsATriangle := true 6. IsATriangle := false
7. IsATriangle?
8. (a=b and b=c)?
9. “Equilateral”
13. “Not a triangle”
10. (a≠b and a≠c and b≠c)?
11. “Scalene” 12. “Equilateral”
14. end
4. (a<=b+c or b<=a+c or c<=b+c)?yes
no
yes
yes
yes
no
no
no
4.37Kiểm thử và Đảm bảo Chất lượng Phần mềm
Program graphs & testing
infeasible paths: combinations that cannot occur (dependent nodes)dead code
# paths: possibly infinite (loops!)
coverage criteria:...
4.38Kiểm thử và Đảm bảo Chất lượng Phần mềm
Program graphs & testing
coverage criteria:statement: each node is executedbranch/decision: each edge is executed(multiple) condition:
let p_1, p_2, ... be the smallest parts in condition c (atomic predicates)condition coverage: each condition is executed so often that each atomic predicate p_i has evaluated to both truth valuesmultiple condition coverage: each condition is executed so often that the atomic predicates have evaluated to all possible truth value combinations
all paths
single fault
multiple fault
4.39Kiểm thử và Đảm bảo Chất lượng Phần mềm
Subsume relations
A B: Criterion A subsumes criterion B if A tests at least what B tests
branch coverage
multiple condition coverageall paths
statement coverage
branch+condition coverage
condition coverage
Note: multiple condition coverage tests more, but may show fewer failures than branch+condition coverage! (Frankl & Weyuker, Provable Improvements on Branch Testing, IEEE Trans. Softw. Eng.:19(10), 1993)
4.40Kiểm thử và Đảm bảo Chất lượng Phần mềm
Testing the triangle program
each test represents a path: a=b=c=1: “Equilateral”coverage:
edges
5. IsATriangle := true 6. IsATriangle := false
7. IsATriangle?
8. (a=b and b=c)?
9. “Equilateral”
13. “Not a triangle”
10. (a≠b and a≠c and b≠c)?
11. “Scalene” 12. “Equilateral”
14. end
4. (a<=b+c or b<=a+c or c<=b+c)?yes
no
yes
yes
yes
no
no
no
Exercise (5 min): Construct input combinations,
such that each edge is executed.Compare program output values
to required output values.Which path is not feasible?
4.41Kiểm thử và Đảm bảo Chất lượng Phần mềm
Observations
How to construct tests & get everything covered?Guidelines:
Start with test generation based on black box methodMeasure white box coverage for black box test setUse tools!Which coverage criterion? How many errors may remain?A bit stronger than branch coverage is very effective, e.g. branch + condition coverageIndustrial practice:
varies from statement coverage (100%?) to multiple condition coverage (?%)
4.42Kiểm thử và Đảm bảo Chất lượng Phần mềm
What we haven’t discussed
Loops:once and zero timesmaximal + minimal number of times...
Alternative program graphs:LCSAJ blocks per node
Stepwise testing: graph condensationDiscrepancies between graph and requirements:
feasible paths not requiredrequired paths not present/feasible
4.43Kiểm thử và Đảm bảo Chất lượng Phần mềm
Outline (Complementary)
Data-based black box testingStructure-based white box testingData/structure-based white box testingStructure-based black box testing
4.44Kiểm thử và Đảm bảo Chất lượng Phần mềm
Data/structure-based criteria
variable x of the program:define occurrence: x gets a value (assignment)use occurrence: the value of x is used
computational (in the value for assignment)predicate (in a condition)
criteria:for each definition occurrence of x at node n, for
each/some use occurrence that can be reached from n, each/some path is executed
all definitions criterion
4.45Kiểm thử và Đảm bảo Chất lượng Phần mềm
Data/structure-based criteria
variable x of the program:define occurrence: x gets a value (assignment)use occurrence: the value of x is used
computational (in the value for assignment)predicate (in a condition)
criteria:for each definition occurrence of x at node n, for
each/some use occurrence that can be reached from n, each/some path is executed
all uses criterion
4.46Kiểm thử và Đảm bảo Chất lượng Phần mềm
Data/structure-based criteria
variable x of the program:define occurrence: x gets a value (assignment)use occurrence: the value of x is used
computational (in the value for assignment)predicate (in a condition)
criteria:for each definition occurrence of x at node n, for
each/some use occurrence that can be reached from n, each/some path is executed
all definition-use paths criterion
4.47Kiểm thử và Đảm bảo Chất lượng Phần mềm
Structure-based white box testing summary
Coverage: poor to very goodbut even for weak criteria 100% not always possible!
#tests: few to very many
Usage: hard to generate tests, easier to measure coverage
When to use:in combination with black box generation methodselaborate methods only for vital parts of softwareif tools are available
4.48Kiểm thử và Đảm bảo Chất lượng Phần mềm
Outline
Data-based black box testingStructure-based white box testingData/structure-based white box testingStructure-based black box testing
4.49Kiểm thử và Đảm bảo Chất lượng Phần mềm
Dimensions of software testing
1. What is tested?a. Software characteristics (design/embedded?/language/...)
b. Requirements (what property/quality do we want from the software)
c. Purpose (what development phase, what kind of errors, what next step)
2. How to test?a. Method (adequacy criterion: how to generate how many tests)
b. Assumptions (limitations, simplifications, heuristics)
c. Organization (documentation, implementation, execution:platform, interactive, batch)
d. Who tests? (programmer, user, third party)
3. How are the results evaluated?
4.50Kiểm thử và Đảm bảo Chất lượng Phần mềm
Dimensions + concept map
1a
1b
1c2a2b
2c
2d
3
4.51Kiểm thử và Đảm bảo Chất lượng Phần mềm
Test purpose
Software development phase:
implementationcode
detaileddesign
specification
requirements acceptancetest
systemtest
integrationtest
unittest
4.52Kiểm thử và Đảm bảo Chất lượng Phần mềm
Test method dimensions
data-based
structure-based
black box white boxerror seedingtypical errors
efficiencycoverage
4.53Kiểm thử và Đảm bảo Chất lượng Phần mềm
Classifying structure-based black box testing
all criteriarequirements: behaviour, defined in FSMsoftware characteristics: can be modelled as
an FSM purpose: unit testing, functional mistakesmethod: black box, structure-based,
coverage, efficiencyassumption: ...
4.54Kiểm thử và Đảm bảo Chất lượng Phần mềm
Running example: musical toy
input:event: a button is
pushed
output: event: sound changes
requirements:FSM (behaviour)
testing: ...
Rhythm
Quit
Song
1. PlayMusic2. inputs pushRhythm, pushSong,
pushQuit;3. outputs startRhythm, startSong, Quit;4. state Rhythm, Song: Boolean;5. begin6. while (true) do7. if (pushQuit) then8. Rhythm, Song := false, false;9. if (Rhythm and Song) then output
Quit fi10. elsif (pushRhythm) then11. if (Rhythm) then output Rhythm fi;12. Rhythm := true13. elsif (pushSong) then14. if (Song) then output Song fi; 15. Song := true fi16. od;17. end
4.55Kiểm thử và Đảm bảo Chất lượng Phần mềm
FSM of musical toy requirements
1. PlayMusic2. inputs pushRhythm, pushSong,
pushQuit;3. outputs startRhythm, startSong, Quit;4. state Rhythm, Song: Boolean;5. begin6. while (true) do7. if (pushQuit) then8. Rhythm, Song := false, false;9. if (Rhythm and Song) then output
Quit fi10. elsif (pushRhythm) then11. if (Rhythm) then output Rhythm fi;12. Rhythm := true13. elsif (pushSong) then14. if (Song) then output Song fi; 15. Song := true fi16. od;17. end
Rhythm Song
Both
Off
PushRhythm?/StartRhythm!
PushRhythm?/StartRhythm!
PushRhythm?/-
PushRhythm?/-
PushSong?/StartSong!
PushSong?/StartSong!
PushSong?/-
PushQuit?/Quit!
PushQuit?/-
PushSong?/-
4.56Kiểm thử và Đảm bảo Chất lượng Phần mềm
What is an FSM?
Rhythm Song
Both
Off
PushRhythm?/StartRhythm!
PushRhythm?/StartRhythm!
PushRhythm?/-
PushRhythm?/-
PushSong?/StartSong!
PushSong?/StartSong!
PushSong?/-
PushQuit?/Quit!
PushQuit?/-
PushSong?/-
FSM (Finite State Machine), or Mealy machine: <S,I,O,δ,λ>
states S (usually has a start state)alphabets: inputs I, outputs Otransfer function: δ:S×I→Soutput function λ:S×I→Ospecial no-output label: ‘-’assumptions!
suitable for: precise, abstract modelsinterfaces, protocols, embedded software, ...formal test generation
4.57Kiểm thử và Đảm bảo Chất lượng Phần mềm
Assumptions for FSM modelling
Rhythm Song
Both
Off
PushRhythm?/StartRhythm!
PushRhythm?/StartRhythm!
PushRhythm?/-
PushRhythm?/-
PushSong?/StartSong!
PushSong?/StartSong!
PushSong?/-
PushQuit?/Quit!
PushQuit?/-
PushSong?/-
FSM must bedeterministic + input-enabled:
δ and λ are proper functionsfinite:
I,O,S all finitestrongly connected:
each state can be reached from any state
reduced:non-equal states have different observable behaviour
4.58Kiểm thử và Đảm bảo Chất lượng Phần mềm
Two FSM models:
SpecificationFSM model of requirementssatisfies restrictions previous slideis used for test generation
ImplementationFSM model of software to be testedsatisfies restrictions except ‘reduced’not really needed, but:
justification for FSM-based testing#states must be estimated
implementation?
4.59Kiểm thử và Đảm bảo Chất lượng Phần mềm
Testing Impl versus Spec:
test = sequence of inputs: (1) a? b? a? (2) b? c? a?expected outcome = sequence of outputs
expected: (1) e! e! e! (2) e! - -actually: (1) e! e! e! (2) e! e! e!
Impl implements Spec if paths(Impl) = paths(Spec)
?
s0
s1
t0
t1t2a?/-
a?/e!b?/e!
b?/e!
c?/-
c?/- c?/-
a?/e!
a?/-
b?/e!
b?/e!c?/e!
c?/-
a?/-c?/-
4.60Kiểm thử và Đảm bảo Chất lượng Phần mềm
Testing Impl versus Spec:
Impl implements Spec if paths(Impl) = paths(Spec)possible errors:1. wrong output (computational mistake)2. extra/missing states (extra/missing features)3. transition to wrong state (computational mistake)which paths to test??
?
s0
s1
t0
t1t2a?/-
a?/e!b?/e!
b?/e!
c?/-
c?/- c?/-
a?/e!
a?/-
b?/e!
b?/e!c?/e!
c?/-
a?/-c?/-
4.61Kiểm thử và Đảm bảo Chất lượng Phần mềm
Test generation from Spec
test each transition: <s,i?>1. go to state s from arbitrary state
(synchronizing sequence + transferring sequence)2. apply input i?
(test transition)3. check output λ(s,i)
(test transition)4. check if in state δ(s,i)
(UIO-sequence/distinguishing sequence/W-set/...)
test fails iff something wrong in step 3. or 4.
s0
s1a?/-
a?/e!b?/e!
b?/e!
c?/-
c?/-
4.62Kiểm thử và Đảm bảo Chất lượng Phần mềm
Test generation from Spec
test each transition: <s,i?>1. go to state s from arbitrary state
(synchronizing sequence + transferring sequence)2. apply input i?
(test transition)3. check output λ(s,i)
(test transition)4. check if in state δ(s,i)
(UIO-sequence/distinguishing sequence/W-set/...)
test fails iff something wrong in step 3. or 4.
s0
s1a?/-
a?/e!b?/e!
b?/e!
c?/-
c?/-
a? b?
4.63Kiểm thử và Đảm bảo Chất lượng Phần mềm
Test generation from Spec
test each transition: <s,i?>1. go to state s from arbitrary state
(synchronizing sequence + transferring sequence)2. apply input i?
(test transition)3. check output λ(s,i)
(test transition)4. check if in state δ(s,i)
(UIO-sequence/distinguishing sequence/W-set/...)
test fails iff something wrong in step 3. or 4.
s0
s1a?/-
a?/e!b?/e!
b?/e!
c?/-
c?/-
for s0: (empty)for s1: a?
4.64Kiểm thử và Đảm bảo Chất lượng Phần mềm
Test generation from Spec
test each transition: <s,i?>1. go to state s from arbitrary state
(synchronizing sequence + transferring sequence)2. apply input i?
(test transition)3. check output λ(s,i)
(test transition)4. check if in state δ(s,i)
(UIO-sequence/distinguishing sequence/W-set/...)
test fails iff something wrong in step 3. or 4.
s0
s1a?/-
a?/e!b?/e!
b?/e!
c?/-
c?/-
e.g. for <s1,b?>: b?
4.65Kiểm thử và Đảm bảo Chất lượng Phần mềm
Test generation from Spec
test each transition: <s,i?>1. go to state s from arbitrary state
(synchronizing sequence + transferring sequence)2. apply input i?
(test transition)3. check output λ(s,i)
(test transition)4. check if in state δ(s,i)
(UIO-sequence/distinguishing sequence/W-set/...)
test fails iff something wrong in step 3. or 4.
s0
s1a?/-
a?/e!b?/e!
b?/e!
c?/-
c?/-
e.g. for <s1,b?>: e!
4.66Kiểm thử và Đảm bảo Chất lượng Phần mềm
Test generation from Spec
test each transition: <s,i?>1. go to state s from arbitrary state
(synchronizing sequence + transferring sequence)2. apply input i?
(test transition)3. check output λ(s,i)
(test transition)4. check if in state δ(s,i)
(UIO-sequence/distinguishing sequence/W-set/...)
test fails iff something wrong in step 3. or 4.
s0
s1a?/-
a?/e!b?/e!
b?/e!
c?/-
c?/-
a?output for state s1: -
4.67Kiểm thử và Đảm bảo Chất lượng Phần mềm
Test generation from Spec
The total set of tests:
s0
s1a?/-
a?/e!b?/e!
b?/e!
c?/-
c?/-
<s0,a>
...<s0,b>
state verification
expected output
transferring sequence
synchronisingsequence
transition
Homework: to be completed!
4.68Kiểm thử và Đảm bảo Chất lượng Phần mềm
What we haven’t (yet) discussed
Why is it correct?Non-deterministic systemsData parametersLiterature: Lee and Yannakakis, “Principles and Methods of Testing Finite State machines—A survey”, Proc. of the IEEE:84(8), 1996.Separate input or output transitionsOther implementation criteria (subsume rel)Multi-process/distributed testingSummary, guidelines
4.69Kiểm thử và Đảm bảo Chất lượng Phần mềm
Test generation from Spec
test each transition: <s,i?>1. go to state s from arbitrary state
(synchronizing sequence + transferring sequence)2. apply input i?
(test transition)3. check output λ(s,i)
(test transition)4. check if in state δ(s,i)
(UIO-sequence/distinguishing sequence/W-set/...)
test fails iff something wrong in step 3. or 4.
s0
s1a?/-
a?/e!b?/e!
b?/e!
c?/-
c?/-
4.70Kiểm thử và Đảm bảo Chất lượng Phần mềm
Test generation assumptions
FSM = <S,I,O,δ,λ>Spec FSM:
is deterministic, complete, reducedhas a synchronizing sequencehas an UIO sequence for each state
Impl FSM :is deterministic, completehas no more states than Spec
Generalised δ,λ functions:δ(s, i_1 i_2 ... i_n) = t (target state of a sequence of inputs)λ(s, i_1 i_2 ... i_n) = o_1 o_2 ... o_n (sequence of outputs)
4.71Kiểm thử và Đảm bảo Chất lượng Phần mềm
Synchronising sequence
sequence of inputsbrings FSM to one designated statedetermine through successor graph:
node: set of states S’={s,t,...} (initially: S)edge: from S’ to S’’ with i/o where
S’’ = { δ(s,i) | s∈S’ and λ(s,i)=o }synchronising sequence σ
all paths for σ from S end in same state {s}σ does not always exist!!often there is a reliable resetif exists, easy to determine in successor graph
s0
s1a?/-
a?/e!b?/e!
b?/e!
c?/-
c?/-
{s0,s1}
a?/e!
{s1}
b?/e! c?/-
a?/-
4.72Kiểm thử và Đảm bảo Chất lượng Phần mềm
Successor graph for musical toy
synchronising sequence: PushQuit?
PR?/R!
{Rhythm,Both}PS?/S!
{Song,Both}
PS?/-PR?/-
{Both}
{Off,Rhythm,Song,Both}
PQ?/Q!{Off}
PQ?/-
PR?/R!
PR?/-PR?/-
PQ?/Q!PQ?/Q!
PS?/S!PS?/-
PS?/-
{Off,Rhythm,Song,Both}
PQ?/Q!{Off}
PQ?/-
4.73Kiểm thử và Đảm bảo Chất lượng Phần mềm
Another FSM + successor graph
synchronising sequence? no!(see Lee & Yannakakis for proof construction)
s1
s2 s3
a?/0!
b?/1!
b?/0!
a?/1! a?/0!
b?/1!
FSM:
a?/0!
{s1,s3}
b?/1!
{s2,s3}
b?/0!a?/1!
{s3}
{s1,s2,s3}
b?/0!
{s1}
a?/1!
b?/1!
a?/0!
a?/0!
b?/1!{s2}
b?/0!
successor graph:
4.74Kiểm thử và Đảm bảo Chất lượng Phần mềm
Relax: homing vs. synchronising
synchronising sequence:ignore outputsalways brings FSM to same end statedoes not always exist
homing sequence:check outputsoutputs determine which end statealways existsextra test effort:
check outputs, determine end statetransfer to initial state (or directly to state-under-test)
s0
s1a?/-
a?/e!b?/e!
b?/e!
c?/-
c?/-
4.75Kiểm thử và Đảm bảo Chất lượng Phần mềm
UIO sequence
Unique Input/Output sequence determines which first stateinputs (+ expected outputs per state)determine through successor graph with initial states:
node: set of states S’={s,t,...} (initial: S)path from S’ to S’’ with i1/o1 i2/02 ...
S’’={ s∈S’ | λ(s,i1 i2...) = o1 o2... }UIO sequence for s: path to {s}does not always exist!!
s0
s1a?/-
a?/e!b?/e!
b?/e!
c?/-
c?/-
{s0,s1}
{s0}
a?/e!
{s1}
b?/e! c?/-
a?/-
4.76Kiểm thử và Đảm bảo Chất lượng Phần mềm
For musical toy
UIO sequence for any state: PushRhythm? PushSong?
PS?/S!
{S,B}{S,B}
PS?/-PQ?/Q!
{O}{R,S,B}
PQ?/-
{O}{O}
{current states}{possible initial states}
{S,B}{O,R}
PR?/R!
{R,B}{O,S}
PR?/-
{B}{O}
{O,R,S,B}{O,R,S,B}
PS?/S! PS?/-PS?/-
{R,B}{R,B}
{B}{S}
{B}{R}
{B}{B}
PS?/S!
PR?/R!
{R,B}{O,S}
PR?/-
{B}{O}
{O,R,S,B}{O,R,S,B}
PS?/S! PS?/-PS?/-
{R,B}{R,B}
{B}{S}
{B}{R}
{B}{B}
PS?/S!
4.77Kiểm thử và Đảm bảo Chất lượng Phần mềm
Yet another FSM
UIO sequences?s1: b?/0! s3: a?/1! s2: no UIO sequence!
s1
s2 s3
a?/0!
b?/1!
b?/0!
b?/1! a?/1!
a?/0!
FSM:
a?/0!
{s1}{s1,s2}
b?/1!
{s2}{s2,s3}
{s1}{s2,s3}
b?/1!a?/0!a?/0!
b?/0!a?/1!
{s1,s2,s3}{s1,s2,s3}
{s3}{s1}
{s3}{s3}
b?/0!
successor graphwith initial states:
{s3}{s1,s2}
b?/0!a?/1!
{s1,s2,s3}{s1,s2,s3}
{s3}{s1}
{s3}{s3}
4.78Kiểm thử và Đảm bảo Chất lượng Phần mềm
Relax: UIO vs. W-set
UIO:1 sequence of inputs/outputs per statedoes not always exist
W-set:set W(s) of input (+ output) sequences for state sfor each distinct pair s,s’: W(s) contains a sequence that distinguishes s from s’always existsextra test effort:
large number of sequencesto establish state verification for s, repeatedly go back to s to test all sequences in W(s)
s0
s1a?/-
a?/e!b?/e!
b?/e!
c?/-
c?/-
4.79Kiểm thử và Đảm bảo Chất lượng Phần mềm
Other relaxations (1)
Spec/Impl FSM:has data parameters (EFSM):
only test control structure (mediocre coverage)compute complete FSM (huge effort, very many tests)combine with data-based black box methods
is non-deterministic: randomize testing algorithm
is incomplete: test only specified parts
sta?/e!
ua?/e!
s in(y)? [NOT a]/e! [x:=y] t
s
u
a?/0! t
b?/1!
4.80Kiểm thử và Đảm bảo Chất lượng Phần mềm
Other relaxations (2)
Impl FSM :has n more states than Spec:
for complete coverage: extra test effort factor |I|n
No FSM model possible:Labelled Transition System (LTS) modelinput/output conformance (ioco) methodsee KUN/UTwente courses on testing:
http://www.cs.kun.nl/~tretmans/testtechnieken/
sa? t
b?f!
ub?
4.81Kiểm thử và Đảm bảo Chất lượng Phần mềm
Structure-based black box testing summary
Coverage: complete (formal) to rather good (relaxations)
#tests: moderate to very manyUsage: hard to model FSM and implement testsWhen to use:
when requirements = (non-trivial) behaviourwhen complete coverage is essentialunit/system/acceptance testingtools for test generation (commercially) available
1Kiểm thử và Đảm bảo Chất lượng Phần mềm
Software Testing and SQA
Chapter 5
Kiểm thử tích hơp
5.2Kiểm thử và Đảm bảo Chất lượng Phần mềm
Nội dung chính
Kiểm thử tích hơpPhân loại kiểm thửCách tiếp cận tích hợpNgữ nghĩa của hệ sinh kiểm thử và tínhphủ (coverage)Thí dụ
5.3Kiểm thử và Đảm bảo Chất lượng Phần mềm
Mục đích
Giai đoạn phát triển phần mềm:
Lập trình
Thiết kế chi tiết
Đặc tả
Các yêu cầu Kiểm thử chấp nhận
Kiểm thử hệ thống
Kiểm thử tích hợp
Kiểm thử đơn vị
5.4Kiểm thử và Đảm bảo Chất lượng Phần mềm
Phân loại kiểm thử tích hợp
Các yêu cầu: “Giao diện giữa các đơn vị CT đươc thực hiện đúng đắn “
Đặc trưng phần mềm : Truy cập phần mềm tạimức mô đun, không phải tới chính các đơn vịchương trình
Mục đích: Kiểm thử tích hơp, giao diện/các lỗiphối hợp
Giả định: Các đơn vị đã được kiểm thử trước đó
5.5Kiểm thử và Đảm bảo Chất lượng Phần mềm
Sự tích hơp phần mềm được kiểmthử
Sự hơp thành các đơn vị chương trìnhCác đơn vị chương trình đã được kiểmthử trướcGiao diện phải được kiểm thử
Không có liên quan đến hành vi mức hệthống ! Lời gọi thủ tục : Cấu tạo đồ thị gọi
Để đảm bảo kiểm thử thực tế, mọi đơnvị CT cần bao gồm biểu diễn về
Các con của nó có mặt trong đồ thị lời gọiÍt ra có một cha trong đồ thị gọi
Main
GF
A C
E
D
B
5.6Kiểm thử và Đảm bảo Chất lượng Phần mềm
Tiếp cận tích hợp: big-bang
Đặt các đơn vị cùng với nhau khichúng cùng một phần (1 sessionDựa vào các kết quả kiểm thử củađơn vị CT (relatively) Có ít các kiểm thửRất khó cô lập các lỗi
Main
GF
A C
E
D
B
5.7Kiểm thử và Đảm bảo Chất lượng Phần mềm
Tiếp cận tích hợp: top-down
start with top unit: Mainreplace each unit called by Main with a stub:
dummy softwarereacts as original unit wouldeasy to program
1st session: test interfaces for Main
Main
GFE
s s
s
s
5.8Kiểm thử và Đảm bảo Chất lượng Phần mềm
Tiếp cận tích hợp: top-down
second session: unit A...many stubs requiredmany test sessions required
1 session per unit
easy to isolate errors
Main
GFs
A s
s
s
5.9Kiểm thử và Đảm bảo Chất lượng Phần mềm
Tiếp cận tích hợp: bottom-up
start with lowest-level unit: Ereplace unit calling E with a driver:
dummy softwaremakes calls as original unit wouldharder to program
many drivers requiredusually fewer than #stubs for top-down
many test sessions required1 session per unit
easy to isolate errors
Main
GF
d C
E
D
B
5.10Kiểm thử và Đảm bảo Chất lượng Phần mềm
Tiếp cận tích hợp: sandwich
mixture of top-down and bottom-upsome stubs, some drivers requiredtest interfaces between portions already testedmoderate #test sessions requirederrors harder to isolate
Main
GF
A C
E
D
B
top-down bottom-up
5.11Kiểm thử và Đảm bảo Chất lượng Phần mềm
Tiếp cận tích hợp: pair-wise
per session, test a pair of units: C/Dsome drivers, some stubs required
fewer than bottom-up or top-down
many test sessions required1 session per pair
easy to isolate errors
d
ss
A C
E
D
B
5.12Kiểm thử và Đảm bảo Chất lượng Phần mềm
What is an integration test?
input values + driver/stub/unit set-upin the composed program graph: unit-paths with call/return markersm1 <Main calls C> c1 <C calls F> f1
<F returns C> c2 ...
Main
GF
A C
E
D
B
Main1
5
2
3
46
F
5
2
3
4
6
C1
2
3
4
m1
m2
c1
c2
f1
5.13Kiểm thử và Đảm bảo Chất lượng Phần mềm
Sinh và phủ các kiểm thử tích hợp
Coverage in the call-graph:minimally/usually: all edges (each call-return interface is tested at least once)maximally: all feasible paths
Test generation requires(white box) the composed program graph(grey box) composition of units’ call/return models, e.g. FSMs
Main
GF
A C
E
DB
Main 1
5
23
46
F
52
3
4
6
C 1
234
5.14Kiểm thử và Đảm bảo Chất lượng Phần mềm
Thực hiện ví dụ : bank machine
1
Bank Balance Machine
WELCOME
insert your card
2 3
4 5 6
7 8 9
0 Cancel
card
5.15Kiểm thử và Đảm bảo Chất lượng Phần mềm
Đặc tả mức hệ thống FSM
Savings?/Screen6,CardOut!
ValidCardIn?/Screen2!
InvalidPIN?/Screen4!
Cancel?/Screen7,CardOut!
CardRemoved?/Screen1!
ValidPIN?/Screen5!
InvalidCardIn?/CardOut! idle
process PIN
process balance
end session
Checking?/Screen6,CardOut!
5.16Kiểm thử và Đảm bảo Chất lượng Phần mềm
Thông báo màn hình của máyNgân hàng
Bank Balance Machine
WELCOME
Insert your card.
Enter your PIN:(Personal
IndentificationNumber)
____
(press Cancel to clear)
Invalid PIN!Please enter your
PIN again:
____
(press Cancel to clear)
Identification failed!
Your card is retained. Contact your bank.
Operation cancelled!
Please take your card.Thank you.
Your balance isbeing printed on
a receipt.
Please take yourreceipt and card.
Thank you.
Select your account:
checkingssavings
(press Cancel to abort)
Screen 1 Screen 4Screen 3Screen 2
Screen 5 Screen 7Screen 6
5.17Kiểm thử và Đảm bảo Chất lượng Phần mềm
Phân rã chức năng ...
... Không giúp đỡ gì vì đồ thị-gọi nhìn kháchoàn toàn !
Bank Machine
PIN ValidationCard TypeValidation
Central BankCommunication
ScreenManagement
CardInput/Output
ReceiptsCardManagement
BalanceManagement
5.18Kiểm thử và Đảm bảo Chất lượng Phần mềm
Đồ thị - gọi máy ngân hàng
Main
PrintScreen
CardIn
ValidateCard
CardOut
ValidatePIN
PrintReceipt
GetPIN
ContactBank
ManageBalance
1Kiểm thử và Đảm bảo Chất lượng Phần mềm
Software Testing and SQA
Chapter 6Phần mềm khả hiện
(More Reliable Software)Một cách nhìn tổng quan về Nhanh
hơn và Rẻ hơn
6.2Kiểm thử và Đảm bảo Chất lượng Phần mềm
Most Important Software Problem
1. Customers demand (in order):A. More reliable softwareB. Faster
C. Cheaper2. Your success in meeting demands affects
market share, profitability, your career3. Demands conflict, causing risk and
overwhelming pressure
Introduction
6.3Kiểm thử và Đảm bảo Chất lượng Phần mềm
Software Reliability Engineering –Developed to Address the Problem
1. It differs from other approaches by being primarily quantitative.
2. You add and integrate software reliability engineering (SRE) with other good processes and practices; you do not replace them.
3. With SRE you control the development process, it doesn’t control you:A. Development process is not externally imposed.B. You use quantitative information to choose the most
cost-effective software reliability strategies for your situation.
Introduction
6.4Kiểm thử và Đảm bảo Chất lượng Phần mềm
Outline
1. Nature of software reliability engineering (SRE)
2. SRE process (with illustration)
Introduction
6.5Kiểm thử và Đảm bảo Chất lượng Phần mềm
Reliability and Availability Definitions1. Reliability: the probability that a system or a
capability of a system will continue to function without failure for a specified period in a specified environment. The period may be specified in natural or time units.A. Natural unit: unit other than time related to amount of
processing performed by software-based product, such as runs, pages of output, transactions, telephone calls, jobs, semiconductor wafers, queries, or API calls
B. Failure intensity (FI): failures per natural or time unit, an alternative way of expressing reliability
2. Availability: the average (over time) probability that a system or a capability of a system is currently functional in a specified environment
Nature of SRE
6.6Kiểm thử và Đảm bảo Chất lượng Phần mềm
How Does SRE Work?
Nature of SRE – How Does SRE Work?
1. Increase effective resourcesA. Quantitatively characterize
expected useB. Focus resources on most
used and most critical functions
C. Maximize test effectiveness by making test highly representative of field
Increase in Effective
Resources
OriginalResources
6.7Kiểm thử và Đảm bảo Chất lượng Phần mềm
How Does SRE Work?
Nature of SRE – How Does SRE Work?
2. Apply resources to maximize customer valueA. Set quantitative
objectives for major quality characteristics (reliability and/or availability, delivery time, price)
Rel / Avail
Time Price
6.8Kiểm thử và Đảm bảo Chất lượng Phần mềm
How Does SRE Work?
Nature of SRE – How Does SRE Work?
B. Choose software reliability strategies to meet objectives
C. Track reliability in system test against objective as one of the release criteria
Added CustomerValue - Focus
Added CustomerValue -
Matching Needs
Original CustomerValue
OriginalCustomer Value
6.9Kiểm thử và Đảm bảo Chất lượng Phần mềm
SRE - A Proven, Standard, Widespread Best Practice
Nature of SRE - Proven, Standard, Widespread Best Practice
1. Proven practiceA. Example: AT&T International Definity PBX
[6, pp 167-8]a. Reduced customer-reported problems by factor of
10b. Reduced system test interval by factor of 2c. Reduced total development time by 30%d. No serious service outages in 2 years of
deployment
6.10Kiểm thử và Đảm bảo Chất lượng Phần mềm
SRE - A Proven, Standard, Widespread Best Practice
Nature of SRE - Proven, Standard, Widespread Best Practice
B. AT&T Best Current Practice since 5/91 (based on widespread practice, documented strong benefit/cost ratio, probing review) [6, pp 219-254]
2. Standard practiceA. McGraw-Hill handbook [6]B. AIAA standard since 1993C. IEEE standards under development
6.11Kiểm thử và Đảm bảo Chất lượng Phần mềm
SRE - A Proven, Standard, Widespread Best Practice
Nature of SRE - Proven, Standard, Widespread Best Practice
3. Widespread practiceA. Users include Alcatel, AT&T, Bellcore, CNES
(France), ENEA (Italy), Ericsson Telecom, Hewlett Packard, Hitachi, IBM, NASA’s Jet Propulsion Laboratory, Lockheed-Martin, Lucent Technologies, Microsoft, Mitre, Nortel, Saab Military Aircraft, Tandem Computers, the US Air Force, and the US Marine Corps.
B. Over 50 published articles by users of software reliability engineering, continuing to grow ([1]; [3], pp. 371 - 374)
6.12Kiểm thử và Đảm bảo Chất lượng Phần mềm
SRE Is Widely Applicable
Nature of SRE - Widely Applicable
1. Technically speaking, you can apply SRE to any software-based product, beginning at start of any release cycle.
2. Economically speaking, the complete SRE process may be impractical for small components (involving perhaps less than 2 staff months of effort), unless used in a large number of products. It may still be worthwhile to implement it in abbreviated form.
3. Independent of development technology and platform.
4. SRE requires no changes in architecture, design, or code - but it may suggest changes that would be beneficial.
6.13Kiểm thử và Đảm bảo Chất lượng Phần mềm
Activities of SRE Process and Relationto Software Development Process
SRE Process
Design andImplementation
Requirementsand Architecture Test
5. Execute Test4. Prepare for Test
6. Guide Test
1. List AssociatedSystems
2. Implement OperationalProfiles
3. Engineer “Just Right”Reliability
6.14Kiểm thử và Đảm bảo Chất lượng Phần mềm
Illustration - FONE FOLLOWER (FF) -Product Description
1. Subscriber calls FF, enters planned phone numbers (forwardees) to which calls are to be forwarded vs time.
2. FF forwards incoming calls (voice or fax) from network to subscriber as per program. Incomplete voice calls go to pager (if subscriber has one) and then voice mail.
SRE Process
6.15Kiểm thử và Đảm bảo Chất lượng Phần mềm
Activities of SRE Process and Relationto Software Development Process
SRE Process - List Associated Systems
Design andImplementation
Requirementsand Architecture Test
5. Execute Test4. Prepare for Test
6. Guide Test
1. List AssociatedSystems
2. Implement OperationalProfiles
3. Engineer “Just Right”Reliability
6.16Kiểm thử và Đảm bảo Chất lượng Phần mềm
List Associated Systems of Product
SRE Process - List Associated Systems
1. Base product2. Major variations of base product (for
substantially different environments, platforms, or configurations)
3. Frequently used supersystems of base product or variations
6.17Kiểm thử và Đảm bảo Chất lượng Phần mềm
Activities of SRE Process and Relationto Software Development Process
SRE Process - Implement OPs
Design andImplementation
Requirementsand Architecture Test
5. Execute Test4. Prepare for Test
6. Guide Test
1. List AssociatedSystems
2. Implement OperationalProfiles
3. Engineer “Just Right”Reliability
6.18Kiểm thử và Đảm bảo Chất lượng Phần mềm
Operation
Operation: major system logical task performed for initiator, which returns control to system when complete.Illustrations - FF:
Process fax call, Phone number entry, Audit section of phone number database
SRE Process - Implement OPs - Concepts
6.19Kiểm thử và Đảm bảo Chất lượng Phần mềm
Operational Profile
Operational profile (OP): complete set of operations with probabilities of occurrenceIllustration - FF:
SRE Process - Implement OPs - Concepts
OperationOccur.Prob.
Process voice call, no pager, ans. 0.21Process voice call, pager, ans. 0.19Process fax call 0.17Process voice call, pager, ans. on page 0.13
•••
1
6.20Kiểm thử và Đảm bảo Chất lượng Phần mềm
Develop Operational Profiles –Step by StepDevelop operational profile for base product and each variation (supersystems have same operational profiles as those of their related base product or variations).
SRE Process - Implement OPs – Develop OPs
6.21Kiểm thử và Đảm bảo Chất lượng Phần mềm
Determine Operational Profile
1. Identify initiators of operations [1]2. Create operations list [1]3. Review operations list [1]4. Determine occurrence rates [1]5. Determine occurrence probabilities [1]Steps 1, 2, 3 are mostly the same across base product and variations. New release often requires only slight change from previous release, all steps.
SRE Process - Implement OPs – Develop OPs
6.22Kiểm thử và Đảm bảo Chất lượng Phần mềm
Apply Operational Profiles
Apply operational profile and criticality information to increase efficiency of:1. Development of all developed software2. Test of all associated systems
SRE Process - Implement OPs - Apply OPs
6.23Kiểm thử và Đảm bảo Chất lượng Phần mềm
Apply Operational Profiles to Increase Development Efficiency
For developed software in base product and each variation:1. Review functionality to be implemented
(Reduced Operation Software or ROS)2. Suggest operations where looking for
opportunities for reuse will be most cost-effective3. Plan a more competitive release strategy
(operational development)
SRE Process - Implement OPs - Apply OPs
6.24Kiểm thử và Đảm bảo Chất lượng Phần mềm
Operational Development -Illustration
Proportion of operations developed
Proportion of use represented
SRE Process - Implement OPs - Apply OPs
Release 1
Release 2
Release 3
6.25Kiểm thử và Đảm bảo Chất lượng Phần mềm
Apply Operational Profiles to Increase Development Efficiency
4. Allocate development resources: A. Among operations - for system engineering,
architectural design, requirements reviews, design reviews
B. Among modules - for code, code reviews, unit test ([3], pp. 118 - 119)
SRE Process - Implement OPs - Apply OPs
6.26Kiểm thử và Đảm bảo Chất lượng Phần mềm
Apply Operational Profiles to Increase Test Efficiency
1. Allocate new test cases of release among new operations of base product and variations (Prepare for Test activity)
2. Allocate test time (Prepare for Test and Execute Test activities)
SRE Process - Implement OPs - Apply OPs
6.27Kiểm thử và Đảm bảo Chất lượng Phần mềm
Activities of SRE Process and Relationto Software Development Process
SRE Process - Engineer “Just Right” Reliability
Design andImplementation
Requirementsand Architecture Test
5. Execute Test4. Prepare for Test
6. Guide Test
1. List AssociatedSystems
2. Implement OperationalProfiles
3. Engineer “Just Right”Reliability
6.28Kiểm thử và Đảm bảo Chất lượng Phần mềm
Failure and Fault
SRE Process - Engineer “Just Right” Reliability
Failure Fault
Departure of systembehavior in executionfrom user needs
Defect in systemimplementation thatcauses the failure whenexecuted
User-oriented Developer-oriented
Failure Fault
Departure of systembehavior in executionfrom user needs
Defect in systemimplementation thatcauses the failure whenexecuted
User-oriented Developer-oriented
6.29Kiểm thử và Đảm bảo Chất lượng Phần mềm
Engineer “Just Right” Reliability -Step by Step
1. Define failure consistently over life of product release and clarify with examples [1]
2. Choose common reference measure for all failure intensities [1]
3. Set system FIO for each associated system [1]4. For any software you develop:
A. Find developed software FIO [1]B. Choose software reliability strategies to meet developed
software FIO and schedule objectives with lowest development cost [1]
SRE Process - Engineer “Just Right” Reliability
6.30Kiểm thử và Đảm bảo Chất lượng Phần mềm
Activities of SRE Process and Relationto Software Development Process
SRE Process - Prepare for Test
Test
2. Implement OperationalProfiles
Design andImplementation
Requirementsand Architecture
5. Execute Test4. Prepare for Test
6. Guide Test
1. List AssociatedSystems
3. Engineer “Just Right”Reliability
6.31Kiểm thử và Đảm bảo Chất lượng Phần mềm
Prepare for Test1.Specify new test cases for new operations
A. Distribute new test cases to new operations based on operational profile [1]Illustration - FF:
Allocate 17% of test cases to Proc. fax call operationB. Detail new test cases for each new operation by
selecting from possible choices of input variable values with equal probability [1]Illustration - FF:
Forwardee = Local calling area
2.Specify test procedure, based on the test operational profile [1]
SRE Process - Prepare for Test
6.32Kiểm thử và Đảm bảo Chất lượng Phần mềm
Activities of SRE Process and Relationto Software Development Process
SRE Process - Execute Test
Design andImplementation
Requirementsand Architecture Test
5. Execute Test4. Prepare for Test
6. Guide Test
1. List AssociatedSystems
2. Implement OperationalProfiles
3. Engineer “Just Right”Reliability
6.33Kiểm thử và Đảm bảo Chất lượng Phần mềm
Execute Test
1.Determine and allocate test time among associated systems and types of test (feature, load, regression) [1]
*2. Invoke test in accordance with operational profile [1]
3. Identify system failures and when they occurred - use data in Guide Test [1]
SRE Process - Execute Test
6.34Kiểm thử và Đảm bảo Chất lượng Phần mềm
Invoke Test
SRE Process - Execute Test
F
Track reliability growthF = Feature testR = Regression test
Certify reliability
Supersystem
Variations
SupersystemLoadTestF
Base ProductLoadTestF
RR RRRRR RRR
RRRR
6.35Kiểm thử và Đảm bảo Chất lượng Phần mềm
Activities of SRE Process and Relationto Software Development Process
SRE Process - Guide Test
Design andImplementation
Requirementsand Architecture Test
5. Execute Test4. Prepare for Test
6. Guide Test
1. List AssociatedSystems
2. Implement OperationalProfiles
3. Engineer “Just Right”Reliability
6.36Kiểm thử và Đảm bảo Chất lượng Phần mềm
Guide Test
Process system failure data gathered in test to:
*1. Track reliability growth of developed software of base product and variations
*2. Certify reliability of:A. Base product and variations that customers
will acceptance testB. Supersystems
3.Guide product release
SRE Process - Guide Test
6.37Kiểm thử và Đảm bảo Chất lượng Phần mềm
Track Reliability Growth
1. Execute CASRE software reliability estimation program to obtain FI / FIO ratio
2. Plot FI / FIO ratio against time*3. Interpret plot
SRE Process - Guide Test - Track Reliability Growth
6.38Kiểm thử và Đảm bảo Chất lượng Phần mềm SRE Process - Guide Test - Track Reliability Growth
Interpret Plot : Illustration - FF
FI/FIO
Mcalls
0
2
4
6
8
10
12
14
16
18
0 0.1 0.2 0.3 0.4 0.5
Conventional test
Operational-profile-driven test
6.39Kiểm thử và Đảm bảo Chất lượng Phần mềm SRE Process - Guide Test - Track Reliability Growth
Interpret Plot : Illustration - FF
0
2
4
6
8
10
12
14
16
18
0 0.1 0.2 0.3 0.4 0.5
FI/FIO
Mcalls
Solutions:1. Defer features2. Rebalance major
quality characteristics3. Increase work hours
Scheduledtest
completion
6.40Kiểm thử và Đảm bảo Chất lượng Phần mềm SRE Process - Guide Test - Track Reliability Growth
Interpret Plot : Illustration - FF
0
2
4
6
8
10
12
14
16
18
0 0.1 0.2 0.3 0.4 0.5
FI/FIO
Mcalls
Investigatesignificant
upwardtrends
Possible causes:1. Poor change control2. Poor control of test
execution, resultingin test selectionprobabilities varyingin time
6.41Kiểm thử và Đảm bảo Chất lượng Phần mềm SRE Process - Guide Test - Track Reliability Growth
Mcalls
Interpret Plot : Illustration - FF
0
2
4
6
8
10
12
14
16
18
0 0.1 0.2 0.3 0.4 0.5
FI/FIO
Terminatetest
6.42Kiểm thử và Đảm bảo Chất lượng Phần mềm
Certify Reliability – Dem. Chart:Illus. - FF – BP Supersystem
SRE Process - Guide Test - Certify Reliability
12
0 10862 4
1614
0246
810
Normalized units
Failurenumber
Continue
Accept
Reject
Fail.No.
Mcalls at
FailureNormalized
Units
123
0.003750.006250.025
0.751.25
5
Failure intensity objective:200 failures / Mcalls
6.43Kiểm thử và Đảm bảo Chất lượng Phần mềm
SRE and You
1. SRE gives you a powerful way to engineer software-based products so you can be confident in the availability and reliability of the software-based product you deliver as you deliver it in minimum time with maximum efficiency.
2. With SRE you control the process; it doesn’t control you.
3. SRE is a vital skill for being competitive.
Conclusion - SRE and You
6.44Kiểm thử và Đảm bảo Chất lượng Phần mềm
To Explore Further1. More Reliable Software Faster and Cheaper, two
day course, described on internet athttp://members.aol.com/JohnDMusa/FLweb.htm
2. Software Reliability Engineering website: http://members.aol.com/JohnDMusa/Overview, briefing for managers, bibliography of articles by software reliability engineering users, course information, useful references, Question of the Month.
3. Musa, J. D., Software Reliability Engineering: More Reliable Software, Faster Development and Testing, ISBN 0-07-913271-5, McGraw-Hill, 1998. Detailed, extensive treatment of practice.
Conclusion - Explore
6.45Kiểm thử và Đảm bảo Chất lượng Phần mềm
To Explore Further4. Musa, Iannino, Okumoto; Software
Reliability: Measurement, Prediction, Application, ISBN 0-07-044093-X, McGraw-Hill, 1987. Practice plus extensive theoretical background.
5. Musa, J.D., More Reliable Software Faster and Cheaper. Overview of software reliability engineering, suitable for managers and anyone wanting a fast and broad but not detailed understanding of the topic. May be downloaded from:http://members.aol.com/JohnDMusa/ARTweb.htm
Conclusion - Explore
6.46Kiểm thử và Đảm bảo Chất lượng Phần mềm
To Explore Further
6. Lyu, M. (Editor), Handbook of Software Reliability Engineering , ISBN 0-07-039400-8, McGraw-Hill, 1996. Particularly useful for CD/ROM of CASRE program.
7. IEEE Computer Society Technical Committee on Software Reliability Engineering. Professional organization of the field: publishes newsletter, sponsors ISSRE annual international conference): membership application at http://www.tcse.org
Conclusion - Explore
1Kiểm thử và Đảm bảo Chất lượng Phần mềm
Software Testing and SQA
Chapter 7Software Testing in Industry
(By Maurice Siteur)
7.2Kiểm thử và Đảm bảo Chất lượng Phần mềm
Dimensions + concept map
1a
1b
1c2a2b
2c
2d
3
7.3Kiểm thử và Đảm bảo Chất lượng Phần mềm
Program
TMapTesting in embedded software
CMM/TMM (process improvement)Test tools
7.4Kiểm thử và Đảm bảo Chất lượng Phần mềm
Testing
Testing is a process of planning, preparation and measuring, with the aim to view the characteristics of a product, that makes it possible to make judgements about the quality of that product.
7.5Kiểm thử và Đảm bảo Chất lượng Phần mềm
Test structure
OrganizationOrganization
Life cycleLife cycle
InfrastructureInfrastructure
TechniquesTechniquesTT
LL
II OO
7.6Kiểm thử và Đảm bảo Chất lượng Phần mềm
TMap
Master test plan
Unit testUnit
integrationtest
System testFunctionalacceptance
test
ProductionAcceptance
test
7.7Kiểm thử và Đảm bảo Chất lượng Phần mềm
What makes testing difficult
Specifications are not always rightThere are bugs in the software (you should test for that)Testing everything is impossible!!Reproducing tests
7.8Kiểm thử và Đảm bảo Chất lượng Phần mềm
What makes embedded systems different
Hardware and software (not available when you would like it to be)Mechanics, electronics, optics, ….
Stubs and drivers needed
Performance, timing, synchronisation, resource usageRegulatory requirements
7.9Kiểm thử và Đảm bảo Chất lượng Phần mềm
Embedded software testing - example
Unit testing programs/componentsUnit integration testingSystem testSoftware acceptance testHardware component testHardware integration testSoftware/Hardware integration testManufacturers acceptance testClient acceptance test
7.10Kiểm thử và Đảm bảo Chất lượng Phần mềm
TMap essentials
Test planning‘Risk based’ testingTest designTest reporting
7.11Kiểm thử và Đảm bảo Chất lượng Phần mềm
Test planning
Looks like normal plan of approach
Approach/strategybased on risk'sUse of quality attributes
Here test coverage is determinedCoverage levelsTheory of finite state machines
7.12Kiểm thử và Đảm bảo Chất lượng Phần mềm
Test design
Logical and physical
Determination of test cases
Test caseDescriptionStarting pointInputExpected result
Regression test / re use of test cases
7.13Kiểm thử và Đảm bảo Chất lượng Phần mềm
Test reporting
Record findings of test executionReport on progressEnd report
ConclusionsFindings
7.14Kiểm thử và Đảm bảo Chất lượng Phần mềm
Guidelines
System development 75%, testing 25%
Preparation 20%Specification 35%Execution 40%Completion 5%
Planning and control 10% on top
Test automation can turn these figures up side down.
7.15Kiểm thử và Đảm bảo Chất lượng Phần mềm
Process improvement
Software process improvementCapability Maturity Model
Test process improvementTest Maturity ModelTPI (Homework)
7.16Kiểm thử và Đảm bảo Chất lượng Phần mềm
CMM
7.17Kiểm thử và Đảm bảo Chất lượng Phần mềm
Level 5: Optimization, Defect Prevention andQuality Control
Level 4: Management and Measurement
Level 3: Integration
Level 2: Phase Definition
Level 1: Initial
- Test proces optimization- Quality control- Application of process data for defect prevention
- Software quality evaluation- Establish a test measurement program- Establish an organization-wide review program
- Control and monitor the test process- Integrate testing into the software lifecycle- Establish a technical training program- Establish a software test organization
- Institutionalize basic testing techniques and methods- Initiate a test planning process- Develop testing and debugging goals
TMM
7.18Kiểm thử và Đảm bảo Chất lượng Phần mềm
Tool Support of testing activitiesSystem development
RequirementsAcceptance criteria
Specifications
Coding
Test execution• Capture and playback• Test drivers• Load en performance testing• Monitoring
Test planning
Generating testsetsfor system- and acc.testing
Generating testsetson the basis of design
Test management & control• Test planning• Configuration management• Defect tracking / incident management• Progress control
Design
White box testing• Debugging• Test coverage
Generating testsetsfor progr.test
Static testing
Dynamic testing
Test repository
Static testing
Static testing
Aids• Test design• Test data manipulation
7.19Kiểm thử và Đảm bảo Chất lượng Phần mềm
Tool selection and implementation
Identify your test activities and see where tools can be supportiveCreation of long listCreation of a shortlistArrange demo’sEvaluation of selected toolsReview and select toolPilot projectRoll out
Are you ready for CAST?
7.20Kiểm thử và Đảm bảo Chất lượng Phần mềm
Dimensions of software testing
1. What is tested?a. Software characteristics (design/embedded?/language/...)
b. Requirements (what property/quality do we want from the software)
c. Purpose (what development phase, what kind of errors, what next step)
2. How to test?a. Method (adequacy criterion: how to generate how many tests)
b. Assumptions (limitations, simplifications, heuristics)
c. Organization (documentation, implementation, execution:platform, interactive, batch)
d. Who tests? (programmer, user, third party)
3. How are the results evaluated?
7.21Kiểm thử và Đảm bảo Chất lượng Phần mềm
Dimensions + concept map
1a
1b
1c2a2b
2c
2d
3
7.22Kiểm thử và Đảm bảo Chất lượng Phần mềm
Homework
Which CMM and TMM level has the following organization?
- 300 employees, 20 IT- Working on getting documentation done- Project plans are made- They do a survey on testing tool- Test planning is on its way- Test design is made during acceptance testWhat would you suggest to improve testing using TMM?
7.23Kiểm thử và Đảm bảo Chất lượng Phần mềm
Homework
Try to make a case for yourself what automated testing would cost and manual testing. (Assume the organisation made the initial investment)
1Kiểm thử và Đảm bảo Chất lượng Phần mềm
Software Testing and SQA
Chương 8
Ước lượng chi phíphần mềm
8.2Kiểm thử và Đảm bảo Chất lượng Phần mềm
Mục tiêu
Giới thiệu các khái niệm và nền tảng về chi phí và giá cả phần mềmMô tả 3 độ đo để đánh giá năng suất phầnmềmGiải thích tại sao nên sử dụng các kỹ thuậtkhác nhau để đánh giá ước lượng phần mềmMô tả các nguyên lý của giải thuật cho môhình ước lượng chi phí COCOMO 2
8.3Kiểm thử và Đảm bảo Chất lượng Phần mềm
Các chủ đề chính
Năng suất phần mềm Software productivityKỹ thuật ước lượng Estimation techniquesMô hình hoá chi phí giải thuật Algorithmic cost modellingKhoảng thời gian và thực hiện Project duration and staffing
8.4Kiểm thử và Đảm bảo Chất lượng Phần mềm
Các vấn đề ước lượng cơ bản
Mất bao nhiêu chi phí cần thiết để hoànthành một hoạt động?Mất bao nhiêu thời gian để hoàn thành mộthoạt động?Tổng chi phí cho một hoạt động là gì?ước lượng dự án và lịch biểu là các hoạt độngquản lý có liên hệ với nhau.
8.5Kiểm thử và Đảm bảo Chất lượng Phần mềm
Các thành phần chi phí phần mềm
Các chi phí về phần cứng và phần mềm.Chi phí đi lại và đào tạo.Các chi phí nỗ lực (Nhân tố nổi trội trong hầu hếtcác dự án)
Tiền lương các kỹ sư làm việc trong dự án;Chi phí bảo hiểm và xã hội.
Các chi phí phải trả trướcChi phí xây nhà, lò sưởi, chiếu sáng.Chi phí mạng và truyền thông.Chi phí các phương tiện công trình công cộng dùngchung (thư viện, nhà ăn, v.v...).
8.6Kiểm thử và Đảm bảo Chất lượng Phần mềm
Chi phí và giá cả
Ước lượng được đánh giá để phát hiện giá cảcho các nhà phát triển làm ra các hệ thốngphần mềm.Không có mối quan hệ đơn giản nào giữa chi phí phát triển và giá cả mà khách hàng phảitrả..Hội đồng tổ chức, nền kinh tế, chính trị vàcác điều kiện kinh doanh có ảnh hưởng đángkể đến giá phải trả.
8.7Kiểm thử và Đảm bảo Chất lượng Phần mềm
Các nhân tố giá cả phần mềmCơ hội thị trường A development organisation may quote a low price because it
wishes to move into a new segment of the software market. Accepting a low profit on one project may give the opportunity of more profit later. The experience gained may allow new products to be developed.
Sự không chắc chắn của luợng lượng chi phí
If an organisation is unsure of its cost estimate, it may increase its price by some contingency over and above its normal profit.
Các điều khoản hơp đồng
A customer may be willing to allow the developer to retain ownership of the source code and reuse it in other projects. The price charged may then be less than if the software source code is handed over to the customer.
Thay đổi yêu cầu If the requirements are likely to change, an organisation may lower its price to win a contract. After the contract is awarded, high prices can be charged for changes to the requirements.
Sức mạnh về tài chính
Developers in financial difficulty may lower their price to gain a contract. It is better to make a smaller than normal profit or break even than to go out of business.
8.8Kiểm thử và Đảm bảo Chất lượng Phần mềm
Độ đo tỷ lệ mà các kỹ sư phát triển phầnmềm đưa ra ra phần mềm và các tài liệu liênquan.Không định hướng tới chất luợng mà thôngqua đảm bảo chất lượng là một nhân tố đánhgiá năng suất.Một cách cần thiết, chúng ta muốn đo chứcnăng của sản phẩm đưa ra trên đơn vị thờigian.
Năng suất phần mềm
8.9Kiểm thử và Đảm bảo Chất lượng Phần mềm
Độ đo liên quan đến kích cỡ : Dựa vào đầu racủa quá trình làm phần mềm. Có thể là sódòng của mã nguồn, các lệnh mã đích v.v....Độ đo liên quan đến chức năng: Dựa vào ướclượng các chức năng của sản phẩm phânphối. Điểm chức năng là cách tốt nhất để đocho loại này.
Độ đo năng suất
8.10Kiểm thử và Đảm bảo Chất lượng Phần mềm
Estimating the size of the measure (e.g. how many function points).Estimating the total number of programmer months that have elapsed.Estimating contractor productivity (e.g. documentation team) and incorporating this estimate in overall estimate.
Measurement problems
8.11Kiểm thử và Đảm bảo Chất lượng Phần mềm
What's a line of code?The measure was first proposed when programs were typed on cards with one line per card;How does this correspond to statements as in Java which can span several lines or where there can be several statements on one line.
What programs should be counted as part of the system?This model assumes that there is a linear relationship between system size and volume of documentation.
Lines of code
8.12Kiểm thử và Đảm bảo Chất lượng Phần mềm
The lower level the language, the more productive the programmer
The same functionality takes more code to implement in a lower-level language than in a high-level language.
The more verbose the programmer, the higher the productivity
Measures of productivity based on lines of code suggest that programmers who write verbose code are more productive than programmers who write compact code.
Productivity comparisons
8.13Kiểm thử và Đảm bảo Chất lượng Phần mềm
System development times
Analysis Design Coding Testing Documentation
Assembly codeHigh-level language
3 weeks3 weeks
5 weeks5 weeks
8 weeks4 weeks
10weeks
6 weeks
2 weeks2 weeks
Size Effort Productivity
Assembly codeHigh-level language
5000 lines1500 lines
28 weeks20 weeks
714 lines/month300 lines/month
8.14Kiểm thử và Đảm bảo Chất lượng Phần mềm
Function points
Based on a combination of program characteristicsexternal inputs and outputs;user interactions;external interfaces;files used by the system.
A weight is associated with each of these and the function point count is computed by multiplying each raw count by the weight and summing all values.
UFC = ∑(number of elements of given type) × (weight)
8.15Kiểm thử và Đảm bảo Chất lượng Phần mềm
Function points
The function point count is modified by complexity of the projectFPs can be used to estimate LOC depending on the average number of LOC per FP for a given language
LOC = AVC * number of function points; AVC is a language-dependent factor varying from 200-300 for assemble language to 2-40 for a 4GL;
FPs are very subjective. They depend on the estimator
Automatic function-point counting is impossible.
8.16Kiểm thử và Đảm bảo Chất lượng Phần mềm
Object points
Object points (alternatively named application points) are an alternative function-related measure to function points when 4Gls or similar languages are used for development.Object points are NOT the same as object classes.The number of object points in a program is a weighted estimate of
The number of separate screens that are displayed;The number of reports that are produced by the system;The number of program modules that must be developed to supplement the database code;
8.17Kiểm thử và Đảm bảo Chất lượng Phần mềm
Object point estimation
Object points are easier to estimate from a specification than function points as they are simply concerned with screens, reports and programming language modules.They can therefore be estimated at a fairly early point in the development process.At this stage, it is very difficult to estimate the number of lines of code in a system.
8.18Kiểm thử và Đảm bảo Chất lượng Phần mềm
Real-time embedded systems, 40-160 LOC/P-month.Systems programs , 150-400 LOC/P-month.Commercial applications, 200-900 LOC/P-month.In object points, productivity has been measured between 4 and 50 object points/month depending on tool support and developer capability.
Productivity estimates
8.19Kiểm thử và Đảm bảo Chất lượng Phần mềm
Factors affecting productivity
Applicationdomainexperience
Knowledge of the application domain is essential for effectivesoftware development. Engineers who already understand adomain are likely to be the most productive.
Process quality The development process used can have a s ignificant effect onproductivity. This is covered in Chapter 28.
Project size The larger a project, the more time required for teamcommunications. Less time is available for development soindividual productivity is reduced.
Technologysupport
Good support technology such as C ASE tools, configurationmanagement systems, etc. can improve productivity.
Workingenvironment
As I discussed in Chapter 25, a q uiet working environment withprivate work areas contributes to improved productivity.
8.20Kiểm thử và Đảm bảo Chất lượng Phần mềm
All metrics based on volume/unit time are flawed because they do not take quality into account.Productivity may generally be increased at the cost of quality.It is not clear how productivity/quality metrics are related.If requirements are constantly changing then an approach based on counting lines of code is not meaningful as the program itself is not static;
Quality and productivity
8.21Kiểm thử và Đảm bảo Chất lượng Phần mềm
Estimation techniques
There is no simple way to make an accurate estimate of the effort required to develop a software system
Initial estimates are based on inadequate information in a user requirements definition;The software may run on unfamiliar computers or use new technology;The people in the project may be unknown.
Project cost estimates may be self-fulfillingThe estimate defines the budget and the product is adjusted to meet the budget.
8.22Kiểm thử và Đảm bảo Chất lượng Phần mềm
Changing technologies
Changing technologies may mean that previous estimating experience does not carry over to new systems
Distributed object systems rather than mainframe systems;Use of web services;Use of ERP or database-centred systems;Use of off-the-shelf software;Development for and with reuse;Development using scripting languages;The use of CASE tools and program generators.
8.23Kiểm thử và Đảm bảo Chất lượng Phần mềm
Estimation techniques
Algorithmic cost modelling.Expert judgement.Estimation by analogy.Parkinson's Law.Pricing to win.
8.24Kiểm thử và Đảm bảo Chất lượng Phần mềm
Estimation techniquesAlgorithmiccost modelling
A model based on historical cost information that relates some softwaremetric (usually its size) to the project cost is used. An estimate is madeof that metric and the model predicts the effort required.
Expertjudgement
Several experts on the proposed software development techniques andthe application domain are consulted. They each estimate the projectcost. These estimates are compared and discussed. The estimationprocess iterates until an agreed estimate is reached.
Estimation byanalogy
This technique is applicable when other projects in the same applicationdomain have been completed. The cost of a new project is estimated byanalogy with these completed projects. Myers (Myers 1989) gives avery clear description of this approach.
ParkinsonÕsLaw
ParkinsonÕs Law states that work expands to fill the time available. Thecost is determined by available resources rather than by objectiveassessment. If the software has to be delivered in 12 months and 5people are available, the effort required is estimated to be 60 person-months.
Pricing to win The software cost is estimated to be whatever the customer hasavailable to spend on the project. The estimated effort depends on thecustomerÕs budget and not on the software functionality.
8.25Kiểm thử và Đảm bảo Chất lượng Phần mềm
Pricing to win
The project costs whatever the customer has to spend on it.Advantages:
You get the contract.
Disadvantages: The probability that the customer gets the system he or she wants is small. Costs do not accurately reflect the work required.
8.26Kiểm thử và Đảm bảo Chất lượng Phần mềm
Top-down and bottom-up estimation
Any of these approaches may be used top-down or bottom-up.Top-down
Start at the system level and assess the overall system functionality and how this is delivered through sub-systems.
Bottom-upStart at the component level and estimate the effort required for each component. Add these efforts to reach a final estimate.
8.27Kiểm thử và Đảm bảo Chất lượng Phần mềm
Top-down estimation
Usable without knowledge of the system architecture and the components that might be part of the system.Takes into account costs such as integration, configuration management and documentation.Can underestimate the cost of solving difficult low-level technical problems.
8.28Kiểm thử và Đảm bảo Chất lượng Phần mềm
Bottom-up estimation
Usable when the architecture of the system is known and components identified.This can be an accurate method if the system has been designed in detail.It may underestimate the costs of system level activities such as integration and documentation.
8.29Kiểm thử và Đảm bảo Chất lượng Phần mềm
Estimation methods
Each method has strengths and weaknesses.Estimation should be based on several methods.If these do not return approximately the same result, then you have insufficient information available to make an estimate.Some action should be taken to find out more in order to make more accurate estimates.Pricing to win is sometimes the only applicable method.
8.30Kiểm thử và Đảm bảo Chất lượng Phần mềm
Pricing to win
This approach may seem unethical and un-businesslike.However, when detailed information is lacking it may be the only appropriate strategy.The project cost is agreed on the basis of an outline proposal and the development is constrained by that cost.A detailed specification may be negotiated or an evolutionary approach used for system development.
8.31Kiểm thử và Đảm bảo Chất lượng Phần mềm
Algorithmic cost modelling
Cost is estimated as a mathematical function of product, project and process attributes whose values are estimated by project managers:
Effort = A × SizeB × M
A is an organisation-dependent constant, B reflects the disproportionate effort for large projects and M is a multiplier reflecting product, process and people attributes.
The most commonly used product attribute for cost estimation is code size.Most models are similar but they use different values for A, B and M.
8.32Kiểm thử và Đảm bảo Chất lượng Phần mềm
Estimation accuracy
The size of a software system can only be known accurately when it is finished.Several factors influence the final size
Use of COTS and components;Programming language;Distribution of system.
As the development process progresses then the size estimate becomes more accurate.
8.33Kiểm thử và Đảm bảo Chất lượng Phần mềm
Estimate uncertainty
8.34Kiểm thử và Đảm bảo Chất lượng Phần mềm
The COCOMO model
An empirical model based on project experience.Well-documented, ‘independent’ model which is not tied to a specific software vendor.Long history from initial version published in 1981 (COCOMO-81) through various instantiations to COCOMO 2.COCOMO 2 takes into account different approaches to software development, reuse, etc.
8.35Kiểm thử và Đảm bảo Chất lượng Phần mềm
COCOMO 81
Projectcomplexity
Formula Description
Simple PM = 2.4 (KDSI)1.05 × M Well-understood applicationsdeveloped by small teams.
Moderate PM = 3.0 (KDSI)1.12 × M More complex projects whereteam members may have limitedexperience of related systems.
Embedded PM = 3.6 (KDSI)1.20 × M Complex projects where thesoftware is part of a stronglycoupled complex of hardware,software, regulations andoperational procedures.
8.36Kiểm thử và Đảm bảo Chất lượng Phần mềm
COCOMO 2COCOMO 81 was developed with the assumption that a waterfall process would be used and that all software would be developed from scratch.Since its formulation, there have been many changes in software engineering practice and COCOMO 2 is designed to accommodate different approaches to software development.
8.37Kiểm thử và Đảm bảo Chất lượng Phần mềm
COCOMO 2 models
COCOMO 2 incorporates a range of sub-models that produce increasingly detailed software estimates.The sub-models in COCOMO 2 are:
Application composition model. Used when software is composed from existing parts.Early design model. Used when requirements are available but design has not yet started.Reuse model. Used to compute the effort of integrating reusable components.Post-architecture model. Used once the system architecture has been designed and more information about the system is available.
8.38Kiểm thử và Đảm bảo Chất lượng Phần mềm
Use of COCOMO 2 models
8.39Kiểm thử và Đảm bảo Chất lượng Phần mềm
Application composition model
Supports prototyping projects and projects where there is extensive reuse.Based on standard estimates of developer productivity in application (object) points/month.Takes CASE tool use into account.Formula is
PM = ( NAP × (1 - %reuse/100 ) ) / PROD
PM is the effort in person-months, NAP is the number of application points and PROD is the productivity.
8.40Kiểm thử và Đảm bảo Chất lượng Phần mềm
Object point productivity
DeveloperÕs experienceand capability
Very low Low Nominal High Very high
ICASE maturity andcapability
Very low Low Nominal High Very high
PROD (NOP/month) 4 7 13 25 50
8.41Kiểm thử và Đảm bảo Chất lượng Phần mềm
Early design model
Estimates can be made after the requirements have been agreed.Based on a standard formula for algorithmic models
PM = A × SizeB × M where
M = PERS × RCPX × RUSE × PDIF × PREX × FCIL × SCED;A = 2.94 in initial calibration, Size in KLOC, B varies from 1.1 to 1.24 depending on novelty of the project, development flexibility, risk management approaches and the process maturity.
8.42Kiểm thử và Đảm bảo Chất lượng Phần mềm
MultipliersMultipliers reflect the capability of the developers, the non-functional requirements, the familiarity with the development platform, etc.
RCPX - product reliability and complexity;RUSE - the reuse required;PDIF - platform difficulty;PREX - personnel experience;PERS - personnel capability;SCED - required schedule;FCIL - the team support facilities.
8.43Kiểm thử và Đảm bảo Chất lượng Phần mềm
The reuse model
Takes into account black-box code that is reused without change and code that has to be adapted to integrate it with new code.There are two versions:
Black-box reuse where code is not modified. An effort estimate (PM) is computed.White-box reuse where code is modified. A size estimate equivalent to the number of lines of new source code is computed. This then adjusts the size estimate for new code.
8.44Kiểm thử và Đảm bảo Chất lượng Phần mềm
Reuse model estimates 1
For generated code:PM = (ASLOC * AT/100)/ATPRODASLOC is the number of lines of generated codeAT is the percentage of code automatically generated.ATPROD is the productivity of engineers in integrating this code.
8.45Kiểm thử và Đảm bảo Chất lượng Phần mềm
Reuse model estimates 2
When code has to be understood and integrated:
ESLOC = ASLOC * (1-AT/100) * AAM.ASLOC and AT as before.AAM is the adaptation adjustment multiplier computed from the costs of changing the reused code, the costs of understanding how to integrate the code and the costs of reuse decision making.
8.46Kiểm thử và Đảm bảo Chất lượng Phần mềm
Post-architecture levelUses the same formula as the early design model but with 17 rather than 7 associated multipliers.The code size is estimated as:
Number of lines of new code to be developed;Estimate of equivalent number of lines of new code computed using the reuse model;An estimate of the number of lines of code that have to be modified according to requirements changes.
8.47Kiểm thử và Đảm bảo Chất lượng Phần mềm
This depends on 5 scale factors (see next slide). Their sum/100 is added to 1.01A company takes on a project in a new domain. The client has not defined the process to be used and has not allowed time for risk analysis. The company has a CMM level 2 rating.
Precedenteness - new project (4)Development flexibility - no client involvement - Very high (1)Architecture/risk resolution - No risk analysis - V. Low .(5)Team cohesion - new team - nominal (3)Process maturity - some control - nominal (3)
Scale factor is therefore 1.17.
The exponent term
8.48Kiểm thử và Đảm bảo Chất lượng Phần mềm
Exponent scale factorsPrecedentedness Reflects the previous experience of the organisation with this type of
project. Very low means no previous experience, Extra high meansthat the organisation is completely familiar with this applicationdomain.
Developmentflexibility
Reflects the degree of flexibility in the development process. Verylow means a prescribed process is used; Extra high means that theclient only sets general goals.
Architecture/riskresolution
Reflects the extent of risk analysis carried out. Very low means littleanalysis, Extra high means a complete a thorough risk analysis.
Team cohesion Reflects how well the development team know each other and worktogether. Very low means very difficult interactions, Extra highmeans an integrated and effective team with no communicationproblems.
Process maturity Reflects the process maturity of the organisation. The computationof this value depends on the CMM Maturity Questionnaire but anestimate can be achieved by subtracting the CMM process maturitylevel from 5.
8.49Kiểm thử và Đảm bảo Chất lượng Phần mềm
Product attributes Concerned with required characteristics of the software product being developed.
Computer attributes Constraints imposed on the software by the hardware platform.
Personnel attributes Multipliers that take the experience and capabilities of the people working on the project into account.
Project attributes Concerned with the particular characteristics of the software development project.
Multipliers
8.50Kiểm thử và Đảm bảo Chất lượng Phần mềm
Effects of cost drivers
Exponent value 1.17System size (including factors for reuseand requirements volatility)
128, 000 DSI
Initial COCOMO estimate withoutcost drivers
730 person-months
Reliability Very high, multiplier = 1.39Complexity Very high, multiplier = 1.3Memory constraint High, multiplier = 1.21Tool use Low, multiplier = 1.12Schedule Accelerated, multiplier = 1.29Adjusted COCOMO estimate 2306 person-months
Reliability Very low, multiplier = 0.75Complexity Very low, multiplier = 0.75Memory constraint None, multiplier = 1Tool use Very high, multiplier = 0.72Schedule Normal, multiplier = 1Adjusted COCOMO estimate 295 person-months
8.51Kiểm thử và Đảm bảo Chất lượng Phần mềm
Algorithmic cost models provide a basis for project planning as they allow alternative strategies to be compared.Embedded spacecraft system
Must be reliable;Must minimise weight (number of chips);Multipliers on reliability and computer constraints > 1.
Cost componentsTarget hardware;Development platform;Development effort.
Project planning
8.52Kiểm thử và Đảm bảo Chất lượng Phần mềm
Management options
8.53Kiểm thử và Đảm bảo Chất lượng Phần mềm
Management option costs
Option RELY STOR TIME TOOLS LTEX Total effort Software cost Hardwarecost
Total cost
A 1.39 1.06 1.11 0.86 1 63 949393 100000 1049393B 1.39 1 1 1.12 1.22 88 1313550 120000 1402025C 1.39 1 1.11 0.86 1 60 895653 105000 1000653D 1.39 1.06 1.11 0.86 0.84 51 769008 100000 897490E 1.39 1 1 0.72 1.22 56 844425 220000 1044159F 1.39 1 1 1.12 0.84 57 851180 120000 1002706
8.54Kiểm thử và Đảm bảo Chất lượng Phần mềm
Option choice
Option D (use more experienced staff) appears to be the best alternative
However, it has a high associated risk as experienced staff may be difficult to find.
Option C (upgrade memory) has a lower cost saving but very low risk.Overall, the model reveals the importance of staff experience in software development.
8.55Kiểm thử và Đảm bảo Chất lượng Phần mềm
Project duration and staffing
As well as effort estimation, managers must estimate the calendar time required to complete a project and when staff will be required.Calendar time can be estimated using a COCOMO 2 formula
TDEV = 3 × (PM)(0.33+0.2*(B-1.01))
PM is the effort computation and B is the exponent computed as discussed above (B is 1 for the early prototyping model). This computation predicts the nominal schedule for the project.
The time required is independent of the number of people working on the project.
8.56Kiểm thử và Đảm bảo Chất lượng Phần mềm
Staffing requirements
Staff required can’t be computed by diving the development time by the required schedule.The number of people working on a project varies depending on the phase of the project.The more people who work on the project, the more total effort is usually required.A very rapid build-up of people often correlates with schedule slippage.
8.57Kiểm thử và Đảm bảo Chất lượng Phần mềm
Key points
There is not a simple relationship between the price charged for a system and its development costs.Factors affecting productivity include individual aptitude, domain experience, the development project, the project size, tool support and the working environment.Software may be priced to gain a contract and the functionality adjusted to the price.
8.58Kiểm thử và Đảm bảo Chất lượng Phần mềm
Key pointsDifferent techniques of cost estimation should be used when estimating costs. The COCOMO model takes project, product, personnel and hardware attributes into account when predicting effort required.Algorithmic cost models support quantitative option analysis as they allow the costs of different options to be compared.The time to complete a project is not proportional to the number of people working on the project.
1Kiểm thử và Đảm bảo Chất lượng Phần mềm
Software Testing and SQA
Chapter 10
Quản lý cấu hình
10.2Kiểm thử và Đảm bảo Chất lượng Phần mềm
Mục tiêu
Diễn tả tầm quan trọng của quản lý cấu hìnhQLCH (CM)Mô tả các hoạt động chính của QLCH: Lập kếhoạch, quản lý thay đổi, quản lý phiên bản vàXD kiến trúcThảo luận về các công cụ CASE để hỗ trợ tiến trình QLCH
10.3Kiểm thử và Đảm bảo Chất lượng Phần mềm
Các chủ đề chính của QLCH
Lập kế hoạch QLCHQuản lý thay đổiQuản lý phiên bản và phát hànhKiến trúc hệ thốngCac công cụ CASE cho QLCH
10.4Kiểm thử và Đảm bảo Chất lượng Phần mềm
Các phiên bản mới của HT phần mềm được tạo ra khi HT thay đổi :
Khi thay đổi máy tính/Hệ điều hành OS;Yêu cầu các chức năng khác đi;Các yêu cầu thay đổi do khách hàng đưa ra cho phù hợp (may đo).
QLCH liên quan đến quản lý hệ thống phần mềm tiến hoá:
Sự thay đổi hệ thống là hoạt động của một nhóm đội ngũ;QLCH nhằm điều khiển chi phí và nỗ lực thay đổi một hệthống.
Quản lý cấu hình
10.5Kiểm thử và Đảm bảo Chất lượng Phần mềm
Quản lý cấu hình (Tiếp)
Liên quan đến sự phát triển và ứng dụng của các thủ tục và các chuẩn nhằm quản lý sản phẩm phần mềm tiến hoá.QLCH có thể xem như một phần của quátrình quản lý chất lượng (tổng quát hơn).Khi đưa ra QLCH, Các hệ thống phần mềm đôi khi còn được gọi đường cơ sở như chúng đang ở thời điểm xuất phát cho việc phát triển sau này.
10.6Kiểm thử và Đảm bảo Chất lượng Phần mềm
Những họ hệ thống
10.7Kiểm thử và Đảm bảo Chất lượng Phần mềm
Các chuẩn QLCH
CM should always be based on a set of standards which are applied within an organisation.Standards should define how items are identified, how changes are controlled and how new versions are managed.Standards may be based on external CM standards (e.g. IEEE standard for CM).Some existing standards are based on a waterfall process model - new CM standards are needed for evolutionary development.
10.8Kiểm thử và Đảm bảo Chất lượng Phần mềm
Concurrent development and testing
A time (say 2pm) for delivery of system components is agreed.A new version of a system is built from these components by compiling and linking them.This new version is delivered for testing using pre-defined tests.Faults that are discovered during testing are documented and returned to the system developers.
10.9Kiểm thử và Đảm bảo Chất lượng Phần mềm
Frequent system building
It is easier to find problems that stem from component interactions early in the process.This encourages thorough unit testing -developers are under pressure not to ‘break the build’.A stringent change management process is required to keep track of problems that have been discovered and repaired.
10.10Kiểm thử và Đảm bảo Chất lượng Phần mềm
All products of the software process may have to be managed:
Specifications;Designs;Programs;Test data;User manuals.
Thousands of separate documents may be generated for a large, complex software system.
Configuration management planning
10.11Kiểm thử và Đảm bảo Chất lượng Phần mềm
Defines the types of documents to be managed and a document naming scheme.Defines who takes responsibility for the CM procedures and creation of baselines.Defines policies for change control and version management.Defines the CM records which must be maintained.
The CM plan
10.12Kiểm thử và Đảm bảo Chất lượng Phần mềm
The CM plan
Describes the tools which should be used to assist the CM process and any limitations on their use.Defines the process of tool use.Defines the CM database used to record configuration information.May include information such as the CM of external software, process auditing, etc.
10.13Kiểm thử và Đảm bảo Chất lượng Phần mềm
Large projects typically produce thousands of documents which must be uniquely identified.Some of these documents must be maintained for the lifetime of the software.Document naming scheme should be defined so that related documents have related names.A hierarchical scheme with multi-level names is probably the most flexible approach.
PCL-TOOLS/EDIT/FORMS/DISPLAY/AST-INTERFACE/CODE
Configuration item identification
10.14Kiểm thử và Đảm bảo Chất lượng Phần mềm
Configuration hierarchy
10.15Kiểm thử và Đảm bảo Chất lượng Phần mềm
All CM information should be maintained in a configuration database.This should allow queries about configurations to be answered:
Who has a particular system version?What platform is required for a particular version?What versions are affected by a change to component X?How many reported faults in version T?
The CM database should preferably be linked to the software being managed.
The configuration database
10.16Kiểm thử và Đảm bảo Chất lượng Phần mềm
CM database implementation
May be part of an integrated environment to support software development.
The CM database and the managed documents are all maintained on the same system
CASE tools may be integrated with this so that there is a close relationship between the CASE tools and the CM tools.More commonly, the CM database is maintained separately as this is cheaper and more flexible.
10.17Kiểm thử và Đảm bảo Chất lượng Phần mềm
Software systems are subject to continual change requests:
From users;From developers;From market forces.
Change management is concerned with keeping track of these changes and ensuring that they are implemented in the most cost-effective way.
Change management
10.18Kiểm thử và Đảm bảo Chất lượng Phần mềm
Request change by completing a change request formAnalyze change request if change is valid then Assess how change might be implemented Assess change cost Submit request to change control board if change is accepted then repeat make changes to software submit changed software for quality approval until software quality is adequate create new system version else reject change request else reject change request
The change management process
10.19Kiểm thử và Đảm bảo Chất lượng Phần mềm
The definition of a change request form is part of the CM planning process.This form records the change proposed, requestor of change, the reason why change was suggested and the urgency of change (from requestor of the change).It also records change evaluation, impact analysis, change cost and recommendations (System maintenance staff).
Change request form
10.20Kiểm thử và Đảm bảo Chất lượng Phần mềm
Change request form
10.21Kiểm thử và Đảm bảo Chất lượng Phần mềm
A major problem in change management is tracking change status.Change tracking tools keep track the status of each change request and automatically ensure that change requests are sent to the right people at the right time.Integrated with E-mail systems allowing electronic change request distribution.
Change tracking tools
10.22Kiểm thử và Đảm bảo Chất lượng Phần mềm
Changes should be reviewed by an external group who decide whether or not they are cost-effective from a strategic and organizational viewpoint rather than a technical viewpoint. Should be independent of project responsible for system. The group is sometimes called a change control board.The CCB may include representatives from client and contractor staff.
Change control board
10.23Kiểm thử và Đảm bảo Chất lượng Phần mềm
This is a record of changes applied to a document or code component.It should record, in outline, the change made, the rationale for the change, who made the change and when it was implemented.It may be included as a comment in code. If a standard prologue style is used for the derivation history, tools can process this automatically.
Derivation history
10.24Kiểm thử và Đảm bảo Chất lượng Phần mềm
Component header information
// BANKSEC project (IST 6087)//// BANKSEC-TOOLS/AUTH/RBAC/USER_ROLE//// Object: currentRole// Author: N. Perwaiz// Creation date: 10th November 2002//// © Lancaster University 2002//// Modification history// Version ModifierDate Change Reason// 1.0 J. Jones 1/12/2002 Add header Submitted to CM// 1.1 N. Perwaiz 9/4/2003New field Change req. R07/02
10.25Kiểm thử và Đảm bảo Chất lượng Phần mềm
Invent an identification scheme for system versions.Plan when a new system version is to be produced.Ensure that version management procedures and tools are properly applied.Plan and distribute new system releases.
Version and release management
10.26Kiểm thử và Đảm bảo Chất lượng Phần mềm
Version An instance of a system which is functionally distinct in some way from other system instances.Variant An instance of a system which is functionally identical but non-functionally distinct from other instances of a system.Release An instance of a system which is distributed to users outside of the development team.
Versions/variants/releases
10.27Kiểm thử và Đảm bảo Chất lượng Phần mềm
Version identification
Procedures for version identification should define an unambiguous way of identifying component versions.There are three basic techniques for component identification
Version numbering;Attribute-based identification;Change-oriented identification.
10.28Kiểm thử và Đảm bảo Chất lượng Phần mềm
Simple naming scheme uses a linear derivation
V1, V1.1, V1.2, V2.1, V2.2 etc.
The actual derivation structure is a tree or a network rather than a sequence.Names are not meaningful. A hierarchical naming scheme leads to fewer errors in version identification.
Version numbering
10.29Kiểm thử và Đảm bảo Chất lượng Phần mềm
Version derivation structure
10.30Kiểm thử và Đảm bảo Chất lượng Phần mềm
Attributes can be associated with a version with the combination of attributes identifying that version
Examples of attributes are Date, Creator, Programming Language, Customer, Status etc.
This is more flexible than an explicit naming scheme for version retrieval; However, it can cause problems with uniqueness - the set of attributes have to be chosen so that all versions can be uniquely identified.In practice, a version also needs an associated name for easy reference.
Attribute-based identification
10.31Kiểm thử và Đảm bảo Chất lượng Phần mềm
Attribute-based queries
An important advantage of attribute-based identification is that it can support queries so that you can find ‘the most recent version in Java’ etc.The query selects a version depending on attribute values
AC3D (language =Java, platform = XP, date = Jan 2003).
10.32Kiểm thử và Đảm bảo Chất lượng Phần mềm
Change-oriented identification
Integrates versions and the changes made to create these versions.Used for systems rather than components.Each proposed change has a change set that describes changes made to implement that change.Change sets are applied in sequence so that, in principle, a version of the system that incorporates an arbitrary set of changes may be created.
10.33Kiểm thử và Đảm bảo Chất lượng Phần mềm
Releases must incorporate changes forced on the system by errors discovered by users and by hardware changes.They must also incorporate new system functionality.Release planning is concerned with when to issue a system version as a release.
Release management
10.34Kiểm thử và Đảm bảo Chất lượng Phần mềm
System releases
Not just a set of executable programs.May also include:
Configuration files defining how the release is configured for a particular installation;Data files needed for system operation;An installation program or shell script to install the system on target hardware;Electronic and paper documentation;Packaging and associated publicity.
Systems are now normally released on optical disks (CD or DVD) or as downloadable installation files from the web.
10.35Kiểm thử và Đảm bảo Chất lượng Phần mềm
Customer may not want a new release of the system
They may be happy with their current system as the new version may provide unwanted functionality.
Release management should not assume that all previous releases have been accepted. All files required for a release should be re-created when a new release is installed.
Release problems
10.36Kiểm thử và Đảm bảo Chất lượng Phần mềm
Release decision making
Preparing and distributing a system release is an expensive process.Factors such as the technical quality of the system, competition, marketing requirements and customer change requests should all influence the decision of when to issue a new system release.
10.37Kiểm thử và Đảm bảo Chất lượng Phần mềm
System release strategyFactor Description
Technical quality ofthe system
If serious system faults are reported which affect the way in whichmany customers use the system, it may be necessary to issue a faultrepair release. Howeve r, minor system faults may be repaired by issuingpatches (often distributed over the Internet) that can be applied to thecurrent release of the system.
Platform changes You may have to create a new release of a software application when anew version of the operating system platform is released.
LehmanÕs fifth law(see Chapter 21)
This suggests that the increment of functionality that is included in eachrelease is approximately constant. Therefore, if there has been a systemrelease with significant new functionality, then it may have to befollowed by a repair release.
Competition A new system release may be necessary because a co mpeting product isavailable.
Marketingrequirements
The marketing department of an organisation may have made acommitment for releases to be available at a particular date.
Customer changeproposals
For customised systems, customers may have made and paid for aspecific set of system change proposals and they expect a system releaseas soon as these have been implemented.
10.38Kiểm thử và Đảm bảo Chất lượng Phần mềm
Release creation
Release creation involves collecting all files and documentation required to create a system release.Configuration descriptions have to be written for different hardware and installation scripts have to be written.The specific release must be documented to record exactly what files were used to create it. This allows it to be re-created if necessary.
10.39Kiểm thử và Đảm bảo Chất lượng Phần mềm
The process of compiling and linking software components into an executable system.Different systems are built from different combinations of components.This process is now always supported by automated tools that are driven by ‘build scripts’.
System building
10.40Kiểm thử và Đảm bảo Chất lượng Phần mềm
Do the build instructions include all required components?
When there are many hundreds of components making up a system, it is easy to miss one out. This should normally be detected by the linker.
Is the appropriate component version specified?
A more significant problem. A system built with the wrong version may work initially but fail after delivery.
Are all data files available?The build should not rely on 'standard' data files. Standards vary from place to place.
System building problems
10.41Kiểm thử và Đảm bảo Chất lượng Phần mềm
Are data file references within components correct?
Embedding absolute names in code almost always causes problems as naming conventions differ from place to place.
Is the system being built for the right platformSometimes you must build for a specific OS version or hardware configuration.
Is the right version of the compiler and other software tools specified?
Different compiler versions may actually generate different code and the compiled component will exhibit different behaviour.
System building problems
10.42Kiểm thử và Đảm bảo Chất lượng Phần mềm
�System building
10.43Kiểm thử và Đảm bảo Chất lượng Phần mềm
CASE tools for configuration management
CM processes are standardised and involve applying pre-defined procedures.Large amounts of data must be managed.CASE tool support for CM is therefore essential.Mature CASE tools to support configuration management are available ranging from stand-alone tools to integrated CM workbenches.
10.44Kiểm thử và Đảm bảo Chất lượng Phần mềm
CM workbenches
Open workbenchesTools for each stage in the CM process are integrated through organisational procedures and scripts. Gives flexibility in tool selection.
Integrated workbenchesProvide whole-process, integrated support for configuration management. More tightly integrated tools so easier to use. However, the cost is less flexibility in the tools used.
10.45Kiểm thử và Đảm bảo Chất lượng Phần mềm
Change management tools
Change management is a procedural process so it can be modelled and integrated with a version management system.Change management tools
Form editor to support processing the change request forms;Workflow system to define who does what and to automate information transfer;Change database that manages change proposals and is linked to a VM system;Change reporting system that generates management reports on the status of change requests.
10.46Kiểm thử và Đảm bảo Chất lượng Phần mềm
Version management tools
Version and release identificationSystems assign identifiers automatically when a new version is submitted to the system.
Storage management.System stores the differences between versions rather than all the version code.
Change history recordingRecord reasons for version creation.
Independent development Only one version at a time may be checked out for change. Parallel working on different versions.
Project supportCan manage groups of files associated with a project rather than just single files.
10.47Kiểm thử và Đảm bảo Chất lượng Phần mềm
Delta-based versioning
10.48Kiểm thử và Đảm bảo Chất lượng Phần mềm
System building
Building a large system is computationally expensive and may take several hours.Hundreds of files may be involved.System building tools may provide
A dependency specification language and interpreter;Tool selection and instantiation support;Distributed compilation;Derived object management.
10.49Kiểm thử và Đảm bảo Chất lượng Phần mềm
Component dependencies
10.50Kiểm thử và Đảm bảo Chất lượng Phần mềm
Configuration management is the management of system change to software products.A formal document naming scheme should be established and documents should be managed in a database.The configuration data base should record information about changes and change requests.A consistent scheme of version identification should be established using version numbers, attributes or change sets.
Key points
10.51Kiểm thử và Đảm bảo Chất lượng Phần mềm
Key points
System releases include executable code, data, configuration files and documentation.System building involves assembling components into a system…CASE tools are available to support all CM activitiesCASE tools may be stand-alone tools or may be integrated systems which integrate support for version management, system building and change management.