Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE...
-
Upload
osborne-berry -
Category
Documents
-
view
220 -
download
4
Transcript of Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE...
![Page 1: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/1.jpg)
Empirical Software Quality Engineering:Experience and Recent Development
Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRCProfessor, CSE & EMIS, SMU, Dallas, Texas, USA
西北工业大学计算机学院陕西省百人计划特聘教授
Presentation given on 2013-07 at Tianjin University/天津大学
Copyright © 2010 NSF Net-Centric I/UCRC. All rights reserved.
![Page 2: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/2.jpg)
2
Agenda
Quality and Empirical Observations
Quality Improvement Risk identification and prioritization
Risk resolution and quality improvement
Empirical Validation
New Trends and Directions Net-Centric, Service-Oriented, Cloud Computing, etc.
Questions, comments…
Net-Centric Software & Systems I/UCRC2013/07@Tianjin University, China
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
![Page 3: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/3.jpg)
3
Software Quality
• Most intuitively associate with low defect• Different perspectives:
• user, service provider, infrastructure/data center, developer, tester, manager, 3rd party, etc.
• Different attributes: • capability, reliability, usability, safety, etc.
• Could be domain specific, e.g., • reliability, security, and usability for the web domain
• Grouping of quality attributions possible, e.g., • Dependability include reliability, availability, security,
etc.• This presentation focuses on reliability and
related quality attributes
Net-Centric Software & Systems I/UCRC2013/07@Tianjin University, China
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
![Page 4: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/4.jpg)
4
Quality Attributes/Distribution for an IBM Product
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
Net-Centric Software & Systems I/UCRC
• ODC (orthogonal defect classification) categories
• Observation: highly uneven distributions
2013/07@Tianjin University, China
![Page 5: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/5.jpg)
5
Quality Attributes/Distribution for an IBM Product
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
Net-Centric Software & Systems I/UCRC
• Dependability attributes (rows)
• Stakeholders of information systems (colomns)
• Observation: highly uneven (qualitative) interests/importance
2013/07@Tianjin University, China
![Page 6: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/6.jpg)
6
Quantification of Software Quality
Some quality attributes may be quantifiable, e.g.,
• Reliability: probability of failure-free operations for a specific time period or input set under a specific environment• Quantified: probability of ….• Customer perspective: failure view• Time, input, and environment characterization
• Others may not be quantifiable, e.g.,• Safety may be defined as the “property of being accident
free” (Leveson, “Safeware”, Addison-Wesley, 1995)
Net-Centric Software & Systems I/UCRC
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
2013/07@Tianjin University, China
![Page 7: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/7.jpg)
7
Quantification of Software Quality
• Quantification requires data and measurement• Defect measurement• Other measurement
• Defect concepts and measurement• Failure: behavioral deviation from expectation• Fault: internal problem in software that may lead to
failure• Error: missing/incorrect human action leading to fault
injection• Collectively referred to as defect
• Example of reliability measurement:• Failure measurement• Time or input space measurement
Net-Centric Software & Systems I/UCRC
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
2013/07@Tianjin University, China
![Page 8: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/8.jpg)
8
Quantification: Measurement
Net-Centric Software & Systems I/UCRC
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
• Fault distribution by type (fault class) for a large telecom system
• Histogram or “Pareto chart”
• Pareto principle or 80:20 rule originator
• Observation:• Highly skewed
distribution
2013/07@Tianjin University, China
![Page 9: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/9.jpg)
9
Quantification: Measurement
Net-Centric Software & Systems I/UCRC
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
• Defect & complexity measurement for an IBM product
• Box plot used• Mean• Distribution
boundaries• Observation:
• Highly skewed distribution, again
2013/07@Tianjin University, China
![Page 10: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/10.jpg)
10
Quantification: Measurement
Net-Centric Software & Systems I/UCRC
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
• Cross referencing for SMU Engineering School web site
• Scatter plot used• # = dot size• Distribution
visualization• Observation:
• Highly skewed distribution, again
2013/07@Tianjin University, China
![Page 11: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/11.jpg)
11
Quantification: Measurement
Net-Centric Software & Systems I/UCRC
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
• Relating metric to change for Mozilla
• Scatter plot to visualize distribution and correlation
• Observation:• skewed
/clustered distribution, again
2013/07@Tianjin University, China
![Page 12: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/12.jpg)
12
Quantification: Measurement
Net-Centric Software & Systems I/UCRC
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
• Number of transactions processed for each test run for an IBM RDBMS product
• Plot used• Time/sequence• Distribution
• Observation:• Highly skewed
distribution, again
2013/07@Tianjin University, China
![Page 13: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/13.jpg)
13
Quantification of Software Quality
• Some quality attributes may not be quantifiable, e.g.,• Safety may be defined as the “property of being accident
free” (Leveson, “Safeware”, Addison-Wesley, 1995)
• But, some quantitative characterization still possible:• Empirical data: about 90% of software safety problems can be traced to
interface/interaction related problem sources
• Hazard analysis to allow selective hazard resolution• Risk analysis: prob(hazard), prob(hazard-accident link), worst-damage
• Also, importance rating of individual dependability attributes by stakeholders example earlier
Net-Centric Software & Systems I/UCRC
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
2013/07@Tianjin University, China
![Page 14: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/14.jpg)
14
Quantifiable Quality Improvement
• Need a baseline• Defect measurement: distribution and characterization• Other measurement: internal char. + external relations
• Improvement paradigms• QIP by Basili: baseline-change assessment-implement• Process maturity work from SEI: moving up CMM/CMMI
levels• Other approaches
• Key observation based on empirical evidence: • non-uniform distribution, non-uniform risk• Implication: risk-based approach should be more
effective and efficient than other approaches, if….
Net-Centric Software & Systems I/UCRC
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
2013/07@Tianjin University, China
![Page 15: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/15.jpg)
15
What Is Risk?
• Most intuitively associate with undesirable, unacceptable, or less desirable consequences• Low reliability• High defect (or defect density)• Possible safety problems (and their likelihood)• Others
• Two definitions: mean vs. variance• Mean used mostly in daily life, and also commonly
used in computing and software engineering• Variance to capture (problems associated with)
uncertainty in financial/insurance industries, etc.
Net-Centric Software & Systems I/UCRC
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
2013/07@Tianjin University, China
![Page 16: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/16.jpg)
16
What to Do about Risk?
• 80:20 rule or Pareto principle:• Highly uneven distribution, a key characterization
based on empirical evidence• see (representative) examples earlier
• Both consequences (risk itself) and contributing/associated factors/entities (risk factors)
• What to do: Risk identification and resolution• Associating risk with other entities
• e.g., relating defect density to complexity • What to do after identification?
• Remedial/preventive actions
Net-Centric Software & Systems I/UCRC
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
2013/07@Tianjin University, China
![Page 17: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/17.jpg)
17
Risk Types/Perspectives
• Similar to quality perspectives, primarily internal vs. external risks
• External risk to customers/users: reliability risks, etc.• Reliability affected by
• Internal faults: count, density, distribution• Usage, characterized by OP (operational profiles)
• Risk-based reliability improvement• Internal risk to software organizations:
• Defect risk identification & reduction.• Metrics-defect predictive modeling.• ODC and extensions.
Net-Centric Software & Systems I/UCRC
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
2013/07@Tianjin University, China
![Page 18: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/18.jpg)
18
Agenda
Quality and Empirical Observations
Quality Improvement Risk identification and prioritization
Risk resolution and quality improvement
Empirical Validation
New Trends and Directions Net-Centric, Service-Oriented, Cloud Computing, etc.
Questions, comments…
Net-Centric Software & Systems I/UCRC2013/07@Tianjin University, China
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
![Page 19: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/19.jpg)
19
Empirical/Risk-Based Quality Improvement
• Understanding the application domain and data
• Selecting quality/risk perspectives and related attributes (risk factors)
• Quantification and measurement
• Risk identification
• Risk resolution
• Empirical validation
• Implementation and deployment
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
Net-Centric Software & Systems I/UCRC2013/07@Tianjin University, China
![Page 20: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/20.jpg)
20
Related Approach: Basili’s QIP/GQM/EF
• QIP (quality improvement paradigm):• Step 1: understand baseline• Step 2: change then assess impact• Step 3: package for improvement
• EF (experience factory), separate from development organization, to support QIP implementation
• GQM (goals/questions/metrics) to guide measurement/analysis activities
• Our ESQE: more streamlined process with more attention to empirical data/analysis/validation and focus on risks and closing the loop
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
Net-Centric Software & Systems I/UCRC2013/07@Tianjin University, China
![Page 21: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/21.jpg)
21
Our ESQE: Evolution
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
Net-Centric Software & Systems I/UCRC2013/07@Tianjin University, China
Our ESQE:
•streamlined process
•empirical data/analysis/validation
•risk focus
•close loop
![Page 22: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/22.jpg)
22
Our ESQE: Details
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
Net-Centric Software & Systems I/UCRC2013/07@Tianjin University, China
![Page 23: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/23.jpg)
23
Approach: Risk Identification/Resolution
• Risk identification (where to focus)• Associating risk with other entities
• Risk = what we care about, trying to avoid• Others might be readily available or available early• Risk areas and their characteristics
• Establishing predictive relations• Many techniques can be used (later, with examples)
• Risk resolution (what to do)• Remedial actions for current project• Preventive/proactive actions for future projects
• Focus on high-risk/high-leverage areas
Net-Centric Software & Systems I/UCRC
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
2013/07@Tianjin University, China
![Page 24: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/24.jpg)
24
Risk Identification Techniques
• Qualitative risk identification techniques • Causal analysis: human intensive root cause analysis and variations• Risk indices, etc.• Suffering from various shortcomings, and not the focus of this
presentation
• Quantitative risk identification techniques:• Old/traditional statistical: correlation, regression, etc.• New (statistical): PCA/DA, TBM, etc.• AI/learning: NN, OSR, etc.• Survey in Tian/SQE book• Focus of this presentation, with many examples to come
Net-Centric Software & Systems I/UCRC
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
2013/07@Tianjin University, China
![Page 25: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/25.jpg)
25
Application Domain• Government: defense, aerospace and aeronautics
• NASA/SEL (Software Engineering Laboratory)• Early work (pre-1992) at U. Maryland
• Axiomatic framework for complexity
• Selection procedure
• Used for better effort prediction, improved results from earlier work by Selby and Porter
• Recent work (more details later)• Lockheed-Martin: quality and productivity improvement
• Raytheon and Boeing: components and services improvement
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
Net-Centric Software & Systems I/UCRC2013/07@Tianjin University, China
![Page 26: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/26.jpg)
26
Application Domain• Commercial software
• IBM (1992-1995): in house work • IBM-SMU collaboration: continuation after 1995• Other companies: HP, TI, etc., associated with our NSF Net-
Centric I/UCRC
• Application of risk-based approach:• UBST (usage-based statistical testing) guided by OP
(operational profiles) for compilers• Reliability modeling and improvement for DBMS etc.• ODC (orthogonal defect classification) and extensions• Defect/metrics analysis and modeling
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
Net-Centric Software & Systems I/UCRC2013/07@Tianjin University, China
![Page 27: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/27.jpg)
27
Application Domain• Telecommunications software and systems
• Nortel, Verizon, and telecommunications companies affiliated with NSF Net-Centric I/UCRC
• Application of risk-based approach to Nortel (pre-2002):• Some replication of IBM work
• formal hypothesis testing HC-HD?• characterization of HD modules
• Application to Verizon:
• E-commerce service quality improvement
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
Net-Centric Software & Systems I/UCRC2013/07@Tianjin University, China
![Page 28: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/28.jpg)
28
Application Domain• Web-based systems
• Web quality characterization and measurement• Web testing, particularly usage-based, risk-focused• Diversity: .edu, .org, .com, social networking, e-commerce, etc.
• Open source software• Some replication of IBM/Nortel work
• Also in connection to web-based systems• Safety-critical systems:
• Nuclear industry case studies• Also in connection with government/DoD/NASA work
• Others ( new trend/direction later)
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
Net-Centric Software & Systems I/UCRC2013/07@Tianjin University, China
![Page 29: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/29.jpg)
29
Risk/Reliability Focus: Important Usage
• Focus on functions/modules/features with• High usage frequency and importance
• Non-uniform testing effort
• Usage-based statistical testing (UBST)
• Other focused quality assurance, etc.
• UBST = OP-guided testing• Capture user/usage information• Usage model = Operational prole (OP)• SRMs: Testing results to estimate reliability• New applications in web, etc.
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
Net-Centric Software & Systems I/UCRC2013/07@Tianjin University, China
![Page 30: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/30.jpg)
30
Risk/Reliability Focus: Important Usage
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
Net-Centric Software & Systems I/UCRC
Traditional testing techniques: – functional (black-box) or structural (white-box) coverage
New testing techniques: – usage-based statistical testing (UBST)
– risk-based testing, etc.
Testing effectiveness/efficiency captured by defect discovery profile
desirable: curve bent towards upper-left corner
Claim: Risk-based testing more effective/efficient
2013/07@Tianjin University, China
![Page 31: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/31.jpg)
31
UBST with Operational Profiles (OP)
Net-Centric Software & Systems I/UCRC
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
• Web usage modeling• High-level: Musa’s
operational profile • Item and prob.• Sorted• Focused
testing• Medium-level• Low-level
• Example OP for SMU/SEAS
2013/07@Tianjin University, China
![Page 32: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/32.jpg)
32
UBST Guided by OP
Net-Centric Software & Systems I/UCRC
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
• Web usage modeling• High-level• Medium-level:
Markov chains (UMMs)
• Navigation• Transition
probabilities• Low-level
• Example OP for SMU/SEAS
2013/07@Tianjin University, China
![Page 33: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/33.jpg)
33
UBST for Web Applications• Top-level: Musa’s operational profile (list of operations/hits sorted by hit count)
• Middle-level: Unified Markov Models (UMMs) to capture navigation
information for major operations and/or clusters of web-based functions
• Bottom-level: Pre-existing coverage testing of individual web components
Net-Centric Software & Systems I/UCRC
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
2013/07@Tianjin University, China
![Page 34: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/34.jpg)
34
Risk-Based Testing: TBRM => Failure Clusters
Tree-based reliability model(TBRM, Tian 1995)
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
Net-Centric Software & Systems I/UCRC
– Relate input and timing to failures by
recursive
partitioning
– Subset reliability id. low reliability subsets for remedial actions
=> reliability
improvement
2013/07@Tianjin University, China
![Page 35: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/35.jpg)
35
Risk-Based Testing: TBRM for IBM Products
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
Net-Centric Software & Systems I/UCRC
Commercial software products from IBM– A, B, C: TBRM not used
– D: TBRM in system test
– similar pattern for
subsequent products Normalized comparison:
– Compare shape only
– Quantitative ρ:
purification level
D: ρ=0.94
A,B,C: ρ=0.65, 0.53, 0.45
2013/07@Tianjin University, China
![Page 36: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/36.jpg)
36
Risk-Based Testing of Web Applications
Web sites and web-based applications become important part of modern life and business– Massive user population
– Initially document-focused
– More “computation” capabilities added in e-commerce/cloud-computing and other new web-based applications
– Reliability, security, and usability as primary quality attributes
– Heterogeneous components: diverse types of testing Risk-based web testing:
– Web log and defect data analysis
– Testing prioritized by defect rate ranking
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
Net-Centric Software & Systems I/UCRC2013/07@Tianjin University, China
![Page 37: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/37.jpg)
37
Risk-Based Web Testing Techniques Data sources
– Web logs: Server access and error logs, application etc.
– Defect database and repositories
– Additional data collection possible: e.g., ODC (orthogonal defect classification) adapted to the web (Ma and Tian, 2007)
Risk analysis and identification: – Risk score = defect rate, two variations
#failures/#accesses #unique-failures / #files/documents (or #unique-accesses)
– Testing prioritized by defect rate ranking
– Combine data from multiple sources (Li/Alaeddine/Tian, 2010)
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
Net-Centric Software & Systems I/UCRC2013/07@Tianjin University, China
![Page 38: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/38.jpg)
38
Test Prioritization Using Combined Data
• Fault view: unique failures from web logs
• Failure view: impact/usage model => falures from faults
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
Net-Centric Software & Systems I/UCRC2013/07@Tianjin University, China
![Page 39: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/39.jpg)
39
Risk/Safety Focus: Problem Sources
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
Net-Centric Software & Systems I/UCRC
• Physical and logical frames
• Focus on interface and interaction problems
• Empirical evidence for it
2013/07@Tianjin University, China
![Page 40: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/40.jpg)
40
Risk/Safety Improvement
Net-Centric Software & Systems I/UCRC
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
• Analyze sources of hazard: frame inconsistencies and sub-types
• Derive systematic assertions or prescriptive specifications
• Dynamically check these assertions
2013/07@Tianjin University, China
![Page 41: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/41.jpg)
41
Risk Focus: Defect-Prone Areas
Net-Centric Software & Systems I/UCRC
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
• Analyze metrics-defect relations• Correlation analysis• Regression models:
linear model here• Impact: behavior
modification (later: HT to confirm)
• Shortcomings: many, leading to other risk id. models
2013/07@Tianjin University, China
![Page 42: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/42.jpg)
42
Risk Focus: Defect Reduction
Net-Centric Software & Systems I/UCRC
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
• TBDM (Tree-based defect model)
• High-defect clusters
• Characterization• Lead to
remedial actions
• Drive quality improvement
2013/07@Tianjin University, China
![Page 43: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/43.jpg)
43
Agenda
Quality and Empirical Observations
Quality Improvement Risk identification and prioritization
Risk resolution and quality improvement
Empirical Validation
New Trends and Directions Net-Centric, Service-Oriented, Cloud Computing, etc.
Questions, comments…
Net-Centric Software & Systems I/UCRC2013/07@Tianjin University, China
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
![Page 44: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/44.jpg)
44
General Validation Issues• Case studies: empirical basis
• In lab: controlled environment
• In production/operational environment
• Controlled (scientific!) experiment/simulation• Experimentation: role of research labs
• Simulation: less expensive, some web results presented earlier
• Data and analysis via analysis of variance, hypothesis testing, etc.
• Implementation and deployment
• All results need to be carefully analyzed
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
Net-Centric Software & Systems I/UCRC2013/07@Tianjin University, China
![Page 45: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/45.jpg)
45
Validation: Result Analysis• General analysis
• Distribution comparison: mean, variance, etc.• Trend analysis, etc.• Careful with causal relation vs. accidental correlation
• Formal hypothesis testing (HT)• More rigorous, powerful and better statistical validity
• HT study by Koru and Tian (TSE 2005):• High-defect (HD) modules vs. high-complexity (HC) modules• HD and HC statistically different• Complexity ranking of HD: 60-80%
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
Net-Centric Software & Systems I/UCRC2013/07@Tianjin University, China
![Page 46: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/46.jpg)
46
Validation: HT by Koru/Tian (TSE 2005)
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
Net-Centric Software & Systems I/UCRC2013/07@Tianjin University, China
|z| critical values at 1.96 and 2.58 for significance levels of 5% and 1% respectively=>All null hypothesis rejected
![Page 47: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/47.jpg)
47
Validation: HT by Koru/Tian (TSE 2005)
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
Net-Centric Software & Systems I/UCRC2013/07@Tianjin University, China
|z| critical values at 1.96 and 2.58 for significance levels of 5% and 1% respectively=>All null hypothesis rejected
![Page 48: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/48.jpg)
48
Validation: HT by Koru/Tian (TSE 2005)
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
Net-Centric Software & Systems I/UCRC2013/07@Tianjin University, China
HC vs HM clusters=>Majority HC different from HM but inconclusive HT result;HC metrics ranking around 80th percentile
![Page 49: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/49.jpg)
49
Risk-Based Testing: Validation
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
Net-Centric Software & Systems I/UCRC
Commercial software products from IBM– A, B, C: TBRM not used
– D: TBRM in system test
– similar pattern for
subsequent products Normalized comparison:
– Compare shape only
– Quantitative ρ:
purification level
D: ρ=0.94
A,B,C: ρ=0.65, 0.53, 0.45
2013/07@Tianjin University, China
![Page 50: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/50.jpg)
50
Risk-Based Web Testing: Experiments Diverse web sites/applications
– Academic web site: seas.smu.edu– Open source project site: kde.org– Small company catalog showroom web site– Large telecommunications company service ordering site– Social networking web site(s)
Experimental setup: training (1st ½) vs. testing (2nd ½) data– Training data log/defect analysis: test prioritization– Testing data: simulated behavior by processing actual data
Defect profile compared to those based on coverage/checklists/hierarchies/etc.
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
Net-Centric Software & Systems I/UCRC2013/07@Tianjin University, China
![Page 51: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/51.jpg)
51
Risk-Based Web Testing: Validation
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
Net-Centric Software & Systems I/UCRC
Academic web site:
seas.smu.edu– risk-based testing
better than
directory level:increasingdecreasingrandom
dictionary order + reverse order
2013/07@Tianjin University, China
![Page 52: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/52.jpg)
52
Risk-Based Web Testing: Validation
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
Net-Centric Software & Systems I/UCRC
Open source web site kde.org– risk-based testing better
– additional analysis pre/post
significant change
2013/07@Tianjin University, China
![Page 53: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/53.jpg)
53
Risk-Based Web Testing: Validation
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
Net-Centric Software & Systems I/UCRC
Small company catalog showroom on the web– risk-based testing better than other orders (similar results)
2013/07@Tianjin University, China
![Page 54: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/54.jpg)
54
Risk-Based Web Testing: Validation
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
Net-Centric Software & Systems I/UCRC
Large telecommunications company online service ordering– risk-based testing better (similar results)
2013/07@Tianjin University, China
![Page 55: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/55.jpg)
55
Risk-Based Web Testing: Validation
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
Net-Centric Software & Systems I/UCRC
Risk identification for a social networking web site
Risk AreaError number(Training)
Error Percent(Training)
Error number(Testing)
Error Percent(Testing)
None 0 0% 11 10.48%Crash 5 1.53% 3 2.85%Function Failure 266 81.35% 74 70.48%Function Workaround 36 11.00% 5 4.76%User Interface 20 6.12% 8 7.62%minorreq 0 0.00% 4 3.81%All 327 100.00% 105 100.00%
Social networking web site– Risk-based testing better than other orders in later ranking too
2013/07@Tianjin University, China
![Page 56: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/56.jpg)
56
Agenda
Quality and Empirical Observations
Quality Improvement Risk identification and prioritization
Risk resolution and quality improvement
Empirical Validation
New Trends and Directions Net-Centric, Service-Oriented, Cloud Computing, etc.
Questions, comments…
Net-Centric Software & Systems I/UCRC2013/07@Tianjin University, China
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
![Page 57: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/57.jpg)
57
New Trend and New Research• Net-centric operations (NCO) and net-enabled
capabilities (NEC)• US DoD and defense industry (UK and Europe too)• Goal is to achieve information superiority through (dynamic)
integration of people, (changeable, often hostile) environment, and (fluid) infrastructure
• Service-oriented computing and service-oriented architecture (SOA)• Commercial industry driven, particularly web-based business• Similar concerns as NCO, quality/performance/etc. through SLA
(service level agreement)
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
Net-Centric Software & Systems I/UCRC2013/07@Tianjin University, China
![Page 58: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/58.jpg)
58
New Trend and New Research• Cloud computing and related software/system issues:
• Service provided via cloud instead of locally
• Shares many similar concerns with NCO and particularly SOA, but with more of a focus over the overall infrastructure instead of services only
• Recent (starting Sept. 2011) project funded by NSF• Instrumentation to measure dependability and quality of cloud
computing systems
• Collaboration among SMU, UTD and UNT, founding members of NSF Net-Centric I/UCRC
• Total funding $1.4+ million over 3 years
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
Net-Centric Software & Systems I/UCRC2013/07@Tianjin University, China
![Page 59: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/59.jpg)
59
New Trend and New Research• Many common issues among NCO, SOA, and CLOUD:
• Complex, dynamically composed systems
• Integration of diverse components (including people too)
• Quality/dependability measurement, evaluation, and assurance at both the component and system level
• Representing concerns of our NSF I/UCRC member companies:• NCO/SOA/CLOUD for Lockheed-Martin, Raytheon, Boeing, etc.
• SOA/CLOUD for EDS, TI, Cisco, etc.
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
Net-Centric Software & Systems I/UCRC2013/07@Tianjin University, China
![Page 60: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/60.jpg)
60
Existing Solutions• V-model for products to reduced V-model for
components used in NCO/SOA/CLOUD
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
Net-Centric Software & Systems I/UCRC2013/07@Tianjin University, China
![Page 61: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/61.jpg)
61
Existing Solutions• Support for measurement, analysis, and reuse
through experience factory for whole products
Net-Centric Software & Systems I/UCRC
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
2013/07@Tianjin University, China
![Page 62: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/62.jpg)
62
Our Solution for NCO/SOA/Cloud
• Evaluator for NCO/SOA/CLOUD system components and services based on multiple perspectives and realistic application scenarios
• A testbed for application scenario modeling/simulation• Normal scenario captured in operational profiles for testing• Perturbation and extreme/hostile condition modeling via boundary
extension, mutation, fault inject, security threat simulation, etc.
• A multi-facet/perspective evaluator• External quality/dependability attribute identification/evaluation• Internal quality characterization and empirical based mapping• Multivariate optimization using data envelopment analysis• Quantify relative value and tradeoffs for optimal decision making
Net-Centric Software & Systems I/UCRC
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
2013/07@Tianjin University, China
![Page 63: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/63.jpg)
63
Our Solution: Tool Suite Architecture
Net-Centric Software & Systems I/UCRC
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
2013/07@Tianjin University, China
![Page 64: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/64.jpg)
64
Our Solution: Experimental Plan
• Overall: incremental/iterative approach• Risk-/opportunity-based identification of increments/iterations
• Continuous adjustment and improvement
• Each increment/iteration includes the following steps• Stakeholder identification
• Stakeholder concerns and priorities to derive context-sensitive external quality/dependability attribute definitions as well as value assessments
• Data collection and testbed/evaluator construction
• Experimental self-validation
Net-Centric Software & Systems I/UCRC
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
2013/07@Tianjin University, China
![Page 65: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/65.jpg)
65
Quality/Dependability Evaluation: Reliability
• Operational reliability captured by failure rate per time-unit or op. instances
• Reliability growth potential evaluated by unique failure sequence analysis
• Time/operation measurement: #users, #sessions, #hits, #bytes for web services
Net-Centric Software & Systems I/UCRC
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
2013/07@Tianjin University, China
![Page 66: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/66.jpg)
66
Defect Analysis and Prioritization• Defect analysis/prioritization based on web logs and other project documents
• Adaptation of ODC (orthogonal defect classification) to SOA/NCO/etc.
• Key attributes: response code, file type, trigger/referrer, time,
• Work by Alaeddine and Tian
Net-Centric Software & Systems I/UCRC
0%5%
10%15%20%25%30%35%
% o
f T
ota
l
Fault class
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
2013/07@Tianjin University, China
![Page 67: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/67.jpg)
67
Multivariate OptimizationDEA (data envelopment analysis) in HASE’2011 paper by Siok &Tian
Net-Centric Software & Systems I/UCRC
InputsInputs OutputsOutputs
Software Reliability At ReleaseDefect Density-1
Defects Removed
EfficiencyEfficiencyOutput / Input
Software Total SizeSoftware Change SizeSoftware Development HoursSoftware Test HoursSoftware Schedule
InputsInputs OutputsOutputs
Software Reliability At ReleaseDefect Density-1
Defects Removed
EfficiencyEfficiencyOutput / Input
Software Total SizeSoftware Change SizeSoftware Development HoursSoftware Test HoursSoftware Schedule
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
2013/07@Tianjin University, China
![Page 68: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/68.jpg)
68
Summary
Quality and Empirical Observations
Quality Improvement Risk identification and prioritization
Risk resolution and quality improvement
Empirical Validation
New Trends and Directions Net-Centric, Service-Oriented, Cloud Computing, etc.
Questions, comments…
Net-Centric Software & Systems I/UCRC2013/07@Tianjin University, China
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大
This approach is proven to be effective
This new research looks promising
![Page 69: Empirical Software Quality Engineering: Experience and Recent Development Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRC Professor, CSE.](https://reader036.fdocument.pub/reader036/viewer/2022062518/56649e215503460f94b0d2db/html5/thumbnails/69.jpg)
69
Agenda
Quality and Empirical Observations
Quality Improvement Risk identification and prioritization
Risk resolution and quality improvement
Empirical Validation
New Trends and Directions Net-Centric, Service-Oriented, Cloud Computing, etc.
Questions, comments…
Net-Centric Software & Systems I/UCRC2013/07@Tianjin University, China
Empirical Software Quality EngineeringJeff Tian, SMU & 西工大