Icst2012 zaman
25
A Large Scale Empirical Study on User-Centric Performance Analysis Shahed Zaman Ahmed E. Hassan Software Analysis and Intelligence Lab (SAIL) Queen’s University 1 Bram Adams MCIS École Polytechnique de Montréal Canada
Transcript of Icst2012 zaman
1
A Large Scale Empirical Study on User-
Centric Performance Analysis
Shahed Zaman Ahmed E. HassanSoftware Analysis and Intelligence Lab (SAIL) Queen’s University
Bram AdamsMCISÉcole Polytechnique de Montréal Canada
2
What is this study about?
Users
10 Requestsper user
Software System
1,000 Requests
10 RequestsBad Response Time
Requests
3
Scenario-Centric View
Users
Software System
1,000 Requests
10 RequestsBad Response Time
10 Requestsper user
1% bad request instance
4
User-Centric View
Users
Software System
1,000 Requests
10 RequestsBad Response Time
10 Requestsper user
0% bad request instance
50% bad request instance
1% bad request instance
User’s Perspective System’s Perspective
5
Data used in this study• 3 systems• 13 most used scenarios
Factor Enterprise System 1
Enterprise System 2
Dell DVD store
Functionality Telecommunications E-commerce
Vendor’s Business Model
Commercial Open-source
Size Ultra Large Large Small
Complexity Complex Complex Simple
6
Performance Load Test
Load Generator(s)
Requests
Requests
Requests
Software System Under Test
7
Performance Load Test
Runs for hours or days
Data
Performance AnalysisLoad Generator(s)
Requests
Requests
Requests
Software System Under Test
8
Current Practice of Performance Analysis
Data Values
Dens
ity Version 1
Data Values
Dens
ity Version 2
9
Load Generator(s)
Requests
Requests
Requests
Software System Under Test
Problem with Current Practice
Data aggregated for all users!!
Improved
Degraded
10
Scenario-Centric View
Load Generator(s)
Requests
Requests
Requests
Software System Under Test
PASS
11
User-Centric View
Load Generator(s)
Requests
Requests
Requests
Software System Under Test
User Group 1
U.G. # 2
U.G. # 3
U.G. # 4
FAILED
12
% of bad instances
Median
Median±
St.Deviation
Resp
onse
Tim
e
Request Instance #
13
% of bad instances
Median
Median±
St.Deviation
Bad instances = Out of “Median ± Standard Deviation” = 6/20 = 30%
Resp
onse
Tim
e
Request Instance #
14
Study
Users
Requests
Software System
User’s Perspective System’s PerspectiveVS
Data
Performance Analysis
15
Our Study Dimensions
Overall Trend Consistency
vs
16
Overall Performance
Old New% of bad instances 16.48 0.21
Scenario Centric View User Centric View
17
Our Study Dimensions
Overall Trend Consistency
vs
10 out of 13 use-cases showed a different view
18
Performance Trend Over TimeScenario Centric View User Centric View
OldNew
0 15 30 45 60
30
35
4
0
4
5Re
spon
se T
ime
Running Time0 20 40 60 80 100 120 140
40
60
80
100
1
20Re
spon
se T
ime
Instance # for a user
OldNew
Performance Trend Over Time
0 2 4 6 8 10 12 14
6
000
60
50 6
100
Resp
onse
Tim
e (m
ean)
Instance # for a user0 100 200 300 400 500 600
5850
59
00 5
950
600
0 6
050
610
0
Resp
onse
Tim
e
Running Time
System’s perspective
User’s perspective
19
20
Our Study Dimensions
Overall Trend Consistency
vs
10 out of 13 use-cases showed a different view
8 out of 13 use-cases showed a different
view
21
Performance Consistency
Scenario Centric View
0 500 1500 2500 3500
200
400
600
800
Re
spon
se T
ime
Running Time
OldNew
User Centric View
0 5 10 15 20 25 30-100
0
0 5
00
15
00
Resp
onse
Tim
e
Instance # for user
OldNew
NewOld
22
Our Study Dimensions
Overall Trend Consistency
vs
10 out of 13 use-cases showed a different view
8 out of 13 use-cases showed a different
view
All 13 use-cases showed a different
view
23
Trend
8 out of 13 use-cases showed a different
view
Our Study Dimensions
Overall
10 out of 13 use-cases showed a different view
Consistency
vs
All 13 use-cases showed a different
view
VS
Consistency vs Overall Performance
Inconsistent Consistent Bad Good
24
0 10 20 30 40 50 60 70
0
100
00
200
00
3000
0Va
rianc
e
% of bad instances
OldNew
Consistency vs Overall Performance
Overall Performance experience
Bad Good
1 2
3 4
C
I
cons
isten
cy
25
A complementary useful view
