ReportPortal.io: пример CI/CD c метриками на основе продукта с...

Software quality assurance days

20 Международная конференция

по вопросам качества ПО

sqadays.com

Минск. 24–26 ноября 2016

Дмитрий ГуменюкEPAM Systems. Минск, Беларусь

ReportPortal.io: пример CI/CD c метриками на

основе продукта с Открытым Кодом

http://ReportPortal.ioпример CI/CD с метриками, на основе продукта с Открытым Кодом

Проект Менеджер

10 лет с компанией EPAM Systems.

Разработчик в прошлом

Возглавляю разработку инициатив внутри

Центра Компетенций Тестирования.

Report Portal – Product Owner.

DZMITRY HUMIANIUK

ДМИТРИЙ ГУМЕНЮК



10 лет с компанией EPAM Systems.

Разработчик в прошлом




DZMITRY HUMIANIUK

ДМИТРИЙ ГУМЕНЮК


10 лет с компанией EPAM Systems. Разработчик в прошлом





OPEN SOURCEDGPL v3.0

GITHUB.COM/ REPORTPORTAL


МЕТРИКИ

МЕТРИКИ “ПОЖАРНЫЕ ИЗВЕЩАТЕЛИ”

Name Why it is important to monitor

this

What decision can be made Periodi

city

Calculation

Execution

Frequenc

y

To understand and evaluate

how often test cases executed.

Goal: Increase execution ratio

per day/build.

Configure automated execution

of TA after commits (smoke,

acceptance) and regressions

right after nightly builds.

Triggers

1 execution per week or less

(???)

Target

1 per commit

Interpretation

code change should be

verified not to affect

functionality or other code

base.

Each

day

Measures

1. test executions count

Formula

Unit of measure

Number

% of

results

analyzed

To understand and evaluate

percentage of reviewed fail

reports. If fail report not

reviewed - there is not any

value

Triggers

<60%

Target

100%

Interpretation

failed report(test cases)

should be review

Each

day

Measures

1. No of not analyzed failed test cases (TI)

2. No of failed due to product bug (PB)

3. No of failed due to automation issues

(AB)

4. No of failed due to system issue (SI)

Formula

100% - To_Investigate / (PB+AB+SI+TI)

Unit of measure

%



Name Why it is important to

monitor this

What decision can be made Period

icity

Calculation

% of

product

issues

(stability of

automatio

n)

To understand and

evaluate percentage of

failed test which

corresponds to

production issues

Triggers

trend not being positive

Target

>90%

Interpretation

trend of fails due to test automation

instability should decrease (automation

issues, system issue, not analyzed failed

test cases)

Measures



3. No of failed due to automation issues (AB)


Formula

PB / (PB+AB+SI+TI)

Unit of measure

%

% of

automatio

n issues

To understand and


failed test which

corresponds to test

automation issues (code

issues, test case invalid)

Triggers


Target

<5%

Interpretation

trend of fails due to test automation

instability should decrease

Measures





Formula

AB / (PB+AB+SI+TI)

Unit of measure

%



Name Why it is important to

monitor this

What decision can be made Perio

dicity

Calculation

% of

system

issues

To understand and


failed test which

corresponds to system

issues (no connectivity,

lack of resources and

etc.)

Triggers


Target

<5%

Interpretation

trend of fails due to system issue

instability should decrease

Measures





Formula

SI / (PB+AB+SI+TI)

Unit of measure

%

Execution

Time

To understand and

evaluate time of each

execution cycle.

Triggers

Regression cycles > 15h

acceptance cycles >1h

Target

regression <8h

acceptance <1h

Interpretation

time for execution should not go

over described limits

Measures

1. Execution duration of regression, hours (Reg)

2. Execution duration of acceptance, hours (Acp)

Formula

Reg < 15h

Acp < 1h

Unit of measure

Hours



ReportPortal.io CommunityM eet-up

14 December 2016

19:00Imaguru, M insk


http:/ / ReportPortal.io

GITHUB.COM / REPORTPORTAL

Report Portal Community

ReportPortal.io Community

ReportPortal_io

ReportPortal.io: пример CI/CD c метриками на основе продукта с...

Education

Transcript of ReportPortal.io: пример CI/CD c метриками на основе продукта с...