Scrum QUALITY KPIs

KPI Name / Definition

Representation

Tool

Maturity Levels

KPI Calculation

On Hover

Remarks

Defect Injection Rate

measures the total number of defects (bugs) linked to closed stories which are part of a Iteration.

Line Graph

• Y- Axis - Percentage

• X- Axis - Sprints

• Aggregation Method - Average

• Positive trend indication - Decrease

Agile Project Management

• Jira

• Azure Boards

M1: > 175%

M2: 175% -125%

M3: 125%-75%

M4: 75-25%

M5: 25%

No. of defects tagged to all stories closed in an iteration/ Total no. of stories closed in the iteration.

• It is calculated as a ‘Percentage’. Lower the percentage, better is the ‘Quality’.

• Maturity of the KPI is calculated based on the average of last 5 iterations that corresponds with the maturity scale

Sprint Name: <<Percentage Value>>

Defects: <<Value>>

Closed Stories: <<Value>>

1. Defects are considered based on the field ‘Issue type to identify defect’ which falls under Jira/Azure Mappings--> Defect Mappings

2. Stories are considered based on the field ‘Defect Injection Rate - Issue Types with Linked Defect’ which falls under Jira/Azure Mappings--> Issue type mappings

3. Defects that have status defined as per ‘Defect Rejection status’ or ‘Resolution type for rejection’ under Workflow status mapping are excluded from calculation

First time pass rate

measures the percentage of tickets that pass QA first time (without stimulating a return transition or not having a linked defect)

Line Graph with multi-select filter

• Y- Axis - Percentage

• X- Axis - Sprints

• Aggregation Method - Average

• Positive trend indication - Increase

Agile Project Management

• Jira

• Azure Boards

M1 <25% 

M2 >=25-50% , 

M3 - >=50-75% , 

M4 - >=75-90%  , 

M5 - >=90%  , 

No. of issues closed in a iteration which do not have a return transition or any defects tagged/ Total no. of issues closed in the iteration.

• It is calculated as a ‘Percentage’. Higher the percentage, better is the ‘Quality’.

• Maturity of the KPI is calculated based on the average of last 5 values that corresponds with the maturity scale.

Sprint Name: <<Percentage Value>>

FTP Stories: <<Value>>

Closed stories: <<Value>>

1. Stories are considered based on the field ‘Issues types included in FTPR’ which falls under Jira/Azure Mappings--> Issue type mappings.

2. Issues closed status are considered based field ‘Issue Delivered Status - Velocity’ under Workflow status mapping. This field is mandatory

3. Defects that have status defined as per ‘Resolution type for rejection’ or ‘Defect Rejection status’ under Workflow status mapping and ‘Defect priority to exclude from FTPR' or 'RCA values to exclude from FTPR' are excluded from calculations.

Defect Density

measures ratio of the total number of defects (bugs) linked to closed stories and the size of a closed stories

Line Graph

• Y- Axis - Percentage

• X- Axis - Sprints

• Aggregation Method - Average

• Positive trend indication - Decrease

Agile Project Management

• Jira

• Azure Boards

M1: > 90%

M2: 60% -90%

M3: 25%-60%

M4: 10-25%

M5: <10%

No. of defects tagged to all stories closed in a sprint/ Total size of stories closed in the iteration

• The KPI is applicable only if the estimation is being done in 'Story Points'

• It is calculated as a ‘Percentage’. Lower the percentage, better is the ‘Quality’

• Maturity of the KPI is calculated based on the average of last 5 values that corresponds with the maturity scale

Sprint Name: <<Percentage Value>>

Defects: <<Value>>

Total Size of Closed stories : <<Value>>

1. Defects are considered based on the field ‘Issue type to identify defect’ which falls under Jira/Azure Mappings--> Defect Mappings

2. Stories are considered based on the field ‘QA Defect Density - Issue Types with Linked Defect’ which falls under Jira/Azure Mappings--> Issue type mappings.

3. Defects that have status defined as per ‘Defect Rejection status’ or ‘Resolution type for rejection’ under Workflow status mapping are excluded from calculation.

Defect Seepage Rate

measures the percentage of defects leaked from the current testing stage to the subsequent stage

Line Graph

• Y- Axis - Percentage

• X- Axis - Sprints

• Aggregation Method - Average

• Positive trend indication - Decrease

Agile Project Management

• Jira

• Azure Boards

M1 - >=90% 

M2 - >=75-90%

M3 - >=50-75% , 

M4 >=25-50% , 

M5 <25%   

No. of  valid defects reported at a stage (e.g. UAT)/ Total no. of defects reported in the current stage and previous stage (UAT & QA)

• It is calculated as a ‘Percentage’. Lesser the percentage, better is the ‘Quality’

• Maturity of the KPI is calculated based on the average of last 5 values that corresponds with the maturity scale

Sprint Name: <<Percentage Value>>

Escaped Defects: <<Value>>

Total Defects : <<Value>>

1. Escaped Defects are considered based on the field ‘UAT Defect Identification’ by Labels/CustomField configuration which falls under Jira/Azure Mappings--> Defect Mappings

2. Stories are considered based on the field ‘Defect Seepage Rate - Issue Types with Linked Defect’ which falls under Jira/Azure Mappings--> Issue type mappings.

3. Total defects is the count of all linked defects to the closed stories

Defect Removal Efficiency

gives a measure of efficiency of the development team in closing defects raised because of new functionalities within the iteration

Line Graph

• Y- Axis - Percentage

• X- Axis - Sprints

• Aggregation Method - Average

• Positive trend indication - Increase

Agile Project Management

• Jira

• Azure Boards

M1 <25% 

M2 >=25-50% , 

M3 - >=50-75% , 

M4 - >=75-90%  , 

M5 - >=90%  

No. of defects in the iteration that are fixed/ Total no. of defects in a iteration

• It is calculated as a ‘Percentage’. Higher the percentage, better is the ‘Quality’

• Maturity of the KPI is calculated based on the average of last 5 values that corresponds with the maturity scale

Sprint Name: <<Percentage Value>>

Closed Defects: <<Value>>

Total Defects : <<Value>>

1. Stories are considered based on the field ‘Defect Removal Rate - Issue Types with Linked Defect’ which falls under Jira/Azure Mappings--> Issue type mappings.

2. Defects that have status defined as per ‘Defect Removal Status’ under Workflow status mapping are considering as fixed defects.

3. Defects that have status defined as per ‘Defect Rejection status’ or ‘Resolution type for rejection’ under Workflow status mapping are excluded from calculation.

Defect Rejection Rate

measures the effectiveness of testing team

Line Graph

• Y- Axis - Percentage

• X- Axis - Sprints

• Aggregation Method - Average

• Positive trend indication - Decrease

Agile Project Management

• Jira

• Azure Boards

M1 - >=75%  ,

M2 - >=50-75%  , 

M3 - >=30-50% , 

M4 >=10-30% , 

M5 <10% 

No. of defects tagged to stories in the iteration that are rejected / Total no. of defects tagged to stories in a iteration

• It is calculated as a ‘Percentage’. Lesser the percentage, better is the ‘Quality’

• Maturity of the KPI is calculated based on the average of last 5 values that corresponds with the maturity scale

Sprint Name: <<Percentage Value>>

Rejected Defects: <<Value>>

Total Defects : <<Value>>

1. Stories are considered based on the field ‘Defect Rejection Rate - Issue Types with Linked Defect’ which falls under Jira/Azure Mappings--> Issue type mappings.

2. Defects that have status defined as per ‘Defect Rejection status’ or ‘Resolution type for rejection’ under Workflow status mapping are considered as Rejected defects

3. Total defects is the count of all linked defects to the closed stories

Defect Count by Priority

measures number of defects for each priority defined in a project

Line chart

• Y- Axis - Count

• X- Axis - Sprints

• Aggregation Method - Sum

• Positive trend indication - Decrease

Agile Project Management

• Jira

• Azure Boards

No Maturity Trend

No. of defects linked to stories in the iteration grouped by priority

• It is calculated as ‘Count’. Lower the count, better is the ‘Quality’

Sprint Name:

P1: <<Value>>

P2: <<Value>>

P3: <<Value>>

P4: <<Value>>

1. Stories are considered based on the field ‘Defect Count - Issue Types with Linked Defect--> Issue type mappings.

2. Defects that have status defined as per ‘Defect Rejection status’ or ‘Resolution type for rejection’ under Workflow status mapping are excluded from calculation

3. Total defects is the count of all linked defects to the closed stories

Defect Count By RCA

measures number of defects along with the root cause of defects

Line chart

• Y- Axis - Count

• X- Axis - Sprints

• Aggregation Method - Sum

• Positive trend indication - Decrease

Agile Project Management

• Jira

• Azure Boards

No Maturity Trend

No. of defects linked to stories in the iteration grouped by Root Cause

• It is calculated as ‘Count’. Lower the count, better is the ‘Quality’

Sprint Name:

RCA1: <<Value>>

RCA2: <<Value>>

RCA3: <<Value>>

RCA4: <<Value>>

1. Stories are considered based on the field ‘Defect Count - Issue Types with Linked Defect--> Issue type mappings.

2. Defects that have status defined as per ‘Defect Rejection status’ or ‘Resolution type for rejection’ under Workflow status mapping are excluded from calculation

3. Total defects is the count of all linked defects to the closed stories

Regression Automation Coverage

measures progress of automation of regression test suite in terms of percentage automated out of total regression testcases

Line Graph

• Y- Axis - Percentage

• X- Axis - Sprints

• Aggregation Method - Average

• Positive trend indication - Increase

Agile Project Management

• Zephyr

• Jira Test

• Data Upload (Release 8.0.0)

M1 - <20%

M2 - 20-40%

M3 - 40-60%

M4 - 60% - 80% , 

M5 - > 80%

No. of regression test cases automated till the end of iteration/Total no. of regression test cases available till the end of iteration

• It is calculated as a ‘Percentage’. Higher the percentage, better is the ‘Quality’

• Maturity of the KPI is calculated based on the average of last 5 values that corresponds with the maturity scale

Sprint Name: <<Percentage Value>>

Regression test cases automated: <<Value>>

Total Regression test cases: <<Value>>

1. Following tools can be set up to get data on Regression Automation Coverage kpi

   1. Zephyr (Server/Cloud) with Jira setup

   2. Jira Test (Zephyr squad/X-ray) with Jira setup

2. Total regression testcases and automated testcases will be fetched based on the configured field values.

3. Testcases may or may not be linked to stories.

In-Sprint Automation Coverage

measures progress of automation of test cases linked to stories within the iteration  

Line Graph

• Y- Axis - Percentage

• X- Axis - Sprints

• Aggregation Method - Average

• Positive trend indication - Increase

Agile Project Management

• Zephyr

• Jira Test

M1 - <20%

M2 - 20-40%

M3 - 40-60%

M4 - 60% - 80% , 

M5 - > 80%

No. of test cases linked to stories in iteration that are automated/Total test cases linked to stories in iteration

• It is calculated as a ‘Percentage’. Higher the percentage, better is the ‘Quality’

• Maturity of the KPI is calculated based on the average of last 5 values that corresponds with the maturity scale

Sprint Name: <<Percentage Value>>

Linked test cases automated: <<Value>>

Total linked test cases: <<Value>>

1. Following tools can be set up to get data on Regression Automation Coverage kpi

   1. Zephyr (Server/Cloud) with Jira setup

   2. Jira Test (Zephyr squad/X-ray) with Jira setup

2. Total test cases will be fetched based on the testcases of linked stories which falls under the iteration and automated test cases will be calculated based on configured field values.

Test Execution and pass percentage

measures the percentage of test cases that have been executed out of the total test cases with an iteration & the percentage that have passed.

Line Graph + Column chart

• Y- Axis - Percentage

• X- Axis - Sprints

• Aggregation Method - Average

• Positive trend indication - Increase

• Jira

• KnowHOW

M1 - <20%

M2 - 20-40%

M3 - 40-60%

M4 - 60% - 80%  

M5 - > 80%

No. of executed test cases out of total test cases in the latest execution of a sprint

No. of pass test cases out of executed test cases in  the latest execution of a sprint

• It is calculated in ‘Percentage’. Higher the percentage, better is the ‘Quality’

• Maturity of the KPI is calculated based on the average of last 5 values that corresponds with the maturity scale

Sprint Name: <<Percentage Value>>

Test Cycle: 

Executed test cases: <<Value>>

Pass test cases: <<Value>>

1. The information of total, executed and pass test cases needs to be uploaded in KnowHOW (Settings--> Upload data) for every iteration

2. The maturity value defined is for the pass percentage

Unit Test Coverage

is a measurement of the amount of code that is run by unit tests - either lines, branches, or methods

Line Graph with single-select filter

• Y- Axis - Percentage

• X- Axis - Weeks

• Aggregation Method - Average

• Positive trend indication - Increase

Code Quality

• Sonarqube

• Sonar Cloud

M1 - <20%

M2 - 20-40%

M3 - 40-60%

M4 - 60% - 80%  

M5 - > 80%

The calculation is done directly in Sonarqube

• It is calculated in ‘Percentage’. Higher the percentage, better is the ‘Quality’

• Maturity of the KPI is calculated based on the latest value

Date Range: << Percentage Value>>

1. Multiple branches can be configured for the same project

Sonar Violations

highlight count of issues that exist every time a piece of code breaks a coding rule. The set of coding rules is defined through the associated Quality profile for each programming language in the project.

Line chart

• Y- Axis - Count

• X- Axis - Weeks

• Aggregation Method - Sum

• Positive trend indication - Decrease

Code Quality

• Sonarqube

• Sonar Cloud

No Maturity Trend

The calculation is done directly in Sonarqube. 

Issues are categorized in 3 types: Bug, Vulnerability and Code Smells

All these issues are categorized into 5 type of severity: Blocker, Critical, Major, Minor, Info   

Date Range:

Blocker: <<Value>>

Critical: <<Value>>

Major: <<Value>>

Minor: <<Value>>

Info: <<Value>>

1. Multiple branches can be configured for the same project

Sonar Tech Debt

explains the estimated time required to fix all Maintainability Issues/ code smells

Line Graph

• Y- Axis - Days

• X- Axis - Weeks

• Aggregation Method - Sum

• Positive trend indication - Decrease

Code Quality

• Sonarqube

• Sonar Cloud

M1: >100 days 

M2: 50-100 days

M3: 30-50 days,

M4: 10-30 days,

M5: <10 days

It is calculated as effort to fix all Code Smells. The effort is calculated in minutes and converted to days by assuming 1 Day=8 Hours 

• It is calculated in ‘Days’. Lower the days, better is the ‘Quality’

• Maturity of the KPI is calculated based on the latest value

Date Range:<<Days>>

1. Multiple branches can be configured for the same project

Sonar Code Quality

is graded based on the static and dynamic code analysis procedure built in Sonarqube that analyses code from multiple perspectives.

Bar Graph

• Y-Axis - Code Quality

• X-Axis- Month

• Aggregation Method- Average

• Positive trend indication- Increase

Code Quality

• Sonarqube

• Sonar Cloud

M1: E 

M2: D

M3: C ,

M4: B,

M5: A

Sonar Code Quality is graded based on the static and dynamic code analysis procedure built in Sonarqube that analyses code from multiple perspectives.

• Code Quality in Sonarqube is shown as Grades (A to E).

• A is the highest (best) and

• E is the least

Date Range:<<Months>>

1. Multiple branches can be configured for the same project

Change Failure Rate

measures the proportion of builds (changes) that have failed for whatever reason over a given period of time

Line Graph with single-select filter

• Y- Axis - Percentage

• X- Axis - Weeks

• Aggregation Method - Average

• Positive trend indication - Decrease

Build

• Jenkins

• Bamboo

• Azure Pipelines

• Teamcity

M1: >50% 

M2: 30-50%,

M3: 20-30%,

M4: 10-20%,

M5: <10%

Total no. of failed Builds/Total number of Builds

• It is calculated as a ‘Percentage’. Lower the percentage, better is the ‘Quality’

• Maturity of the KPI is calculated based on the average of last 5 values that corresponds with the maturity scale

*If the KPI data is not available for last 5 weeks, the Maturity level will not be shown

Date Range: <<Percentage>>

Total number of Changes: <<Value>>

Failed Changes: <<Value>>

1. Multiple jobs can be configured for the same project

Created vs Resolved

gives an idea about the speed at which the team closes the raised defects.

Line Graph + Column chart

• Y- Axis - Count

• X- Axis - Sprints

• Aggregation Method - Sum

• Positive trend indication - Increase

Agile Project Management

• Jira

• Azure

M1: <20%

M2: 20-40%,

M3: 40-60%,

M4: 60-80%,

M5: >80%

Created defects consider all defects which are initially tagged to the Sprint or are added to the sprint in between

Closed defects are the completed defects i the sprint out of the created count

Created Defects - <<Value>>

Resolved Defects - <<Value>>

Resolved Percentage -