Test Metrics - Description and Definition

Severity wise Defect distribution % :
Description: This report shows severity wise defect distribution
Definition: Severity 1 defect % = (Total number of Severity 1 defects/Total number of defects) *100 and so on

Status wise Defect distribution % :
Description: This report shows State wise defect distribution
Definition: Closed Defect % = (Total number of Closed Defects /Total number of defects )*100 and so on for other status of the Defects

Regression Defect % :
Description:This shows the ability to keep the product right while fixing defect
Definition: Regression % = ( no of regression bugs )*100 / Total no of bugs

Reopen defect %:
Description: Percentage of reopened bugs against total bugs
Definition: Reopen bugs % = (reopened bugs) * 100/ total bugs

Invalid / Duplicate defect %:
Description:Percentage of invalid and duplicate bugs against total bugs
Definition: Invalid & Duplicate bugs % = (invalid bugs + Duplicate bugs) * 100/ total bugs

Defect distribution %:
Description: This metric shows a defect distribution per different kind of testing eg Functional testing, Regression testing, Usability testing, Documentation, Installation/Uninstallation Testing etc
Definition: % of Defect out of Functional testing =(Number of defects produced by functional testing /The total defects) *100

Root Cause Analysis on Customer found Defect and % distribution:
Description: This report shows a root cause analysis and percentage distribution of the customer found defects
Definition: Distribute it % wise as (Test case missing), (Platform not supported), (Not clear Documentation), (Not clear requirement), (Platform not covered) etc

Cumulative defects fixed per week:
Description: The shows the cumulative volume of defects that gets fixed on a weekly basis. An input to the testing team to plan ahead for the validation effort
Definition: Sum of all defects fixed in the week+sum of all defects fixed till that week.

Defect Validation %:
Description: This report shows (Closed+Reopen) vs Fixed defects
Definition: (No of validated (closed + reopen) defects/No of fixed defects) *100

Defect Validation Rate:
Description:No of defects validated per week. This is a measure of QA productivity in terms of validating the fixed defects
Definition: No of defects validated per week per person = (total no. of validated defects) *40/(Total hours to validate the defects * no. of resources)

Cumulative defects closed per week:
Description: The shows the cumulative volume of defects that gets closed on a weekly basis.
Definition: Sum of all defects closed in the week+sum of all defects closed till that week.

Bug Trend and Forecasting:
Description: This shows the inclination of uncovering a bug based on the past data and thus extrapolate it to forecast a value for future
Definition: E.g. Past 3 weeks moving average of weekly number of bugs

Defect ageing report:
Description: This shows how many bugs are still open on different time period slots.
Definition: No of open bugs with <> 3 months old etc as applicable

Defect Density:
Description: This shows the number of bugs found per per months of dev effort
Definition: (Total no. of bugs) * 160/(Total man hours of planned dev effort)

Defect Removal Efficiency (DRE):
Description: This shows the efficiency of QA in finding bugs against the bugs found by customers
Definition: DRE = (Pre release Sev 1&2 bugs) * 100/ (Pre release sev 1&2 bugs + post Release sev 1&2 bugs)

Test Coverage %:
Description:This report shows the extent to which testing covers the product’s complete functionality
Definition: (Total number of Test cases executed/Total number of Test cases available across all the platforms )*100

Test design per person week:
Description: No of test cases written per week classified into size or complexity based categories. This is a measure of QA productivity during the TC definition stages.
Definition: TC created per week per person = (total no. of TCs created) *40/(Total hours to wrting TCs * no. of resources)

Test Case effectiveness:
Description: This metric shows an indication of the effectiveness of the test cases.
Definition: (Number of Test cases produced defects/. the total number of test cases) *100

Status wise Test Case execution distribution:
Description: This metric gives an indication of status of Test case execution
Definition: "Passed Test Cases %=(No of Passed Test cases/Total Number of Test cases)*100 and so on forFailedBlockedNot yet Executed Test cases"

Cumulative test execution per week:
Description: The shows the cumulative no. of test cases that gets executed on a weekly basis.
Definition: Sum of all test cases executed in the week+sum of all test cases executed till that week.

Test execution per person week:
Description: No of Test cases or Test steps executed per week. This is a measure of QA productivity during the Testing phases.
Definition: TCs executed per week per person = (total no. of TCs executed) *40/(Total hours to execute TCs * no. of resources)

Review Comments on Test Plan:
Description: This shows the quality of the Test Plan
Definition: No of review comments received per Test Plan or comments can be distributed as High, medium, low

Review Comments on Test cases:
Description: This shows the quality of the Test Cases
Definition: No of review comments received per Test Case or comments can be distributed as High, medium, low

Automated Test Cases %:
Description: This report shows Automated vs total Test Cases
Definition: (No of automated Test case/Total no of Test cases to be automated)*100 %

Automation of TC rate:
Description:No of Test cases automated per week. This is a measure of QA productivity in terms of automating the test cases
Definition: TCs automated per week per person = (total no. of TCs automated) *40/(Total hours to automate TCs * no. of resources)

Resource Utilization:
Description: This shows how the time spent by the QA resources have been utilized.
Definition: (Time reported on productive QA task) / Standard time available for the period

Effort variance:
Description: This is the deviation in the actual effort as a percentage of the estimated planned effort.
Definition: (Actual effort hours - Planned effort hours) / Planned effort

Schedule variance:
Description: This is the deviation in the actual completion date from the planned completion date as captured in the project schedule.
Definition: Differential duration / Planned duration