Test Case Success Rate
Introduction
Introducing the success rate for test cases, calculated based on their execution history, holds significant importance in the testing process. The success rate provides QA teams with a quantitative measure of the reliability and effectiveness of the test cases. The test case success rate is a crucial score for measuring test case execution performance, as a high success rate indicates a robust and dependable test case. On the other hand, a low success rate highlights potential issues or areas of improvement, signaling the need for further investigation and refinement.
Formula to Calculate Success Rate
The Success Rate of a test case is calculated based on the pass rate of the test case. It helps the QA team determine the effectiveness of the test case and facilitates test planning.
Success Rate = (Number of times the test case version got passed/Number of times the test case version got executed as Passed and Failed) * 100
The Status definition for Passed and Failed is defined by the admin on the configuration screen.
Success Rate Instance Level Settings
These are the settings for the instance. To activate the AI feature for users, the super administrator must enable it for the instance in the Customization module under General Settings & Audit, then open AI Configuration.
Note
Only the super admin can access and configure Intelligence Configuration.
To enable this feature, go to AI Configuration and select Flaky Score and Success Rate Configuration (Default: Disabled)
There are two settings for Flaky Score and Success Rate, each.
Allow Project Admins to Enable & Configure the Flaky Score for their project
Allow Project Admins to Enable & Configure the Success Rate for this project
Enable the Allow Project Admins to Enable & Configure the Success Rate for this project settings for Success Rate calculations.
 |
Success Rate Settings Project Level Settings
Prerequisites
Project administrators must configure the Success Rate Settings if the option Allow Project Admins to Enable & Configure the Success Rate for their project is enabled from Customization's General Settings and Audit's AI Configuration tab.
Required Permissions
Project Administrators and users with Success Rate - Modify or Success Rate - Generate permissions can configure the Success Rate settings.
Users with only the Success Rate - View permission can view the configuration, but cannot make any changes.
Steps
To enable Success Rate Settings, perform the following steps:
Go to Intelligence.
Select Success Rate Settings.
Open the Configurations tab.
Turn on the Enable Success Rate for the Project.
Once the Success Rate is enabled, the fields for Success Rate Configuration appear editable.
Success Rate Configuration
The settings on this page allow Administrators to define the scope of test executions for which the Success Rate will be calculated. The criteria should be refined to calculate the Success Rate based on the QA reporting methodologies.
AI will calculate the Success Rate based on the filtered test execution records according to the criteria. The existing Success Rate will be reset and recalculated whenever the configuration is modified.
Provide the following details for the Success Rate configuration.
Test Case Executions: Mention the number of the latest test executions of a test case you want to consider while calculating the success rate. 10 test executions are selected by default. The Test Case Execution value should be from 1 to 1000.
Test Case Version: Latest Version or All Versions, while calculating the success rate. The Latest Version is selected by default.
Latest Version: Only the test executions of the latest test case versions will be considered when calculating the Success Rate.
All Versions: The test executions of all test case versions will be considered while calculating the Success Rate.
Execution Time Frame: Mention the days to define the timeframe in which the test executions were executed in the last specified days. By default, 90 days are selected, indicating that test executions carried out during the previous 90 days will be considered when calculating the Success Rate. If left blank, it will be considered as all the days.
Release: Select Release(s) to consider test executions for those releases.
Allreleases are selected by default.Cycle: Select Cycle(s) to consider test executions for those cycles.
Allcycles are selected by default.Platform: Select Platform(s) to consider test executions for those platforms.
Allplatforms are selected by default.Build: Select build(s) to consider the test executions about that particular build(s). \
Allbuilds are selected by default.Test Case Execution Status (All execution statuses other than selected as passed and failed will be considered as Skipped. Skipped status will not be considered for calculation)
As Passed: The test execution with these statuses will be considered
Passedwhen calculating the Success Rate. The default value isPassed.As Failed: The test executions with these statuses will be considered
Failedwhen calculating the Success Rate. The default values areFailedandBlocked.
Click Update to save the Success Rate Settings.
To reset the settings, click the Reset button.
To calculate the success rate, click the Generate button.
The success message pops up.
Note
The Success Rate can be calculated once in 24 hours for each project. Once you generate the Success Rate, the Generate button will remain disabled for the next 24 hours.
When you hover over the Generate button, it displays the details of the user who last generated the Success Rate, along with the timestamp.
 |
Once you generate the Success Rate on the Success Rate Settings screen, you can view the success rate on the test case list view, test case detail page, test case link screens, linked test cases screen, and test case execution screens.
Display of Success Rate
The following table interprets the success ratio in accordance with the display of the derived success rate calculation and color code.
Success Rate Range | Success Ratio |
|---|---|
Between 81 to 100 % | High |
Between 51 to 80 % | Medium |
Between 0-50 % | Low |
Filter Test Cases on Success Rate
The test cases can be filtered by success rate. You can find the Success Rate under the Advanced Filters.
View Success Rate on Test Case List View
The Success Rate is calculated based on the configuration set in Success Rate Settings and is displayed on the Test Case List View.
To view the Success Rate, perform the following steps:
Go to the Test Case module.
Ensure the Success Rate column is enabled in the list view.
About Success Rate
The Success Rate is calculated based on the pass rate of a test case. The Success Rate column provides the following details and actions:
Displays the calculated success percentage for each test case.
Allows you to sort test cases to view those with the highest success rates.
Clicking the Success Rate value opens a Traceability Report, which shows the test case’s association with test executions, issues, and execution results. This helps analyze the Success Rate further.
An information icon beside the column displays the details of when the Success Rate was last generated and by whom.
If the Success Rate Settings are changed, the existing Success Rate values will reset.
View Success Rate on Test Case Detail Page
The generated Success Rate is displayed beside the Test Case Key at the top of the screen.
 |
View Success Rate on Test Case Link Screens
You can show or hide the Success Rate column on the Link Test Cases screen.
View Success Rate on Test Execution Screen
According to the Success Rate Settings set in the Configuration, the Success Rate is calculated and displayed accordingly.
Go to the Test Execution screen.
Test Execution Detail View
 |
If you go further into the Detail View, you can view the Success Rate count at the top of the page.
Test Execution Default View
You can show or hide the Success Rate column on the execution screen.
Click the Success Rate to drill down to view the test executions and defects associated with the test case.