View Flaky Score
The Flaky Score is a risk indicator that QMetry calculates for each test case based on the settings configured in the QMetry Intelligence module. It reflects the probability that a test case produces inconsistent results across executions, and it gives QA managers and testers a quick view of both the risk level and the test execution history.
You can view the Flaky Score in several places across the application: the Test Cases list view, the test case detail page, the Link Test Cases screen, and the Test Execution screen. This topic explains how to display and interpret the score in each location.
Note
Whenever the Flaky Score settings are changed in the QMetry Intelligence module, QMetry resets the existing Flaky Score and recalculates it using the new settings.
Pre-requisites
Before you begin, ensure the following conditions are met:
The Flaky Score settings are configured in the System Settings, and QMetry Intelligence module.
You have access to the Test Cases module and the Test Execution module.
Tracking Flaky Score
You can view the Flaky Score from the following locations:
Screen | Actions available |
|---|---|
Test Cases list view | View the score and its statistics for every test case, then sort, filter, and open the traceability report. |
Test case detail page | View the generated score next to the test case key at the top of the page. |
Link Test Cases screen | Show or hide the score while linking test cases. |
Test Execution screen | Compare pre-execution and post-execution results, and drill down to related executions and defects. |
View the Flaky Score on the Test Cases list view
The Test Cases list view displays the Flaky Score for each test case, along with its supporting statistics. To display the score, do the following:
Go to the Test Cases module.
Open the column settings for the list view, and select the Flaky Score column to make it visible.
Select Save to apply the column to the list view.
After you make the Flaky Score column visible, you can do the following:
Sort on the column to view the test cases with the highest Flaky Scores first.
Filter the test cases by Flaky Score from Advanced Filters.
Select a Flaky Score to open the traceability report for that test case.
Analyze the Traceability Report
When you select a Flaky Score, QMetry opens the Test Executions and Issues report for the test case. This report shows the test executions, issues, and execution results that the test case is associated with, so that you can analyze why the score was assigned.
The report presents two columns:
Test Executions, which lists each execution with its result, release, cycle, platform, build, executed version, execution date, and the person who ran it.
Issues, which lists each linked issue with its status, external key, summary, type, priority, and resolution.
Check when the Score was Last Generated
The information icon next to the Flaky Score column shows when the score was last generated and who generated it. If the score is out of date, QMetry indicates that it can be generated again based on the current configuration settings.
Filter Test Cases on Flaky Score
You can filter the test cases based on the Flaky Score. You can find the Flaky Score under the Advanced Filters.
View the Flaky Score on the Test Case Detail Page
The test case detail page displays the generated Flaky Score next to the Test Case Key at the top of the page. To view it, open any test case from the Test Cases list. The score appears as a badge beside the key, giving you the risk level at a glance while you review the test case steps, requirements, and executions.
View the Flaky Score on the Link Test Cases screen
You can show or hide the Flaky Score column on the Link Test Cases screen. Displaying the column while you link test cases helps you take each test case's risk level into account before you associate it with a requirement, execution, or other entity.
View the Flaky Score on the Test Execution screen
On the Test Execution screen, QMetry calculates the Flaky Score from the settings configured in the Configuration module. The score indicates the tester's probability of risk while running a test case, and it lets you compare pre-execution and post-execution results.
Go to the Test Execution screen to view the score in either of the following views.
Test Execution Detail View
The detail view displays two related columns for each run: the Flaky Score (F.S.) and the success rate (S.R.). When you open a run in the detail view, you can view the Flaky Score count at the top of the page.
Test Execution Default View
In the default view, you can show or hide the Flaky Score column on the execution screen. To analyze a score in more detail, select the Flaky Score to drill down to the test executions and defects that are associated with the test case.