Flaky Test Detection
Flaky Score is an AI-driven feature that evaluates the stability of test cases by analyzing their execution history. It indicates the likelihood of a test case behaving unpredictably.
Flaky test cases can disrupt testing pipelines and undermine confidence in test results. Traditionally, identifying such tests required manual comparison of execution results across multiple runs. With flaky score, this process is automated.
QA Managers can define and configure settings according to their specific testing processes for calculating the flaky score, ensuring its relevance to their testing methodologies.
For example, the following table shows the execution results of the test cases executed multiple times.
Note
The system calculates the Flaky Score only for the executed test cases. It is determined based on the latest number of executions. The Flaky Score ranges between 0 (Not Flaky) and 1 (Flaky).
Test Case Name | Test 1 | Test 2 | Test 3 | Test 4 | Test 5 | Test 6 | Test 7 | Test 8 | Flaky or Non-flaky? |
|---|---|---|---|---|---|---|---|---|---|
Test Case A | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Non-flaky |
Test Case B | Fail | Fail | Fail | Fail | Fail | Fail | Fail | Fail | Non-flaky |
Test Case C | Pass | Pass | Fail | Fail | Pass | Fail | Pass | Fail | Flaky |
Note
The system considers only the final execution status of the test case.
The system calculates Flaky Score only for the test executions of the same project.