How do I find and fix flaky tests?
SmartestQA lists flaky test cases in InSight › Execution › Flaky scenarios, with a score that says how often the result flips between runs (0 stable, 1 flips every time). Report shows the same instability per run as Passed on retry, and you fix the cause in the test case: a wait that ends early, a stable locator, or a Retry Count on the pipeline.
In short
On this page you can
- See which test cases flip between pass and fail in a period, highest score first.
- Read the last ten results of each as pass and fail bars.
- Tell a random failure from a real bug with the AI verdict on the failed step.
- Retry failed test cases 0 to 5 times per pipeline and still see that a retry was needed.
- Copy a prompt that asks your AI assistant to make the test stable.
How to use it
- Open InSight › Execution and pick a Period.
- Read the Flaky scenarios card and table: Score and Last 10.
- Click a row to open the test case; compare a passing and a failing run in Report.
- Fix the step: a check that waits, a second locator or a
data-testid. - Set Retry Count in the pipeline's Test Management only as a safety net.
The short answer#
Open InSight › Execution. The Flaky scenarios card counts the test cases that flip between pass and fail in the chosen period (score ≥ 0.2 over at least 5 runs), and the table below lists up to 10 of them with a Score and their Last 10 results. The score is flips ÷ (n − 1): how many times a result differs from the one before it, divided by the number of runs minus one.
For a single run, Report marks a test case that failed first and passed on a retry as Passed on retry, and the AI can label a failed step Flaky. To fix it, find what differs between the passing and the failing runs. The usual repair is a wait that ends early, or a more stable locator. A pipeline’s Retry Count (0 to 5) keeps a run going, but it does not hide the instability.
The problem this solves#
Most test suites have a few tests that fail now and then for no visible reason. The team learns to run them again, and a red result stops meaning much: “it is probably that test again”. Real bugs can hide behind that habit. Many teams keep the known unstable tests in a spreadsheet or in their heads, or add a retry to their scripts so that the run ends green.
A retry in a script hides the problem without naming it. What is missing is a measure of which tests flip and how often, and a record that a retry was needed.
An example#
This is an illustration, not a customer story. A team tests an HR portal with a nightly pipeline whose Retry Count is 1.
- The QA lead opens InSight › Execution and picks the Period Last 30 days. The Flaky scenarios card shows 3.
- The top row of the table is the test case “Submit leave request” with a Score of 0.80 in red. Its Last 10 bars switch between pass and fail.
- A tester opens last night’s run in Report. The test case is Passed on retry and the run is Passed with warnings. With the Retry selector the tester compares First Attempt (✕) with Retry 1 (✓), then clicks AI Analyze. The failed step gets the verdict Flaky.
- The tester clicks the row in Flaky scenarios to open the test case in Automate. Before the failed step there is a fixed Wait (seconds). The tester replaces it with a Visibility Check, which waits for the element and ends early.
- The next nights pass on the first attempt. As passing runs add up in the period, the score falls, and below 0.2 the test case is no longer listed.
The team fixed a cause where it used to run the test again, and the retry stayed as a safety net that shows when it was used.
With a list kept by hand, with retries in your own scripts, and with SmartestQA#
| Question | A list of known flaky tests | Retries in your own scripts | SmartestQA |
|---|---|---|---|
| How do you learn that a test is flaky? | Someone remembers earlier failures and adds it to the list | Often you do not: the retry makes the run green | A score per test case, flips ÷ (n − 1), listed from 0.2 over at least 5 runs in the period |
| Is a retry visible afterwards? | Only if someone notes it | In the run log, if you read it | The test case is Passed on retry, the run is Passed with warnings, and First Attempt stays available |
| Flaky or a real bug? | A person runs it again and judges | The script cannot tell | The AI gives a failed step a verdict such as Flaky or Real Bug with a confidence. It is advice |
| Who runs a failed test again? | A person | The script, as often as you coded | The pipeline, 0 to 5 times as set in Retry Count. Each retry starts a new browser at the start page |
| Where is the history of one test? | In old reports, one by one | In old CI logs | Last 10 shows the last ten results as pass and fail bars |
| How do you know the fix worked? | Watch the next runs and update the list | Watch the next runs | The score counts flips in the period. Below 0.2 the test case is no longer listed |
Steps#
Open Execution and pick the period
InSight is the first item in the main menu. On Execution, choose a Period; the default is Last 7 days. A test case needs at least 5 finished runs in the period for a score, so pick a period that is long enough, for example Last 30 days.
Read the Flaky scenarios table
Only test cases with at least one pass and one fail in the period are listed. Score runs from 0.00 to 1.00: green below 0.2, amber from 0.2, red from 0.8. Under 5 runs the row says “uncertain · {n} runs”. Last 10 shows the last ten results as pass and fail bars, oldest left.
Check the cause of the failures
On the same tab, Failure clusters group failed steps with the same error message under a cause chip. Locator means “The element was not found, not visible or not clickable. Usually the page changed.” Environment covers connection, agent, browser or session problems, not the product’s behaviour.
Open a failing run in Report
In the test case’s tab, the Retry selector lets you pick First Attempt, Retry 1, Retry 2…, each with ✓ or ✕. Compare the failed attempt with the one that passed. If the failed step has no verdict yet, click AI Analyze.
Each failed step gets a verdict chip. Flaky means the failure looks random or timing-related and may pass on a rerun.
Make the step stable
Click the row in Flaky scenarios to open the test case in Automate. Prefer a check that waits, such as a Visibility Check or a Network Check, over a fixed Wait (seconds): a fixed wait always costs its full time and still fails when the app is slower than usual. For an element that is sometimes not found, give the action a second locator, or ask developers for a
data-testid.Or hand it to your AI assistant
The MCP column of the table copies a prompt that asks your assistant to compare passing and failing runs and make the test stable.
Set a retry as a safety net
In the pipeline, open Test Management and set Retry Count (“Retry failed tests”, 0 to 5 times; 0 in a new project). A retry starts a new browser at the start page. Click Save.
Watch the score fall
The score counts flips between runs in the period. Below 0.2 the test case is no longer listed.
How the flaky score is calculated#
| Results in the period, oldest first | Runs (n) | Flips | Score | Shown as |
|---|---|---|---|---|
| P F P P F P | 6 | 4 | 4 ÷ 5 = 0.80 | Listed, red |
| P P P P P P F | 7 | 1 | 1 ÷ 6 = 0.17 | Not listed (below 0.2) |
| P F P | 3 | 2 | No score | Listed as “uncertain · 3 runs” |
P is a finished run that passed, F is anything else. The table keeps at most 10 test cases, highest score first. The card counts the scored test cases in that list, so it is never more than 10. Dry runs are not counted.
Where a flaky test shows up#
| Screen | What you see | What it means |
|---|---|---|
| InSight › Execution › Flaky scenarios | Score and Last 10 | The result flips between runs in the period. |
| Report › Summary › test cases table | Passed on retry | All last attempts passed, but an earlier attempt failed. |
| Report › Summary › result card | Passed with warnings, “{n} needed a retry” | No test case failed, but some passed only on a retry. |
| Reports list › Result chip | For example 5/6, red | The chip counts every attempt, so the failed first attempt still counts. |
| Report › failed step | Verdict chip Flaky with a confidence | The AI’s advice. The step’s status does not change and nothing is rerun. |
| InSight › Pipeline run strip | A Passed mark | The strip takes each test case’s latest scenario run, so a pass on retry counts as passed. |
Watch it#
The video shows InSight’s Execution tab, with the period, the six summary cards (Flaky scenarios is one of them), the pipeline run strip and the failure clusters, and then the Test Health tab. It does not show the Flaky scenarios table or how a flaky test is fixed.
Video transcript
InSight opens on Execution: what ran in the selected period, and how it went. The period sets the window for every number on the page. Six cards sum it up: success rate, runs, failed runs, flaky scenarios, average run time and time saved. The pipeline run strip shows the last 14 runs of each pipeline. A row of red is where to look. Failure clusters group failed steps that broke for the same reason. Test Health asks a different question: how much of the product is really proven? Automation maturity is one score built from four pillars, each with an 80% target. Fix these first lists the three actions that raise the score most. The coverage funnel shows where requirements drop out on the way to a passing test. Modules × coverage ranks every module by risk, coverage and last results.
What you get#
- A number instead of a feeling. The score is 0 for a stable test case and 1 for one that flips on every run. It is listed from 0.2.
- History at a glance. The last ten results of each listed test case, as pass and fail bars.
- Retries that stay visible. A test case that needed a retry is Passed on retry, not Passed, and the run is Passed with warnings.
- A verdict per failed step. Real Bug, Test / Infrastructure Issue, Flaky or Unclassified, with a confidence from 0 to 100%.
- Slow pages are not treated as broken locators. When an element only appeared late, the run log says “the element appeared late … nothing was changed.”
- Known waiting times. An element step waits the Wait Timeout (default 30 seconds, allowed 1–120), then retries twice. A Visibility Check retries for up to 8 seconds.
- Fixed clusters only after a real pass. A failure cluster turns Fixed when every test case in it passed in a later, different run. A pass on a retry inside the same pipeline run does not count.
Questions and answers#
What counts as a flaky test in SmartestQA?Customers
A test case whose finished runs in the period contain at least one pass and one fail, with a score of 0.2 or more over at least 5 runs. The score is the number of flips divided by the number of runs minus one. See Flaky score.
What does “uncertain · 3 runs” mean?Customers
The test case passed and failed in the period but has fewer than 5 runs, so SmartestQA shows no score. It is listed, but the Flaky scenarios card does not count it. Pick a longer period or wait for more runs. See Flaky scenarios.
Can SmartestQA rerun failed tests automatically?Customers
Yes. Retry Count in the pipeline’s Test Management tab retries failed tests 0 to 5 times; a new project starts at 0. Each retry starts a new browser at the start page. See Test Management.
Do retries hide flaky tests?Customers
No. Report shows the test case as Passed on retry and the run as Passed with warnings, and the Result chip in the Reports list counts the failed attempt. The first attempt stays available under First Attempt. See Test case state.
Does a Dry Run retry a failed test?Customers
No. A Dry Run never retries, whatever the Retry Count is. Dry runs are also not counted in InSight’s Execution numbers, so they do not change the flaky score. See How long a step waits for its element.
How do I know whether a failure is flaky or a real bug?Customers
AI Failure Analysis gives each failed step a verdict. Flaky means the failure looks random or timing-related; Real Bug means the application did something wrong. The verdict is advice: confirm important findings before you act. See The verdicts.
How do I fix a test that fails because the page is sometimes slow?Customers
Most steps already wait for their element, up to the Wait Timeout. Add an explicit wait only when nothing else waits for the state you need, and prefer one that ends early, such as a Visibility Check or a Network Check, over a fixed number of seconds. See Waits.
Does self-healing fix flaky tests?Customers
It fixes one cause: a locator that no longer finds its element after a UI change. Healing starts only after the step’s own search failed with the full wait time, and if the element was only slow the runner changes nothing. See When the runner heals.
Why is a run green in InSight but Failed in the Reports list?Customers
A test case failed on its first try and passed on a retry inside the same run. InSight’s pipeline run strip takes each test case’s latest scenario run, so the mark is Passed. The Reports list counts every attempt. See Pipeline run strip.
Can I export the list of flaky tests?Customers
Yes. Download on the Execution tab creates an Excel workbook with one sheet per section for the selected period, including the flaky and slow test cases. See Download Excel or PDF.
Read more#
- Execution: every card on the tab, including Flaky scenarios, Failure clusters and Slowest scenarios.
- How the InSight numbers work: each formula with a worked example.
- AI failure analysis: the verdicts and what is sent to the AI.
- Make repairs rare:
data-testid, a second locator, a passing run on record. - How do I find out why a test failed?: reading one failed run step by step.
- What happens to my tests when the UI changes?: locator repairs and their review.
Didn’t find your answer?Write to support with your project name and what you see on screen.
info@smartestqa.ai