AI failure analysis
Report has two AI helpers. AI Failure Analysis reads one failed step’s log and screenshot and says whether it looks like a real bug, a test or infrastructure problem, or a flaky failure. AI Report Feedback reviews the whole run.
In short
On this page you can
- Get a verdict, a confidence, a root cause and suggestions for each failed step.
- Tell product bugs from broken locators and slow environments before you debug.
- See verdict counts per run in the list’s Notes column.
- Get a written review of a whole run with Analyze with AI.
- Turn the AI options on or off per project, and control what data is shared.
How to use it
- Open a failed test case tab; analyzed steps already show a verdict chip.
- If not, click AI Analyze above the step table and wait for AI analyzing... to finish.
- Read the AI Failure Analysis card; click a note for root cause, evidence and suggestions.
- On the Summary tab, click Analyze with AI for the run review.
- Act on it: fix the test, open a task for a real bug, or rerun a flaky one.
Two AI helpers#
| AI Failure Analysis | AI Report Feedback | |
|---|---|---|
| Looks at | One failed step: its log text and its screenshot | The whole run: every test case, its steps, descriptions and AI notes |
| Answers | A verdict, a confidence, root cause, evidence, suggestions | Verified scenarios, failures with root cause and impact, recommendations, a performance review for load tests |
| Runs | Automatically after a run with failures, or with AI Analyze | Only when you click Analyze with AI |
| Shown in | The step row, the AI Failure Analysis card, Log Detail, the list’s Notes column | The Summary tab and the PDF |
| Project option | AI Failure Analysis | AI Report Feedback |
The verdicts#
Each verdict has its own icon and colour, a full label and a short label for tables. The colour is never the only signal.
| Verdict | Short | Means | Usually do |
|---|---|---|---|
| Real Bug | Bug | The application did something wrong: a value or message did not match, the app threw an error. | Open a task with the step’s evidence. |
| Test / Infrastructure Issue | Test/Infra | The test or its surroundings failed: element not found or not clickable, timeout, network, a test step that could not be built. | Fix the locator or wait, check the agent and environment. |
| Flaky | Flaky | The failure looks random or timing-related and may pass on a rerun. | Rerun; if it repeats, stabilise the step. Compare with InSight’s flaky list. |
| Unclassified | ? | No verdict: the analysis failed, or neither AI nor the fallback rules recognised the message. | Read the log yourself; try again later. |
The confidence (0–100%) is the AI’s own estimate. When the AI cannot answer, SmartestQA falls back to rules on the error message with fixed confidences, so a 50% or 60% note may come from the rules. See AI confidence.
A verdict is advice, not a test result
The step’s status does not change. A Real Bug verdict does not open a task and a Flaky verdict does not rerun anything. Confirm important findings before you act on them.
Automatic analysis after a run#
When a pipeline run finishes with failures and the project has Enable AI and AI Failure Analysis on (both on by default), SmartestQA analyzes the failed steps by itself, once per run. Test case runs that already have notes are skipped, and a manual analysis is never overwritten. The language follows the pipeline’s locale. So for most failed runs the verdicts are already there when you open the report.
Analyze failed steps yourself#
Open the failed test case
Open its tab and pick the Run and Retry you want. The buttons appear only when that attempt has failed steps.
Click AI Analyze
Tooltip: “Analyze failed steps with AI”. The button shows AI analyzing... while each failed step is sent with its log, its screenshot and the test case name, in your interface language.
Each failed step gets a verdict chip, and the AI Failure Analysis card fills in. The results are saved with this attempt, for everyone.
The button is disabled, with the reason as its tooltip, when:
- the run is still going: “Please wait for execution to complete”;
- this attempt was already analyzed: “Analysis already completed. Results are shown below.” There is no re-analyze for steps;
- AI is off for the project: “Enable 'Enable AI' in Project Panel → AI Options” or “Enable 'AI Failure Analysis' in Project Panel → AI Options”. See Settings.
Read the result#
- In the step table: a short verdict chip with the confidence under the failed step’s status.
- AI Failure Analysis card (right panel): “{n} steps analyzed”, one line per note with verdict, confidence and root cause. Click a line.
- The detail dialog: verdict and confidence, Log Message, Screenshot (Click to enlarge), AI Root Cause, Evidence (the log line that gave it away) and numbered Suggestions.
- Log Detail of the step shows the same notes.
- Reports list, Notes column: one chip per verdict with a count, for example Bug 2 · Flaky 1. Unclassified notes are not counted there.
AI Report Feedback (whole run)#
The card on the Summary tab, AI Report Feedback (“Verified scenarios, failures and next steps for this run”), reviews the run as a whole.
Wait for the run to finish
Before that the card says “Available when the run is completed.”
Click Analyze with AI
“Sending all test case results and descriptions of this run to AI...” A large run takes a while.
The review appears with “Last analyzed: {date}” and who ran it. It is saved with the run and is also printed in the PDF.
Re-analyze when needed
Re-analyze runs it again and replaces the stored review.
| Section | What it says |
|---|---|
| Verified scenarios | Per test case, up to four checks the run really proved. Partially verified: “The case failed later; these checks passed before the failure.” |
| Failures | Per failure: test case, Step, Error type, Root cause, Impact. |
| Recommendations | Up to four next steps. |
| Performance review (JMeter) | For runs with load test results: per test (up to 10) a verdict OK, Warning or Critical, What was tested, Workload model, Key findings, Possible bottlenecks, Test plan (JMX) gaps and Next steps. |
- Every review ends with “Generated by AI from this run’s logs; verify important findings.”
- If Skip Successful Logs was on, the card notes that only logged (failed) steps were reviewed.
- Very large runs are trimmed before sending (see limits); failed steps are always kept. A cut-off answer shows “The AI response was cut off before it finished…”.
Settings and who can change them#
The AI options are per project, in Project Panel › AI Options. Project Panel is open to Admins only (see roles). Each pipeline run keeps a copy of the options it ran with, visible under Pipeline Preferences.
| Option | Default | Effect in Report |
|---|---|---|
| Enable AI | On | Off turns off both helpers. |
| AI Failure Analysis | On | Automatic analysis after a run, and the AI Analyze button. Off disables both. |
| AI Report Feedback | On | The Analyze with AI review on the Summary. |
| Share Sensitive Data | Project choice | When off, e-mail addresses, card numbers, national ID numbers, phone numbers, secrets and tokens are masked before the run review is sent. |
| AI Account | Project choice | Smartest AI uses SmartestQA’s AI provider; Own Account uses your company’s provider and key. |
What is sent to the AI#
- Failure analysis: the failed step’s log text, its screenshot (images over 4 MB are not sent) and the test case name.
- Report feedback: the run’s test cases and steps with their logs, descriptions and AI notes, shortened to fit, with sensitive values masked unless Share Sensitive Data is on. For load tests, the JMeter test plan and its report data.
- Each call is logged with its token use and cost in the project’s AI usage. There is no separate quota for these features.
- Screenshots can show customer data. Run pipelines on test environments with test data.
If something goes wrong#
Didn’t find your answer?Write to support with your project name and what you see on screen.
info@smartestqa.ai