---
title: "AI failure analysis"
section: "Report"
description: "How SmartestQA AI classifies failed steps as real bug, test or infrastructure issue, or flaky, with confidence and root cause; how to run it, read it, turn it on and what data it sends."
url: https://docs.smartestqa.ai/en/report-ai-analysis.html
audience: [customer, admin]
updated: 2026-11-08
keywords: "ai failure analysis verdict real bug test infrastructure issue flaky unclassified confidence root cause evidence suggestions ai analyze automatic analysis ai report feedback analyze with ai re-analyze verified scenarios recommendations performance review sensitive data"
---
# Report: AI failure analysis

Report has two AI helpers. AI Failure Analysis reads one failed step’s log and screenshot and says whether it looks like a real bug, a test or infrastructure problem, or a flaky failure. AI Report Feedback reviews the whole run.

## In short

**On this page you can**

- Get a verdict, a confidence, a root cause and suggestions for each failed step.
- Tell product bugs from broken locators and slow environments before you debug.
- See verdict counts per run in the list’s Notes column.
- Get a written review of a whole run with Analyze with AI .
- Turn the AI options on or off per project, and control what data is shared.

**How to use it**

1. Open a failed test case tab; analyzed steps already show a verdict chip.
2. If not, click AI Analyze above the step table and wait for AI analyzing... to finish.
3. Read the AI Failure Analysis card; click a note for root cause, evidence and suggestions.
4. On the Summary tab, click Analyze with AI for the run review.
5. Act on it: fix the test, open a task for a real bug, or rerun a flaky one.

## Two AI helpers

|  | AI Failure Analysis | AI Report Feedback |
| --- | --- | --- |
| Looks at | One failed step: its log text and its screenshot | The whole run: every test case, its steps, descriptions and AI notes |
| Answers | A verdict, a confidence, root cause, evidence, suggestions | Verified scenarios, failures with root cause and impact, recommendations, a performance review for [load tests](https://docs.smartestqa.ai/en/report-load-tests.html) |
| Runs | Automatically after a run with failures, or with **AI Analyze** | Only when you click [**Analyze with AI**](https://docs.smartestqa.ai/en/report-ai-analysis.html#ai-report-feedback) |
| Shown in | The [step row](https://docs.smartestqa.ai/en/report-steps.html#step-table), the [**AI Failure Analysis**](https://docs.smartestqa.ai/en/project-settings-ai.html#what-each-switch-unlocks) card, [**Log Detail**](https://docs.smartestqa.ai/en/report-steps.html#log-detail), the list’s [Notes column](https://docs.smartestqa.ai/en/report-list.html#columns) | The [Summary tab](https://docs.smartestqa.ai/en/report-summary.html#ai-feedback-box) and the [PDF](https://docs.smartestqa.ai/en/report-summary.html#export-pdf) |
| Project option | **AI Failure Analysis** | **AI Report Feedback** |

## The verdicts

Each verdict has its own icon and colour, a full label and a short label for tables. The colour is never the only signal.

| Verdict | Short | Means | Usually do |
| --- | --- | --- | --- |
| Real Bug | Bug | The application did something wrong: a value or message did not match, the app threw an error. | Open a [task](https://docs.smartestqa.ai/en/report-steps.html#create-task) with the step’s evidence. |
| Test / Infrastructure Issue | Test/Infra | The test or its surroundings failed: element not found or not clickable, timeout, network, a test step that could not be built. | Fix the [locator](https://docs.smartestqa.ai/en/discover-actions.html) or [wait](https://docs.smartestqa.ai/en/automate-steps-flow.html#waits), check the [agent](https://docs.smartestqa.ai/en/agent-pool-agents.html#agent-status) and environment. |
| Flaky | Flaky | The failure looks random or timing-related and may pass on a rerun. | [Rerun](https://docs.smartestqa.ai/en/schedule-runs.html#run-now); if it repeats, [stabilise the step](https://docs.smartestqa.ai/en/maintenance-how-healing-works.html#make-it-stable). Compare with [InSight’s flaky list](https://docs.smartestqa.ai/en/insight-execution.html#flaky-scenarios). |
| Unclassified | ? | No verdict: the analysis failed, or neither AI nor the fallback rules recognised the message. | Read the log yourself; try again later. |

The confidence (0–100%) is the AI’s own estimate. When the AI cannot answer, SmartestQA falls back to rules on the error message with fixed confidences, so a 50% or 60% note may come from the rules. See [AI confidence](https://docs.smartestqa.ai/en/report-numbers.html#ai-confidence).

> **AI: A verdict is advice, not a test result**
>
> The step’s status does not change. A Real Bug verdict does not open a [task](https://docs.smartestqa.ai/en/tasks.html) and a Flaky verdict does not rerun anything. Confirm important findings before you act on them.

## Automatic analysis after a run

When a pipeline run finishes with failures and the project has [**Enable AI**](https://docs.smartestqa.ai/en/project-settings-ai.html#enable-ai) and [**AI Failure Analysis**](https://docs.smartestqa.ai/en/project-settings-ai.html#what-each-switch-unlocks) on (both on by default), SmartestQA analyzes the failed steps by itself, once per run. Test case runs that already have notes are skipped, and a manual analysis is never overwritten. The language follows the [pipeline’s locale](https://docs.smartestqa.ai/en/schedule-settings.html#test-management). So for most failed runs the verdicts are already there when you open the report.

## Analyze failed steps yourself

1. **Open the failed test case**

   Open its [tab](https://docs.smartestqa.ai/en/report-steps.html#case-tabs) and pick the [**Run** and **Retry**](https://docs.smartestqa.ai/en/report-steps.html#runs-and-retries) you want. The buttons appear only when that attempt has failed steps.
2. **Click AI Analyze**

   Tooltip: “Analyze failed steps with AI”. The button shows **AI analyzing...** while each failed step is sent with its log, its screenshot and the test case name, in your interface language.

   Each failed step gets a verdict chip, and the [**AI Failure Analysis**](https://docs.smartestqa.ai/en/project-settings-ai.html#what-each-switch-unlocks) card fills in. The results are saved with this attempt, for everyone.

The button is disabled, with the reason as its tooltip, when:

- the run is still going: “Please wait for execution to complete”;
- this attempt was already analyzed: “Analysis already completed. Results are shown below.” There is no re-analyze for steps;
- AI is off for the project: “Enable 'Enable AI' in Project Panel → AI Options” or “Enable 'AI Failure Analysis' in Project Panel → AI Options”. See [Settings](https://docs.smartestqa.ai/en/report-ai-analysis.html#settings).

## Read the result

- **In the [step table](https://docs.smartestqa.ai/en/report-steps.html#step-table)**: a short verdict chip with the confidence under the failed step’s status.
- **[**AI Failure Analysis**](https://docs.smartestqa.ai/en/project-settings-ai.html#what-each-switch-unlocks) card** (right panel): “{n} steps analyzed”, one line per note with verdict, confidence and root cause. Click a line.
- **The detail dialog**: verdict and confidence, **Log Message**, **Screenshot** (**Click to enlarge**), **AI Root Cause**, **Evidence** (the log line that gave it away) and numbered **Suggestions**.
- **[**Log Detail**](https://docs.smartestqa.ai/en/report-steps.html#log-detail)** of the step shows the same notes.
- **[Reports list, Notes column](https://docs.smartestqa.ai/en/report-list.html#columns)**: one chip per verdict with a count, for example Bug 2 · Flaky 1. Unclassified notes are not counted there.

## AI Report Feedback (whole run)

The card on the [Summary tab](https://docs.smartestqa.ai/en/report-summary.html#ai-feedback-box), [**AI Report Feedback**](https://docs.smartestqa.ai/en/project-settings-ai.html#what-each-switch-unlocks) (“Verified scenarios, failures and next steps for this run”), reviews the run as a whole.

1. **Wait for the run to finish**

   Before that the card says “Available when the run is completed.”
2. **Click Analyze with AI**

   “Sending all test case results and descriptions of this run to AI...” A large run takes a while.

   The review appears with “Last analyzed: {date}” and who ran it. It is saved with the run and is also printed in the [PDF](https://docs.smartestqa.ai/en/report-summary.html#export-pdf).
3. **Re-analyze when needed**

   **Re-analyze** runs it again and replaces the stored review.

| Section | What it says |
| --- | --- |
| **Verified scenarios** | Per test case, up to four checks the run really proved. **Partially verified**: “The case failed later; these checks passed before the failure.” |
| **Failures** | Per failure: test case, **Step**, **Error type**, **Root cause**, **Impact**. |
| **Recommendations** | Up to four next steps. |
| **Performance review (JMeter)** | For runs with [load test results](https://docs.smartestqa.ai/en/report-load-tests.html): per test (up to 10) a verdict **OK**, **Warning** or **Critical**, **What was tested**, **Workload model**, **Key findings**, **Possible bottlenecks**, **Test plan (JMX) gaps** and **Next steps**. |

- Every review ends with “Generated by AI from this run’s logs; verify important findings.”
- If [**Skip Successful Logs**](https://docs.smartestqa.ai/en/schedule-settings.html#test-management) was on, the card notes that only logged (failed) steps were reviewed.
- Very large runs are trimmed before sending (see [limits](https://docs.smartestqa.ai/en/report-faq.html#limits)); failed steps are always kept. A cut-off answer shows “The AI response was cut off before it finished…”.

## Settings and who can change them

The AI options are per project, in [**Project Panel** › **AI Options**](https://docs.smartestqa.ai/en/project-settings-ai.html). Project Panel is open to Admins only (see [roles](https://docs.smartestqa.ai/en/company-settings-roles.html)). Each pipeline run keeps a copy of the options it ran with, visible under [Pipeline Preferences](https://docs.smartestqa.ai/en/report-summary.html#pipeline-preferences).

| Option | Default | Effect in Report |
| --- | --- | --- |
| [Enable AI](https://docs.smartestqa.ai/en/project-settings-ai.html#enable-ai) | On | Off turns off both helpers. |
| [AI Failure Analysis](https://docs.smartestqa.ai/en/project-settings-ai.html#what-each-switch-unlocks) | On | Automatic analysis after a run, and the **AI Analyze** button. Off disables both. |
| AI Report Feedback | On | The [**Analyze with AI**](https://docs.smartestqa.ai/en/report-ai-analysis.html#ai-report-feedback) review on the Summary. |
| Share Sensitive Data | Project choice | When off, e-mail addresses, card numbers, national ID numbers, phone numbers, secrets and tokens are masked before the run review is sent. |
| AI Account | Project choice | [**Smartest AI**](https://docs.smartestqa.ai/en/project-settings-ai.html#ai-account) uses SmartestQA’s AI provider; [**Own Account**](https://docs.smartestqa.ai/en/project-settings-ai.html#own-account) uses your company’s provider and key. |

## What is sent to the AI

- **Failure analysis**: the failed step’s log text, its screenshot (images over 4 MB are not sent) and the test case name.
- **Report feedback**: the run’s test cases and steps with their logs, descriptions and AI notes, shortened to fit, with sensitive values masked unless [**Share Sensitive Data**](https://docs.smartestqa.ai/en/project-settings-ai.html#what-each-switch-unlocks) is on. For load tests, the [JMeter test plan](https://docs.smartestqa.ai/en/automate-steps-load.html) and its report data.
- Each call is logged with its token use and cost in the project’s AI usage. There is no separate quota for these features.
- Screenshots can show customer data. Run pipelines on test environments with [test data](https://docs.smartestqa.ai/en/automate-test-data.html).

## If something goes wrong

- [AI Analyze is disabled / “AI Analysis is disabled…”](https://docs.smartestqa.ai/en/report-troubleshooting.html#fix-ai-disabled)
- [“Analysis already completed. Results are shown below.”](https://docs.smartestqa.ai/en/report-troubleshooting.html#fix-ai-already)
- [Every verdict is Unclassified](https://docs.smartestqa.ai/en/report-troubleshooting.html#fix-ai-unclassified)
- [“AI analysis results could not be saved.”](https://docs.smartestqa.ai/en/report-troubleshooting.html#fix-ai-save)
- [“AI or AI Report Feedback is turned off for this project…”](https://docs.smartestqa.ai/en/report-troubleshooting.html#fix-ai-feedback-disabled)
- [“The AI could not produce a review right now…”](https://docs.smartestqa.ai/en/report-troubleshooting.html#fix-ai-feedback-failed)
