1. Docs
  2. Use cases
  3. Use cases
  4. How do I find out why a test failed?

How do I find out why a test failed?

Open the run in Report. The Summary lists the failed test cases first with the first failed step of each, the step’s Log Detail shows its message and screenshot, and AI Failure Analysis gives a verdict with a root cause: Real Bug, Test / Infrastructure Issue or Flaky.

8 min read Updated CustomersAdmins

In short

On this page you can

  • See at a glance which test cases of a run failed and at which step.
  • Read the failed step’s log message and screenshot.
  • Get an AI verdict with a confidence, a root cause and suggestions.
  • Open a task from the failed step with its evidence.
  • Export the run as PDF or send it by e-mail.

How to use it

  1. Open Report and click the run with a red result.
  2. On Summary, read the verdict and the Failure reason of each test case.
  3. Click the failed test case, then the failed step, to open Log Detail.
  4. Read the AI Failure Analysis card, or click AI Analyze.
  5. Click Task to report it, or fix the test.

The short answer#

Every pipeline run has a report. It answers three questions in order: which test cases failed, at which step, and why. For most failed runs the AI verdicts are already there when you open the report, because SmartestQA analyzes the failed steps by itself when a run finishes with failures.

The problem this solves#

A red test says that something is wrong, not what. In many teams the first step is to open a long log from the night’s run and search for the first error. The second step is often to run the test again and watch it.

What is missing is the evidence from the moment of the failure, and a first sorting. Before anyone can fix anything, someone has to decide whether the application is wrong, the test is wrong, or the failure will not happen again.

An example#

This is an illustration, not a customer story. A team tests a banking app in the browser with a pipeline of four test cases that runs every night.

  1. In the morning a tester opens Report. The newest run is on top; its Result chip reads 3/4, and Notes already counts one AI verdict, because the failed step was analyzed when the run finished.
  2. The tester clicks the run. The Summary shows Failed, and the transfer test case is the first row of the table. Its Failure reason starts with “Step 6:”.
  3. The tester clicks the test case and then the failed step. Log Detail shows the full Log Message and the screenshot of the page at that moment: the balance after the transfer is not the expected one.
  4. The step carries the verdict Real Bug with a confidence. In the AI Failure Analysis card the tester clicks the note and reads AI Root Cause, Evidence and the numbered Suggestions.
  5. The verdict does not open a task by itself. The tester compares it with the screenshot, agrees, and clicks Task to open a task with the step’s evidence.

The tester did not rerun the test to see what happened: the step, its message, its screenshot and a first verdict were in the report of the night’s run.

With a log, by running it again, and with SmartestQA#

QuestionReading the log of your own scriptsRunning the test again by handSmartestQA
Which tests failed, and where?Search the log for the first error. Fine for a small suiteKnown only for the test you rerunThe Summary lists failed test cases first, with the first failed step of each
What did the screen show?Only if the script saved a screenshotYou watch it, if the failure happens againThe step’s Log Detail has its message and screenshot
Is it the application or the test?A person decides from the error textA person decides from what they seeAn AI verdict with a confidence: Real Bug, Test / Infrastructure Issue, Flaky or Unclassified. It is advice, not a test result
What if it does not fail the second time?The log of the first run is all there isThe evidence is goneFor a test case that passed on retry, First Attempt in the Retry selector shows the failure
How does it reach a developer?Copy the error into a ticketDescribe what you sawTask opens a task with the step’s evidence
How do I share the whole run?Send the log file or a link to the CI jobWrite a summaryExport PDF, or Share Report to e-mail the same PDF

Steps#

  1. Find the run

    Open Report. One row is one pipeline run, newest on top. The Result chip shows passed test case runs out of the planned total, for example 3/4, and Notes counts the AI verdicts on its failed steps. Click the run. See Find a run.

  2. Read the Summary

    The large word is the verdict: Passed, Passed with warnings, Failed or Running. The line under it says why, for example “7 of 8 test cases passed · 1 needed a retry”. The test cases table puts problems first, and Failure reason shows the first failed step of each as “Step 4: …”.

  3. Open the failed test case

    Click its row. Its tab lists every step with status, timing and a screenshot. If the test case ran more than once, pick the Run and the Retry you want to look at.

  4. Open the failed step

    Click the step row to open Log Detail: the full Log Message, the timings and the screenshot of what the browser showed. See Log Detail.

  5. Read the AI verdict

    A failed step that AI has analyzed shows a verdict chip with a confidence. The AI Failure Analysis card lists one line per note; click a line for AI Root Cause, Evidence and numbered Suggestions. If there is no verdict yet, click AI Analyze.

  6. Act on it

    For a Real Bug, click Task to open a task with the step’s evidence. For a Test / Infrastructure Issue, fix the locator or the wait and check the agent and environment. For Flaky, rerun; if it repeats, stabilise the step. See The verdicts.

  7. Share it

    Export PDF downloads the report and Share Report e-mails the same PDF. Both are enabled once the run has finished.

Watch it#

The video reads two runs: a passed run of four test cases, step by step, and then a failed run from the list to the failing step and its AI Failure Analysis.

Video transcript

When the run is done, open it from Report. Four of four passed. The summary shows the verdict, the duration and one row per test case. Each case has its own tab, in run order: Setup first, Teardown last. Open a case to read every step, with its timing and a screenshot. Click a step for the detail: status, timings, log message and what the browser showed. The Teardown ran last and deleted the note the tests created. Report lists every run: pipeline, where it ran, duration, result and status. A red result means at least one test case failed. Open that run. The summary gives the verdict and counts what passed, failed and did not run. The test cases table shows each result with its failure reason. Open the failed case: its tab lists every step with status, timing and a screenshot. Click the failing step for its log detail and what the browser showed. AI Failure Analysis says whether the product, the test or the environment is to blame.

Read a passed run and a failed run, down to the step that broke.

What you get#

  • Problems first. The Summary sorts test cases as failed, passed on retry, running, not run, then passed.
  • Evidence for every step. Each step keeps its log, a screenshot, its duration and, for API steps, the response.
  • Four AI verdicts, each with a confidence from 0 to 100%: Real Bug, Test / Infrastructure Issue, Flaky and Unclassified.
  • A review of the whole run. AI Report Feedback lists verified scenarios, failures with root cause and impact, and up to four recommendations. It runs when you click Analyze with AI.
  • A file to send. One PDF with the summary, the steps and the failure screenshots.
  • Kept results. Reports, step logs and screenshots are not deleted automatically.

A verdict is advice, not a test result

The step’s status does not change. A Real Bug verdict does not open a task and a Flaky verdict does not rerun anything. Confirm important findings before you act on them.

Questions and answers#

Do I have to start the AI analysis myself?Customers

Usually not. When a pipeline run finishes with failures and the project has Enable AI and AI Failure Analysis on, the failed steps are analyzed once per run. AI Analyze does the same by hand for one attempt. See Automatic analysis after a run.

How do I know whether the product or the test is to blame?Customers

Read the verdict. Real Bug means the application did something wrong, such as a value or message that did not match. Test / Infrastructure Issue means the test or its surroundings failed: an element not found, a timeout, the network. See The verdicts.

The run says Completed. Did everything pass?Customers

Not necessarily. Always read the Result chip: 4/5 with status Completed means one test case did not finish. See What each column shows.

What does “Passed on retry” mean?Customers

The test case failed first and passed on an automatic retry. The last attempt decides the result; pick First Attempt in the Retry selector to see the failure. See Test case state.

Why is my dry run not in Report?Customers

Dry runs from Automate show a live log only and are not saved in Report. Run the test case in a pipeline to get a report. See The report record and the live log.

Many runs fail. How do I see the pattern?Customers

Use Report for one run and InSight for the pattern. InSight groups failed steps that broke for the same reason into failure clusters and lists flaky scenarios.

Read more#

Didn’t find your answer?Write to support with your project name and what you see on screen.

info@smartestqa.ai