---
title: "How the numbers work"
section: "Report"
description: "How every number and status in SmartestQA Report is calculated: passed/total, duration, run status, Disconnected, run verdict, Passed on retry, attempts, steps passed, Total and Actual time, AI confidence, load test verdict and comparison."
url: https://docs.smartestqa.ai/en/report-numbers.html
audience: [customer, admin]
updated: 2026-11-08
keywords: "report numbers formula result chip passed total duration run status completed failed disconnected 2 hours verdict passed with warnings passed on retry flaky attempts steps passed total actual ai confidence load verdict p90 sla comparison"
---
# Report: How the numbers work

Every number, status and colour in Report, how it is calculated, and an example you can check against your own runs.

## In short

**On this page you can**

- See exactly what the Result chip 5/6 counts.
- Understand why the list can say Failed while the Summary says Passed with warnings .
- Learn when a run turns Disconnected .
- Read Attempts , Duration , Total and Actual correctly.
- Check how a load step’s verdict and its comparison are decided.

**How to use it**

1. Find the number you see on screen in the headings below.
2. Read its formula box: what is counted and what is not.
3. Recompute the worked example with your own run.
4. If your result differs, check the notes under the example, then the troubleshooting page.

## Result chip (Reports list)

**passed / total** — Shown in: Reports list › Result

```text
attempt  = one execution of one test case (each iteration and each automatic retry is its own attempt)
passed   = attempts that ended Completed
failed   = attempts that ended Failed or Disconnected
           + planned attempts that never started because the run was stopped
total    = planned attempts: each test case × its execute count (or its data rows),
           requirement rows count their test cases, called pipelines count theirs
           raised to passed + failed when more attempts ran than were planned (retries)
pending  = total − passed − failed − running
colour   = the run status (green Completed, red Failed, amber Running, blue New, grey Disconnected)
```

**Example:** a pipeline has Login, Search with a 3-row [data source](https://docs.smartestqa.ai/en/data-source-datasets.html), and Checkout: total = 1 + 3 + 1 = 5. Search row 2 fails, is [retried](https://docs.smartestqa.ai/en/schedule-settings.html#test-management) once and passes. Attempts: Login ✓, Search 1 ✓, Search 2 ✕, Search 2 retry ✓, Search 3 ✓, Checkout ✓, so passed = 5, failed = 1. That is 6 attempts for 5 planned, so total becomes 6: the chip shows 5/6, red, because one attempt failed.

The [Summary](https://docs.smartestqa.ai/en/report-summary.html#test-cases-table) judges each test case by its last attempt instead, so the same run is **Passed with warnings** there, with Search as **Passed on retry**. The two views answer different questions: the chip counts every attempt, the Summary says whether each test case finally passed.

**[Manual pipelines](https://docs.smartestqa.ai/en/report-summary.html#manual-runs):** passed manual cases ÷ all manual cases of the run.

## Duration and Took

**Wall-clock time** — Shown in: Reports list › Duration, Summary › Took

```text
start    = start of the run’s first attempt
end      = end of the run’s last attempt          (empty while any attempt runs)
duration = end − start                             shown as 1h 2m 5s · 4m 12s · 38s
while running: list shows "..."; Summary shows the time so far + "so far"
```

**Example:** the first test case starts at 10:00:05, three [run in parallel](https://docs.smartestqa.ai/en/schedule-settings.html#test-management), and the last ends at 10:12:47. Duration is 12m 42s, not the sum of the three test cases. Sorting the list by Duration uses the same value.

## Run status (Reports list)

**One status for the whole run** — Shown in: Reports list › Status

```text
the first rule that matches, over all attempts of the run:
  any attempt Running            → Running
  any attempt In progress        → IN_PROGRESS (manual pipelines, shown as is)
  any attempt Failed             → Failed
  any attempt Completed          → Completed
  any attempt Disconnected       → Disconnected
  otherwise                      → New
a stopped run with test cases that never ran → Failed instead of Completed
```

| Attempts | Status | Result chip |
| --- | --- | --- |
| 4 Completed | Completed | 4/4 |
| 3 Completed, 1 Failed | Failed | 3/4 |
| 4 Completed, 1 Disconnected | Completed | 4/5 |
| 2 Completed, 1 Running | Running | 2/3 |
| Stopped after 2 of 5, both Completed | Failed | 2/5 |

The third row is why you should read the chip as well as the status: a disconnected attempt does not make the run Failed.

## The 2-hour rule (Disconnected)

**When a run is treated as dead** — Shown in: Reports list, report header, InSight

```text
on screen:  Running or New, and started more than 120 minutes ago   → shown as Disconnected
            finished attempts, but the run was never closed,
            and its last attempt ended more than 120 minutes ago   → treated as finished
on the server: attempts still Running more than 2 hours after start are saved as Disconnected
            (when the Reports list loads, and by a check every 4 hours)
```

**Example:** a run starts at 08:00 on [your agent](https://docs.smartestqa.ai/en/agent-pool-agents.html#agent-status) and the laptop goes to sleep at 08:20. At 09:30 the row still says Running. From 10:00 on it shows **Disconnected**; [**Export PDF**](https://docs.smartestqa.ai/en/report-summary.html#export-pdf), [**Share Report**](https://docs.smartestqa.ai/en/report-summary.html#share-report), tags and **AI Analyze** unlock, and the dead attempt counts as failed.

A legitimate run longer than 2 hours keeps running as long as its test cases keep finishing: the second rule looks at the last end, not the start.

## Run verdict (Summary)

**Passed · Passed with warnings · Failed** — Shown in: Summary › result card

```text
Running                 = the run, or any test case, is still running
No test case ran        = the run has no test case
Failed                  = a test case failed (its last attempt)
Passed with warnings    = no test case failed, but some passed only on a retry,
                          some did not run, or steps ended with a warning
Passed                  = every test case passed on its first attempt and no step warned
sub-line                = "{passed} of {total} test cases passed" (passed on retry counts as passed)
                          · "{n} needed a retry" · "{n} not run" · "{n} step(s) passed with warnings"
```

| Test cases | Warning steps | Verdict | Sub-line |
| --- | --- | --- | --- |
| 8 passed | 0 | Passed | 8 of 8 test cases passed |
| 7 passed, 1 passed on retry | 0 | Passed with warnings | 8 of 8 · 1 needed a retry |
| 8 passed | 2 | Passed with warnings | 8 of 8 · 2 step(s) passed with warnings |
| 6 passed, 2 failed | 0 | Failed | 6 of 8 test cases passed |

A warning step is one that ended with a warning instead of failing, for example a [Network step whose rule is set to warn](https://docs.smartestqa.ai/en/automate-steps-network.html#check).

## Test case state (Summary table)

**One state per test case** — Shown in: Summary › Test cases › Result, status chips

```text
last attempts = for each iteration, the attempt with the highest retry number
Failed           = a last attempt failed
                   or the run finished but an attempt never completed (disconnected, stopped,
                   or fewer iterations than planned)
Running          = a last attempt is still running, or planned iterations are missing while the run goes
Passed on retry  = all last attempts passed, but an earlier attempt failed
Passed           = all last attempts passed, every planned iteration ran, no earlier failure
Not run          = no attempt was recorded
order in the table: Failed, Passed on retry, Running, Not run, Passed (all passed: slowest first)
```

**Example:** Search planned for 3 [data rows](https://docs.smartestqa.ai/en/schedule-settings.html#data-loop). Row 1 ✓; row 2 ✕ then retry ✓; row 3 ✓. Last attempts: ✓ ✓ ✓, with an earlier failure: **Passed on retry**. If row 3 had never run before the run was stopped, the state would be Failed.

## Attempts and Duration (Summary table)

**Per test case** — Shown in: Summary › Test cases

```text
Attempts       = all attempts of the test case in this run (iterations + retries)
Duration       = Σ over iterations of (end − start) of the last attempt     "–" if one is missing
Failure reason = first failed step of the last attempt, by step number, cut at 400 characters
                 (loaded for up to 10 failed rows per page)
```

**Example:** Search ran 3 iterations and iteration 2 was retried: Attempts = 4. Last attempts took 20 s, 25 s and 18 s: Duration = 63 s. The failed first try of iteration 2 is not in Duration.

## Steps chip and case tab status

**passed steps / all steps** — Shown in: test case tab, above the step table; tab icons

```text
chip   = step rows with status Success ÷ all step rows of the chosen run and retry
tone   = red if any row failed · green if all succeeded · grey otherwise (warnings)
tab    = after the run: all rows Success → Passed · any row failed → Failed
                        otherwise → Passed with warnings
         while the run goes: from the attempts (Failed, Running, or Passed when all planned iterations passed)
```

**Example:** 14 rows, 12 success, 1 failed, 1 warning: chip 12/14 in red, tab Failed. Without the failed row: 13/14 in grey, tab **Passed with warnings**. One step can write several rows (the action plus an auto-heal or AI note), so the total can be higher than the number of steps in the test case.

## Total and Actual

**Two times per step** — Shown in: step table, Log Detail

```text
Total  = time when the step’s result was recorded − time the step started
         (finding the element, retries, waits, the action, the screenshot)
Actual = time the action itself took, where the step measures it
         (wait time, query time, the JMeter run …); empty when not measured
format: under 1 s → 350ms · under 60 s → 2.41s · 1m 5s · 1h 2m · days
```

**Example:** a [`Wait 5` step](https://docs.smartestqa.ai/en/automate-steps-browser.html#wait-seconds) shows Actual 5.00s and Total 5.38s; the extra 0.38 s is the screenshot and logging. A click whose [locator needed fallbacks](https://docs.smartestqa.ai/en/automate-steps-browser.html#finding-elements) can show a Total of 40 s: the runner tried the first locator for the full [wait timeout](https://docs.smartestqa.ai/en/project-settings-test-management.html#timeouts) before moving on.

## AI confidence

**How sure the verdict is** — Shown in: verdict chips, AI Failure Analysis card

```text
AI answered            → verdict and confidence (0–100) as the AI returned them
AI could not answer    → rules on the error message:
   step could not be built                         Test / Infrastructure Issue   90%
   network, connection, DNS                        Test / Infrastructure Issue   65%
   assertion, does not match                       Real Bug                       60%
   timeout, wait, could not be loaded              Test / Infrastructure Issue   55%
   element not found, stale, not clickable         Test / Infrastructure Issue   50%
   JavaScript error, exception                     Real Bug                       45%
   anything else                                   Unclassified                   25%
request failed         → Unclassified, 0%
```

**Example:** “Element not found: css=#submit” while the [AI provider](https://docs.smartestqa.ai/en/project-settings-ai.html#ai-account) is unreachable gives [**Test / Infrastructure Issue**](https://docs.smartestqa.ai/en/report-ai-analysis.html#verdicts) at 50%. The same message analyzed by the AI might come back as [Flaky](https://docs.smartestqa.ai/en/report-ai-analysis.html#verdicts) at 72% after it compared the screenshot. The rule results never say Flaky.

## Load step verdict

**Passed · Passed with warnings · Failed** — Shown in: Load steps table, load step verdict strip

```text
error rate   = failed requests ÷ all requests × 100           (2 decimals)
Failed       = any failed request (no answer, 4xx/5xx, failed assertion)
               or any k6 threshold missed
Warning      = requests slower than their target (SLA misses)
               or saturation users felt: P95 rose by ≥ 100 ms AND to ≥ 1.5 × its base
               or a target measured nothing
               or the load generator was under pressure
Passed       = none of the above
functional check note = peak users ≤ 1 or fewer than 30 requests
run verdict  = the worst step (Failed > Warning > Passed)
```

**Example:** 12,000 requests, 0 failed, 340 slower than the 800 ms [target](https://docs.smartestqa.ai/en/schedule-settings.html#mobile-desktop-load): **Passed with warnings**, “340 requests slower than their target”. With 6 failed requests it is Failed, “6 requests failed (0.05%)”.

## Comparison with the previous run

**Better · Worse · Similar** — Shown in: Compared with a previous run, vs previous run

```text
times, throughput  change % = (this − baseline) ÷ baseline × 100
                   similar when |change| < 10 %, or for times when the difference < 20 ms
error rate         difference in points; similar under 0.5 pt
Apdex              absolute difference; similar under 0.02
overall            Worse if P90 or error rate got worse · Better if one got better · else Similar
"compare with care" = peak users differ by more than 10 % or duration by more than 20 %
```

**Example:** P90 119 ms → 131 ms is +10 %, but only 12 ms: Similar. P90 400 ms → 520 ms is +30 % and 120 ms: Worse.

## Manual pipeline counts

**Manual runs** — Shown in: manual pipeline Summary

```text
Total Test Cases      = manual cases in the run
Executed Test Cases   = cases with status Passed or Failed
Successful Test Cases = Passed      Failed Test Cases = Failed      Pending Test Cases = New
Run Ratio             = Runned Cases (Passed + Failed) vs Pending Cases (New)
```

**Example:** 20 cases: 12 Passed, 3 Failed, 5 New. Executed 15, Success - Failed Ratio 12 : 3 : 5 pending, Run Ratio 15 : 5, list chip 12/20.
