How the numbers work
Every number, status and colour in Report, how it is calculated, and an example you can check against your own runs.
In short
On this page you can
- See exactly what the Result chip
5/6counts. - Understand why the list can say Failed while the Summary says Passed with warnings.
- Learn when a run turns Disconnected.
- Read Attempts, Duration, Total and Actual correctly.
- Check how a load step’s verdict and its comparison are decided.
How to use it
- Find the number you see on screen in the headings below.
- Read its formula box: what is counted and what is not.
- Recompute the worked example with your own run.
- If your result differs, check the notes under the example, then the troubleshooting page.
Result chip (Reports list)#
attempt = one execution of one test case (each iteration and each automatic retry is its own attempt)
passed = attempts that ended Completed
failed = attempts that ended Failed or Disconnected
+ planned attempts that never started because the run was stopped
total = planned attempts: each test case × its execute count (or its data rows),
requirement rows count their test cases, called pipelines count theirs
raised to passed + failed when more attempts ran than were planned (retries)
pending = total − passed − failed − running
colour = the run status (green Completed, red Failed, amber Running, blue New, grey Disconnected)
Example: a pipeline has Login, Search with a 3-row data source, and Checkout: total = 1 + 3 + 1 = 5. Search row 2 fails, is retried once and passes. Attempts: Login ✓, Search 1 ✓, Search 2 ✕, Search 2 retry ✓, Search 3 ✓, Checkout ✓, so passed = 5, failed = 1. That is 6 attempts for 5 planned, so total becomes 6: the chip shows 5/6, red, because one attempt failed.
The Summary judges each test case by its last attempt instead, so the same run is Passed with warnings there, with Search as Passed on retry. The two views answer different questions: the chip counts every attempt, the Summary says whether each test case finally passed.
Manual pipelines: passed manual cases ÷ all manual cases of the run.
Duration and Took#
start = start of the run’s first attempt end = end of the run’s last attempt (empty while any attempt runs) duration = end − start shown as 1h 2m 5s · 4m 12s · 38s while running: list shows "..."; Summary shows the time so far + "so far"
Example: the first test case starts at 10:00:05, three run in parallel, and the last ends at 10:12:47. Duration is 12m 42s, not the sum of the three test cases. Sorting the list by Duration uses the same value.
Run status (Reports list)#
the first rule that matches, over all attempts of the run: any attempt Running → Running any attempt In progress → IN_PROGRESS (manual pipelines, shown as is) any attempt Failed → Failed any attempt Completed → Completed any attempt Disconnected → Disconnected otherwise → New a stopped run with test cases that never ran → Failed instead of Completed
| Attempts | Status | Result chip |
|---|---|---|
| 4 Completed | Completed | 4/4 |
| 3 Completed, 1 Failed | Failed | 3/4 |
| 4 Completed, 1 Disconnected | Completed | 4/5 |
| 2 Completed, 1 Running | Running | 2/3 |
| Stopped after 2 of 5, both Completed | Failed | 2/5 |
The third row is why you should read the chip as well as the status: a disconnected attempt does not make the run Failed.
The 2-hour rule (Disconnected)#
on screen: Running or New, and started more than 120 minutes ago → shown as Disconnected
finished attempts, but the run was never closed,
and its last attempt ended more than 120 minutes ago → treated as finished
on the server: attempts still Running more than 2 hours after start are saved as Disconnected
(when the Reports list loads, and by a check every 4 hours)
Example: a run starts at 08:00 on your agent and the laptop goes to sleep at 08:20. At 09:30 the row still says Running. From 10:00 on it shows Disconnected; Export PDF, Share Report, tags and AI Analyze unlock, and the dead attempt counts as failed.
A legitimate run longer than 2 hours keeps running as long as its test cases keep finishing: the second rule looks at the last end, not the start.
Run verdict (Summary)#
Running = the run, or any test case, is still running
No test case ran = the run has no test case
Failed = a test case failed (its last attempt)
Passed with warnings = no test case failed, but some passed only on a retry,
some did not run, or steps ended with a warning
Passed = every test case passed on its first attempt and no step warned
sub-line = "{passed} of {total} test cases passed" (passed on retry counts as passed)
· "{n} needed a retry" · "{n} not run" · "{n} step(s) passed with warnings"
| Test cases | Warning steps | Verdict | Sub-line |
|---|---|---|---|
| 8 passed | 0 | Passed | 8 of 8 test cases passed |
| 7 passed, 1 passed on retry | 0 | Passed with warnings | 8 of 8 · 1 needed a retry |
| 8 passed | 2 | Passed with warnings | 8 of 8 · 2 step(s) passed with warnings |
| 6 passed, 2 failed | 0 | Failed | 6 of 8 test cases passed |
A warning step is one that ended with a warning instead of failing, for example a Network step whose rule is set to warn.
Test case state (Summary table)#
last attempts = for each iteration, the attempt with the highest retry number
Failed = a last attempt failed
or the run finished but an attempt never completed (disconnected, stopped,
or fewer iterations than planned)
Running = a last attempt is still running, or planned iterations are missing while the run goes
Passed on retry = all last attempts passed, but an earlier attempt failed
Passed = all last attempts passed, every planned iteration ran, no earlier failure
Not run = no attempt was recorded
order in the table: Failed, Passed on retry, Running, Not run, Passed (all passed: slowest first)
Example: Search planned for 3 data rows. Row 1 ✓; row 2 ✕ then retry ✓; row 3 ✓. Last attempts: ✓ ✓ ✓, with an earlier failure: Passed on retry. If row 3 had never run before the run was stopped, the state would be Failed.
Attempts and Duration (Summary table)#
Attempts = all attempts of the test case in this run (iterations + retries)
Duration = Σ over iterations of (end − start) of the last attempt "–" if one is missing
Failure reason = first failed step of the last attempt, by step number, cut at 400 characters
(loaded for up to 10 failed rows per page)
Example: Search ran 3 iterations and iteration 2 was retried: Attempts = 4. Last attempts took 20 s, 25 s and 18 s: Duration = 63 s. The failed first try of iteration 2 is not in Duration.
Steps chip and case tab status#
chip = step rows with status Success ÷ all step rows of the chosen run and retry
tone = red if any row failed · green if all succeeded · grey otherwise (warnings)
tab = after the run: all rows Success → Passed · any row failed → Failed
otherwise → Passed with warnings
while the run goes: from the attempts (Failed, Running, or Passed when all planned iterations passed)
Example: 14 rows, 12 success, 1 failed, 1 warning: chip 12/14 in red, tab Failed. Without the failed row: 13/14 in grey, tab Passed with warnings. One step can write several rows (the action plus an auto-heal or AI note), so the total can be higher than the number of steps in the test case.
Total and Actual#
Total = time when the step’s result was recorded − time the step started
(finding the element, retries, waits, the action, the screenshot)
Actual = time the action itself took, where the step measures it
(wait time, query time, the JMeter run …); empty when not measured
format: under 1 s → 350ms · under 60 s → 2.41s · 1m 5s · 1h 2m · days
Example: a Wait 5 step shows Actual 5.00s and Total 5.38s; the extra 0.38 s is the screenshot and logging. A click whose locator needed fallbacks can show a Total of 40 s: the runner tried the first locator for the full wait timeout before moving on.
AI confidence#
AI answered → verdict and confidence (0–100) as the AI returned them AI could not answer → rules on the error message: step could not be built Test / Infrastructure Issue 90% network, connection, DNS Test / Infrastructure Issue 65% assertion, does not match Real Bug 60% timeout, wait, could not be loaded Test / Infrastructure Issue 55% element not found, stale, not clickable Test / Infrastructure Issue 50% JavaScript error, exception Real Bug 45% anything else Unclassified 25% request failed → Unclassified, 0%
Example: “Element not found: css=#submit” while the AI provider is unreachable gives Test / Infrastructure Issue at 50%. The same message analyzed by the AI might come back as Flaky at 72% after it compared the screenshot. The rule results never say Flaky.
Load step verdict#
error rate = failed requests ÷ all requests × 100 (2 decimals)
Failed = any failed request (no answer, 4xx/5xx, failed assertion)
or any k6 threshold missed
Warning = requests slower than their target (SLA misses)
or saturation users felt: P95 rose by ≥ 100 ms AND to ≥ 1.5 × its base
or a target measured nothing
or the load generator was under pressure
Passed = none of the above
functional check note = peak users ≤ 1 or fewer than 30 requests
run verdict = the worst step (Failed > Warning > Passed)
Example: 12,000 requests, 0 failed, 340 slower than the 800 ms target: Passed with warnings, “340 requests slower than their target”. With 6 failed requests it is Failed, “6 requests failed (0.05%)”.
Comparison with the previous run#
times, throughput change % = (this − baseline) ÷ baseline × 100
similar when |change| < 10 %, or for times when the difference < 20 ms
error rate difference in points; similar under 0.5 pt
Apdex absolute difference; similar under 0.02
overall Worse if P90 or error rate got worse · Better if one got better · else Similar
"compare with care" = peak users differ by more than 10 % or duration by more than 20 %
Example: P90 119 ms → 131 ms is +10 %, but only 12 ms: Similar. P90 400 ms → 520 ms is +30 % and 120 ms: Worse.
Manual pipeline counts#
Total Test Cases = manual cases in the run Executed Test Cases = cases with status Passed or Failed Successful Test Cases = Passed Failed Test Cases = Failed Pending Test Cases = New Run Ratio = Runned Cases (Passed + Failed) vs Pending Cases (New)
Example: 20 cases: 12 Passed, 3 Failed, 5 New. Executed 15, Success - Failed Ratio 12 : 3 : 5 pending, Run Ratio 15 : 5, list chip 12/20.
Didn’t find your answer?Write to support with your project name and what you see on screen.
info@smartestqa.ai