1. Docs
  2. Build tests
  3. Report
  4. How the numbers work

How the numbers work

Every number, status and colour in Report, how it is calculated, and an example you can check against your own runs.

9 min read Updated CustomersAdmins

In short

On this page you can

  • See exactly what the Result chip 5/6 counts.
  • Understand why the list can say Failed while the Summary says Passed with warnings.
  • Learn when a run turns Disconnected.
  • Read Attempts, Duration, Total and Actual correctly.
  • Check how a load step’s verdict and its comparison are decided.

How to use it

  1. Find the number you see on screen in the headings below.
  2. Read its formula box: what is counted and what is not.
  3. Recompute the worked example with your own run.
  4. If your result differs, check the notes under the example, then the troubleshooting page.

Result chip (Reports list)#

passed / totalShown in: Reports list › Result
attempt  = one execution of one test case (each iteration and each automatic retry is its own attempt)
passed   = attempts that ended Completed
failed   = attempts that ended Failed or Disconnected
           + planned attempts that never started because the run was stopped
total    = planned attempts: each test case × its execute count (or its data rows),
           requirement rows count their test cases, called pipelines count theirs
           raised to passed + failed when more attempts ran than were planned (retries)
pending  = total − passed − failed − running
colour   = the run status (green Completed, red Failed, amber Running, blue New, grey Disconnected)

Example: a pipeline has Login, Search with a 3-row data source, and Checkout: total = 1 + 3 + 1 = 5. Search row 2 fails, is retried once and passes. Attempts: Login ✓, Search 1 ✓, Search 2 ✕, Search 2 retry ✓, Search 3 ✓, Checkout ✓, so passed = 5, failed = 1. That is 6 attempts for 5 planned, so total becomes 6: the chip shows 5/6, red, because one attempt failed.

The Summary judges each test case by its last attempt instead, so the same run is Passed with warnings there, with Search as Passed on retry. The two views answer different questions: the chip counts every attempt, the Summary says whether each test case finally passed.

Manual pipelines: passed manual cases ÷ all manual cases of the run.

Duration and Took#

Wall-clock timeShown in: Reports list › Duration, Summary › Took
start    = start of the run’s first attempt
end      = end of the run’s last attempt          (empty while any attempt runs)
duration = end − start                             shown as 1h 2m 5s · 4m 12s · 38s
while running: list shows "..."; Summary shows the time so far + "so far"

Example: the first test case starts at 10:00:05, three run in parallel, and the last ends at 10:12:47. Duration is 12m 42s, not the sum of the three test cases. Sorting the list by Duration uses the same value.

Run status (Reports list)#

One status for the whole runShown in: Reports list › Status
the first rule that matches, over all attempts of the run:
  any attempt Running            → Running
  any attempt In progress        → IN_PROGRESS (manual pipelines, shown as is)
  any attempt Failed             → Failed
  any attempt Completed          → Completed
  any attempt Disconnected       → Disconnected
  otherwise                      → New
a stopped run with test cases that never ran → Failed instead of Completed
AttemptsStatusResult chip
4 CompletedCompleted4/4
3 Completed, 1 FailedFailed3/4
4 Completed, 1 DisconnectedCompleted4/5
2 Completed, 1 RunningRunning2/3
Stopped after 2 of 5, both CompletedFailed2/5

The third row is why you should read the chip as well as the status: a disconnected attempt does not make the run Failed.

The 2-hour rule (Disconnected)#

When a run is treated as deadShown in: Reports list, report header, InSight
on screen:  Running or New, and started more than 120 minutes ago   → shown as Disconnected
            finished attempts, but the run was never closed,
            and its last attempt ended more than 120 minutes ago   → treated as finished
on the server: attempts still Running more than 2 hours after start are saved as Disconnected
            (when the Reports list loads, and by a check every 4 hours)

Example: a run starts at 08:00 on your agent and the laptop goes to sleep at 08:20. At 09:30 the row still says Running. From 10:00 on it shows Disconnected; Export PDF, Share Report, tags and AI Analyze unlock, and the dead attempt counts as failed.

A legitimate run longer than 2 hours keeps running as long as its test cases keep finishing: the second rule looks at the last end, not the start.

Run verdict (Summary)#

Passed · Passed with warnings · FailedShown in: Summary › result card
Running                 = the run, or any test case, is still running
No test case ran        = the run has no test case
Failed                  = a test case failed (its last attempt)
Passed with warnings    = no test case failed, but some passed only on a retry,
                          some did not run, or steps ended with a warning
Passed                  = every test case passed on its first attempt and no step warned
sub-line                = "{passed} of {total} test cases passed" (passed on retry counts as passed)
                          · "{n} needed a retry" · "{n} not run" · "{n} step(s) passed with warnings"
Test casesWarning stepsVerdictSub-line
8 passed0Passed8 of 8 test cases passed
7 passed, 1 passed on retry0Passed with warnings8 of 8 · 1 needed a retry
8 passed2Passed with warnings8 of 8 · 2 step(s) passed with warnings
6 passed, 2 failed0Failed6 of 8 test cases passed

A warning step is one that ended with a warning instead of failing, for example a Network step whose rule is set to warn.

Test case state (Summary table)#

One state per test caseShown in: Summary › Test cases › Result, status chips
last attempts = for each iteration, the attempt with the highest retry number
Failed           = a last attempt failed
                   or the run finished but an attempt never completed (disconnected, stopped,
                   or fewer iterations than planned)
Running          = a last attempt is still running, or planned iterations are missing while the run goes
Passed on retry  = all last attempts passed, but an earlier attempt failed
Passed           = all last attempts passed, every planned iteration ran, no earlier failure
Not run          = no attempt was recorded
order in the table: Failed, Passed on retry, Running, Not run, Passed (all passed: slowest first)

Example: Search planned for 3 data rows. Row 1 ✓; row 2 ✕ then retry ✓; row 3 ✓. Last attempts: ✓ ✓ ✓, with an earlier failure: Passed on retry. If row 3 had never run before the run was stopped, the state would be Failed.

Attempts and Duration (Summary table)#

Per test caseShown in: Summary › Test cases
Attempts       = all attempts of the test case in this run (iterations + retries)
Duration       = Σ over iterations of (end − start) of the last attempt     "–" if one is missing
Failure reason = first failed step of the last attempt, by step number, cut at 400 characters
                 (loaded for up to 10 failed rows per page)

Example: Search ran 3 iterations and iteration 2 was retried: Attempts = 4. Last attempts took 20 s, 25 s and 18 s: Duration = 63 s. The failed first try of iteration 2 is not in Duration.

Steps chip and case tab status#

passed steps / all stepsShown in: test case tab, above the step table; tab icons
chip   = step rows with status Success ÷ all step rows of the chosen run and retry
tone   = red if any row failed · green if all succeeded · grey otherwise (warnings)
tab    = after the run: all rows Success → Passed · any row failed → Failed
                        otherwise → Passed with warnings
         while the run goes: from the attempts (Failed, Running, or Passed when all planned iterations passed)

Example: 14 rows, 12 success, 1 failed, 1 warning: chip 12/14 in red, tab Failed. Without the failed row: 13/14 in grey, tab Passed with warnings. One step can write several rows (the action plus an auto-heal or AI note), so the total can be higher than the number of steps in the test case.

Total and Actual#

Two times per stepShown in: step table, Log Detail
Total  = time when the step’s result was recorded − time the step started
         (finding the element, retries, waits, the action, the screenshot)
Actual = time the action itself took, where the step measures it
         (wait time, query time, the JMeter run …); empty when not measured
format: under 1 s → 350ms · under 60 s → 2.41s · 1m 5s · 1h 2m · days

Example: a Wait 5 step shows Actual 5.00s and Total 5.38s; the extra 0.38 s is the screenshot and logging. A click whose locator needed fallbacks can show a Total of 40 s: the runner tried the first locator for the full wait timeout before moving on.

AI confidence#

How sure the verdict isShown in: verdict chips, AI Failure Analysis card
AI answered            → verdict and confidence (0–100) as the AI returned them
AI could not answer    → rules on the error message:
   step could not be built                         Test / Infrastructure Issue   90%
   network, connection, DNS                        Test / Infrastructure Issue   65%
   assertion, does not match                       Real Bug                       60%
   timeout, wait, could not be loaded              Test / Infrastructure Issue   55%
   element not found, stale, not clickable         Test / Infrastructure Issue   50%
   JavaScript error, exception                     Real Bug                       45%
   anything else                                   Unclassified                   25%
request failed         → Unclassified, 0%

Example: “Element not found: css=#submit” while the AI provider is unreachable gives Test / Infrastructure Issue at 50%. The same message analyzed by the AI might come back as Flaky at 72% after it compared the screenshot. The rule results never say Flaky.

Load step verdict#

Passed · Passed with warnings · FailedShown in: Load steps table, load step verdict strip
error rate   = failed requests ÷ all requests × 100           (2 decimals)
Failed       = any failed request (no answer, 4xx/5xx, failed assertion)
               or any k6 threshold missed
Warning      = requests slower than their target (SLA misses)
               or saturation users felt: P95 rose by ≥ 100 ms AND to ≥ 1.5 × its base
               or a target measured nothing
               or the load generator was under pressure
Passed       = none of the above
functional check note = peak users ≤ 1 or fewer than 30 requests
run verdict  = the worst step (Failed > Warning > Passed)

Example: 12,000 requests, 0 failed, 340 slower than the 800 ms target: Passed with warnings, “340 requests slower than their target”. With 6 failed requests it is Failed, “6 requests failed (0.05%)”.

Comparison with the previous run#

Better · Worse · SimilarShown in: Compared with a previous run, vs previous run
times, throughput  change % = (this − baseline) ÷ baseline × 100
                   similar when |change| < 10 %, or for times when the difference < 20 ms
error rate         difference in points; similar under 0.5 pt
Apdex              absolute difference; similar under 0.02
overall            Worse if P90 or error rate got worse · Better if one got better · else Similar
"compare with care" = peak users differ by more than 10 % or duration by more than 20 %

Example: P90 119 ms → 131 ms is +10 %, but only 12 ms: Similar. P90 400 ms → 520 ms is +30 % and 120 ms: Worse.

Manual pipeline counts#

Manual runsShown in: manual pipeline Summary
Total Test Cases      = manual cases in the run
Executed Test Cases   = cases with status Passed or Failed
Successful Test Cases = Passed      Failed Test Cases = Failed      Pending Test Cases = New
Run Ratio             = Runned Cases (Passed + Failed) vs Pending Cases (New)

Example: 20 cases: 12 Passed, 3 Failed, 5 New. Executed 15, Success - Failed Ratio 12 : 3 : 5 pending, Run Ratio 15 : 5, list chip 12/20.

Didn’t find your answer?Write to support with your project name and what you see on screen.

info@smartestqa.ai
© 2026 Selenity Technology · SmartestQAsmartestqa.aiThis page as MarkdownChecked against smarttest-fe predev 2a8c79cc (ReportTable.js, reportStatus.js, testRunRows.js, ReportLogDetailListTable.js, JmeterReportPage/stepVerdict.js, JmeterCompareCard.jsx, ManualReportPage/*), smarttest-be 633c2a03 (TestLogsServiceImpl calculateFinalExecutionSummary, aggregateSuiteStatus, StuckTestLogWatchdog, GeminiAIServiceImpl), smarttest-testsuite 017fcec (test_log_dispatcher.py)