1. Docs
  2. Build tests
  3. InSight
  4. How the numbers work

How the numbers work

Every InSight number is calculated live from your project when you open or refresh the tab. This page gives each formula, what is counted and what is not, and an example you can check.

14 min read Updated CustomersAdmins

In short

On this page you can

  • See which runs and test cases each Execution card counts.
  • Recompute Time saved and the flaky score by hand.
  • Understand when a failure cluster is Fixed.
  • See how the four pillars and Automation maturity are weighted.
  • Know why the classic view’s numbers differ from the tabs.

How to use it

  1. Find the card or column name in the headings below.
  2. Read the formula box: inputs, the rule, rounding.
  3. Check the worked example against your own numbers.
  4. If they still differ, see InSight troubleshooting.

Execution tab#

Period and comparison#

Which runs are in the periodApplies to: every Execution number except Run environment, the strip and the weekly trend
current period  = first day 00:00 → last day 24:00, both days included (d days, at most 365)
previous period = the d days just before
a scenario run belongs to the period in which it started
counted         = every reported test case run: pipeline runs, test cases run on their own,
                  manual pipeline runs, load tests        not counted: dry runs

Example: Last 30 days on 14 November covers 16 October – 14 November (30 days). The change on each card is against 16 September – 15 October.

Runs of test cases that were later deleted still count in the cards, but not in the flaky, slow and cluster lists.

Success rate#

Success rateShown in: summary card, weekly trend, Results by requirement
finished      = scenario runs that are no longer Running
passed        = finished runs with a passing status (Completed)
success rate  = passed ÷ finished × 100        one decimal;  "—" when nothing finished
change        = this period − previous period, in percentage points (green when up)

Example: 200 finished scenario runs, 184 passed: 92.0%. The previous period had 180 of 200 (90.0%): the change shows +2 in green.

Disconnected runs, and manual pipeline cases still New or In progress, count as finished but not passed: they lower the rate without being failed runs.

Runs and Failed runs#

Runs · Failed runsShown in: summary cards, weekly trend
run          = all scenario runs of one pipeline run; a test case started on its own is one run
Runs         = runs in the period                      change: grey (no better direction)
Failed runs  = runs with at least one scenario run Failed or Error     change: green when down
trend        = the same value for each of the last 8 weeks

Example: three pipelines ran every night for 7 days (21 runs) and you started 4 test cases on their own: Runs = 25. Three runs had a failed test case: Failed runs = 3. A run whose only problem was a disconnected test case is not a failed run.

Flaky score and Flaky scenarios#

flips ÷ (n − 1)Shown in: Flaky scenarios card and table
results  = the test case’s finished runs in the period, oldest first: P (passed) or F (anything else)
listed   = only when the results contain at least one P and one F
flips    = how many times a result differs from the one before it
score    = flips ÷ (n − 1), two decimals            n < 5 → "uncertain · n runs"
kept     = score ≥ 0.2, or uncertain;  at most 10, highest score first (uncertain ones rank with the highest)
colour   = < 0.2 green · ≥ 0.2 amber · ≥ 0.8 red
card     = scored test cases in that list (uncertain ones not counted), so never more than 10

Example: results P F P P F P: n = 6, flips P→F, F→P, P→F, F→P = 4, score 4 ÷ 5 = 0.80, red. A test case with P P P P P P F has one flip in 7 runs, 0.17: not listed. A test case with only 3 runs, P F P, is listed as “uncertain · 3 runs”.

Average run time#

Average run timeShown in: summary card
per run   = latest end − earliest start of its scenario runs (runs with both times)
average   = mean over the runs of the period, whole seconds
change    = (this − previous) ÷ previous × 100 %, green when shorter

Example: three runs of 10, 12 and 14 minutes: 12 min 0 s. Previous period 15 min: change −20%, green.

Time saved#

(steps × 30 s − automation time) ÷ 3600Shown in: summary card
passing runs  = finished scenario runs in the period that passed
steps         = every step row those runs wrote
by hand       = steps × 30 s
automation    = the sum of those runs’ own durations
time saved    = max(0, by hand − automation) ÷ 3600, rounded to 0.1 h      "—" with no passing run

Example: 40 passing runs wrote 25 step rows each: 1,000 × 30 s = 30,000 s by hand. They ran for 4 hours, 14,400 s. Time saved: (30,000 − 14,400) ÷ 3600 = 4.3 h.

Pipelines with Skip Successful Logs write few rows for passing runs, so they add little to this number.

Run environment#

Agents · stuck runs · last runShown in: Run environment row (always now)
Agents on / total  = your company’s agents that are online / all of them (not only this project’s)
stuck              = scenario runs still Running more than 2 hours after they started,
                     started in the last 30 days
Last run           = start of the project’s newest run

Example: your company has 3 agents and 1 is running: “Agents 1 / 3 online”, green dot. When an agent is online is explained in When is an agent Online?

Pipeline run strip#

One mark per pipeline runShown in: Pipeline run strip
window  = at least the last 56 days (longer when the period is longer), up to now
marks   = the pipeline’s last 14 runs in that window, oldest left
per mark, each test case’s latest scenario run in that pipeline run:
   Running  if any is still running
   Failed   else if any failed
   Passed   else if all passed
   neutral  otherwise (for example only disconnected runs)
title     = passed / total test cases
Avg time  = mean duration of the marks shown
order     = pipelines whose last run failed, then other pipelines that ran, then never ran; newest first

Example: a nightly pipeline of 10 test cases: last night 9 passed, 1 failed on its first try and passed on a retry inside the same run. Its latest scenario run passed, so the mark is Passed with 10/10, while the Reports list shows that run as Failed (it counts every attempt; see Report’s Result chip).

Failure clusters, Fixed and causes#

Same cause + same signatureShown in: Failure clusters, Causes
input      = failed steps of failed scenario runs in the period
signature  = the error’s first line; IDs and numbers → #, quoted values → …, at most 160 characters
cause      = first match: Environment (connection, agent, browser, session, 502/503/504…)
                        → Assertion (assert, expected, mismatch, does not match…)
                        → Locator (element, selector, xpath, not found/visible/clickable, stale…)
                        → API (status code, http, response, json, 4xx) → Other
cluster    = steps with the same cause and signature
failures   = failed steps in it      scenarios = distinct test cases
Fixed      = every test case in it passed in a later, different run after its last failure here
             (a pass on a retry inside the same pipeline run does not count; runs after the period do)
order      = most scenarios, then most failures; 6 on the card, up to 100 in the dialog
Causes     = one cause per failed scenario run, from one of its failed steps

Example: “Expected 3 items but found 0” and “Expected 12 items but found 0” both become Expected # items but found #: one Assertion cluster. If both test cases pass in tomorrow’s run, the cluster turns Fixed and leaves the list.

Failed runs without a failed step row (for example the browser never started) are not in clusters or causes. Failures by action type and by page count failed steps, top 6.

Slowest scenarios#

p50 · p95 · ChangeShown in: Slowest scenarios
input   = finished runs of the test case in the period with a duration (failed ones included)
pX      = sort the durations; take the value at position ⌈X% × n⌉
Change  = (p95 − previous period’s p95) ÷ previous p95 × 100, whole %     "—" when no previous
colour  = above +10% red · below −10% green
list    = top 8 by p95

Example: 20 runs: p50 is the 10th shortest, p95 the 19th. If p95 was 50 s last period and is 60 s now, Change is +20%, red.

Results by requirement#

Per sub-requirementShown in: Results by requirement
listed        = sub-requirements whose linked test cases have finished runs in the period
Success rate  = passed ÷ finished runs of those test cases
Status        = from each linked test case’s latest run: any failed → Failed, all passed → Passed, else Not run
Risk          = its own severity, else its main requirement’s
order         = highest risk first, then lowest success rate; top 10
bar colour    = ≥ 90% green · ≥ 60% amber · below red

Example: a HIGH sub-requirement with 2 test cases ran 10 times in the period, 7 passed: 70%, amber. Its last runs: one passed, one failed: Status Failed. It is listed above a LOW one at 40%.

Open tasks#

open · new · from runsShown in: Open tasks
open           = tasks whose status is not closed, resolved or done (any date)
new in n days  = open tasks created in the period
from runs      = open tasks created from a test run
list           = the 6 newest open tasks

Example: 14 open tasks, 3 created this week, 9 created by pipelines with Automatic Task: “14 open · 3 new in 7 days · 9 from runs”. A task opened and closed in the same week is not counted.

Weekly trend#

8 × 7 daysShown in: Weekly trend, card trend lines
weeks = 8 blocks of 7 days, the last one ending today
per week: Runs, Failed runs, Success rate (same formulas as the cards)

Example: on a Thursday, each bar runs Friday to Thursday. The weekly trend ignores the period you picked.

Test Health tab#

The four pillars#

Coverage pillarsShown in: pillar cards, Progress
SUBs            = sub-requirements, except those marked not automatable
real scenario   = a test case that is not a template and not tagged "planned"
proving         = has a step that checks a result, in itself or in a template it calls (3 levels):
                  assertion with an actual value · visibility check with a text · AI step in validation mode ·
                  API request with an extractor · assert-text step (mobile, desktop) · JMeter or k6 step
proven SUB      = a linked real scenario is proving and passed its last run
risky SUB       = severity HIGH or CRITICAL (its own, else its main requirement’s), or a linked scenario tagged "critical"

Requirement coverage = proven SUBs ÷ SUBs
Scenario coverage    = SUBs with a linked real scenario that is in a pipeline and has run ÷ SUBs
Risk coverage        = proven risky SUBs ÷ risky SUBs
Action coverage      = actions used by at least one step ÷ all actions
one decimal · "not measured yet" when the divisor is 0
change               = today − the value one week ago, in points

Example: 60 SUBs; 42 have a scenario that asserts the result and passed last time: requirement coverage 42 / 60 = 70.0%. 51 have a scenario in a pipeline that ran: scenario coverage 85.0%. 20 are HIGH or CRITICAL and 10 of them are proven: risk coverage 50.0%. 180 of 300 actions are used: action coverage 60.0%.

A scenario that passes but checks nothing never proves a SUB. That is the most common reason for “all green, low coverage”.

Automation maturity#

Weighted averageShown in: Automation maturity card, Progress
maturity = (0.35 × requirement + 0.25 × scenario + 0.25 × risk + 0.15 × action)
           ÷ (the sum of the weights of the pillars that are measured)          rounded to a whole number
band     = ≥ 80 Good · 50–79 Needs work · < 50 Low          "—" when no pillar is measured
target   = the 80% marker on the pillar cards
"pulling it down most" = the pillar with the lowest percentage

Example with the pillars above: 0.35 × 70 + 0.25 × 85 + 0.25 × 50 + 0.15 × 60 = 24.5 + 21.25 + 12.5 + 9 = 67.25 → 67, Needs work; pulling it down most: risk coverage. A project with no risky SUB has no risk pillar: (24.5 + 21.25 + 9) ÷ 0.75 = 73.

Funnels#

Each step ⊂ the one aboveShown in: Coverage funnel, Scenario funnel
Coverage funnel (SUBs):    all → with a real scenario → … in a pipeline → … that has run
                           → … that is proving and passed
Scenario funnel:           real scenarios except load tests → linked to a SUB → in a pipeline
                           → last run within 30 days → proving → passed its last run
in a pipeline              = a direct row, the test cases of a requirement row, or of a suite;
                             test cases of a pipeline that is only called from another pipeline do not count
biggest loss               = the largest drop between two steps next to each other

Example: 60 → 55 → 48 → 34 → 30: the biggest loss is In a pipeline → Ran, 14 sub-requirements. The funnel’s last step needs a pipeline, so it can be lower than the requirement pillar’s 42.

Modules × coverage#

Per moduleShown in: Modules × coverage
module         = a top-level folder of the Requirements tree (root main requirements: one row each)
Coverage bar   = proven SUBs (green) · with a scenario, not proven (blue) · no scenario (grey)
SUB result     = no scenario → not covered · any scenario failed → failed
                 · all passed → passed · otherwise → not run
Last results   = SUBs per result: failed, not run, passed (not covered ones left out)
module status  = any SUB failed → failed · all passed → passed · all not covered → not covered · else not run
Action coverage = used actions on the pages of the App Map module with the same name
order          = highest risk first, then most unproven SUBs

Example: module Orders has 12 SUBs: 7 proven, 3 with a scenario but no proof, 2 without: bar 58% green, 25% blue, 17% grey. One SUB failed: the module is failed.

Pipeline level#

Draft → Executed → Proving → AutomatedShown in: Pipeline maturity
Draft      = no test cases and no called pipelines, or it has never run
Executed   = its last run was not all passed, or none of its test cases is proving
Proving    = last run all passed and it proves something, but no active schedule
Automated  = Proving + an active schedule
last run   = the newest run in the last 30 days (passed / total per test case)

Example: “Checkout smoke” has 6 test cases, its last run passed 6/6, two of them assert a result, and it runs daily at 06:00: Automated. Remove the schedule and it becomes Proving; let one test fail and it becomes Executed with “Fix the failed ones (1)”.

Fix these first: the gain#

from % → to %Shown in: Fix these first
to    = the pillar’s value if the recommendation is done (capped at the divisor)
gain  = to − from, in points;  at most 3 items, biggest gain first
the Schedule recommendation always has gain 0 and comes last

Example: requirement coverage 42 / 60. “Write scenarios for 6 unlinked sub-requirements” shows Requirement coverage 70% → 80%, because 48 / 60 = 80%.

Trust issues and stale tests#

Counts of real scenariosShown in: Trust issues, Stale tests, Gaps
check no result        = not proving (see the list above)
long                   = more than 60 steps
not run for 30+ days   = last run older than 30 days        never run = no run at all
requirement changed    = the SUB was edited after its scenarios’ newest run
deleted action         = a step points at an action that no longer exists
gap lists              = up to 100 rows each, riskiest first

Example: a login scenario whose last step is “Click Sign In” passes without checking the dashboard appeared: it is counted under “scenarios check no result” until you add a visibility check with text.

Progress#

Rebuilt past pointsShown in: Progress chart
points   = one per day up to 31 days, otherwise one every 7 days back from the end; the last is today (live)
a past point counts what existed at the end of that day:
   records created by then · each test case’s last reported run before then ·
   proof from the date its first checking step was created · pipelines with their current test cases ·
   actions from the date a step first used them
not reflected: deleted records, later edits of steps or links

Example: on 1 October you added assertions to 10 scenarios. The Progress line jumps on 1 October, even though the scenarios existed before. Deleting a requirement today does not lower last month’s points.

Classic view numbers#

Cards of ?classic=1Shown in: the classic Execution layout
Success Rate          = step rows with status Success ÷ all step rows × 100 (one decimal)
Executions            = scenario runs (not pipeline runs)
Auto-Heal Actions     = actions marked as auto-healed
Time Saved            = the sum of all step durations, shown in days, hours or minutes
Defects by Automation = tasks reported by automated tests
date filter           = applies when both dates are set (by creation date)
Weekly Metrics        = per 7-day week: test cases created, sub-requirements created,
                        average requirement AI score (above 0), step success rate,
                        coverage (SUBs with a test case ÷ SUBs)
User Statistics       = per user of your company: Test Case Count (created in this project),
                        Pending Cases (tasks assigned to them that are Open or In Progress, any date)

Example: a run with 10 scenario runs of 20 steps, 190 steps successful: the classic Success Rate is 95.0% (per step), while Execution’s Success rate is per scenario run and could be 80% if 2 of the 10 failed. Classic “Time Saved” is the automation’s own step time, not time saved compared with manual testing.

Didn’t find your answer?Write to support with your project name and what you see on screen.

info@smartestqa.ai
© 2026 Selenity Technology · SmartestQAsmartestqa.aiThis page as MarkdownChecked against smarttest-be 633c2a03 (quality/ExecutionHealthService.java, QualityHealthService.java, QualityRules.java, PipelineSchedule.java, RequirementFolders.java, service/impl/RequirementDashboardService.java, repository/IUserStatisticsRepository.java), smarttest-fe predev 2a8c79cc (execution/execModel.js, periods.js, test-health/healthModel.js)