How the numbers work
Every InSight number is calculated live from your project when you open or refresh the tab. This page gives each formula, what is counted and what is not, and an example you can check.
In short
On this page you can
- See which runs and test cases each Execution card counts.
- Recompute Time saved and the flaky score by hand.
- Understand when a failure cluster is Fixed.
- See how the four pillars and Automation maturity are weighted.
- Know why the classic view’s numbers differ from the tabs.
How to use it
- Find the card or column name in the headings below.
- Read the formula box: inputs, the rule, rounding.
- Check the worked example against your own numbers.
- If they still differ, see InSight troubleshooting.
Execution tab#
Period and comparison#
current period = first day 00:00 → last day 24:00, both days included (d days, at most 365)
previous period = the d days just before
a scenario run belongs to the period in which it started
counted = every reported test case run: pipeline runs, test cases run on their own,
manual pipeline runs, load tests not counted: dry runs
Example: Last 30 days on 14 November covers 16 October – 14 November (30 days). The change on each card is against 16 September – 15 October.
Runs of test cases that were later deleted still count in the cards, but not in the flaky, slow and cluster lists.
Success rate#
finished = scenario runs that are no longer Running passed = finished runs with a passing status (Completed) success rate = passed ÷ finished × 100 one decimal; "—" when nothing finished change = this period − previous period, in percentage points (green when up)
Example: 200 finished scenario runs, 184 passed: 92.0%. The previous period had 180 of 200 (90.0%): the change shows +2 in green.
Disconnected runs, and manual pipeline cases still New or In progress, count as finished but not passed: they lower the rate without being failed runs.
Runs and Failed runs#
run = all scenario runs of one pipeline run; a test case started on its own is one run Runs = runs in the period change: grey (no better direction) Failed runs = runs with at least one scenario run Failed or Error change: green when down trend = the same value for each of the last 8 weeks
Example: three pipelines ran every night for 7 days (21 runs) and you started 4 test cases on their own: Runs = 25. Three runs had a failed test case: Failed runs = 3. A run whose only problem was a disconnected test case is not a failed run.
Flaky score and Flaky scenarios#
results = the test case’s finished runs in the period, oldest first: P (passed) or F (anything else) listed = only when the results contain at least one P and one F flips = how many times a result differs from the one before it score = flips ÷ (n − 1), two decimals n < 5 → "uncertain · n runs" kept = score ≥ 0.2, or uncertain; at most 10, highest score first (uncertain ones rank with the highest) colour = < 0.2 green · ≥ 0.2 amber · ≥ 0.8 red card = scored test cases in that list (uncertain ones not counted), so never more than 10
Example: results P F P P F P: n = 6, flips P→F, F→P, P→F, F→P = 4, score 4 ÷ 5 = 0.80, red. A test case with P P P P P P F has one flip in 7 runs, 0.17: not listed. A test case with only 3 runs, P F P, is listed as “uncertain · 3 runs”.
Average run time#
per run = latest end − earliest start of its scenario runs (runs with both times) average = mean over the runs of the period, whole seconds change = (this − previous) ÷ previous × 100 %, green when shorter
Example: three runs of 10, 12 and 14 minutes: 12 min 0 s. Previous period 15 min: change −20%, green.
Time saved#
passing runs = finished scenario runs in the period that passed steps = every step row those runs wrote by hand = steps × 30 s automation = the sum of those runs’ own durations time saved = max(0, by hand − automation) ÷ 3600, rounded to 0.1 h "—" with no passing run
Example: 40 passing runs wrote 25 step rows each: 1,000 × 30 s = 30,000 s by hand. They ran for 4 hours, 14,400 s. Time saved: (30,000 − 14,400) ÷ 3600 = 4.3 h.
Pipelines with Skip Successful Logs write few rows for passing runs, so they add little to this number.
Run environment#
Agents on / total = your company’s agents that are online / all of them (not only this project’s)
stuck = scenario runs still Running more than 2 hours after they started,
started in the last 30 days
Last run = start of the project’s newest run
Example: your company has 3 agents and 1 is running: “Agents 1 / 3 online”, green dot. When an agent is online is explained in When is an agent Online?
Pipeline run strip#
window = at least the last 56 days (longer when the period is longer), up to now marks = the pipeline’s last 14 runs in that window, oldest left per mark, each test case’s latest scenario run in that pipeline run: Running if any is still running Failed else if any failed Passed else if all passed neutral otherwise (for example only disconnected runs) title = passed / total test cases Avg time = mean duration of the marks shown order = pipelines whose last run failed, then other pipelines that ran, then never ran; newest first
Example: a nightly pipeline of 10 test cases: last night 9 passed, 1 failed on its first try and passed on a retry inside the same run. Its latest scenario run passed, so the mark is Passed with 10/10, while the Reports list shows that run as Failed (it counts every attempt; see Report’s Result chip).
Failure clusters, Fixed and causes#
input = failed steps of failed scenario runs in the period
signature = the error’s first line; IDs and numbers → #, quoted values → …, at most 160 characters
cause = first match: Environment (connection, agent, browser, session, 502/503/504…)
→ Assertion (assert, expected, mismatch, does not match…)
→ Locator (element, selector, xpath, not found/visible/clickable, stale…)
→ API (status code, http, response, json, 4xx) → Other
cluster = steps with the same cause and signature
failures = failed steps in it scenarios = distinct test cases
Fixed = every test case in it passed in a later, different run after its last failure here
(a pass on a retry inside the same pipeline run does not count; runs after the period do)
order = most scenarios, then most failures; 6 on the card, up to 100 in the dialog
Causes = one cause per failed scenario run, from one of its failed steps
Example: “Expected 3 items but found 0” and “Expected 12 items but found 0” both become Expected # items but found #: one Assertion cluster. If both test cases pass in tomorrow’s run, the cluster turns Fixed and leaves the list.
Failed runs without a failed step row (for example the browser never started) are not in clusters or causes. Failures by action type and by page count failed steps, top 6.
Slowest scenarios#
input = finished runs of the test case in the period with a duration (failed ones included) pX = sort the durations; take the value at position ⌈X% × n⌉ Change = (p95 − previous period’s p95) ÷ previous p95 × 100, whole % "—" when no previous colour = above +10% red · below −10% green list = top 8 by p95
Example: 20 runs: p50 is the 10th shortest, p95 the 19th. If p95 was 50 s last period and is 60 s now, Change is +20%, red.
Results by requirement#
listed = sub-requirements whose linked test cases have finished runs in the period Success rate = passed ÷ finished runs of those test cases Status = from each linked test case’s latest run: any failed → Failed, all passed → Passed, else Not run Risk = its own severity, else its main requirement’s order = highest risk first, then lowest success rate; top 10 bar colour = ≥ 90% green · ≥ 60% amber · below red
Example: a HIGH sub-requirement with 2 test cases ran 10 times in the period, 7 passed: 70%, amber. Its last runs: one passed, one failed: Status Failed. It is listed above a LOW one at 40%.
Open tasks#
open = tasks whose status is not closed, resolved or done (any date) new in n days = open tasks created in the period from runs = open tasks created from a test run list = the 6 newest open tasks
Example: 14 open tasks, 3 created this week, 9 created by pipelines with Automatic Task: “14 open · 3 new in 7 days · 9 from runs”. A task opened and closed in the same week is not counted.
Weekly trend#
weeks = 8 blocks of 7 days, the last one ending today per week: Runs, Failed runs, Success rate (same formulas as the cards)
Example: on a Thursday, each bar runs Friday to Thursday. The weekly trend ignores the period you picked.
Test Health tab#
The four pillars#
SUBs = sub-requirements, except those marked not automatable
real scenario = a test case that is not a template and not tagged "planned"
proving = has a step that checks a result, in itself or in a template it calls (3 levels):
assertion with an actual value · visibility check with a text · AI step in validation mode ·
API request with an extractor · assert-text step (mobile, desktop) · JMeter or k6 step
proven SUB = a linked real scenario is proving and passed its last run
risky SUB = severity HIGH or CRITICAL (its own, else its main requirement’s), or a linked scenario tagged "critical"
Requirement coverage = proven SUBs ÷ SUBs
Scenario coverage = SUBs with a linked real scenario that is in a pipeline and has run ÷ SUBs
Risk coverage = proven risky SUBs ÷ risky SUBs
Action coverage = actions used by at least one step ÷ all actions
one decimal · "not measured yet" when the divisor is 0
change = today − the value one week ago, in points
Example: 60 SUBs; 42 have a scenario that asserts the result and passed last time: requirement coverage 42 / 60 = 70.0%. 51 have a scenario in a pipeline that ran: scenario coverage 85.0%. 20 are HIGH or CRITICAL and 10 of them are proven: risk coverage 50.0%. 180 of 300 actions are used: action coverage 60.0%.
A scenario that passes but checks nothing never proves a SUB. That is the most common reason for “all green, low coverage”.
Automation maturity#
maturity = (0.35 × requirement + 0.25 × scenario + 0.25 × risk + 0.15 × action)
÷ (the sum of the weights of the pillars that are measured) rounded to a whole number
band = ≥ 80 Good · 50–79 Needs work · < 50 Low "—" when no pillar is measured
target = the 80% marker on the pillar cards
"pulling it down most" = the pillar with the lowest percentage
Example with the pillars above: 0.35 × 70 + 0.25 × 85 + 0.25 × 50 + 0.15 × 60 = 24.5 + 21.25 + 12.5 + 9 = 67.25 → 67, Needs work; pulling it down most: risk coverage. A project with no risky SUB has no risk pillar: (24.5 + 21.25 + 9) ÷ 0.75 = 73.
Funnels#
Coverage funnel (SUBs): all → with a real scenario → … in a pipeline → … that has run
→ … that is proving and passed
Scenario funnel: real scenarios except load tests → linked to a SUB → in a pipeline
→ last run within 30 days → proving → passed its last run
in a pipeline = a direct row, the test cases of a requirement row, or of a suite;
test cases of a pipeline that is only called from another pipeline do not count
biggest loss = the largest drop between two steps next to each other
Example: 60 → 55 → 48 → 34 → 30: the biggest loss is In a pipeline → Ran, 14 sub-requirements. The funnel’s last step needs a pipeline, so it can be lower than the requirement pillar’s 42.
Modules × coverage#
module = a top-level folder of the Requirements tree (root main requirements: one row each)
Coverage bar = proven SUBs (green) · with a scenario, not proven (blue) · no scenario (grey)
SUB result = no scenario → not covered · any scenario failed → failed
· all passed → passed · otherwise → not run
Last results = SUBs per result: failed, not run, passed (not covered ones left out)
module status = any SUB failed → failed · all passed → passed · all not covered → not covered · else not run
Action coverage = used actions on the pages of the App Map module with the same name
order = highest risk first, then most unproven SUBs
Example: module Orders has 12 SUBs: 7 proven, 3 with a scenario but no proof, 2 without: bar 58% green, 25% blue, 17% grey. One SUB failed: the module is failed.
Pipeline level#
Draft = no test cases and no called pipelines, or it has never run Executed = its last run was not all passed, or none of its test cases is proving Proving = last run all passed and it proves something, but no active schedule Automated = Proving + an active schedule last run = the newest run in the last 30 days (passed / total per test case)
Example: “Checkout smoke” has 6 test cases, its last run passed 6/6, two of them assert a result, and it runs daily at 06:00: Automated. Remove the schedule and it becomes Proving; let one test fail and it becomes Executed with “Fix the failed ones (1)”.
Fix these first: the gain#
to = the pillar’s value if the recommendation is done (capped at the divisor) gain = to − from, in points; at most 3 items, biggest gain first the Schedule recommendation always has gain 0 and comes last
Example: requirement coverage 42 / 60. “Write scenarios for 6 unlinked sub-requirements” shows Requirement coverage 70% → 80%, because 48 / 60 = 80%.
Trust issues and stale tests#
check no result = not proving (see the list above) long = more than 60 steps not run for 30+ days = last run older than 30 days never run = no run at all requirement changed = the SUB was edited after its scenarios’ newest run deleted action = a step points at an action that no longer exists gap lists = up to 100 rows each, riskiest first
Example: a login scenario whose last step is “Click Sign In” passes without checking the dashboard appeared: it is counted under “scenarios check no result” until you add a visibility check with text.
Progress#
points = one per day up to 31 days, otherwise one every 7 days back from the end; the last is today (live) a past point counts what existed at the end of that day: records created by then · each test case’s last reported run before then · proof from the date its first checking step was created · pipelines with their current test cases · actions from the date a step first used them not reflected: deleted records, later edits of steps or links
Example: on 1 October you added assertions to 10 scenarios. The Progress line jumps on 1 October, even though the scenarios existed before. Deleting a requirement today does not lower last month’s points.
Classic view numbers#
?classic=1Shown in: the classic Execution layoutSuccess Rate = step rows with status Success ÷ all step rows × 100 (one decimal)
Executions = scenario runs (not pipeline runs)
Auto-Heal Actions = actions marked as auto-healed
Time Saved = the sum of all step durations, shown in days, hours or minutes
Defects by Automation = tasks reported by automated tests
date filter = applies when both dates are set (by creation date)
Weekly Metrics = per 7-day week: test cases created, sub-requirements created,
average requirement AI score (above 0), step success rate,
coverage (SUBs with a test case ÷ SUBs)
User Statistics = per user of your company: Test Case Count (created in this project),
Pending Cases (tasks assigned to them that are Open or In Progress, any date)
Example: a run with 10 scenario runs of 20 steps, 190 steps successful: the classic Success Rate is 95.0% (per step), while Execution’s Success rate is per scenario run and could be 80% if 2 of the 10 failed. Classic “Time Saved” is the automation’s own step time, not time saved compared with manual testing.
Didn’t find your answer?Write to support with your project name and what you see on screen.
info@smartestqa.ai