---
title: "How the numbers work"
section: "InSight"
description: "How SmartestQA InSight calculates success rate, runs, failed runs, flaky score, average run time, time saved, failure clusters, p50 and p95, the four coverage pillars, maturity, funnels and pipeline levels."
url: https://docs.smartestqa.ai/en/insight-numbers.html
audience: [customer, admin]
updated: 2026-11-08
keywords: "insight formula success rate runs failed runs flaky score flips average run time time saved 30 seconds per step stuck runs pipeline strip failure cluster signature fixed causes p50 p95 requirement coverage scenario coverage risk coverage action coverage maturity weights funnel pipeline level recommendation gain progress classic view"
---
# InSight: How the numbers work

Every InSight number is calculated live from your project when you open or refresh the tab. This page gives each formula, what is counted and what is not, and an example you can check.

## In short

**On this page you can**

- See which runs and test cases each Execution card counts.
- Recompute Time saved and the flaky score by hand.
- Understand when a failure cluster is Fixed .
- See how the four pillars and Automation maturity are weighted.
- Know why the classic view’s numbers differ from the tabs.

**How to use it**

1. Find the card or column name in the headings below.
2. Read the formula box: inputs, the rule, rounding.
3. Check the worked example against your own numbers.
4. If they still differ, see InSight troubleshooting.

## Execution tab

### Period and comparison

**Which runs are in the period** — Applies to: every Execution number except Run environment, the strip and the weekly trend

```text
current period  = first day 00:00 → last day 24:00, both days included (d days, at most 365)
previous period = the d days just before
a scenario run belongs to the period in which it started
counted         = every reported test case run: pipeline runs, test cases run on their own,
                  manual pipeline runs, load tests        not counted: dry runs
```

**Example:** **Last 30 days** on 14 November covers 16 October – 14 November (30 days). The change on each card is against 16 September – 15 October.

Runs of test cases that were later deleted still count in the cards, but not in the flaky, slow and cluster lists.

### Success rate

**Success rate** — Shown in: summary card, weekly trend, Results by requirement

```text
finished      = scenario runs that are no longer Running
passed        = finished runs with a passing status (Completed)
success rate  = passed ÷ finished × 100        one decimal;  "—" when nothing finished
change        = this period − previous period, in percentage points (green when up)
```

**Example:** 200 finished scenario runs, 184 passed: 92.0%. The previous period had 180 of 200 (90.0%): the change shows +2 in green.

[Disconnected runs](https://docs.smartestqa.ai/en/schedule-numbers.html#disconnected), and [manual pipeline cases](https://docs.smartestqa.ai/en/report-summary.html#manual-runs) still New or In progress, count as finished but not passed: they lower the rate without being failed runs.

### Runs and Failed runs

**Runs · Failed runs** — Shown in: summary cards, weekly trend

```text
run          = all scenario runs of one pipeline run; a test case started on its own is one run
Runs         = runs in the period                      change: grey (no better direction)
Failed runs  = runs with at least one scenario run Failed or Error     change: green when down
trend        = the same value for each of the last 8 weeks
```

**Example:** three pipelines [ran every night](https://docs.smartestqa.ai/en/schedule-timing.html) for 7 days (21 runs) and you started 4 test cases on their own: Runs = 25. Three runs had a failed test case: Failed runs = 3. A run whose only problem was a [disconnected](https://docs.smartestqa.ai/en/schedule-numbers.html#disconnected) test case is not a failed run.

### Flaky score and Flaky scenarios

**flips ÷ (n − 1)** — Shown in: Flaky scenarios card and table

```text
results  = the test case’s finished runs in the period, oldest first: P (passed) or F (anything else)
listed   = only when the results contain at least one P and one F
flips    = how many times a result differs from the one before it
score    = flips ÷ (n − 1), two decimals            n < 5 → "uncertain · n runs"
kept     = score ≥ 0.2, or uncertain;  at most 10, highest score first (uncertain ones rank with the highest)
colour   = < 0.2 green · ≥ 0.2 amber · ≥ 0.8 red
card     = scored test cases in that list (uncertain ones not counted), so never more than 10
```

**Example:** results P F P P F P: n = 6, flips P→F, F→P, P→F, F→P = 4, score 4 ÷ 5 = 0.80, red. A test case with P P P P P P F has one flip in 7 runs, 0.17: not listed. A test case with only 3 runs, P F P, is listed as “uncertain · 3 runs”.

### Average run time

**Average run time** — Shown in: summary card

```text
per run   = latest end − earliest start of its scenario runs (runs with both times)
average   = mean over the runs of the period, whole seconds
change    = (this − previous) ÷ previous × 100 %, green when shorter
```

**Example:** three runs of 10, 12 and 14 minutes: 12 min 0 s. Previous period 15 min: change −20%, green.

### Time saved

**(steps × 30 s − automation time) ÷ 3600** — Shown in: summary card

```text
passing runs  = finished scenario runs in the period that passed
steps         = every step row those runs wrote
by hand       = steps × 30 s
automation    = the sum of those runs’ own durations
time saved    = max(0, by hand − automation) ÷ 3600, rounded to 0.1 h      "—" with no passing run
```

**Example:** 40 passing runs wrote 25 step rows each: 1,000 × 30 s = 30,000 s by hand. They ran for 4 hours, 14,400 s. Time saved: (30,000 − 14,400) ÷ 3600 = 4.3 h.

Pipelines with [**Skip Successful Logs**](https://docs.smartestqa.ai/en/schedule-settings.html#test-management) write few rows for passing runs, so they add little to this number.

### Run environment

**Agents · stuck runs · last run** — Shown in: Run environment row (always now)

```text
Agents on / total  = your company’s agents that are online / all of them (not only this project’s)
stuck              = scenario runs still Running more than 2 hours after they started,
                     started in the last 30 days
Last run           = start of the project’s newest run
```

**Example:** your company has 3 [agents](https://docs.smartestqa.ai/en/agent-pool-agents.html#agent-list) and 1 is running: “Agents 1 / 3 online”, green dot. When an agent is online is explained in [When is an agent Online?](https://docs.smartestqa.ai/en/quick-start-connect.html#agent-online-rule)

### Pipeline run strip

**One mark per pipeline run** — Shown in: Pipeline run strip

```text
window  = at least the last 56 days (longer when the period is longer), up to now
marks   = the pipeline’s last 14 runs in that window, oldest left
per mark, each test case’s latest scenario run in that pipeline run:
   Running  if any is still running
   Failed   else if any failed
   Passed   else if all passed
   neutral  otherwise (for example only disconnected runs)
title     = passed / total test cases
Avg time  = mean duration of the marks shown
order     = pipelines whose last run failed, then other pipelines that ran, then never ran; newest first
```

**Example:** a nightly pipeline of 10 test cases: last night 9 passed, 1 failed on its first try and passed on a [retry](https://docs.smartestqa.ai/en/report-steps.html#runs-and-retries) inside the same run. Its latest scenario run passed, so the mark is **Passed** with 10/10, while the [Reports list](https://docs.smartestqa.ai/en/report-list.html) shows that run as Failed (it counts every attempt; see [Report’s Result chip](https://docs.smartestqa.ai/en/report-numbers.html#result-chip)).

### Failure clusters, Fixed and causes

**Same cause + same signature** — Shown in: Failure clusters, Causes

```text
input      = failed steps of failed scenario runs in the period
signature  = the error’s first line; IDs and numbers → #, quoted values → …, at most 160 characters
cause      = first match: Environment (connection, agent, browser, session, 502/503/504…)
                        → Assertion (assert, expected, mismatch, does not match…)
                        → Locator (element, selector, xpath, not found/visible/clickable, stale…)
                        → API (status code, http, response, json, 4xx) → Other
cluster    = steps with the same cause and signature
failures   = failed steps in it      scenarios = distinct test cases
Fixed      = every test case in it passed in a later, different run after its last failure here
             (a pass on a retry inside the same pipeline run does not count; runs after the period do)
order      = most scenarios, then most failures; 6 on the card, up to 100 in the dialog
Causes     = one cause per failed scenario run, from one of its failed steps
```

**Example:** “Expected 3 items but found 0” and “Expected 12 items but found 0” both become `Expected # items but found #`: one **Assertion** cluster. If both test cases pass in tomorrow’s run, the cluster turns **Fixed** and leaves the list.

Failed runs without a failed step row (for example the browser never started) are not in clusters or causes. Failures by action type and by page count failed steps, top 6.

### Slowest scenarios

**p50 · p95 · Change** — Shown in: Slowest scenarios

```text
input   = finished runs of the test case in the period with a duration (failed ones included)
pX      = sort the durations; take the value at position ⌈X% × n⌉
Change  = (p95 − previous period’s p95) ÷ previous p95 × 100, whole %     "—" when no previous
colour  = above +10% red · below −10% green
list    = top 8 by p95
```

**Example:** 20 runs: p50 is the 10th shortest, p95 the 19th. If p95 was 50 s last period and is 60 s now, Change is +20%, red.

### Results by requirement

**Per sub-requirement** — Shown in: Results by requirement

```text
listed        = sub-requirements whose linked test cases have finished runs in the period
Success rate  = passed ÷ finished runs of those test cases
Status        = from each linked test case’s latest run: any failed → Failed, all passed → Passed, else Not run
Risk          = its own severity, else its main requirement’s
order         = highest risk first, then lowest success rate; top 10
bar colour    = ≥ 90% green · ≥ 60% amber · below red
```

**Example:** a HIGH sub-requirement with 2 test cases ran 10 times in the period, 7 passed: 70%, amber. Its last runs: one passed, one failed: Status Failed. It is listed above a LOW one at 40%.

### Open tasks

**open · new · from runs** — Shown in: Open tasks

```text
open           = tasks whose status is not closed, resolved or done (any date)
new in n days  = open tasks created in the period
from runs      = open tasks created from a test run
list           = the 6 newest open tasks
```

**Example:** 14 [open tasks](https://docs.smartestqa.ai/en/tasks-statuses.html#defaults), 3 created this week, 9 created by pipelines with [Automatic Task](https://docs.smartestqa.ai/en/tasks-from-runs.html#auto-task): “14 open · 3 new in 7 days · 9 from runs”. A task opened and closed in the same week is not counted.

### Weekly trend

**8 × 7 days** — Shown in: Weekly trend, card trend lines

```text
weeks = 8 blocks of 7 days, the last one ending today
per week: Runs, Failed runs, Success rate (same formulas as the cards)
```

**Example:** on a Thursday, each bar runs Friday to Thursday. The weekly trend ignores the period you picked.

## Test Health tab

### The four pillars

**Coverage pillars** — Shown in: pillar cards, Progress

```text
SUBs            = sub-requirements, except those marked not automatable
real scenario   = a test case that is not a template and not tagged "planned"
proving         = has a step that checks a result, in itself or in a template it calls (3 levels):
                  assertion with an actual value · visibility check with a text · AI step in validation mode ·
                  API request with an extractor · assert-text step (mobile, desktop) · JMeter or k6 step
proven SUB      = a linked real scenario is proving and passed its last run
risky SUB       = severity HIGH or CRITICAL (its own, else its main requirement’s), or a linked scenario tagged "critical"

Requirement coverage = proven SUBs ÷ SUBs
Scenario coverage    = SUBs with a linked real scenario that is in a pipeline and has run ÷ SUBs
Risk coverage        = proven risky SUBs ÷ risky SUBs
Action coverage      = actions used by at least one step ÷ all actions
one decimal · "not measured yet" when the divisor is 0
change               = today − the value one week ago, in points
```

**Example:** 60 SUBs; 42 have a scenario that [asserts the result](https://docs.smartestqa.ai/en/automate-steps-assertions.html#assertion-step) and passed last time: requirement coverage 42 / 60 = 70.0%. 51 have a scenario [in a pipeline](https://docs.smartestqa.ai/en/schedule-create.html#add-content) that ran: scenario coverage 85.0%. 20 are HIGH or CRITICAL and 10 of them are proven: risk coverage 50.0%. 180 of 300 [actions](https://docs.smartestqa.ai/en/discover-actions.html) are used: action coverage 60.0%.

A scenario that passes but checks nothing never proves a SUB. That is the most common reason for “all green, low coverage”.

### Automation maturity

**Weighted average** — Shown in: Automation maturity card, Progress

```text
maturity = (0.35 × requirement + 0.25 × scenario + 0.25 × risk + 0.15 × action)
           ÷ (the sum of the weights of the pillars that are measured)          rounded to a whole number
band     = ≥ 80 Good · 50–79 Needs work · < 50 Low          "—" when no pillar is measured
target   = the 80% marker on the pillar cards
"pulling it down most" = the pillar with the lowest percentage
```

**Example** with the pillars above: 0.35 × 70 + 0.25 × 85 + 0.25 × 50 + 0.15 × 60 = 24.5 + 21.25 + 12.5 + 9 = 67.25 → 67, Needs work; pulling it down most: risk coverage. A project with no risky SUB has no risk pillar: (24.5 + 21.25 + 9) ÷ 0.75 = 73.

### Funnels

**Each step ⊂ the one above** — Shown in: Coverage funnel, Scenario funnel

```text
Coverage funnel (SUBs):    all → with a real scenario → … in a pipeline → … that has run
                           → … that is proving and passed
Scenario funnel:           real scenarios except load tests → linked to a SUB → in a pipeline
                           → last run within 30 days → proving → passed its last run
in a pipeline              = a direct row, the test cases of a requirement row, or of a suite;
                             test cases of a pipeline that is only called from another pipeline do not count
biggest loss               = the largest drop between two steps next to each other
```

**Example:** 60 → 55 → 48 → 34 → 30: the biggest loss is In a pipeline → Ran, 14 sub-requirements. The funnel’s last step needs a pipeline, so it can be lower than the requirement pillar’s 42.

### Modules × coverage

**Per module** — Shown in: Modules × coverage

```text
module         = a top-level folder of the Requirements tree (root main requirements: one row each)
Coverage bar   = proven SUBs (green) · with a scenario, not proven (blue) · no scenario (grey)
SUB result     = no scenario → not covered · any scenario failed → failed
                 · all passed → passed · otherwise → not run
Last results   = SUBs per result: failed, not run, passed (not covered ones left out)
module status  = any SUB failed → failed · all passed → passed · all not covered → not covered · else not run
Action coverage = used actions on the pages of the App Map module with the same name
order          = highest risk first, then most unproven SUBs
```

**Example:** [module](https://docs.smartestqa.ai/en/analyze-folders.html#modules) Orders has 12 SUBs: 7 proven, 3 with a scenario but no proof, 2 without: bar 58% green, 25% blue, 17% grey. One SUB failed: the module is failed.

### Pipeline level

**Draft → Executed → Proving → Automated** — Shown in: Pipeline maturity

```text
Draft      = no test cases and no called pipelines, or it has never run
Executed   = its last run was not all passed, or none of its test cases is proving
Proving    = last run all passed and it proves something, but no active schedule
Automated  = Proving + an active schedule
last run   = the newest run in the last 30 days (passed / total per test case)
```

**Example:** “Checkout smoke” has 6 test cases, its last run passed 6/6, two of them [assert a result](https://docs.smartestqa.ai/en/automate-steps-assertions.html), and it [runs daily at 06:00](https://docs.smartestqa.ai/en/schedule-timing.html#set-schedule): **Automated**. Remove the schedule and it becomes Proving; let one test fail and it becomes Executed with “Fix the failed ones (1)”.

### Fix these first: the gain

**from % → to %** — Shown in: Fix these first

```text
to    = the pillar’s value if the recommendation is done (capped at the divisor)
gain  = to − from, in points;  at most 3 items, biggest gain first
the Schedule recommendation always has gain 0 and comes last
```

**Example:** requirement coverage 42 / 60. “Write scenarios for 6 unlinked sub-requirements” shows Requirement coverage 70% → 80%, because 48 / 60 = 80%.

### Trust issues and stale tests

**Counts of real scenarios** — Shown in: Trust issues, Stale tests, Gaps

```text
check no result        = not proving (see the list above)
long                   = more than 60 steps
not run for 30+ days   = last run older than 30 days        never run = no run at all
requirement changed    = the SUB was edited after its scenarios’ newest run
deleted action         = a step points at an action that no longer exists
gap lists              = up to 100 rows each, riskiest first
```

**Example:** a login scenario whose last step is “Click Sign In” passes without checking the dashboard appeared: it is counted under “scenarios check no result” until you add a [visibility check with text](https://docs.smartestqa.ai/en/automate-steps-browser.html#visibility-check).

### Progress

**Rebuilt past points** — Shown in: Progress chart

```text
points   = one per day up to 31 days, otherwise one every 7 days back from the end; the last is today (live)
a past point counts what existed at the end of that day:
   records created by then · each test case’s last reported run before then ·
   proof from the date its first checking step was created · pipelines with their current test cases ·
   actions from the date a step first used them
not reflected: deleted records, later edits of steps or links
```

**Example:** on 1 October you added [assertions](https://docs.smartestqa.ai/en/automate-steps-assertions.html) to 10 scenarios. The Progress line jumps on 1 October, even though the scenarios existed before. Deleting a requirement today does not lower last month’s points.

## Classic view numbers

**Cards of ?classic=1** — Shown in: the classic Execution layout

```text
Success Rate          = step rows with status Success ÷ all step rows × 100 (one decimal)
Executions            = scenario runs (not pipeline runs)
Auto-Heal Actions     = actions marked as auto-healed
Time Saved            = the sum of all step durations, shown in days, hours or minutes
Defects by Automation = tasks reported by automated tests
date filter           = applies when both dates are set (by creation date)
Weekly Metrics        = per 7-day week: test cases created, sub-requirements created,
                        average requirement AI score (above 0), step success rate,
                        coverage (SUBs with a test case ÷ SUBs)
User Statistics       = per user of your company: Test Case Count (created in this project),
                        Pending Cases (tasks assigned to them that are Open or In Progress, any date)
```

**Example:** a run with 10 scenario runs of 20 steps, 190 steps successful: the classic Success Rate is 95.0% (per step), while Execution’s [Success rate](https://docs.smartestqa.ai/en/insight-numbers.html#success-rate) is per scenario run and could be 80% if 2 of the 10 failed. Classic “Time Saved” is the automation’s own step time, not time saved compared with manual testing.
