1. Docs
  2. Build tests
  3. Analyze
  4. How the numbers work

How the numbers work

Every number and colour in Analyze, how it is calculated, and an example you can check against your own project.

5 min read Updated CustomersAdmins

In short

On this page you can

  • Read the Quality Score and its label correctly.
  • Understand why the Score Breakdown bars do not average to the score.
  • Check how a scenario’s coverage % and the Coverage: n% chip are counted.
  • Decode the counts on wizard tabs, buttons and folders.
  • See how Analyze data drives the roadmap and Test Health percentages.

How to use it

  1. Find the number you see in the headings below.
  2. Read its formula in the grey box.
  3. Recalculate the worked example with your own figures.
  4. If the number still looks wrong, check the matching entry in Troubleshooting.

Quality Score#

Quality ScoreShown in: Assessment (AI) of a main requirement
score  = the AI's score for the main requirement and how well its subs cover it
         limited to 0–100; 75 if the AI returns no score
shown  = score rounded to a whole percent
label  = Excellent ≥ 80 · Good ≥ 60 · Fair ≥ 40 · Needs Improvement < 40
colour = green ≥ 80 · orange ≥ 60 · red < 60

The score is not calculated from the sub-requirement scores. It is the model’s single verdict on the main requirement, made while reading all its subs.

AI scoreShownLabelColour
86.086%Excellentgreen
72.472%Goodorange
55.055%Fairred
38.038%Needs Improvementred

Note that 40–59 is labelled Fair but drawn in red, and the suggestion icons follow the same colours.

Score Breakdown#

Score BreakdownShown in: Assessment (AI), main requirements with sub-requirements
"Coverage Score (50% weight)"            = the Quality Score (same number)
"Sub Requirements Average (50% weight)"  = mean of the sub scores the AI returned
"n/m sub-requirements assessed"          n = subs that got a score, m = all subs
Quality Score is NOT (Coverage + Average) ÷ 2. The "50% weight" in the labels does not apply.

Example: a main requirement has 4 subs. The AI returns main 72 and sub scores 80, 60 and 90, and skips the fourth.

  • Quality Score and Coverage Score: 72%.
  • Sub Requirements Average: (80 + 60 + 90) ÷ 3 = 76.7, shown 77%.
  • “3/4 sub-requirements assessed”, the skipped one listed under “1 sub-requirement(s) not yet assessed:”, and the summary ends with “Note: 1 sub(s) could not be scored by the LLM.”

A main requirement without subs has no breakdown; its score is the AI’s score for the main alone.

Score of a sub-requirement#

Sub-requirement scoreShown in: Assessment (AI) with a sub selected
sub score = the AI's score for that sub in the main's last run, limited to 0–100
labels and colours as for the Quality Score

A sub keeps the score from the last run of its main requirement. A sub added after that run has no score until you analyze the main again.

Scenario coverage in the AI wizard#

a step is covered (✓) when it uses an existing action, or calls another test case
scenario %   = round(covered steps ÷ steps × 100)
sub, requirement and top chip "Coverage: n%" = round(Σ covered steps ÷ Σ steps × 100)
               over every scenario beneath, selected or not
colour       = green at 100 · orange ≥ 50 · red < 50
before steps exist the scenario chip says "outline" (or "failed"), not 0%
ScenarioStepsCoveredShown
Pay with saved Visa66100% green
Expired card shows error5360% orange
3-D Secure cancelled4125% red
Coverage chip151067% orange

The chip is 10 ÷ 15, not the average of 100, 60 and 25 (62). Record the actions for the ✗ steps and click Check Coverage: every step is matched again against your current actions. See Fix uncovered steps.

The note under the list says scenarios with at least 70% coverage are selected automatically. In this version every scenario starts selected; untick weak ones yourself.

Wizard counts#

WhereCounts
Badge on Requirements & ScenariosAll scenarios in the outline
Badge on Review & CreateScenarios that have steps
Generate steps (n)Selected scenarios (those that already have steps are skipped when it runs)
Continue to review (n)Selected scenarios that have steps; disabled at 0
Create (x req, y TC)x = selected main requirements, y = selected scenarios
“n requirements and m test cases created successfully!”n = main requirements + their sub-requirements; m = test cases
“Outline ready: n requirement(s), m scenario(s)”n = main requirements, m = scenarios

Example: 2 main requirements with 3 subs each are created with 7 test cases. The button read Create (2 req, 7 TC); the message says “8 requirements and 7 test cases created successfully!” (2 mains + 6 subs).

Folder count#

Requirements in this folderShown in: the number next to a folder in the tree
count = main requirements in the folder and in all its sub folders
        (sub-requirements and documents are not counted)

Example: module Payments holds 3 main requirements and a sub folder Refunds with 2 more. Payments shows 5, Refunds shows 2, even if the five mains have 20 sub-requirements between them.

Numbers shown elsewhere that come from Analyze#

Setup Roadmap · Write requirements, Write test casesShown in: Discover › Setup Roadmap
Write requirements  = 50% × (modules with a main requirement ÷ modules) + 50% if any sub exists
                      (no modules → the module part counts as complete)
                      done when every module has one and at least one sub exists
Write test cases    = subs with a test case that is not a template ÷ subs   (at least 10%)
                      done when every sub has one
unfinished steps show at most 99%

Example: 4 modules, 2 of them with a main requirement, and some subs: 50 × 2/4 + 50 = 75%. 20 subs, 15 with tests: 75%. A module counts when a main requirement sits in its folder (or is named like the module). More on the roadmap: Progress meters.

Test Health · Requirement coverage and related pillarsShown in: InSight › Test Health
Requirement coverage = subs with a linked scenario that checks a result and passed its last run ÷ subs
Scenario coverage    = subs with a real scenario (no template) that is in a pipeline and has run ÷ subs
Risk                 = High/Critical subs (or with a 'critical' tagged scenario) proven by a passing scenario
                       ÷ those risky subs
Maturity             = 35% requirement + 25% scenario + 25% risk + 15% action
percentages are rounded to 1 decimal; subs marked "not automatable" are left out of every base

Example: 10 subs. 8 have linked test cases; 6 of those check a result and passed last time. Requirement coverage = 6 ÷ 10 = 60.0%. Linking a test case alone does not move this number: it must check a result and pass.

Sub-requirements are marked “not automatable” by an AI assistant through MCP, with a reason; see MCP.

Didn’t find your answer?Write to support with your project name and what you see on screen.

info@smartestqa.ai