FAQ, MCP & limits
Answers to common questions about InSight, how AI assistants work with these numbers, and the limits of every list.
In short
On this page you can
- Find out what InSight counts and leaves out.
- Learn which assumptions are fixed: 30 s per step, the 80% target, the maturity weights.
- Use the copied MCP prompts and the
smartest_statustool with an assistant. - Check list sizes and time windows before you trust an empty card.
- Give a five-minute InSight demo.
How to use it
- Filter the questions by your role with the buttons above the list.
- Open a question; follow its link for the formula or the screen.
- For assistants, connect one and paste a copied prompt.
- Check the limits table when a list looks shorter than expected.
Questions and answers#
Do Admins and Users see different numbers?Admins
No. Anyone with access to the project sees the same InSight, and can save and delete Test Health reports. Access to a project is granted per user; see roles.
Are dry runs counted?Customers
No. Dry runs leave no report, so InSight never sees them. Pipeline runs, test cases started on their own, manual pipeline runs and load tests are counted. See the report record and the live log.
Do automatic retries change the success rate?Customers
Yes. Each attempt is a scenario run, so a test case that failed and then passed on a retry adds one failed and one passed run, and counts toward its flaky score. The pipeline run strip, however, judges each test case by its latest attempt. See Strip marks.
How do load tests count?Customers
In Execution they are runs like any other. In Test Health a JMeter or k6 step counts as a check, so a load test can prove a sub-requirement, but load tests are left out of the scenario funnel.
Why does “Agents online” include agents from other projects?Admins
Agents belong to the company, not to a project, and any project of the company can run on them. The count is the company’s.
Can we change the 30 seconds per step, the 80% target or the maturity weights?Customers
No, they are fixed: 30 s of manual work per step for Time saved, 80% as the pillar target, and weights of 35 / 25 / 25 / 15 for maturity. Quote them with the number when you report it.
Can I see Test Health for one module only?Customers
On screen, use Modules × coverage. An AI assistant can ask for one module’s status through MCP.
How far back can I look?Customers
Execution takes any range of up to 365 days. Progress can go back to the project’s first record; its past points are rebuilt and approximate. For an exact picture of a past day, save a Test Health report on that day.
Where are User Statistics and the weekly metrics chart?Customers
In the classic layout at /dashboard/default?classic=1. Its numbers are counted differently; see Classic view and its numbers.
For AI assistants (MCP)#
InSight is built to hand work to an assistant connected through the SmartestQA MCP server.
- Copied prompts. Every Copy MCP prompt button copies a ready task. Test Health prompts start with a line such as
SmartestQA project 42 · PROVE · sub:318, which the assistant reads as a command for that project, key and record, then works from there. smartest_statusreturns the Test Health numbers for the project or one module: coverage, the funnel, the sub-requirements automation cannot prove, and the open work as keys in priority order.smartest_runsreads runs and logs for the Execution questions; see Report › MCP.- With an API key or MCP sign-in, an assistant can read Test Health and Execution, but not Progress history or saved reports; those need a login in the app.
Connect an assistant from Device FarmTest AgentsConnect to Claude; see Discover › MCP.
What is the Test Health of project "Customer Portal"? Start on the first open item and tell me what you changed.
Which failure cluster in the last 7 days hits the most scenarios, and is it a product bug or a test problem?
List the risky sub-requirements in module Orders that no passing scenario proves, and write scenarios for the first three.Limits#
| Item | Limit |
|---|---|
| Execution period | 1–365 days, default 7 |
| Pipeline run strip | 14 runs per pipeline, looking back at least 56 days |
| Failure clusters | 6 on the card · 100 in the dialog · signature 160 characters |
| Flaky scenarios | 10 · score from 5 runs · last 10 results shown |
| Slowest scenarios | 8 |
| Results by requirement | 10 |
| Failures by action type / by page | 6 each |
| Open tasks listed | 6 |
| Weekly trend | 8 weeks of 7 days |
| Stuck runs | Running over 2 hours, started in the last 30 days |
| Fix these first | 3 |
| Gaps, “check no result” list, not automatable list | 100 rows each |
| Long scenario | over 60 steps |
| Recent run (funnel, pipeline last run, stale) | 30 days |
| Autonomous agents card | last 7 days |
| Progress points | daily up to 31 days, weekly beyond · default 90 days |
| Saved reports | 100 newest listed · name up to 200 characters |
Didn’t find your answer?Write to support with your project name and what you see on screen.
info@smartestqa.ai