Can agents watch my test suite for me?
Yes. Autonomous agents read your runs, reports, defects and app map on a schedule and report what they found: Daily triage groups yesterday’s failures by cause, and Coverage gap report lists what is not tested. An agent run on the SmartestQA server never changes anything: every change waits as a proposal for a person to apply.
In short
On this page you can
- Have yesterday’s failed tests grouped by cause before the team starts work.
- Get a daily or weekly report by e-mail, Slack or Teams.
- See what is not tested, in priority order.
- Read what each run did, step by step, with its tokens and cost.
- Apply or dismiss the changes an agent proposes.
How to use it
- Open Agents and click Install new agent.
- Pick an agent, check Trigger, Reports and Limits, and install it.
- Switch it on with the toggle on its card and click Run now once.
- Open the run on the Runs tab and read Summary, Outcome and the steps.
- Click Apply or Dismiss on each proposal.
The short answer#
Agents come after the order of work. Once you have tests and pipelines, an agent keeps watching them so nobody has to check every morning by hand. Each agent has instructions, a set of tools it may call, a trigger and limits. It runs on a schedule, after a deploy, or when you click Run now, and it leaves a run you can read from start to finish.
The problem this solves#
A suite that runs every night produces results every morning, and somebody has to read them. In many teams one person opens the night’s runs, looks at each red test, sorts them by cause and writes a summary in the team chat. When that person is away, the reds pile up.
Some teams write a script that posts the pass and fail counts to a channel. That tells the team that something is red, not why, and not which failures belong together.
An example#
This is an illustration, not a customer story. A team tests the web application of a delivery service with pipelines that run every night.
- The QA lead opens Agents, clicks Install new agent and picks Daily triage. The QA lead checks Trigger, Reports and Limits and installs it. The message is “Agent created as paused. Review the settings, then activate it.”
- The QA lead switches the agent on with the toggle on its card. The next day at 06:00 it runs: it groups the red test cases since the previous day by the same error on the same page, classifies each group and names the next action. It writes nothing.
- A tester opens the run on the Runs tab and reads Summary and Outcome: seven red test cases in two groups, one classified as a locator change and one as a real bug.
- The team also installs Library maintenance, which proposes tags and fixes, switches it on and clicks Run now. The run starts on the SmartestQA server. The server records each change the agent wants as a PROPOSAL step instead of making it, and the run shows “2 proposals are waiting. An applied proposal runs as you, with the same arguments.”
- The tester opens each proposal, reads what will be sent, clicks Apply on one and Dismiss on the other. The applied call runs as the tester, with the tester’s permissions.
The morning sorting is done before the team starts work, and nothing in the test library changed without a person clicking Apply.
By hand, with your own summary script, and with SmartestQA#
| Question | A person checks every morning | Your own script that posts a summary | SmartestQA |
|---|---|---|---|
| Who reads the night’s failures? | One person, when there is time. Fine for a small suite | The script counts them; a person still reads them | Daily triage, every day at 06:00 |
| Are failures with the same cause grouped? | Yes, by the person’s judgement | Only as far as the script was written to do it | Red test cases are grouped by the same error on the same page, and each group is classified |
| Can it change my tests? | Yes, the person can | Only if you write that into the script | Not in a run on the SmartestQA server: every change is recorded as a proposal until a person clicks Apply |
| Can I see what it did? | Ask the person | Read the script’s log | Each run lists its steps, call by call, with duration, tokens and cost |
| What does it cost to run? | The person’s time each morning | The time to write and maintain the script | AI usage, capped per agent by Max steps per run, Max cost per run (USD) and Daily budget (USD) |
Steps#
Open the catalog
Open Agents in the main menu (the page title is Autonomous Agents) and click Install new agent. Agents are grouped by rhythm: after each deploy, daily, weekly and on demand. See The catalog.
Pick an agent
For a suite that runs every night, start with Daily triage: every day at 06:00 it groups every red test case since the previous day by the same error on the same page, classifies each group and names the next action. Coverage gap report runs every Monday at 07:00 and lists untested points on the app map and sub-requirements without test cases.
Check the settings and install
The settings dialog opens with the agent’s defaults. Look at least at Trigger (when it runs and in which time zone), Reports (who gets the result) and Limits. Click Install.
“Agent created as paused. Review the settings, then activate it.” The card appears on the Agents tab.
Try it once
A paused agent cannot be run. Switch it on with the toggle on its card and click Run now. If the result is useful, leave it on; its trigger takes over.
Read the run
Open the Runs tab and click the run. The top row shows the status, trigger, start, duration, steps, tokens, cost and the model. Summary is the agent’s conclusion in plain words; Outcome is the full result, as a report or as JSON. The steps list what the agent did, call by call. See Read a run.
Apply or dismiss proposals
When a run would change something, such as a locator, a defect, a tag or a test case, the server records the exact call as a proposal instead of making it. Apply runs it now, as you, with your permissions. Dismiss drops it. See Apply or dismiss proposals.
Watch it#
The video shows the review side: the four numbers at the top of Autonomous Agents, the Runs tab, and one run with its top row, Summary, Outcome and steps. It does not show an agent being installed.
Video transcript
Autonomous Agents starts with four numbers: active agents, runs today, proposals to review and spend. The Runs tab lists every run with its agent, status, trigger, duration, steps and cost. Click a run to open its detail. The top row shows trigger, start, duration, steps, tokens, cost and the model that ran. Summary is the agent's conclusion in plain words. Outcome holds the full result, as a report or as JSON. Steps lists what the agent did, call by call. Proposed changes would wait here for you to apply or dismiss.
What you get#
- Failures explained every morning. Daily triage classifies each group of failures, for example as a real bug, a locator change, test data, the environment or flaky. It writes nothing.
- The same picture for everyone. Daily QA report and Weekly management report go to e-mail with a PDF, Slack, Teams, Discord or a webhook.
- What is not tested. Coverage gap report lists the gaps in priority order.
- Checks after a deploy. Deploy agents look for a new deploy every hour and run only when one has arrived.
- Control. New agents start paused. An agent can call only the tools ticked in its settings; anything else is refused and logged.
- Limits. Each agent has Max steps per run, Max cost per run (USD) and Daily budget (USD); the run stops at whichever comes first.
- A full record. The Runs tab keeps the latest 100 runs of the project with status, trigger, duration, steps, proposals and cost, and each run shows the settings it used.
Questions and answers#
Can an agent change my tests without asking?Customers
Not when the SmartestQA server runs it. Every call that would change something is recorded as a proposal, whatever the agent’s autonomy setting says, and nothing changes until a person clicks Apply. See Suggest only or Apply automatically.
Which agent should I install first?Customers
The catalog marks the ones SmartestQA recommends first as Core: Deploy verification, Deploy impact analysis, Daily triage, Daily QA report and Coverage gap report. Daily triage and the two reports need no repository. See The catalog.
How do I keep the AI cost of an agent under control?Customers
With its limits. A run stops at Max steps per run or Max cost per run (USD), and when the agent has spent its Daily budget (USD) the next run is recorded as Budget blocked and does not start. See Limits.
Can I write my own agent?Customers
Yes. A Custom agent takes its instructions in plain words in five boxes, starting with What should the agent do?. Safe rules such as “Never delete anything” are ticked by default. See Instructions.
Where does an agent run?Customers
On the SmartestQA server with the project’s AI connection, or as a scheduled task in Claude Desktop. Some agents work on the server only. See Runs on.
How does an agent know which pipeline is my regression suite?Customers
By tag, not by name. Agents find pipelines by the tags smoke, regression, api and performance and do not touch your other pipelines. See Tag a pipeline.
Who can use agents?Customers
Every role, in projects they can open. An agent can be refused tools that its user’s role has no permission for. See Autonomous Agents.
Read more#
- Autonomous Agents: what an agent is and how a run works.
- Install an agent: every agent of the catalog and when it runs.
- Agent settings: instructions, tools, trigger, reports and limits.
- Runs and proposals: read a run, apply or dismiss what it proposes.
- How do I build, run and schedule a regression suite?: the pipelines the agents watch.
Didn’t find your answer?Write to support with your project name and what you see on screen.
info@smartestqa.ai