FAQ, MCP & limits
Answers to common questions about Automate, how AI assistants use this module, and its limits.
In short
On this page you can
- Find quick answers about copying, undo, templates, dry runs and roles.
- Let an AI assistant write and dry-run test cases through MCP.
- Look up Automate’s limits.
How to use it
- Filter the questions by your role with the chips.
- Open a question for the short answer and the link to details.
- Developers: connect an assistant and use the example prompts.
Questions and answers#
Can I copy a test case?Customers
Not in the screen. Put the shared steps in a template and add it to each new test case, use Expand Scenarios for variations, or ask an AI assistant through MCP to write a new case from an existing one.
Is there an undo?Customers
Before you save: leave the test case with Exit without saving. After deleting: restore from the Archive. There is no undo for a saved change, and no version history to roll back to; History shows who changed what and when.
Why don’t my dry runs show up in Report?Customers
Dry runs are for trying while you build; they write no report and do not change the tree’s status icon. Run the test case in a pipeline in Schedule to keep results. See Run modes.
Should I record, add steps by hand, or let AI write them?Customers
Record the first time you automate a flow: it creates the actions. After that, Generate from Prompt can reuse those actions for new test cases, and the toolbar adds assertions, API and data steps. Always end with a proof step. See Record or add by hand?
If I change a template, what happens to the tests that use it?Customers
They all use the new steps from their next run. A template is a live reference, not a copy. Check its ×n badge first to see how many tests you affect.
Why did renaming a step change other test cases?Customers
A step’s name is its action’s name, and actions are shared. Renaming it renames the action everywhere. Tests keep working, because they point at the action by ID. See Rename a step.
How long should a test case be?Customers
There is no fixed maximum. Aim for one user goal in 5 to 20 steps; put repeated parts in templates. Short tests fail at a step you can find quickly.
Can one test case run with many sets of data?Customers
Yes, in a pipeline. Read values from a dataset with {{excel:name}}_column and set the execute count to {{excel:name}}_#. A dry run uses the first row. See Run a test case once per data row.
Where do I put passwords and tokens?Customers
In a data source, referenced as {{ds:name}}. Values typed into steps are stored in the test case and visible to the whole project. See Secrets and personal data.
Do mobile and desktop test cases work the same way?Customers
Mostly. The tree, templates, variables and saving are the same. Differences: Record opens an inspector and its steps need Save; there is no start page (mobile uses the configured app, desktop a Target Application); dry runs go to a device or a desktop agent instead of a browser; the step types differ. See Record mobile and desktop steps.
Which browser engine runs my web tests?Customers
Selenium or Playwright, chosen per project in the project settings (Script Language). Most steps behave the same. Python steps keep separate code for each engine, and a dry run refuses to start when the code for the active engine is empty.
Can two people work on the same test case?Admins
Not safely at the same time: there is no locking and the last save wins. Split work by test case or folder. See Two people editing.
Who can do what in Automate?Admins
| Action | User | Admin |
|---|---|---|
| Create, edit, save, dry run test cases and folders | Yes, in projects they can access | Yes |
| Archive items | Only items they created entirely | Yes |
| Archive a module, move a module under another folder | No | Yes |
| Restore from the Archive | What they archived | Yes |
| Delete for good, Empty archive | No | Yes |
| Turn AI on (Project Panel › AI Options) | No | Yes |
Roles in general: Roles: who can do what and Roles & permissions.
Does archiving a test case free up licence capacity?Admins
No. Your company’s licence can limit how many test cases you create, and its counter counts test cases created, not test cases that exist. An Admin sees the licence in Company Settings › Billing.
What does Automate send to the AI provider?Admins
Prompts, test case names and steps, action and page names, page descriptions and notes; for AI steps and AI Analyze, information about the current page. Secret-looking values are masked in description requests. See What is sent to the AI.
For AI assistants (MCP)#
An assistant connected through the SmartestQA MCP server can build and run test cases for you. Connect it from Device FarmTest AgentsConnect to Claude; see Connect an assistant and Discover › MCP.
| Tool | What it does |
|---|---|
smartest_context | Where the project stands and the next step. |
smartest_case_rules | The rules for writing steps, which the assistant reads before writing. |
smartest_write_case | Writes a planned scenario’s steps and dry-runs it. A red run returns the failed step; after the third red run the scenario is marked blocked and the assistant asks you how to go on. |
smartest_run_case, smartest_run_status | Run a written scenario again, unchanged; read the result of a dry run that was still running. |
smartest_runs, smartest_heal | Why a run went red and its history; healed locators. |
smartest_propose, smartest_apply | Other changes to records: a preview and a diff first, applied only after you approve. |
The assistant dry-runs on your own agent unless you tell it to use the server. Desktop scenarios run only on an agent. More prompts: Example prompts.
In project "Customer Portal", write a test case for "reset password with an expired link" in module Accounts and dry run it until it passes.
Why did test case 1484 fail last night? Fix the locator if it is a locator problem.
Add an assertion that checks the order total to every test case in folder Checkout that has no proof step.Limits#
| Item | Limit |
|---|---|
| Steps per test case | No fixed limit |
| Step name, folder name | 255 characters |
| Description | 1,000 characters |
| Step timeout | 1–300 seconds |
| Stored variable value | 8,000 characters |
| Wait for recorded web steps | 20 seconds |
| Template list page size | 5 · 15 · 25 · 100 |
| Dry-run screenshots kept | 1 hour |
| Dry-run live log | 5,000 lines or 2 million characters per run, 30 minutes, 50 runs per project |
| One live-log message | 4,000 characters |
| Run on Server queue | 20 waiting runs |
| Browser kept open for AI Analyze (agent) | 15 minutes |
| Recent runs listed | 10 |
| Expand Scenarios | 2–4 variations |
| Generate Test Cases with AI | Up to 30 per request |
| AI descriptions | 8 test cases per AI call, 600 per update, 60 steps per case read |
Questions not answered here: write to info@smartestqa.ai.
Didn’t find your answer?Write to support with your project name and what you see on screen.
info@smartestqa.ai