FAQ, MCP & limits
Answers to the questions customers and admins ask about Schedule, how AI assistants run and schedule pipelines, and the limits that apply.
In short
On this page you can
- Find quick answers about running, scheduling and sharing pipelines.
- Filter the questions for customers or admins.
- Let an AI assistant list, run, follow and schedule pipelines through MCP.
- Check the limits: pipelines, threads, retries, dates, queues and tables.
How to use it
- Pick your audience in the filter bar.
- Open the question closest to yours.
- Follow its link for the step-by-step page.
- Check the Limits table before you plan a big setup.
Questions and answers#
What is the difference between a dry run and a pipeline run?Customers
A dry run in Automate tries one test case while you build it: live log only, nothing saved, it cannot be stopped. A pipeline run executes a saved list of cases with saved settings and writes a report with screenshots, video and logs. Use dry runs to build, pipelines to test (comparison).
Do the cases of a pipeline run in parallel?Customers
The cases of one run execute one after another, in run-list order; that is what makes Setup, Before, After, Teardown and Same session work. To finish sooner, split the cases into several pipelines and start them together: on the server they take separate slots, and on Run on Your Agent each ticked agent runs a whole pipeline. The Thread Count setting is capped by your company licence’s parallel threads (limits). Mind shared test users: two runs logging in with the same user can log each other out.
If I change a test case, do I have to update the pipeline?Customers
No. A pipeline points at the test case, so the next run uses its current steps. New typed values appear in Pipeline Parameters marked NEW; values you changed for this pipeline stay changed. A step you deleted shows DEL until you save.
Can one test case be in several pipelines?Customers
Yes, and that is the usual set-up: the same login case in a smoke pipeline and a nightly regression. Within one pipeline a case can appear once; to run it more often use a repeat or a data loop (Data & loop).
How do I log in once and run all tests in that session?Customers
Put the login case first and turn on Same session on the cases after it: they continue in the same browser without opening the start page again (web). Or mark the login case Before to log in fresh before every test (Same session, Phases).
How do I run the same tests on DEV and on TEST?Customers
Test cases follow the pipeline’s Environment. Make two pipelines with the same run list, one on DEV and one on TEST, or switch the environment and save before you run. The start page needs an address for both (Environment).
Can I run a pipeline every hour, or every 15 minutes during office hours?Customers
Yes, with Custom (Cron): 0 * * * * every hour, */15 8-18 * * 1-5 every 15 minutes on weekdays from 08:00 to 18:45. Leave enough time for one run to finish before the next (Cron expressions).
In which time zone does my schedule run?Customers
On the SmartestQA server’s clock. Check Next run on Scheduled Runs, which is shown in your time, and adjust if needed (Time zone).
How do I pause a schedule for a week?Customers
Set Period to Real-time and save. When you are back, choose the Period again, check the start and end dates and save (Pause a schedule).
Can my laptop agent run the nightly pipeline?Customers
Only while it is on, awake and connected. A sleeping laptop makes the run fail to start or end Disconnected. For web apps the internet can reach, use Run on Server; otherwise a VM or desktop that stays on (Run on Your Agent).
Who receives the report e-mail?Customers
The addresses in Email To and Email CC of the pipeline, when Email Notification is on, always or only on failure. They start as the address of whoever created the project; change them per pipeline. Team chat messages are set in Integrations (E-mail notifications).
Can every deployment start the tests automatically?Customers
Yes. Copy the command from Execute Pipeline via CLI into a step after the deployment, poll the run for its exit code, and fail the build on 1 (CLI and CI/CD). Ready-made CI integrations are described in Integrations › CI/CD.
Who can create, run, stop and delete pipelines?Admins
Everyone with access to the project can create, edit, run and stop pipelines and manage triggers and queues. Deleting a pipeline needs Admin or the pipeline’s creator. A User without access to a project sees “No Project Access”; the Admin grants access per project (Roles).
What counts against our licence?Admins
Every new pipeline adds one to the pipeline counter, also an empty one, and deleting does not take it back. Your company’s licence can limit how many pipelines you create; at the limit you see “Company license limit has been reached. The maximum allowed usage for this operation has been exceeded.” (fix). The licence also caps Thread Count. An Admin sees the licence in ConfigurationCompany SettingsBilling (Billing). Reuse pipelines by changing their run list instead of creating new ones.
Does our data leave the network when pipelines run?Admins
With Run on Server the browser runs on SmartestQA servers and reaches your app over the internet. With Run on Your Agent it runs on your machine inside your network. In both cases results, screenshots and videos are stored in your SmartestQA project. Use test data (Which one to choose).
Which addresses do we allow in our firewall for server runs?Admins
Company Settings › Network & capacity lists Our egress IPs with a copy button. If it says “Not defined yet.”, ask support (Run on Server).
Our server runs often wait. Can we get more capacity?Admins
Check Company Settings › Network & capacity: running and waiting runs, your shared-pool limit and any dedicated servers. SmartestQA can raise the limit or set up dedicated servers; ask info@smartestqa.ai. Meanwhile, agents add capacity you control (Server queue).
Is a webhook trigger safe?Admins
The URL contains a secret token; anyone with the URL can start the pipeline, and only its inputs can be changed from outside. Share it only with the calling system, regenerate it if it leaks, and switch the trigger off when unused (Triggers).
For AI assistants (MCP)#
An assistant connected through the SmartestQA MCP server works with pipelines through three tools. Connect one from Device FarmTest AgentsConnect to Claude (how).
| Tool | Does |
|---|---|
smartest_pipeline | list the project’s pipelines with their tags; run one (it refuses while another run of the project is going, because runs share test users, unless you confirm); status of a run, waiting up to 50 seconds; schedule a pipeline that has passed once with a 5-field cron on the server clock (the assistant asks you for the time and time zone). |
smartest_module_pipelines | Builds and fills a module’s standard pipelines, “<module> - functional”, “- api” and “- security”, plus the project’s Smoke, E2E and Known defects pipelines, from the module’s scenarios and their tags. A new pipeline is scheduled after its first green run. |
smartest_runs | Reports, a run per scenario, step logs with the failed step, pass/fail history of a case or pipeline, what is running now, and stop (after asking you). |
Run the "Checkout – nightly" pipeline in project "Web Shop" and tell me which case failed and why.
Schedule pipeline 42 every weekday at 02:30 our time (we are in Istanbul).
Build the pipelines for the Orders module and start the functional one.
Which pipelines failed in the last seven days, and is it always the same step?Limits#
| Item | Limit |
|---|---|
| Pipelines | Can be limited by your company’s licence. Each create counts, deleting does not refund. |
| Thread Count | Your company licence’s parallel threads |
| Retry Count | 0–5 |
| Wait Timeout · Page Load Timeout | 1–120 s each |
| End date, Daily / Weekly / Monthly | At most 1 year from today |
| Dates, any Period | Year 2000 to 10 years ahead; end after start |
| Cron expression | 5 fields; minute is the smallest step |
| Test case per pipeline | Once; test cases and requirements cannot be mixed |
| Phases of a sub-pipeline row | Test, Setup or Teardown |
| Agent heartbeat | Offline after 90 s without one |
| Disconnected | 3 min without update (when another case reports); 2 h still running (checked every 4 h) |
| Fallback from dedicated to shared servers | After 30 s, when allowed |
| Queue · Interval | 1 min to 7 days (default 15 min) |
| Queue · Max parallel · Max retries | 1–50 · 0–10 (retry after 1 min) |
| Queue item | Failed if no report in 120 min or running over 8 h |
| E-mail trigger | Checked every 120 s by default, 30 s at the fastest; 20 mails per check |
| Excel / CSV import (queues, tables) | 5,000 rows per file |
| Data table rows | Run one after another |
| Rows per page (lists) | 10 · 25 · 50 · 100 |
Didn’t find your answer?Write to support with your project name and what you see on screen.
info@smartestqa.ai