FAQ, MCP & limits
Answers to common questions about Report, how AI assistants use reports, and the limits that apply.
In short
On this page you can
- Find out how long reports, logs and screenshots are kept.
- Learn who can see, stop, export and share runs.
- Understand why counts include retries and iterations.
- Let an AI assistant read runs, logs and screenshots through MCP.
- Check every limit: rows, logs, PDF images, AI input size.
How to use it
- Filter the questions by your role with the buttons above the list.
- Open a question; follow its link for the full procedure.
- For assistants, connect one and try the example prompts in the MCP section.
- Check the limits table before you plan very large runs.
Questions and answers#
Why is my dry run not in Report?Customers
A dry run only streams a live log, which is not stored. Reports come from pipeline runs. Add the test case to a pipeline in Schedule and run it. See the report record and the live log.
How long are reports kept?Customers
Reports, step logs and screenshots are not deleted automatically. Two things expire: the server error logs behind Last 5 Error Logs after 5 days, and dry-run screenshots after 60 minutes (those never reach Report anyway).
Can we delete a report?Admins
Not from the project, not even as Admin. If a run captured data it should not have, write to info@smartestqa.ai with the project and report ID. Removing a report also removes its steps, screenshots and load-test results.
Can I email a report to several people at once?Customers
The Share dialog takes one address per send. For a team, set the pipeline’s automatic email with several addresses in Email To separated by ;, Always or On Failure. See Automatic emails.
Who can see, stop, export and analyze runs?Admins
Everyone with access to the project: Admins for every project of the company, Users for the projects they were given. There is no separate report permission; both roles can stop runs, export, share, tag and run AI analysis. Changing the AI options needs an Admin (Project Panel). See roles.
Why does the Result say 5/6 when the pipeline has 3 test cases?Customers
The chip counts attempts: each data iteration and each automatic retry is one. Three test cases with a 3-row data source on one of them and one retry make six attempts. See Result chip.
Can I rerun only the failed test cases from the report?Customers
Report has no rerun button. Run the pipeline again in Schedule, or put the failed test cases in a pipeline of their own. Automatic retries are set in the pipeline’s retry count.
Which language is the PDF in?Customers
Your interface language when you click Export PDF or Share Report: English, Turkish, German, Spanish or Arabic. Automatic emails after a run are always in English.
Does AI analysis use credits or a quota?Admins
There is no separate quota for it. Each analysis is a call to the project’s AI account (SmartestQA’s, or your own with Own Account) and is logged with its token use in the AI usage of the project. Automatic analysis runs once per failed run.
What does the AI see?Admins
For a failed step: its log, its screenshot and the test case name. For a run review: all test cases and steps with logs, descriptions and AI notes, with e-mails, card numbers, IDs, phone numbers and tokens masked unless Share Sensitive Data is on. See What is sent.
The AI says Real Bug but it is our test. What now?Customers
A verdict is advice; it changes nothing by itself. Read the Evidence line and the screenshot, fix the test, and run again. Clear test case descriptions also help the run review tell expected from unexpected behaviour.
Why does my mobile run have a real video and my web run only Create Video?Customers
AWS Device Farm and BrowserStack record the device and deliver the video. Other runs keep screenshots, and your browser turns them into a video on demand. See Video.
For AI assistants (MCP)#
An assistant connected through the SmartestQA MCP server reads Report with the smartest_runs tool:
| Action | Returns |
|---|---|
list | The Reports list, newest first (optionally only failed). |
report | One run per test case, by report ID or run ID. |
log | One test case run step by step: the failed step, locators and the failure screenshot. |
case_history, pipeline_history | Pass and fail across runs of one test case or pipeline. |
active, runners | What runs now and runner health. |
stop, mail | Stop a run, or email a report to an address you give. Both ask you first. |
Connect an assistant from Device FarmTest AgentsConnect to Claude; see Discover › MCP.
Why did last night's "Nightly regression" run fail? Show me the failed steps and their screenshots.
Which test cases in project "Customer Portal" failed in more than two of the last ten runs?
Read the failed step of report #482, tell me whether it is a product bug or a test problem, and fix the test if it is the test.Limits#
| Item | Limit |
|---|---|
| Rows per page in the Reports list | 10 · 25 · 100 |
| List refresh while a run is going | every 2 seconds |
| Running or New before a run shows as Disconnected | 2 hours |
| Test cases per page in the Summary table | 25 |
| Failure reasons loaded per page / length | 10 rows · 400 characters |
| Step log text stored | about 8,000 characters |
| Server error logs (Last 5 Error Logs, View Logs) | kept 5 days |
| Recipients per Share Report | 1 |
| PDF step-table thumbnails / image size | 60 images · 4 MB each |
| Video built from screenshots | 1 second per screenshot, not stored |
| Screenshot sent with a failure analysis | up to 4 MB |
| AI Report Feedback input | 100,000 characters · 60 steps per run (failed steps always kept) |
| AI Report Feedback log text per step | 400 characters (passed) · 1,500 (failed) |
| AI Report Feedback output | 4 checks per scenario · 4 recommendations · 10 load tests |
| Chart with AI | first 500,000 result rows |
| Number of reports | No limit. Your company’s licence can limit pipeline runs; an Admin sees it in Company Settings › Billing. |
Didn’t find your answer?Write to support with your project name and what you see on screen.
info@smartestqa.ai