1. Docs
  2. Use cases
  3. Use cases
  4. How do I build, run and schedule a regression suite?

How do I build, run and schedule a regression suite?

A regression suite in SmartestQA is a pipeline: a saved list of test cases that runs in a fixed order. Build it in Schedule, run it once with Save & Execute, set its Period to Daily, Weekly, Monthly or Custom (Cron), and read each run in Report.

10 min read Updated CustomersAdmins

In short

On this page you can

  • Collect the test cases of a regression suite in one pipeline.
  • Log in once in Setup and clean up in Teardown.
  • Run the suite now, on a schedule, or from CI/CD.
  • Stop a run safely and see every past run.
  • Read each run in Report, export it as PDF or send it by e-mail.

How to use it

  1. Open Schedule and click New Pipeline; name it.
  2. Click Add Test Case, tick the cases and click Add to pipeline.
  3. Set the login case to Setup and the clean-up case to Teardown.
  4. Click Save & Execute and check the report.
  5. Choose a Period, set Start Date Time and End Date Time, click Save.

The short answer#

You do not need a script or a separate scheduler. A pipeline holds the test cases of the suite, where they run and with which settings. It starts when you click Run, at the times its Period sets, or from your CI/CD. Every run is kept in Report with its steps, screenshots and logs.

The problem this solves#

Many teams run their regression checks by hand before a release: a list of flows in a document, and a few people who click through them. As the application grows, the round takes longer and gets shortened under time pressure.

Teams that automate often end up with scripts on one machine and a scheduler entry that starts them. Then the questions change: did it run last night, where is the result, and who cleans up the test data when a script stops halfway?

An example#

This is an illustration, not a customer story. A team runs a booking site and wants its twelve main flows checked every night.

  1. A tester opens Schedule, clicks New Pipeline and names it. With Add Test Case the tester ticks the twelve cases and clicks Add to pipeline.
  2. The login case gets the phase Setup and the case that deletes the test bookings gets Teardown. The first case after the login gets Same session, so it continues in the login’s browser.
  3. The tester keeps Run on Server and clicks Save & Execute. The report opens. Its Summary shows the verdict Failed and lists the one failed test case first, with its Failure reason. The tester fixes it and runs the pipeline again until it passes.
  4. The tester sets Period to Daily, Start Date Time to tomorrow at 02:00, an End Date Time, and clicks Save.
  5. Scheduled Runs lists the pipeline. Next run reads 05:00, three hours later than intended, because the server reads the time in its own time zone. The tester sets the time three hours earlier and saves again.
  6. The next morning the night’s run is in Report, and the pipeline’s History lists it on top.

The suite now starts by itself every night, and the result is in one place for the whole team.

By hand, with your own scripts, and with SmartestQA#

QuestionRegression by handYour own scripts and a schedulerSmartestQA
What is the suite?A list of flows in a document. Fine for a short listA folder of scripts in a repositoryA pipeline: a saved list of test cases that runs in a fixed order
Who starts it?The testers, before a releaseA scheduler entry or a CI job you set upRun, the pipeline’s Period, or your CI/CD
Where does it run?On the testers’ machinesOn a machine or runner you maintainOn SmartestQA servers with Run on Server, or with Run on Your Agent
Is test data cleaned up after a failure?If the tester remembersIf the script handles itTeardown runs once at the end and always runs
Where is the result of last night?In notes or a chat messageIn the log or report the scripts write, if someone keeps itIn Report, with steps, screenshots and logs; reports are not deleted automatically
What if a planned run is missed?It is done later, or notDepends on the schedulerThere is no catch-up: the run is skipped and the next one starts at its normal time

Steps#

  1. Create the pipeline

    Open SchedulePipelines and click New Pipeline. SmartestQA creates the pipeline immediately and opens its page. Name it after what it proves.

  2. Add the test cases

    Click Add Test Case. Filter by name, tick cases or a whole folder, then click Add to pipeline. To run everything linked to a requirement instead, use Add Requirement. A pipeline holds test cases or requirements, not both. See Add test cases, requirements or pipelines.

  3. Set the order and the phases

    Drag the rows into the order they should run. Every row starts as a Test. Hover a row, click its phase chip and choose Setup for the login case and Teardown for the clean-up case. Setup runs once at the start; if it fails, the tests are skipped. Teardown runs once at the end and always runs. See Setup, Before, After and Teardown.

  4. Keep one browser session where it helps

    Turn on Same session for a case that should continue in the previous case’s browser, for example after the login. See Continue in the same browser session.

  5. Run it once

    Keep Run on Server, which needs no setup, or choose Run on Your Agent. Click Save & Execute.

    The run is queued and its report opens. Make sure the suite passes before you schedule it: a schedule repeats whatever the pipeline does, failures included.

  6. Put it on a schedule

    A new pipeline has the Period Real-time: it runs only when something starts it. Choose Daily, Weekly, Monthly or Custom (Cron). Start Date Time is the first run, and its time is the time of every run. End Date Time is when the schedule stops. Click Save. See Set a schedule.

  7. Check the next run

    Open ScheduleScheduled Runs. The pipeline is listed with its Period and Next run, shown in your own time.

  8. Read the result

    Open Report and find the run. The Summary sorts test cases as failed, passed on retry, running, not run, then passed, and shows the first failed step next to each. Export PDF and Share Report are enabled once the run has finished. See Run summary, export & share.

Watch it#

The video builds a pipeline of four test cases with Setup, Same session and Teardown, runs it on a server, stops a run, and schedules the pipeline for every night at 02:00.

Video transcript

Schedule lists every pipeline with its test cases, run environment and last result. New Pipeline creates an empty pipeline and opens it. Name it after what it proves. Add Test Case opens the project's test cases. Tick Log in as test user: it will be the Setup. Filter the tree to find the rest: three cases that add, edit and delete a note. Tick all three. The counter now shows four cases selected. Select a case to preview its steps before you add it. Add to pipeline puts the ticked cases into the run list. Every row starts as a normal Test. Hover a row and click its phase chip. Setup runs once at the start. Before and After wrap every test. Teardown runs once at the end. The login becomes Setup: if it fails, the tests are skipped. The delete case becomes Teardown. Teardown always runs, so the note is cleaned up even after a failure. Same session: the case continues in the previous case's browser. No new browser, no start page. Turn it on for every case that should carry on where the login left off. The run list now reads: Setup, two tests, Teardown, all in one browser session. Run on Server needs no setup: the first free server in the shared pool takes the run. Save & Execute saves the pipeline and starts the run. The run is queued, then the report opens on its own. Test Execution Logs follow the run live: the queue, the opened page, then each step. Every step lands in the log as it runs, until the Teardown finishes. Find the pipeline, hover its row and click Run. Run queues the pipeline and opens its report as soon as it starts. Back in Schedule, Last Run Date shows Running, with Stop run beside it. Stop run asks first: the running case finishes, the rest are skipped, and Teardown still runs. The cell shows Stopping until the run ends. Open the pipeline and click History for every past run, with result and status. A new pipeline runs on demand: its Period is Real-time. Open Period to schedule it: Daily, Weekly, Monthly or a cron expression. Choose Daily. The date fields become editable. Start Date Time is the first run, and its time is the time of every run. Here: 02:00. End Date Time is when the schedule stops: at most one year ahead. Save. From now on the pipeline starts by itself every night. Scheduled Runs lists every scheduled pipeline with its period and Next run, in your own time.

Build a pipeline, run it, stop it, and put it on a nightly schedule.

To see how a finished run is read, watch the Report lesson: a passed run and a failed run, down to the step that broke.

What you get#

  • A suite that starts by itself. Daily, weekly or monthly at the time of Start Date Time, or at the times a cron expression of five fields sets.
  • A fixed order with clean-up. Five phases, always run as Setup → Before → Test → After → Teardown. Teardown always runs, so test data is removed even after a failure.
  • Three ways to start it. By hand with Run or Save & Execute, by the schedule, and from a terminal or CI/CD with one command.
  • A report per run. A report appears within seconds of the run starting and fills in live. Each step keeps its log, a screenshot and its duration. Reports are not deleted automatically.
  • History. The pipeline’s History lists every run, newest first, with its result and status. InSight shows the last 14 runs of each pipeline in one strip.
  • Retries when you want them. Retry Count re-runs a failed test 0 to 5 times.

Questions and answers#

What is a regression suite in SmartestQA?Customers

A pipeline in Schedule. It holds test cases, or requirements whose linked test cases it runs, together with the run environment and the settings of the run. You can tag a pipeline; InSight reads the tags to show which kinds of suite a project has, such as Smoke and Regression. See Tag a pipeline.

How often can the suite run?Customers

Daily, Weekly and Monthly run at the time of Start Date Time. Custom (Cron) takes a five-field expression, for example 30 2 * * 1-5 for weekdays at 02:30. Choose an interval longer than one run takes. See Cron expressions.

How long does a schedule keep running?Customers

Until End Date Time. For Daily, Weekly and Monthly the end can be at most one year from today; extend it before it passes to keep the schedule going. After the end date the pipeline stays and can still be run by hand. See Start and end dates.

What happens if a scheduled run is missed?Customers

There is no catch-up: the run is skipped and the next one starts at its normal time. A run on your own agent needs the agent online at that moment. See Missed runs, restarts and long runs.

What happens when one test fails in the middle of the suite?Customers

With Continue On Failure on, the remaining tests still run. With it off, the first failed test skips the rest. In both cases Teardown runs. See What happens when a phase fails.

Can my CI/CD start the suite?Customers

Yes. The pipeline page gives a one-line command for your shell. The run starts like a click on Run, and CI can wait for the result and read an exit code: 0 passed, 1 failed or disconnected. See Run from a terminal or CI/CD.

How does the team get the result?Customers

Every run is in Report for everyone with access to the project. A pipeline can also send the test report by e-mail when a run ends, always or only on failure. See E-mail notifications.

Why is Next run a few hours away from the time I typed?Customers

The server reads the time in its own time zone, and Next run is shown in yours. Trust the Next run column and move the start time by the difference. See Time zone.

Read more#

Didn’t find your answer?Write to support with your project name and what you see on screen.

info@smartestqa.ai