1. Docs
  2. Use cases
  3. Use cases
  4. How do I run tests on every deploy and see what changed?

How do I run tests on every deploy and see what changed?

In SmartestQA your CI starts a pipeline with a ready-made step from Integrations, waits for it and fails the build when a test fails. A second step reports each deploy to Deploys & Repos, which shows the version in every environment, the commits and changed files since the previous deploy, and whether a pipeline ran there afterwards.

11 min read Updated CustomersAdmins

In short

On this page you can

  • Start a pipeline from GitHub Actions, GitLab CI, Jenkins, Azure Pipelines or any shell.
  • Fail the build when a test fails, with a JUnit report in the CI's test tab.
  • Record every deploy per service and environment.
  • See the commits and changed files between two versions.
  • Spot a version that reached an environment and was not tested.

How to use it

  1. Open Integrations, choose your CI tool, pick the Pipeline and click Copy.
  2. Store an agent API key as the CI secret SMARTEST_API_KEY and paste the step into your CI file.
  3. In Deploys & Repos click Add repository and save.
  4. Store the Deploy token as a CI secret and paste the step from CI code & deploy token.
  5. Deploy, click Refresh and read the cell and the deploy detail.

The short answer#

Two CI steps do it. The first is the step that the CI/CD page in Integrations builds for you: it starts a SmartestQA pipeline, checks it every 15 seconds, writes a JUnit report and ends with exit code 0 when every test case passed, 1 when one failed and 2 on a timeout or error. The second is one HTTP request to Deploys & Repos that records the deploy: repository, environment, commit and status.

With both in place, Deploys & Repos shows one row per service and one column per environment. Each cell holds the running commit and version and a test state such as Tested or Not tested. A click opens the deploy detail with the previous version, the commits, the changed files and the pipeline runs that started after the deploy.

The problem this solves#

Most teams deploy more often than they test. A build goes to the test environment several times a day, and the UI tests run when someone remembers to start them, usually the night before a release. When a test then goes red, nobody can say which of the last deploys broke it, so the search starts with a question in the team chat: “what changed since yesterday?”

Two things are missing. The tests are not part of the deploy, so a broken version can sit in an environment untested. And the test tool does not know what was deployed, so a failure cannot be tied to a version, a commit or a changed file.

An example#

This is an illustration, not a customer story. A team runs a web shop with two services, web and api, and deploys both to TEST from GitHub Actions.

  1. They add two steps to the deploy job: the step from the CI/CD page that starts their smoke pipeline, and the deploy notification for each repository.
  2. At 14:10 a developer merges a change to the checkout form. The job deploys web to TEST. In Deploys & Repos the cell for web and TEST shows the new short commit and Not tested.
  3. The same job starts the smoke pipeline and waits. Two of its 20 test cases fail. The step ends with exit code 1, the job goes red, and the cell now reads “2 broken · 18/20”.
  4. The developer clicks the cell. The deploy detail lists the commits and changed files since the previous version, and under Tests after deploy the run that failed. One click opens it in Report at the failing step.
  5. The fix is merged at 14:40. The next deploy runs the same pipeline, it passes, the step ends with 0 and the cell turns to Tested.

Nobody started a test by hand, and nobody had to ask what changed: the failed run and the list of changes are on the same screen.

By hand, with your own scripts, and with SmartestQA#

QuestionTests started by handYour own test scripts in CISmartestQA
Who starts the tests after a deploy?A person, when there is timeThe CI jobThe CI job, with a step the CI/CD page builds for you
Does a red test stop the build?NoYes, if the script returns the right exit codeYes: exit code 0 passed, 1 failed, 2 timeout or error
Where do the browsers run?On someone’s machineOn CI runners you set up and maintainWhere the pipeline is set to run: a SmartestQA server or your agent
Which version was tested?Nobody knows for sureWhatever the job checked out, if you record itEach cell shows the running commit and version with its test state
What changed since the last good version?Ask the teamRead the repository history yourselfThe deploy detail lists commits and changed files next to the runs
Can a version go untested without anyone noticing?YesYes, when the job is skipped or fails earlyThe cell stays amber with Not tested

Steps#

  1. Have a pipeline to run

    Build the pipeline in Schedule and set its environment to DEV, TEST or PRP, because a deploy is matched with pipeline runs by environment. Run a pipeline in that environment after every deploy, for example a smoke pipeline. See Build a pipeline and Before you start.

  2. Generate the CI step

    Open IntegrationsGitHub Actions (or GitLab CI or Jenkins). Under 1. Choose the pipeline pick the Pipeline. Optionally enter Parameters (optional, JSON), which override the pipeline parameters for this run only, and a Timeout (min) between 1 and 1440 (60 by default). See Set up the CI step.

  3. Store the key and paste the step

    Under 2. Add the connection to your CI copy SMARTEST_URL and create the CI secret SMARTEST_API_KEY with an agent API key. Under 3. Add the step pick the CI tool, click Copy and paste the snippet into your CI file. Commit and run the job.

    The log prints “SmartestQA: run started, suiteRuntimeId=…”, progress lines, then a summary with Result, counts and Report. The CI’s test tab shows each test case.

  4. Add the repository

    Open Deploys & ReposRepositories and click Add repository. Pick the provider, paste the Repository URL and type a Service name such as web. Choose Deploy notification only, or Connect with token to also read commits and changed files, and switch on the environments you deploy to. Click Save. See Add a repository.

    “Repository saved.” appears and the CI code & deploy token window opens. The deploy token is shown only once: copy it before you close the window.

  5. Report the deploy from CI

    Store the Deploy token as a CI secret. In the same window pick the tab GitHub Actions, GitLab CI, Jenkins, Shell script or curl and paste the code after your deploy step. It already contains the Webhook URL and the first mapped environment; change the environment per job. See Add the CI step.

  6. Deploy and check

    Run a deploy, then click the Refresh icon on Deploys & ReposOverview: the page does not update by itself.

    The cell for that service and environment shows the short commit, the version and “just now”. Until a pipeline runs there it shows Not tested; after a run without failed test cases it shows Tested.

  7. Read what changed

    Click the cell. The deploy detail shows Change from previous version with the number of commits and files, Tests after deploy with up to 10 pipeline runs, Commits and Show changed files. Click a run to open it in Report. See Deploy detail.

What the CI step returns#

The step waits until the run reaches a final state. Your CI reads the exit code. Full list: Run states and exit codes.

StateMeaningExit code
QUEUED, RUNNING, STOPPINGThe run is waiting, running, or a stop was requestedNone: the step keeps waiting
PASSEDEvery test case passed0
FAILEDAt least one test case failed, or a stopped run left test cases unrun1
DISCONNECTEDA test case has been running for more than 2 hours without news: the agent connection was lost1
Timeout or errorThe step’s timeout passed, the trigger was refused, or the status could not be read 5 times in a row2

What a cell in Deploys & Repos tells you#

Each cell describes the last successful deploy of one service in one environment. How each state is decided: Cell states.

Chip or textColourMeaning
Tested · 18/20GreenThe newest pipeline run after this deploy had no failed test cases. 18 of its 20 test cases passed; the other 2 ended with another result, such as stopped.
2 broken · 18/20RedThe newest run after this deploy has 2 failed test cases.
Test runningBlueThe newest run is still running.
Not testedAmberNo pipeline ran in this environment since the deploy.
No pipeline environmentGreyThe environment is your own, for example STAGING. Pipelines do not run there, so no test state exists.
No deploy yetNoneThe environment is mapped, but no deploy arrived.
Not mappedNoneThe environment is switched off in this repository’s mapping and nothing was deployed there.

What you get#

  • A build that fails on a red test. Exit code 0 means every test case passed, 1 means a failure or a lost agent connection, 2 means a timeout or error. The step prints a link to the SmartestQA report.
  • Test cases in the CI’s test tab. The step writes smartest-junit.xml. Each test case is one entry named name (#id); cases that did not run are skipped.
  • A normal pipeline run. A run started from CI appears in Report, sends the usual notifications and webhooks, and runs where the pipeline is set to run, on a server or an agent.
  • A record of every deploy. Repository, environment, commit, status, version, branch, who deployed it and a link to the CI run. Failed deploys are recorded too, because the ready-made steps run even when the deploy fails.
  • The change between two versions. With a read-only access token the deploy detail lists up to 100 commits and up to 300 changed files since the previous successful deploy in the same environment.
  • Untested releases in view. A cell turns amber with Not tested when a version reached an environment and no pipeline ran there since.
  • A release that is never blocked by the notification. The deploy step ends with || true: if SmartestQA cannot be reached, the deploy still succeeds.

Questions and answers#

Which CI tools does SmartestQA work with?Customers

The CI/CD page in Integrations builds a step for GitHub Actions, GitLab CI, Jenkins, Azure Pipelines and Shell / other CI. The step needs bash and curl, or PowerShell on Windows, where smartest-ci.ps1 is used. See Set up the CI step.

Does a failed test fail my build?Customers

Yes. The SmartestQA CI step ends with exit code 1 when at least one test case failed, when a stopped run left test cases unrun, or when the agent connection was lost. It ends with 0 only when every test case passed. See Run states and exit codes.

Which key does the CI step need, and what can it do?Customers

An agent API key, stored as the CI secret SMARTEST_API_KEY. Runs start with the project access of the user the key belongs to, so anyone with the key can start pipelines in every project that user can open. A dedicated agent for CI lets you revoke CI access alone. See Before you start.

What happens if the pipeline runs longer than the CI timeout?Customers

The CI step stops waiting and ends with exit code 2. The run keeps going in SmartestQA. The timeout is 60 minutes by default and can be set between 1 and 1440. See “timed out after 60 min; the run continues in SmartestQA.”

Can CI pass values such as the environment to the pipeline?Customers

Yes. The CI step takes SMARTEST_PARAMS, a JSON object such as {"ENV":"uat"}, which overrides the pipeline parameters for that run only. The one-line command on the pipeline page sends only the pipeline ID and uses the pipeline’s saved values. See Script settings.

How do I see which commits went into a deploy?Customers

Connect the repository with Connect with token. SmartestQA then reads the commits and changed files between the deploy and the previous successful deploy in the same environment and shows them in the deploy detail. Read-only access is enough. With Deploy notification only you see which version runs where, without a commit list. See Connection modes.

When does a deploy count as tested?Customers

When a pipeline run in the same environment started after the successful deploy and before the next one. The match is by environment and time, for the whole project: a TEST pipeline that only covers the web app also counts for the api deployed to TEST at the same time. See the tested-after window.

Can the deploy notification break my release?Customers

No. The ready-made step in Deploys & Repos ends with || true, so a failed notification never breaks the deploy. The deploy token can only record deploys for its one repository and gives no access to anything else in SmartestQA. See The deploy token.

Can SmartestQA start the tests by itself after a deploy?Customers

Yes, with an Autonomous Agent. Deploy verification runs the pipeline tagged smoke after each deploy, triages new reds and re-checks defects marked resolved. Deploy impact analysis reads the code diff, finds the affected APIs and screens and runs the related scenarios; it needs a repository with an access token. Both check for a new deploy every hour and run only when one has arrived. See After each deploy.

Can another system start a pipeline without waiting for the result?Customers

Yes. A webhook trigger in ScheduleTriggers gives a URL; a POST with a JSON body starts one run and can fill the pipeline’s inputs. Anyone who knows the URL can start the pipeline, so share it only with the calling system. See Triggers.

Read more#

Didn’t find your answer?Write to support with your project name and what you see on screen.

info@smartestqa.ai