---
title: "How do I run tests on every deploy and see what changed?"
section: "Use cases"
description: "Run SmartestQA tests on every deploy: start a pipeline from GitHub Actions, GitLab CI, Jenkins or Azure Pipelines, fail the build on a red test, record each deploy, and see the commits, changed files and test state per environment."
url: https://docs.smartestqa.ai/en/use-case-ci-cd.html
audience: [customer, admin]
updated: 2026-11-12
keywords: "ci cd run tests on every deploy after deployment github actions gitlab ci jenkins azure pipelines fail the build junit exit code deploy webhook deploy token what changed commits changed files tested after deploy not tested smoke test after release deploy verification deploy impact analysis trigger pipeline from ci"
---
# Use cases: How do I run tests on every deploy and see what changed?

In SmartestQA your CI starts a pipeline with a ready-made step from Integrations, waits for it and fails the build when a test fails. A second step reports each deploy to Deploys & Repos, which shows the version in every environment, the commits and changed files since the previous deploy, and whether a pipeline ran there afterwards.

## In short

**On this page you can**

- Start a pipeline from GitHub Actions, GitLab CI, Jenkins, Azure Pipelines or any shell.
- Fail the build when a test fails, with a JUnit report in the CI's test tab.
- Record every deploy per service and environment.
- See the commits and changed files between two versions.
- Spot a version that reached an environment and was not tested.

**How to use it**

1. Open Integrations , choose your CI tool, pick the Pipeline and click Copy .
2. Store an agent API key as the CI secret SMARTEST_API_KEY and paste the step into your CI file.
3. In Deploys & Repos click Add repository and save.
4. Store the Deploy token as a CI secret and paste the step from CI code & deploy token .
5. Deploy, click Refresh and read the cell and the deploy detail.

## The short answer

Two CI steps do it. The first is the step that the **CI/CD** page in Integrations builds for you: it starts a SmartestQA pipeline, checks it every 15 seconds, writes a JUnit report and ends with exit code 0 when every test case passed, 1 when one failed and 2 on a timeout or error. The second is one HTTP request to **Deploys & Repos** that records the deploy: repository, environment, commit and status.

With both in place, Deploys & Repos shows one row per service and one column per environment. Each cell holds the running commit and version and a test state such as **Tested** or **Not tested**. A click opens the deploy detail with the previous version, the commits, the changed files and the pipeline runs that started after the deploy.

## The problem this solves

Most teams deploy more often than they test. A build goes to the test environment several times a day, and the UI tests run when someone remembers to start them, usually the night before a release. When a test then goes red, nobody can say which of the last deploys broke it, so the search starts with a question in the team chat: “what changed since yesterday?”

Two things are missing. The tests are not part of the deploy, so a broken version can sit in an environment untested. And the test tool does not know what was deployed, so a failure cannot be tied to a version, a commit or a changed file.

## An example

This is an illustration, not a customer story. A team runs a web shop with two services, `web` and `api`, and deploys both to TEST from GitHub Actions.

1. They add two steps to the deploy job: the step from the CI/CD page that starts their smoke pipeline, and the deploy notification for each repository.
2. At 14:10 a developer merges a change to the checkout form. The job deploys `web` to TEST. In Deploys & Repos the cell for `web` and TEST shows the new short commit and **Not tested**.
3. The same job starts the smoke pipeline and waits. Two of its 20 test cases fail. The step ends with exit code 1, the job goes red, and the cell now reads “2 broken · 18/20”.
4. The developer clicks the cell. The deploy detail lists the commits and changed files since the previous version, and under [**Tests after deploy**](https://docs.smartestqa.ai/en/deployments-overview.html#deploy-detail) the run that failed. One click opens it in Report at the failing step.
5. The fix is merged at 14:40. The next deploy runs the same pipeline, it passes, the step ends with 0 and the cell turns to **Tested**.

Nobody started a test by hand, and nobody had to ask what changed: the failed run and the list of changes are on the same screen.

## By hand, with your own scripts, and with SmartestQA

| Question | Tests started by hand | Your own test scripts in CI | SmartestQA |
| --- | --- | --- | --- |
| Who starts the tests after a deploy? | A person, when there is time | The CI job | The CI job, with a step the CI/CD page builds for you |
| Does a red test stop the build? | No | Yes, if the script returns the right exit code | Yes: exit code 0 passed, 1 failed, 2 timeout or error |
| Where do the browsers run? | On someone’s machine | On CI runners you set up and maintain | Where the pipeline is set to run: a SmartestQA server or your agent |
| Which version was tested? | Nobody knows for sure | Whatever the job checked out, if you record it | Each cell shows the running commit and version with its test state |
| What changed since the last good version? | Ask the team | Read the repository history yourself | The deploy detail lists commits and changed files next to the runs |
| Can a version go untested without anyone noticing? | Yes | Yes, when the job is skipped or fails early | The cell stays amber with **Not tested** |

## Steps

1. **Have a pipeline to run**

   Build the pipeline in Schedule and set its environment to DEV, TEST or PRP, because a deploy is matched with pipeline runs by environment. Run a pipeline in that environment after every deploy, for example a smoke pipeline. See [Build a pipeline](https://docs.smartestqa.ai/en/schedule-create.html) and [Before you start](https://docs.smartestqa.ai/en/deployments.html#before-you-start).
2. **Generate the CI step**

   Open **Integrations** › **GitHub Actions** (or GitLab CI or Jenkins). Under **1. Choose the pipeline** pick the **Pipeline**. Optionally enter [**Parameters (optional, JSON)**](https://docs.smartestqa.ai/en/integrations-cicd.html#set-up), which override the pipeline parameters for this run only, and a [**Timeout (min)**](https://docs.smartestqa.ai/en/integrations-cicd.html#settings) between 1 and 1440 (60 by default). See [Set up the CI step](https://docs.smartestqa.ai/en/integrations-cicd.html#set-up).
3. **Store the key and paste the step**

   Under **2. Add the connection to your CI** copy `SMARTEST_URL` and create the CI secret `SMARTEST_API_KEY` with an agent API key. Under **3. Add the step** pick the [**CI tool**](https://docs.smartestqa.ai/en/integrations-cicd.html#set-up), click **Copy** and paste the snippet into your CI file. Commit and run the job.

   The log prints “SmartestQA: run started, suiteRuntimeId=…”, progress lines, then a summary with Result, counts and Report. The CI’s test tab shows each test case.
4. **Add the repository**

   Open **Deploys & Repos** › **Repositories** and click [**Add repository**](https://docs.smartestqa.ai/en/deployments-repos.html#add-repo). Pick the provider, paste the [**Repository URL**](https://docs.smartestqa.ai/en/deployments-repos.html#fields) and type a **Service name** such as `web`. Choose [**Deploy notification only**](https://docs.smartestqa.ai/en/deployments-repos.html#connection-modes), or [**Connect with token**](https://docs.smartestqa.ai/en/deployments-repos.html#connection-modes) to also read commits and changed files, and switch on the environments you deploy to. Click **Save**. See [Add a repository](https://docs.smartestqa.ai/en/deployments-repos.html#add-repo).

   “Repository saved.” appears and the [**CI code & deploy token**](https://docs.smartestqa.ai/en/deployments-ci.html#add-ci-step) window opens. The deploy token is shown only once: copy it before you close the window.
5. **Report the deploy from CI**

   Store the [**Deploy token**](https://docs.smartestqa.ai/en/deployments-ci.html#deploy-token) as a CI secret. In the same window pick the tab **GitHub Actions**, **GitLab CI**, **Jenkins**, **Shell script** or **curl** and paste the code after your deploy step. It already contains the **Webhook URL** and the first mapped environment; change the environment per job. See [Add the CI step](https://docs.smartestqa.ai/en/deployments-ci.html#add-ci-step).
6. **Deploy and check**

   Run a deploy, then click the **Refresh** icon on **Deploys & Repos** › **Overview**: the page does not update by itself.

   The cell for that service and environment shows the short commit, the version and “just now”. Until a pipeline runs there it shows **Not tested**; after a run without failed test cases it shows **Tested**.
7. **Read what changed**

   Click the cell. The deploy detail shows [**Change from previous version**](https://docs.smartestqa.ai/en/deployments-overview.html#deploy-detail) with the number of commits and files, [**Tests after deploy**](https://docs.smartestqa.ai/en/deployments-overview.html#deploy-detail) with up to 10 pipeline runs, **Commits** and **Show changed files**. Click a run to open it in Report. See [Deploy detail](https://docs.smartestqa.ai/en/deployments-overview.html#deploy-detail).

## What the CI step returns

The step waits until the run reaches a final state. Your CI reads the exit code. Full list: [Run states and exit codes](https://docs.smartestqa.ai/en/integrations-cicd.html#exit-codes).

| State | Meaning | Exit code |
| --- | --- | --- |
| `QUEUED`, `RUNNING`, `STOPPING` | The run is waiting, running, or a stop was requested | None: the step keeps waiting |
| `PASSED` | Every test case passed | 0 |
| `FAILED` | At least one test case failed, or a stopped run left test cases unrun | 1 |
| `DISCONNECTED` | A test case has been running for more than 2 hours without news: the agent connection was lost | 1 |
| Timeout or error | The step’s timeout passed, the trigger was refused, or the status could not be read 5 times in a row | 2 |

## What a cell in Deploys & Repos tells you

Each cell describes the last successful deploy of one service in one environment. How each state is decided: [Cell states](https://docs.smartestqa.ai/en/deployments-numbers.html#cell-states).

| Chip or text | Colour | Meaning |
| --- | --- | --- |
| **Tested** · 18/20 | Green | The newest pipeline run after this deploy had no failed test cases. 18 of its 20 test cases passed; the other 2 ended with another result, such as stopped. |
| 2 broken · 18/20 | Red | The newest run after this deploy has 2 failed test cases. |
| **Test running** | Blue | The newest run is still running. |
| **Not tested** | Amber | No pipeline ran in this environment since the deploy. |
| [**No pipeline environment**](https://docs.smartestqa.ai/en/deployments-troubleshooting.html#fix-no-pipeline-env) | Grey | The environment is your own, for example STAGING. Pipelines do not run there, so no test state exists. |
| [**No deploy yet**](https://docs.smartestqa.ai/en/deployments-overview.html#cell-content) | None | The environment is mapped, but no deploy arrived. |
| [**Not mapped**](https://docs.smartestqa.ai/en/deployments-overview.html#cell-content) | None | The environment is switched off in this repository’s mapping and nothing was deployed there. |

## What you get

- **A build that fails on a red test.** Exit code 0 means every test case passed, 1 means a failure or a lost agent connection, 2 means a timeout or error. The step prints a link to the SmartestQA report.
- **Test cases in the CI’s test tab.** The step writes `smartest-junit.xml`. Each test case is one entry named name (#id); cases that did not run are skipped.
- **A normal pipeline run.** A run started from CI appears in Report, sends the usual notifications and webhooks, and runs where the pipeline is set to run, on a server or an agent.
- **A record of every deploy.** Repository, environment, commit, status, version, branch, who deployed it and a link to the CI run. Failed deploys are recorded too, because the ready-made steps run even when the deploy fails.
- **The change between two versions.** With a read-only access token the deploy detail lists up to 100 commits and up to 300 changed files since the previous successful deploy in the same environment.
- **Untested releases in view.** A cell turns amber with **Not tested** when a version reached an environment and no pipeline ran there since.
- **A release that is never blocked by the notification.** The deploy step ends with `|| true`: if SmartestQA cannot be reached, the deploy still succeeds.

## Questions and answers

### Q: Which CI tools does SmartestQA work with?

The CI/CD page in Integrations builds a step for GitHub Actions, GitLab CI, Jenkins, Azure Pipelines and Shell / other CI. The step needs `bash` and `curl`, or PowerShell on Windows, where `smartest-ci.ps1` is used. See [Set up the CI step](https://docs.smartestqa.ai/en/integrations-cicd.html#set-up).

_Audience: customer, admin_

### Q: Does a failed test fail my build?

Yes. The SmartestQA CI step ends with exit code 1 when at least one test case failed, when a stopped run left test cases unrun, or when the agent connection was lost. It ends with 0 only when every test case passed. See [Run states and exit codes](https://docs.smartestqa.ai/en/integrations-cicd.html#exit-codes).

_Audience: customer, admin_

### Q: Which key does the CI step need, and what can it do?

An agent API key, stored as the CI secret `SMARTEST_API_KEY`. Runs start with the project access of the user the key belongs to, so anyone with the key can start pipelines in every project that user can open. A dedicated agent for CI lets you revoke CI access alone. See [Before you start](https://docs.smartestqa.ai/en/integrations-cicd.html#before).

_Audience: customer, admin_

### Q: What happens if the pipeline runs longer than the CI timeout?

The CI step stops waiting and ends with exit code 2. The run keeps going in SmartestQA. The timeout is 60 minutes by default and can be set between 1 and 1440. See [“timed out after 60 min; the run continues in SmartestQA.”](https://docs.smartestqa.ai/en/integrations-troubleshooting.html#fix-ci-timeout)

_Audience: customer, admin_

### Q: Can CI pass values such as the environment to the pipeline?

Yes. The CI step takes `SMARTEST_PARAMS`, a JSON object such as `{"ENV":"uat"}`, which overrides the pipeline parameters for that run only. The one-line command on the pipeline page sends only the pipeline ID and uses the pipeline’s saved values. See [Script settings](https://docs.smartestqa.ai/en/integrations-cicd.html#settings).

_Audience: customer, admin_

### Q: How do I see which commits went into a deploy?

Connect the repository with [**Connect with token**](https://docs.smartestqa.ai/en/deployments-repos.html#connection-modes). SmartestQA then reads the commits and changed files between the deploy and the previous successful deploy in the same environment and shows them in the deploy detail. Read-only access is enough. With [**Deploy notification only**](https://docs.smartestqa.ai/en/deployments-repos.html#connection-modes) you see which version runs where, without a commit list. See [Connection modes](https://docs.smartestqa.ai/en/deployments-repos.html#connection-modes).

_Audience: customer, admin_

### Q: When does a deploy count as tested?

When a pipeline run in the same environment started after the successful deploy and before the next one. The match is by environment and time, for the whole project: a TEST pipeline that only covers the web app also counts for the api deployed to TEST at the same time. See [the tested-after window](https://docs.smartestqa.ai/en/deployments-numbers.html#tested-window).

_Audience: customer, admin_

### Q: Can the deploy notification break my release?

No. The ready-made step in Deploys & Repos ends with `|| true`, so a failed notification never breaks the deploy. The deploy token can only record deploys for its one repository and gives no access to anything else in SmartestQA. See [The deploy token](https://docs.smartestqa.ai/en/deployments-ci.html#deploy-token).

_Audience: customer, admin_

### Q: Can SmartestQA start the tests by itself after a deploy?

Yes, with an Autonomous Agent. **Deploy verification** runs the pipeline tagged `smoke` after each deploy, triages new reds and re-checks defects marked resolved. **Deploy impact analysis** reads the code diff, finds the affected APIs and screens and runs the related scenarios; it needs a repository with an access token. Both check for a new deploy every hour and run only when one has arrived. See [After each deploy](https://docs.smartestqa.ai/en/ai-agents-catalog.html#on-deploy).

_Audience: customer, admin_

### Q: Can another system start a pipeline without waiting for the result?

Yes. A webhook trigger in **Schedule** › **Triggers** gives a URL; a POST with a JSON body starts one run and can fill the pipeline’s inputs. Anyone who knows the URL can start the pipeline, so share it only with the calling system. See [Triggers](https://docs.smartestqa.ai/en/schedule-triggers-queues.html#triggers).

_Audience: customer, admin_

## Read more

- [CI/CD](https://docs.smartestqa.ai/en/integrations-cicd.html): the ready-made step, script settings, run states and exit codes.
- [Run from a terminal or CI/CD](https://docs.smartestqa.ai/en/schedule-runs.html#cli): the one-line command on the pipeline page.
- [Deploys & Repos](https://docs.smartestqa.ai/en/deployments.html): the screen, the building blocks and best practices.
- [Send deploys from CI](https://docs.smartestqa.ai/en/deployments-ci.html): the deploy token, the webhook fields and statuses.
- [Overview](https://docs.smartestqa.ai/en/deployments-overview.html): running versions, deploy history and the deploy detail.
- [Deploys & Repos problems](https://docs.smartestqa.ai/en/deployments-troubleshooting.html): a 401 from the webhook, a deploy that stays “Not tested”.
- [How do I build, run and schedule a regression suite?](https://docs.smartestqa.ai/en/use-case-regression.html): the pipeline your CI starts.
