- Docs
- Administration
- Integrations
- CI/CD
CI/CD: GitHub Actions, GitLab CI, Jenkins, Azure Pipelines
The CI/CD page builds a ready-made step for your CI. The step starts a SmartestQA pipeline, waits for it, writes a JUnit report for the CI’s test tab and fails the build when a test fails.
In short
On this page you can
- Start any pipeline of the project from GitHub Actions, GitLab CI, Jenkins or Azure Pipelines.
- Override pipeline parameters for one run, for example
{"ENV":"uat"}. - See each test case in the CI’s test tab from
smartest-junit.xml. - Fail the build when a test fails, with a link to the SmartestQA report.
- Use the same script from any CI that has bash and curl, or PowerShell.
How to use it
- Under 1. Choose the pipeline pick the pipeline, optional parameters and a timeout.
- Store an agent API key as the CI secret
SMARTEST_API_KEY. - Under 3. Add the step pick your CI tool and click Copy.
- Paste the step into your CI file and commit.
- Run the CI job and open the report link it prints.
How it works#
- 1StartThe step calls
POST /api/pipelines/runwith the API key and gets a run id - 2WaitIt checks
/api/ci/runs/idevery 15 seconds - 3ReportIt downloads the JUnit file and prints the result and report link
- 4Exit0 passed, 1 failed, 2 timeout or error
The run is a normal pipeline run: it appears in Report, sends the usual notifications and webhooks, and runs where the pipeline is set to run (server or agent). The run states follow the same rules as the Reports screen.
Before you start#
- A pipeline in the project. Without one the page says “This project has no pipelines yet. Create one first, then come back here.” with Go to Pipelines.
- An agent API key. Use the key of your own agent, or of a dedicated agent for CI (Device FarmTest Agents, Copy API Key; see Install an agent). Runs start with the project access of the user the key belongs to.
- Network. The CI runner must reach the SmartestQA API address shown as
SMARTEST_URL. The step needsbashandcurl(or PowerShell on Windows);jqorpython3are used when present.
Keep the key out of the file
Store the API key as a secret in your CI: GitHub repository secret, GitLab masked variable, Jenkins credential (Secret text, id smartest-api-key) or Azure Pipelines secret variable. Never paste it into the YAML. Anyone with the key can start pipelines in every project that user can open. A dedicated agent for CI lets you revoke CI access alone.
Set up the CI step#
Choose the pipeline
Open IntegrationsGitHub Actions (or GitLab CI or Jenkins). Under 1. Choose the pipeline pick the Pipeline. Optionally enter Parameters (optional, JSON), which override the pipeline parameters for this run only, and a Timeout (min) between 1 and 1440 (60 by default).
Add the connection to your CI
Under 2. Add the connection to your CI copy
SMARTEST_URL. Create the CI secretSMARTEST_API_KEYwith the agent key (Open Agents takes you to the agents).Add the step
Under 3. Add the step pick the CI tool: GitHub Actions, GitLab CI, Jenkins, Azure Pipelines or Shell / other CI. Click Copy and paste the snippet into your CI file.
Run it
Commit and run the CI job.
The log prints “SmartestQA: run started, suiteRuntimeId=…”, progress lines, then a summary with Result, counts and Report. The test tab shows each test case.
- name: 'SmartestQA · Nightly regression'
env:
SMARTEST_URL: 'https://api.example.com'
SMARTEST_API_KEY: ${{ secrets.SMARTEST_API_KEY }}
PIPELINE_ID: '318'
SMARTEST_TIMEOUT_MIN: '60'
run: |
curl -fsSL "$SMARTEST_URL/install/smartest-ci.sh" -o smartest-ci.sh
bash smartest-ci.sh
- name: SmartestQA test results
if: always()
uses: mikepenz/action-junit-report@v5
with:
report_paths: smartest-junit.xml| CI tool | Where the key goes | Test report |
|---|---|---|
| GitHub Actions | Repository secret SMARTEST_API_KEY | A check from action-junit-report, plus a summary table in the job summary |
| GitLab CI | Settings › CI/CD › Variables (masked) | artifacts: reports: junit, shown in the merge request |
| Jenkins | Manage Jenkins › Credentials, Secret text, id smartest-api-key | junit post step |
| Azure Pipelines | Secret pipeline variable | PublishTestResults@2 |
| Shell / other CI | Your CI’s secret store, exported as SMARTEST_API_KEY | smartest-junit.xml in the work folder |
Windows agents use smartest-ci.ps1 instead; the Jenkins and Azure snippets include the PowerShell line as a comment.
Script settings#
| Variable | Default | Meaning |
|---|---|---|
SMARTEST_URL | — | SmartestQA API address |
SMARTEST_API_KEY | — | Agent API key, sent as Authorization: Bearer |
PIPELINE_ID | — | The pipeline’s number |
SMARTEST_PARAMS | none | JSON object of parameter overrides |
SMARTEST_TIMEOUT_MIN | 60 | How long the step waits |
SMARTEST_POLL_SEC | 15 | Seconds between status checks |
SMARTEST_JUNIT | smartest-junit.xml | Report file; - writes none |
Run states and exit codes#
| State | Meaning | Exit code |
|---|---|---|
QUEUED | The run is waiting to start | — (keeps waiting) |
RUNNING / STOPPING | Test cases are running, or a stop was requested | — (keeps waiting) |
PASSED | Every test case passed | 0 |
FAILED | At least one test case failed, or a stopped run left test cases unrun | 1 |
DISCONNECTED | A test case has been running for more than 2 hours without news: the agent connection was lost | 1 |
| Timeout or error | The step’s timeout passed, the trigger was refused, or the status could not be read 5 times in a row | 2 |
On a timeout the run keeps going in SmartestQA; only the CI step stops waiting. In the JUnit file each test case is one <testcase> named name (#id): failed and disconnected cases are failures with the first error, cases that did not run are skipped, and cases still running at the timeout are errors.
Related#
- To tell SmartestQA that a deploy happened (so it can react), see Deployments & Repos.
- To start a pipeline from another system without waiting, use a webhook trigger in Schedule.
- To announce the result in chat, add Slack, Teams or WhatsApp.
If something goes wrong#
Didn’t find your answer?Write to support with your project name and what you see on screen.
info@smartestqa.ai