1. Docs
  2. Administration
  3. Integrations
  4. CI/CD

CI/CD: GitHub Actions, GitLab CI, Jenkins, Azure Pipelines

The CI/CD page builds a ready-made step for your CI. The step starts a SmartestQA pipeline, waits for it, writes a JUnit report for the CI’s test tab and fails the build when a test fails.

4 min read Updated CustomersAdmins

In short

On this page you can

  • Start any pipeline of the project from GitHub Actions, GitLab CI, Jenkins or Azure Pipelines.
  • Override pipeline parameters for one run, for example {"ENV":"uat"}.
  • See each test case in the CI’s test tab from smartest-junit.xml.
  • Fail the build when a test fails, with a link to the SmartestQA report.
  • Use the same script from any CI that has bash and curl, or PowerShell.

How to use it

  1. Under 1. Choose the pipeline pick the pipeline, optional parameters and a timeout.
  2. Store an agent API key as the CI secret SMARTEST_API_KEY.
  3. Under 3. Add the step pick your CI tool and click Copy.
  4. Paste the step into your CI file and commit.
  5. Run the CI job and open the report link it prints.

How it works#

  1. 1StartThe step calls POST /api/pipelines/run with the API key and gets a run id
  2. 2WaitIt checks /api/ci/runs/id every 15 seconds
  3. 3ReportIt downloads the JUnit file and prints the result and report link
  4. 4Exit0 passed, 1 failed, 2 timeout or error

The run is a normal pipeline run: it appears in Report, sends the usual notifications and webhooks, and runs where the pipeline is set to run (server or agent). The run states follow the same rules as the Reports screen.

Before you start#

  • A pipeline in the project. Without one the page says “This project has no pipelines yet. Create one first, then come back here.” with Go to Pipelines.
  • An agent API key. Use the key of your own agent, or of a dedicated agent for CI (Device FarmTest Agents, Copy API Key; see Install an agent). Runs start with the project access of the user the key belongs to.
  • Network. The CI runner must reach the SmartestQA API address shown as SMARTEST_URL. The step needs bash and curl (or PowerShell on Windows); jq or python3 are used when present.

Keep the key out of the file

Store the API key as a secret in your CI: GitHub repository secret, GitLab masked variable, Jenkins credential (Secret text, id smartest-api-key) or Azure Pipelines secret variable. Never paste it into the YAML. Anyone with the key can start pipelines in every project that user can open. A dedicated agent for CI lets you revoke CI access alone.

Set up the CI step#

  1. Choose the pipeline

    Open IntegrationsGitHub Actions (or GitLab CI or Jenkins). Under 1. Choose the pipeline pick the Pipeline. Optionally enter Parameters (optional, JSON), which override the pipeline parameters for this run only, and a Timeout (min) between 1 and 1440 (60 by default).

  2. Add the connection to your CI

    Under 2. Add the connection to your CI copy SMARTEST_URL. Create the CI secret SMARTEST_API_KEY with the agent key (Open Agents takes you to the agents).

  3. Add the step

    Under 3. Add the step pick the CI tool: GitHub Actions, GitLab CI, Jenkins, Azure Pipelines or Shell / other CI. Click Copy and paste the snippet into your CI file.

  4. Run it

    Commit and run the CI job.

    The log prints “SmartestQA: run started, suiteRuntimeId=…”, progress lines, then a summary with Result, counts and Report. The test tab shows each test case.

GitHub Actions step (generated, shortened)
- name: 'SmartestQA · Nightly regression'
  env:
    SMARTEST_URL: 'https://api.example.com'
    SMARTEST_API_KEY: ${{ secrets.SMARTEST_API_KEY }}
    PIPELINE_ID: '318'
    SMARTEST_TIMEOUT_MIN: '60'
  run: |
    curl -fsSL "$SMARTEST_URL/install/smartest-ci.sh" -o smartest-ci.sh
    bash smartest-ci.sh
- name: SmartestQA test results
  if: always()
  uses: mikepenz/action-junit-report@v5
  with:
    report_paths: smartest-junit.xml
CI toolWhere the key goesTest report
GitHub ActionsRepository secret SMARTEST_API_KEYA check from action-junit-report, plus a summary table in the job summary
GitLab CISettings › CI/CD › Variables (masked)artifacts: reports: junit, shown in the merge request
JenkinsManage Jenkins › Credentials, Secret text, id smartest-api-keyjunit post step
Azure PipelinesSecret pipeline variablePublishTestResults@2
Shell / other CIYour CI’s secret store, exported as SMARTEST_API_KEYsmartest-junit.xml in the work folder

Windows agents use smartest-ci.ps1 instead; the Jenkins and Azure snippets include the PowerShell line as a comment.

Script settings#

VariableDefaultMeaning
SMARTEST_URL—SmartestQA API address
SMARTEST_API_KEY—Agent API key, sent as Authorization: Bearer
PIPELINE_ID—The pipeline’s number
SMARTEST_PARAMSnoneJSON object of parameter overrides
SMARTEST_TIMEOUT_MIN60How long the step waits
SMARTEST_POLL_SEC15Seconds between status checks
SMARTEST_JUNITsmartest-junit.xmlReport file; - writes none

Run states and exit codes#

StateMeaningExit code
QUEUEDThe run is waiting to start— (keeps waiting)
RUNNING / STOPPINGTest cases are running, or a stop was requested— (keeps waiting)
PASSEDEvery test case passed0
FAILEDAt least one test case failed, or a stopped run left test cases unrun1
DISCONNECTEDA test case has been running for more than 2 hours without news: the agent connection was lost1
Timeout or errorThe step’s timeout passed, the trigger was refused, or the status could not be read 5 times in a row2

On a timeout the run keeps going in SmartestQA; only the CI step stops waiting. In the JUnit file each test case is one <testcase> named name (#id): failed and disconnected cases are failures with the first error, cases that did not run are skipped, and cases still running at the timeout are errors.

If something goes wrong#

Didn’t find your answer?Write to support with your project name and what you see on screen.

info@smartestqa.ai
© 2026 Selenity Technology · SmartestQAsmartestqa.aiThis page as MarkdownChecked against smarttest-fe predev 2a8c79cc (CiCdIntegrationPage, ciSnippets), smarttest-be 633c2a03 (CiRunController, CiRunRules, CiJunitWriter, CiScriptController, scripts/ci/smartest-ci.sh, ApiKeyAuthenticationFilter, PipelineServiceImpl)