1. Docs
  2. Use cases
  3. Use cases
  4. How do I sync automated test results with Xray or Zephyr Scale?

How do I sync automated test results with Xray or Zephyr Scale?

An Admin connects Xray or Zephyr Scale on its page in SmartestQA Integrations and syncs the test cases, because only linked test cases get results. Zephyr Scale then receives a new test cycle with the results after each pipeline run; Xray receives a Test Execution when an export of a run is requested.

11 min read Updated CustomersAdmins

In short

On this page you can

  • Write pipeline results to a Zephyr Scale test cycle after each run.
  • Export a run's results to Xray as a Test Execution.
  • Send SmartestQA test cases to Xray or Zephyr Scale.
  • Bring Xray Tests and Zephyr test cases into SmartestQA.
  • Bring Jira issues linked in Zephyr into Analyze as requirements.

How to use it

  1. Open Integrations, choose Xray or Zephyr Scale and click Setup.
  2. Enter the credentials and the Jira project key, then click Test Connection.
  3. Choose the directions and click Save.
  4. Link the test cases: Send to Xray or Pull from Xray; Get from Zephyr or Send to Zephyr.
  5. Run a pipeline and check the test cycle in Zephyr, or export the run to Xray.

The short answer#

Both tools are connected per project under ConfigurationIntegrations by a Company Admin, and both work on linked test cases: a result is written only for a test case that is mapped to an Xray Test or linked to a Zephyr test case. So the order is always: connect, sync the test cases, then run.

The two differ in how results travel. Zephyr Scale: with Write pipeline results to Zephyr on, linked test cases get a result in a new test cycle after each pipeline, automatically. Xray: results of mapped test cases are sent as a Test Execution, but in this version an export runs when it is requested, by an AI assistant connected through MCP. The Automatic Export line on the Xray page does not start an export by itself.

The problem this solves#

Many teams keep their test cases in a test management tool inside Jira, because that is where managers and auditors look for the test status. The automated tests run somewhere else. After a run, someone opens the tool and sets each test to passed or failed by hand, or the team writes a script that maps result files to test keys and uploads them. The hand-kept status often falls behind, and the script needs care each time a test is renamed or a key is missing.

What is missing is a link between each automated test case and its entry in the tool, kept in one place, so that a result lands on the right test without a mapping file in between.

An example#

This is an illustration, not a customer story. A bank has two test teams, each with its own SmartestQA project and its own Jira project: the online banking team uses Zephyr Scale, the cards team uses Xray.

  1. The Admin of the online banking project connects Zephyr Scale, keeps Write pipeline results to Zephyr on and clicks Send to Zephyr. The automated test cases written in SmartestQA are sent and get the label SmartestQA.
  2. The nightly pipeline runs. Afterwards Zephyr holds a new test cycle named SmartestQA · Nightly regression · plus the date and time, with Pass or Fail for each linked test case. Nobody asked for it.
  3. On the Zephyr page the panel Results written to Zephyr lists the run with its Status. A failed write would show Retry.
  4. The Admin of the cards project connects Xray with Execution Results chosen and clicks Send to Xray. The links appear in the Mappings table.
  5. Their nightly pipeline runs too, and nothing appears in Xray by itself. In the morning a tester asks an AI assistant connected through MCP: “Export the results of last night's "Nightly regression" run to Xray.” Xray gets a Test Execution with PASSED or FAILED per mapped test and up to three error messages in the comment.
  6. A test case added yesterday is not mapped yet, so it is not in the Test Execution. The tester clicks Send to Xray before the next export.

The Zephyr team no longer touches results. The Xray team replaced typing results with one request per run, which someone still has to make.

By hand, with your own upload script, and with SmartestQA#

QuestionResults entered by handYour own upload scriptSmartestQA
How is a test tied to its entry in the tool?By name, by the person who enters the resultBy a key in the test code or a mapping file you maintainBy a link made with Send to Xray or Pull from Xray, Send to Zephyr or Get from Zephyr
Who writes the results to Zephyr Scale?A person, after the runThe script, when the CI job calls itSmartestQA, automatically after each pipeline, in a new test cycle, while Write pipeline results to Zephyr is on
Who writes the results to Xray?A person, after the runThe script, when the CI job calls itOnly on request: an AI assistant connected through MCP exports the run as a Test Execution. The Automatic Export line does not start an export by itself
What happens to a test that is not linked?The person notices, or it is left outThe upload skips it or fails, as codedIt gets no result. Only linked test cases are written
What if a write fails?The person tries againYou read the CI log and run it againZephyr: the panel Results written to Zephyr lists the last 20 runs, with Retry on failed writes

Steps#

  1. Get the credentials

    Xray Cloud: in Global Settings › API Keys create a key for a technical user; you get a Client ID and a Client Secret. Zephyr Scale on Jira Cloud: a Zephyr API access token and the region (US or EU). Zephyr Scale on Jira Data Center: a Jira personal access token and the Jira address. Note the Jira project key. See Xray: before you start and Zephyr Scale: before you start.

  2. Connect Xray

    Open IntegrationsXray and click Setup. Keep Xray Cloud URL, enter Client ID and Client Secret and click Test Connection. Then enter the Jira Project Key and choose the Default Test Type: Manual, Cucumber (BDD) or Generic. Under Sync Direction choose Test Case → Xray, Xray → Test Case and Execution Results, and click Save.

    “Settings saved successfully”, and the page shows Active.

  3. Or connect Zephyr Scale

    Open IntegrationsZephyr Scale and click Setup. Pick Where Zephyr runs, then the region or the Jira address, the Jira project key and the token. Click Test Connection. Three switches are on by default: Bring linked Jira issues to Analyze, Send test cases written in SmartestQA to Zephyr and Write pipeline results to Zephyr. Set Status for passed and Status for failed to the names in your Zephyr (Pass and Fail by default) and click Save.

    Connected with the project name.

  4. Link the test cases

    Xray: click Send to Xray (“Synchronization started. Test cases are being sent to Xray...”) or Pull from Xray; the links appear in the Mappings table. Zephyr: click Get from Zephyr (“Started — this page updates when it finishes.”) or Send to Zephyr. See What each Xray sync does and What each Zephyr direction does.

  5. Run a pipeline

    Run the pipeline that holds the linked test cases, by hand, on its schedule or from CI. See Run, stop & follow.

  6. Check the result in Zephyr

    Zephyr gets a new test cycle named SmartestQA · pipeline · yyyy-MM-dd HH:mm (on Data Center: a test run). Each run is written once. The panel Results written to Zephyr lists the last 20 runs with Pipeline, Results, Status and Retry on failed writes.

  7. Export the run to Xray

    Ask an AI assistant connected through MCP, for example: “Export the results of last night's "Nightly regression" run to Xray.” The assistant exports by pipeline, run or test log. Only test cases mapped to Xray are included. See Test execution results and For AI assistants (MCP).

Xray and Zephyr Scale side by side#

Source: Xray and Zephyr Scale and the limits.

XrayZephyr Scale
EditionsXray CloudJira Cloud (US or EU region) and Jira Data Center
CredentialsClient ID and Client SecretZephyr API access token (Cloud) or Jira personal access token (Data Center)
Test cases inPull from Xray, up to 100 TestsGet from Zephyr, up to 5,000 cases, as planned scenarios
Test cases outSend to Xray: every test caseSend to Zephyr: automated test cases written in SmartestQA
RequirementsNoLinked Jira issues become requirements
Run resultsExported on request, as a Test ExecutionAutomatically after each pipeline, in a new test cycle
Timed syncNone in this version; use the two buttonsNone; results follow pipeline runs, imports and exports run when you click their button
RemoveRemove deletes all mappings; Tests in Xray and test cases in SmartestQA stayRemove deletes the connection and its settings; imported test cases and requirements stay

How a SmartestQA result is written#

SmartestQA resultXray status
Passed (passed, success, completed)PASSED
Failed (failed, error)FAILED
RunningEXECUTING
Anything elseTODO

In Xray the execution comment lists up to three error messages. In Zephyr Scale a passed test case gets the name in Status for passed and a failed one the name in Status for failed: Pass and Fail by default. If you change the status names in Zephyr, update these two fields.

What you get#

  • A Zephyr test cycle per pipeline run. Named SmartestQA · pipeline · yyyy-MM-dd HH:mm, with a result for each linked test case. Each run is written once.
  • An Xray Test Execution per exported run. With PASSED, FAILED, EXECUTING or TODO per test and up to three error messages in the comment.
  • Test cases in both tools. Send to Xray creates a Test named [SmartestQA-TCid] name with one Xray step per SmartestQA step. Send to Zephyr adds the label SmartestQA.
  • Existing cases as a plan. Each Zephyr test case becomes a planned scenario tagged planned and zephyr; the first folder level becomes the module. Running the import again only updates cases that changed.
  • Requirements from Jira. With Bring linked Jira issues to Analyze on, each linked issue becomes a sub-requirement under its epic or parent, and the scenario is linked to it.
  • Clear ownership. Test cases that came from Zephyr stay owned by Zephyr: their name and steps are never written back. Tests that SmartestQA created in Xray are skipped by Pull from Xray.
  • Counters on the page. Xray shows Mapped Tests, Successful Syncs and Failed; Zephyr shows Test cases from Zephyr, Test cases sent to Zephyr, Requirements from Jira and Runs written to Zephyr.

Questions and answers#

Are test results sent to Xray and Zephyr automatically after every run?Customers

To Zephyr Scale, yes: with Write pipeline results to Zephyr on, linked test cases get a result in a new test cycle after each pipeline. To Xray, an export runs when it is requested through an AI assistant connected to SmartestQA by MCP; the Automatic Export line does not start one by itself. See Test execution results.

Does it work with Xray Server or Data Center?Customers

No. In SmartestQA, Xray works only with Xray Cloud. Zephyr Scale works with both Jira Cloud and Jira Data Center. See Does Jira Server or Data Center work?

Why does no Test Execution appear in Xray after a run?Customers

Either no export was requested, or the run holds no mapped test case. Only test cases mapped to Xray are included, so sync them first; a run with none returns “No Xray mappings found for tests in this log”. See No Test Execution appears in Xray after a run.

Why was nothing written to Zephyr for my run?Customers

Results are written only for test cases linked to Zephyr, and only while Write pipeline results to Zephyr is on; with the switch off the panel reads “Writing results is turned off.” Link the cases with Get from Zephyr or Send to Zephyr first. See “No test case in this run is linked to Zephyr”.

How many test cases can I import from Xray or Zephyr?Customers

One Pull from Xray reads up to 100 Tests of the Jira project. One Zephyr import takes up to 5,000 test cases, and descriptions are cut at 1,000 characters. See Limits.

Do I need the Jira integration for Zephyr Scale?Customers

For test cases and results, no. For requirements on Jira Cloud, yes: linked issues are read through the project’s Jira integration, so set up Jira first. See “Requirements were not imported: Zephyr Cloud needs the Jira integration…”

Will a sync overwrite the test cases in my test management tool?Customers

Test cases that came from Zephyr stay owned by Zephyr: SmartestQA never writes their name and steps back. Send to Xray updates the Tests that are already mapped and creates new ones for the rest. Decide which tool owns each test case before the first sync. See Best practices.

Can I use Xray and Zephyr Scale in the same project?Customers

Do not connect Xray and Zephyr to the same Jira project from one SmartestQA project. Every integration belongs to one project. See Do I have to set up each integration for every project?

What does “Another sync is still running” mean?Customers

Only one Zephyr import or export runs at a time per project, and a second click returns this message. A job that has not finished after 45 minutes no longer blocks new ones. See “Another sync is still running”.

Can run reports also go to Confluence?Customers

Yes. With the Confluence integration, Test Reports Sync and Auto-Create Report Pages on, SmartestQA creates one report page for each pipeline run that finishes, under the reports parent page. Send Report publishes every finished run that has no page yet. See What is sent and when.

Read more#

Didn’t find your answer?Write to support with your project name and what you see on screen.

info@smartestqa.ai