1. Docs
  2. Use cases
  3. Use cases
  4. How do failed tests become Jira issues?

How do failed tests become Jira issues?

In SmartestQA a failed test case becomes a task when the pipeline has Auto Task Creation on, and a task becomes a Jira issue when Jira is connected in Integrations with Send Tasks to Jira on. Status, priority, assignee and comments that developers change in Jira come back to the task in Tasks.

12 min read Updated CustomersAdmins

In short

On this page you can

  • Get a task for each failed test case, with the failed steps, the error and the screenshot.
  • Send tasks to Jira as issues with mapped status and priority.
  • Read status, priority, assignee and comments back from Jira.
  • Raise a defect by hand from a failed step in Report.
  • Sync on a timer every 1 to 24 hours, or with one click.

How to use it

  1. Open Integrations, choose Jira and click Setup.
  2. Enter the Jira URL and the token, click Test Connection, then choose the Jira Project.
  3. Keep Send Tasks to Jira and Get Updates from Jira on, check the mappings and click Save.
  4. In the pipeline's Test Settings tick Auto Task Creation and save the pipeline.
  5. After a failed run, open the task in Tasks, assign it and rename it to the real problem.

The short answer#

There are two links in the chain. First, Auto Task Creation in a pipeline’s Test Settings: when a run has finished and at least one test case failed, each failed test case gets a task named “[Auto] Test Failed: test case” in the status Open. SmartestQA calls defects tasks. Second, the Jira integration with Send Tasks to Jira on: saving a task creates a Jira issue, and later changes update that issue.

One detail matters: a task created by hand goes to Jira at once when it is saved, but an automatic task is sent the first time someone changes it, for example when you assign it or rename it in triage. The Send to Jira button sends every task of the project. With Get Updates from Jira on, status, priority, assignee, comments and attachment links come back.

The problem this solves#

Many teams run automated tests in one tool and track defects in Jira. When a test goes red, a tester reads the report, opens Jira, types a summary, copies the error text, attaches a screenshot and picks a priority. On a morning with many red tests this takes a while, and the issues often end up short on detail.

The way back is missing too. A developer moves the issue to Done in Jira, and the test side only learns of it when someone looks. Some teams write a script that files an issue for every failure. That fills the backlog with issues nobody has read yet.

An example#

This is an illustration, not a customer story. A team tests a hotel booking site with a nightly pipeline that has Auto Task Creation on; Jira is connected with Send Tasks to Jira and Get Updates from Jira on, and Enable Auto Sync off.

  1. The nightly run finishes with two failed test cases. Tasks shows two new tasks in Open, each named “[Auto] Test Failed: ” plus the test case name, reported by Automated Tests, with no members. Nothing is in Jira yet.
  2. At 09:00 a tester opens the first task. The description is a read-only report with the failed steps and the error, and the run’s screenshot is the first attachment. Go to Report opens the run.
  3. The tester adds a developer under Members and renames the task to “Payment page rejects a valid card”. This first change sends the task to Jira: a new issue of the default type with the labels smartestqa and auto-synced, moved to the mapped status.
  4. The second failure is a known test data problem. The tester does not change that task, so it has no Jira issue. A click on Send to Jira would send it, because that button sends every task of the project.
  5. While reading the report the tester sees a wrong label that no test covers, clicks New Task and saves it. This task was saved by hand, so its Jira issue is created at once.
  6. In the afternoon the developer moves the first issue to Done in Jira. The tester clicks Get from Jira and the task’s status follows the mapping: Resolved.

Jira now holds only the failures a person has looked at, and nobody copied an error text by hand.

By hand, with your own script, and with SmartestQA#

QuestionIssues written by handYour own script that files issuesSmartestQA
Who records a failure?A tester, after reading the reportThe script, for every red testWith Auto Task Creation on, each failed test case gets a task when the run has finished
When does the Jira issue appear?When the tester has written itRight after the run, for every failureFor an automatic task: the first time someone changes it, or with Send to Jira. For a task saved by hand: at once
What evidence does it carry?What the tester copies overWhat the script was written to attachThe task holds a read-only report with the failed steps and the error, and the run’s screenshot when one was taken. The issue gets the task’s details, comments and attachments
What if the same test fails every night?The tester recognises it and writes nothing newA new issue each night, unless the script searches for an open oneA new task for each run: SmartestQA does not look for an existing task
Do changes in Jira come back?No, someone has to look in JiraOnly if you write that part tooWith Get Updates from Jira on: status, priority and assignee, new comments, and attachments as links

Steps#

  1. Prepare the Jira access

    For Jira Cloud use your Atlassian email and an API token; for Jira Server / Data Center a personal access token. The Jira user needs permission to browse the project, create and edit issues, transition them, comment and add attachments. Use a technical user, not a person. See Before you start.

  2. Connect Jira

    An Admin opens IntegrationsJira and clicks Setup. In Connection Settings enter the Jira URL, pick the Authentication Type, fill Username / Email and API Token, then click Test Connection. Next only works after a successful test.

    A green chip Connected: your Jira name appears.

  3. Choose the project and the defaults

    In Project Selection choose the Jira Project. Default Issue Type, Default Priority and Default Status load from that project; Bug and Medium are preselected when they exist.

  4. Choose the directions and the timer

    In Sync Settings keep Send Tasks to Jira and Get Updates from Jira on, or turn one off. Under Automatic Synchronization you can Enable Auto Sync and pick a Sync Interval of 1, 2, 4, 6, 12 or 24 hours.

  5. Check the mappings and save

    In Field Mapping check Severity → Priority Mapping and Status Mapping, then click Save. See Status and priority mapping.

    The page shows Active, tests the connection and shows Last test: OK.

  6. Switch on Auto Task Creation

    In Schedule, open the pipeline and go to its Test Settings. Tick Auto Task Creation (“Auto-create a task on test failure”) and save the pipeline. The pipeline’s own value wins over the project setting, so switch it on in each pipeline that should create tasks. See Which switch counts.

  7. Let a run fail, then triage

    Run the pipeline or wait for its schedule. When the run has finished, each failed test case has a task in Open, reported by Automated Tests, with no members. Open it in Tasks, add Members and rename it to the real problem. This first change sends the automatic task to Jira. Go to Report opens the run. See What an automatic task contains.

What is sent to Jira, and when#

Source: What is sent and when.

TriggerWhat happens in Jira or in the taskNeeds
A task is createdA new issue of the default type. The summary is the task’s ID in square brackets plus its name; the labels are smartestqa and auto-synced; the description holds the task’s details. The issue is moved to the mapped status, and comments and attachments are added.Integration Active and Send Tasks to Jira. Auto sync is not needed.
A linked task is changedSummary, description and priority of the issue are overwritten; the status is moved if the mapped status changed; new comments and attachments are added.The same
Send to JiraEvery task of the project is sent: new ones are created, linked ones updated.The same
Get from JiraFor linked issues, status, priority and assignee overwrite the task when they differ; new comments are copied; new attachments are added as links. Then the 200 newest unlinked issues of the project, of any type, become new tasks.Get Updates from Jira
Timed syncFirst Send, then Get, for each project whose interval has passed.Automatic Synchronization on

Default status and priority mapping#

You can change every row in the wizard’s last step or in the Configuration Summary panel. The mapping is read backwards for changes that come in from Jira.

SmartestQA taskJira
Status OpenTo Do
Status In ProgressIn Progress
Status ResolvedDone
Status ClosedDone
Priority CriticalHighest
Priority HighHigh
Priority MediumMedium
Priority LowLow

Watch it#

The video shows the task board with one column per status, the List view, a task’s detail, and reporting a bug by hand with New Task. It does not show Jira, and it does not show a task created by a failed run.

Video transcript

Tasks is a board with one column per status: Open, In Progress, Resolved and Closed. A card shows the name, the priority and the task number. List shows the same tasks as a table, with status, priority, type and who reported them. Back on Kanban, open a task. The detail holds the description, the members, attachments and the comments of the team. Close it and the board is where you left it. Tasks holds the defects of the project: found by failed runs, or reported by hand. New Task opens an empty task. Name it after what is wrong, not after where you saw it. Type, Priority and Status start as Functional Bug, Medium and Open. Raise the priority if it blocks users. In Description, write the steps, what you expected and what happened. Save. The task appears in the Open column with its priority and number. Open the card to add members, attachments and comments.

The board, a task's detail, and reporting a bug by hand.

What you get#

  • A task per failed test case. Created once per run, after its last test case has finished. Passed, stopped and skipped test cases create none.
  • The evidence in the task. A read-only report with Test Information, Step Summary, Failed Steps with the error cut after 200 characters, Passed Steps and Error Details. The run’s screenshot is the first attachment when one was taken.
  • A Jira issue per task. With the labels smartestqa and auto-synced, the default issue type and the mapped status and priority.
  • Updates from developers. Status, priority and assignee overwrite the task when they differ; new Jira comments are copied; attachments arrive as links.
  • A timer or a button. Timed sync every 1, 2, 4, 6, 12 or 24 hours, or Send to Jira and Get from Jira by hand.
  • Tasks that never wait for Jira. Problems with the Jira connection never block saving a task in SmartestQA.
  • Your own workflow. The status rows of the mapping are your company’s task statuses, including the ones you add, such as “In Review”.

Questions and answers#

Does SmartestQA create a Jira bug automatically for every failed test?Customers

It creates a task for every failed test case when the pipeline has Auto Task Creation on. With Jira connected and Send Tasks to Jira on, an automatic task is sent to Jira the first time someone changes it, or when you click Send to Jira. A task saved by hand creates the issue at once. See Jira and Azure DevOps.

What happens if the same test fails every night?Customers

SmartestQA does not look for an existing task. A test case that fails in three nightly runs creates three tasks. Switch Auto Task Creation off for pipelines that are red for a known reason. See The same failure creates a task every night.

Does it work with Jira Cloud and with Jira Server or Data Center?Customers

Yes. The Authentication Type offers API Token (Jira Cloud), Personal Access Token (Server/DC) and Basic Authentication for older Jira Server setups. See Before you start.

Is the Jira sync two-way?Customers

Yes, when both Send Tasks to Jira and Get Updates from Jira are on. There is no conflict check: the last write wins. A task edit overwrites the issue summary and description; an incoming sync overwrites the task’s status, priority and assignee. Agree on where each field is edited. See What is sent and when.

Will existing Jira issues be imported into SmartestQA?Customers

Yes, if Get Updates from Jira is on. The first Get from Jira turns up to 200 of the project’s newest issues into tasks, whatever their type: stories and epics too. Connect a Jira project that holds only the issues you want as tasks, or turn Get Updates from Jira off before the first sync. See Best practices.

Why does the Jira status not change when I move the task?Customers

An outgoing status change moves the issue through a Jira transition whose name matches the mapped status. If your Jira workflow has no such transition, the status stays as it is. Check the Status Mapping rows against your workflow. See The Jira status does not change.

Can I raise a defect by hand from a failed step?Customers

Yes. In Report, open the run and the test case with errors and click Task (tooltip “Create a defect task”). The task window opens empty; a task raised this way is not linked to the run and has no Go to Report link, so paste the report’s address into the description. See Raise a task from Report.

A run failed but no task appeared. Why?Customers

Check the pipeline’s own Auto Task Creation switch in Schedule. It is off by default and it wins over the project setting. Tasks are created once per run, after its last test case has finished, and dry runs in Automate never create tasks. See A run failed but no task appeared.

Does it work with Azure DevOps too?Customers

Yes. With Azure DevOps connected, tasks become work items (Bug by default) on the next timed sync or with Send tasks to Azure DevOps, and the state and assignee come back. Unlike Jira, creating a task does not create the work item at once. See Azure DevOps.

How do I stop the Jira sync?Customers

Turn Integration Active off on the Jira page; the switch is saved at once. Jira has no Remove button. Links between tasks and issues are kept for when you turn it on again. See The Jira page.

Read more#

Didn’t find your answer?Write to support with your project name and what you see on screen.

info@smartestqa.ai