1. Docs
  2. Use cases
  3. Use cases
  4. How do I get test results in Slack, Teams or e-mail?

How do I get test results in Slack, Teams or e-mail?

SmartestQA posts pipeline results to Slack or Microsoft Teams once an Admin pastes the channel's webhook URL on the Slack or Microsoft Teams page in Integrations and picks the events. For e-mail, switch on Email Notification in the pipeline's settings in Schedule: the PDF report is mailed when a run ends, always or only on failure.

11 min read Updated CustomersAdmins

In short

On this page you can

  • Post a message to Slack or Teams when a pipeline completes or fails.
  • Mention @here, @channel or a person on failures.
  • Get one daily summary of yesterday's runs and the open tasks.
  • Mail the PDF report after each run, always or only on failure.
  • Send signed JSON events to any other system with a webhook.

How to use it

  1. Create an incoming webhook for the channel in Slack, or a Workflows webhook in Teams.
  2. Open Integrations, choose Slack or Microsoft Teams and click Setup.
  3. Paste the webhook URL and click Test Connection.
  4. Under Notification Preferences choose the events and click Save.
  5. For e-mail, switch on Email Notification in the pipeline and fill Email To.

The short answer#

Chat and e-mail are set in two different places. Slack, Microsoft Teams and WhatsApp are set per project in ConfigurationIntegrations, by an Admin: you paste the channel’s webhook URL, test it, and switch events on or off under Notification Preferences. By default a run that ends posts either Pipeline Completed (every test case passed) or Pipeline Failed (at least one failed or did not run), never both.

E-mail is set in the pipeline’s settings in Schedule: with Email Notification on, SmartestQA sends one e-mail with the PDF report when the run ends, to the addresses in Email To and Email CC. For a single report, Share Report in Report mails the PDF to one address. For any other system, a Generic Webhook sends signed JSON events.

The problem this solves#

Many teams learn about a red nightly run when someone opens the test tool. Often one person checks each morning and posts a line in the team chat, or forwards a screenshot. When that person is away, nobody looks, and a failure surfaces late. Other teams add a script to the end of a CI job that posts to a chat webhook. That works for the runs that job starts, and someone has to maintain it.

What is missing is a result that arrives where the team already reads, without a person in between. Different readers also need different things: the channel needs a short line when something is red, a manager needs the report.

An example#

This is an illustration, not a customer story. A team tests a food delivery app with a nightly regression pipeline and reads a Slack channel called #qa-alerts.

  1. An Admin opens IntegrationsSlack, clicks Setup, pastes the channel’s webhook URL and clicks Test Connection. The channel receives “SmartestQA Slack integration test successful!”.
  2. Under Notification Preferences the Admin turns Pipeline Completed off, keeps Pipeline Failed on, sets Mention on Failure to @here and clicks Save.
  3. In the pipeline’s settings in Schedule a tester switches on Email Notification, chooses On Failure and enters the QA lead and the product owner in Email To, separated by a semicolon.
  4. The night run ends with 3 failed test cases. The channel gets one Pipeline Failed message with the first error and a last line CC: @here. The two recipients each get the e-mail with the PDF report.
  5. The next night every test case passes. The channel stays quiet because Pipeline Completed is off, and no e-mail is sent because the mode is On Failure.
  6. A developer from another team asks for the failed run. The tester opens it in Report, clicks Share Report, types the address and clicks Send: “Email sent successfully!”. The developer needs no SmartestQA account.

The morning check is gone: a red run reaches the channel and two inboxes by itself, and a green night sends nothing.

By hand, with your own script, and with SmartestQA#

QuestionA person posts the resultYour own script posts to a chat webhookSmartestQA
Who sends the result?Whoever checked the runThe script at the end of the CI jobSmartestQA, when the run ends
Can it report failures only?Yes, if the person remembers to lookYes, if you code the conditionYes: turn Pipeline Completed off and keep Pipeline Failed on; for e-mail choose On Failure
What is in the message?What the person typesWhat the script builds from the result filesCompleted: total, passed, failed, pass rate and duration. Failed: the first error. E-mail: the PDF report
Can it call someone’s attention?Yes, by typing a mentionYes, if codedA mention adds a last line CC: value to Pipeline Failed and Critical Task messages
Is there a daily overview?Only if someone compiles itA second script on a timerSend Daily Summary, once a day at the Summary Time hour; skipped when the channel already got a message that day
Can people outside the tool get the report?Yes, as a forwarded file or screenshotOnly what the script attachesYes: the PDF goes to Email To and Email CC, or to one address with Share Report

Steps#

  1. Create the webhook in Slack or Teams

    Slack: create a new application in your Slack workspace, turn on Incoming Webhooks and add a webhook for the channel. The URL starts with https://hooks.slack.com/services/. Teams: open Workflows, choose the template Send webhook alerts to a channel, select the team and channel, save, and copy the webhook URL from the workflow’s details. See Slack and Microsoft Teams.

  2. Enter it in SmartestQA

    Select the project, open IntegrationsSlack and click Setup. Under Webhook Settings paste the Webhook URL and, for your own reference, a Channel Name (optional) such as #qa-alerts. For Teams open IntegrationsMicrosoft Teams, click Setup and paste the Teams Webhook URL.

  3. Test the connection

    Click Test Connection.

    Slack shows Success! and the channel receives “SmartestQA Slack integration test successful!”. The Teams channel receives a card “SmartestQA connected”.

  4. Choose the events and the mention

    Under Notification Preferences turn the events on or off. Pick a Summary Time (whole hours, 00:00 to 23:00) if you use the daily summary. In Slack set Mention on Failure to None, @here or @channel; in Teams type an email or a name in Mention (Email or name). Click Save. See Which events send a message.

    The page shows Active. Send Test Message posts a test with the saved webhook at any time.

  5. Switch on the e-mail report

    In Schedule, open the pipeline’s settings and switch on Email Notification. Choose Always or On Failure (only runs with at least one failed case). Fill Email To and Email CC, with several addresses separated by semicolons. With an empty Email To, nothing is sent. See E-mail notifications.

  6. Run the pipeline

    Start a run or wait for the schedule. When the run ends, the channel gets the message and the recipients get one e-mail with the PDF report. See Run, stop & follow.

  7. Send one report to one person

    In Report, open a finished run and click Share Report. Type the recipient in Email and click Send. The recipient does not need a SmartestQA account; everything is in the PDF. See Share a report by email.

    “Email sent successfully!”

Which event sends a message#

Slack, Teams and WhatsApp share the same switches. Source: Which events send a message.

SwitchSent whenSlack / Teams defaultWhatsApp default
Pipeline StartedA pipeline run starts.OffOff
Pipeline CompletedA run ends and every test case passed. The message shows total, passed, failed, pass rate and duration.OnOn, but skipped while Finished runs: only when a test failed is on
Pipeline FailedA run ends and at least one test case failed or did not run. The message shows the first error.OnOn
Test FailedSaved with the settings, but this version sends no separate message per failed test. Failed tests are reported by Pipeline Failed.OnOff
Task CreatedA task with any severity other than Critical is created.OnOff
Critical TaskA task with severity Critical is created.OnOff
Send Daily SummaryOnce a day at the Summary Time hour.OffOn

Which channel for what#

ChannelWhere you set itWhat arrives
SlackIntegrationsSlackPipeline, task and daily summary messages in one channel.
Microsoft TeamsIntegrationsMicrosoft TeamsThe same messages, as Adaptive Cards, through a Workflows webhook.
WhatsApp BusinessIntegrationsWhatsApp BusinessFailures and the daily summary to up to 20 phones, with templates Meta has approved.
E-mail after each runEmail Notification in the pipeline, or as a project defaultThe PDF report, Always or On Failure, to Email To and Email CC.
E-mail by handShare Report in ReportA short e-mail with the PDF, to one address per send.
Generic WebhookIntegrationsGeneric WebhookSigned JSON events to any HTTP endpoint: pipeline started, completed, failed, defect created, deploy received.

Watch it#

The video is an overview of the Integrations screen: the catalogue, the category row, the Active filter and what the Jira, Xray, Zephyr Scale, Slack, Confluence and Generic Webhook cards do. It does not show a Slack set-up.

Video transcript

Integrations lists everything SmartestQA can connect to. The category row narrows it: defect trackers, test management, notifications, documentation, monitoring, identity and CI/CD. Active shows what is already connected. Nothing is connected in this project yet, so the list is empty. Jira syncs tasks with issues. Xray and Zephyr Scale sync test cases and run results. Slack sends test and task notifications. Confluence publishes reports. Generic Webhook sends signed events to any system, and starts pipelines from outside. A card marked Ready can be set up: open it and follow its steps.

What SmartestQA connects to, and where to start.

What you get#

  • One message per run. A run sends either Completed or Failed, never both. Completed shows total, passed, failed, pass rate and duration; Failed shows the first error.
  • A mention when it is red. With a mention set, SmartestQA adds a last line CC: value to Pipeline Failed and Critical Task messages. Completed, Started, Task Created and daily summary messages carry no mention.
  • A daily summary. One message per day with the tests run yesterday, passed, failed and rate, and the open tasks, at the hour of Summary Time.
  • The PDF report by e-mail. One e-mail per run, when the run ends. Automatic e-mails are always in English.
  • No test data in chat. Chat and WhatsApp messages carry pipeline names, counts and error text, never test data values.
  • Signed events for everything else. Up to 20 webhooks per project. Each delivery is an HTTP POST with a JSON body and an HMAC-SHA256 signature, retried after 1 s, 5 s and 30 s, up to 4 attempts.

Questions and answers#

Who can connect Slack or Teams to SmartestQA?Customers

A Company Admin. Integrations is under ConfigurationIntegrations and Users do not see the menu item. An integration belongs to one project, so select the project first. See Integrations.

Can I get a message only when tests fail?Customers

Yes. On the Slack or Microsoft Teams page turn Pipeline Completed off and keep Pipeline Failed on. For e-mail, choose On Failure: only runs with at least one failed case send the report. See Which events send a message.

Why did no message arrive after a pipeline?Customers

Check that the page shows Active, that the event’s switch is on and that Send Test Message reaches the channel. Integration Active off turns all messages off without losing the webhook. See No message after a pipeline.

Why did the daily summary not arrive?Customers

The summary is skipped when the channel already received a message of any kind that day. If a pipeline message went to the channel at 07:00 and the summary is set for 09:00, that day’s summary is skipped. Pick an hour before the first run of the day. See The daily summary.

Does it work with the old Office 365 connector URL in Teams?Customers

Use a Workflows webhook. If you paste an address of the retired Office 365 connector type, the dialog and the page warn that messages may no longer arrive. See “This is an old Office 365 connector URL…”

Can the report go to several people by e-mail?Customers

Yes, with the pipeline’s automatic e-mail: enter several addresses separated by semicolons in Email To and Email CC. Share Report takes one address, so send once per address. See Automatic emails after each run.

Does the person who gets the report need a SmartestQA account?Customers

No. The e-mail carries the report as a PDF, and everything is in the PDF. There is no public link to a report: a report address only works for people with access to the project. See Share a report by email.

Can I set the e-mail once for the whole project?Customers

You can set a project default under Project PanelTest Management, Email Notification. It is used when a pipeline’s own copy has no value. Changing the project does not change existing pipelines, so open each pipeline in Schedule to change it. See Email Notification.

Can SmartestQA send test results to WhatsApp?Customers

Yes, through WhatsApp Business, to up to 20 numbers. You need a business number, a permanent access token and four approved templates. Messages carry only the project and pipeline names, counts and a report link. See WhatsApp Business.

Can I send results to a system that is not Slack or Teams?Customers

Yes. Under IntegrationsGeneric Webhook click Add webhook, enter the Payload URL and tick the events. Pipeline completed and Pipeline failed are preselected. Webhooks fire whether or not Slack or Teams are set up. See Add a webhook.

Read more#

Didn’t find your answer?Write to support with your project name and what you see on screen.

info@smartestqa.ai