How do I keep passwords and API keys out of my tests?
In SmartestQA a secret lives in Data Source, not in the test: an encrypted key in Data Source › Variables for a password, token or API key, and the connection in Data Source › Connections for a database or Kafka password. The step stores only a reference such as {{ds:adminPassword}}, and run logs and reports show •••••• in place of the value.
In short
On this page you can
- Store passwords, tokens and API keys encrypted; only the creator of a value can show it again.
- Reference a secret in any step as
{{ds:key}}instead of typing it. - Keep database and Kafka passwords in the connection, not in the step.
- See
••••••in the live log, the report and the API request log. - Rotate a secret in one place by typing the new value and saving.
How to use it
- Open Data Source › Variables and click New key.
- Name the key, keep Value comes from on Typed value and switch on Encrypt.
- Type the secret and click Save.
- In the test step, insert the key with the data picker:
{{ds:adminPassword}}. - Run the test and check that the log shows
••••••.
The short answer#
Do not type a secret into a step. Step values are stored with the test case and shown to every project member who opens it. Create a key in Data Source › Variables, switch on Encrypt, and use the key in the step as {{ds:key}}. The value is stored encrypted, only its creator can show it again in the key panel, and the runner receives it for the test run. Before a run writes a log line, a report entry or a live message, it replaces every known secret with ••••••.
Database and Kafka passwords belong in the connection under Data Source › Connections. Steps pick the connection by name, so the password never appears in the step or the test case.
The problem this solves#
Tests need real credentials: the password of a test user, an API token, a database password. The shortest way is to type the value into the step or the script. From then on everyone who opens the test can read it, and it can show up in logs and reports. Many teams also keep a shared document or spreadsheet of test accounts and copy from there.
When a password changes, someone has to find every test that contains it. When a colleague leaves the team, nobody can say which passwords that person has seen. What is missing is one place per secret, tests that hold only its name, and runs that do not print the value.
An example#
This is an illustration, not a customer story. A team tests an online banking web application, and the password of the test user and an API token are typed into the steps of about forty test cases.
- The QA lead installs the Test data audit agent from the catalog on Agents and clicks Run now. It finds the passwords and tokens written into steps and says where each should move. It changes nothing.
- In Data Source › Variables the QA lead clicks New key, names it
adminPassword, leaves Value comes from on Typed value, switches on Encrypt, types the password and clicks Save. The field shows “•••••• (set, encrypted)” and the list shows the kind Encrypted. A second key,apiToken, holds the token. - A tester opens the login steps and picks the key with the picker icon next to the value. The value is now
{{ds:adminPassword}}. The API steps get{{ds:apiToken}}as the Bearer token. - The tester starts a Dry Run. The runner receives the real value and uses it in the step; the live log shows
••••••. - Later the password of the test user is changed. The tester opens the key, types the new value and clicks Save, without seeing the old one. The QA lead, who created the value, can still show it with the eye icon.
The password is now in one key, not in forty test cases, and every test that references the key uses the new value on its next run.
Typed into the test, a separate secret store with your own scripts, and SmartestQA#
| Question | Typed into the test or script | A separate secret store with your own scripts | SmartestQA |
|---|---|---|---|
| Where is the value? | In the test, next to the steps | In the store; the script fetches it at run time | In an encrypted key in Data Source › Variables; the step holds {{ds:key}} |
| Who can read it? | Everyone who can open the test | Whoever the store allows | Only its creator can show it again. Everyone else, Admins included, sees “•••••• (set, encrypted)” and can replace it |
| Does it appear in logs and reports? | It can, unless you filter the output | That depends on whether your script masks it | The live log, the run report and the API request log show ••••••. A screenshot of a page that displays a secret still shows it |
| How do I rotate it? | Find and edit every test | Change it in the store | Type the new value in the key and save; tests use it on their next run |
| Does an AI assistant see it? | Yes, when it reads the test | That depends on your setup | The AI receives key names and whether they are encrypted, never the value. Through MCP an assistant creates the key empty and you fill it in |
Steps#
Create the key
Go to Data Source › Variables and click New key. Type the name the steps will use, for example
adminPassword. Leave Value comes from on Typed value: a dataset or database value cannot be encrypted.Switch on Encrypt
Click Encrypt next to the value. The hint says “Store the value encrypted: it is never shown again and only the test run reads it.”
Type the secret and save
The field hides what you type. Add a Description so others know what the key is for, and click Save.
The field now shows “•••••• (set, encrypted)”. In the list the key’s kind is Encrypted and its value “•••••••• set”.
Use the key in the step
In Automate, select the step and click the picker icon next to the value (“Data Sources - Global Parameters - Datasets”). Pick the key. The step’s value is now
{{ds:adminPassword}}. For an API step, put{{ds:apiToken}}in the Bearer token or a header.Put database and Kafka passwords in the connection
In Data Source › Connections, click New DB or New Kafka Connection. The Password is required for a new connection, stored encrypted and never sent back to the screen. Do not put the password in the Database Url: it would be visible to everyone in the list.
Run and check the log
Start a Dry Run. The runner receives the real value and uses it in the step; the live log shows
••••••.Rotate when needed
Open the key, type the new value and save. Anyone who can edit keys can replace a secret without seeing the old one. Leave the field empty and save to keep the stored secret.
Where each kind of secret belongs#
| Secret | Put it in | The step uses |
|---|---|---|
| Test user password typed into a login form | An encrypted key, for example adminPassword | {{ds:adminPassword}} as the Type value |
| API token or API key | An encrypted key, for example apiToken | {{ds:apiToken}} in the Bearer token or a header |
| Database password | The DB connection | The connection, picked by name |
| Kafka SASL password | The Kafka connection | The connection, picked by name |
| Test mailbox password | The email integration under Integrations | The Mail step’s mailbox |
Who and what can see an encrypted value#
| Where | What appears |
|---|---|
| The key list | “•••••••• set”, or “no value”. Also for its creator. |
| The key panel, for the creator | An eye icon: “Show the value (only you, its creator, can)”. |
| The key panel, for anyone else, Admins included | Only “•••••• (set, encrypted)”. No eye icon. They can replace the value. |
| Test runs | The runner receives the real value and uses it in the step. |
| Live log, run report, API request log | •••••• |
| Send previews on the screen | •••••• is sent in place of the value. Check secured calls with a Dry Run. |
| The AI (App Map, generation) | Key names and “encrypted”, never the value. |
| AI assistants through MCP | Never the value. They cannot write one either. |
| The audit log | Field values of screens that can hold secrets are not recorded; secret field names show ***. |
What you get#
- One place per secret. A step stores the key’s name, not the value. Change the key and every test that uses it picks up the new value on its next run.
- Encryption on the server. Encrypted keys and connection passwords are stored with AES-256-GCM.
- Masking in runs. The value of every encrypted key read during the run, the Authorization, token and API key headers of API steps, and text typed into a password field are replaced with
••••••. - An audit of what slipped through. The Test data audit agent finds passwords, tokens and personal data written into steps, and empty data sources, and says where each should move. It changes nothing.
- AI assistants that never handle a password. Through MCP, an assistant creates the key empty (encrypted for secrets) and asks you to fill it in SmartestQA.
- Masked AI reviews. With Share Sensitive Data off, e-mail addresses, card numbers, national ID numbers, phone numbers, secrets and tokens are masked before the run review is sent to the AI.
Three limits to know
Masking hides values, not screenshots: a screenshot of a page that displays a secret still shows it. Values shorter than 3 characters are not masked. Datasets are not secret: every project member can open a dataset and see all its cells, so keep passwords out of them.
Questions and answers#
Where do I put the password for a login test?Customers
In an encrypted key in Data Source › Variables, for example adminPassword. The Type step’s value is then {{ds:adminPassword}}, and the password itself is not in the test case. See Encrypt a value.
Can an admin see an encrypted value?Customers
No. Only the person who created an encrypted value can show it, with the eye icon in the key panel. An Admin sees “•••••• (set, encrypted)” and can replace the value, but cannot read it. See Who can see an encrypted value.
Do passwords show up in test logs or reports?Customers
No. A run replaces every known secret with •••••• in the live log, the report and the API request log. In the report’s list of variables, values whose name looks like a password or token are masked too. A screenshot of a page that displays a secret still shows it. See Secrets in runs.
What happens if someone types a password straight into a step?Customers
The value is stored with the test case and visible to everyone in the project. Values typed into fields whose name looks like password, secret, token or PIN are shown as •••••• in run logs, but the test case still holds the text. Move it to an encrypted key. See Secrets and personal data.
How do I find passwords that are already written into steps?Customers
Install the Test data audit agent from the catalog on Agents. It is an on-demand agent that you start with Run now. It finds passwords, tokens and personal data written into steps and says where each should move, and it changes nothing. See On demand agents.
How do I rotate a password or token?Customers
Open the key in Variables, type the new value and click Save. Every test that references the key uses the new value on its next run. For a connection, type the new password in the field labelled “Password (leave empty to keep current)”. See Replace or remove a secret.
Does the AI see my passwords?Customers
No. The AI receives key names, their kind, description and whether they are encrypted, never the value. For the AI review of a run, switch Share Sensitive Data off in the project’s AI Options to mask sensitive values before they are sent. See What is sent to the AI.
Can an AI assistant connected through MCP read or write a password?Customers
No. The assistant never asks you for a password and never writes one. When a test needs a credential, it creates the key empty, encrypted for secrets, and you fill it in SmartestQA. It reads connections with their credentials masked. See Other safeguards.
What happens when I copy an encrypted key or switch Encrypt off?Customers
The secret does not travel. A copy is a plain key without a value: type the secret again and switch on Encrypt. If you switch Encrypt off, the field is empty; saving it empty leaves the key without a value. See My key lost its value.
Where does the database password go?Customers
Into the connection in Data Source › Connections. It is stored encrypted and never sent back to the screen: the field stays empty when you open the connection. A failed Test Connection masks it in its message. See Add a database connection.
Read more#
- Secrets and credentials: encrypting, replacing, masking and the checklist.
- Create a key: the key panel, folders and environment variants.
- Connections: database and Kafka connections and their passwords.
- What happens at run time: how a run resolves
{{ds:…}}. - Safe changes through MCP: what an AI assistant may and may not do.
- What is sent to the AI: the project’s AI options, including Share Sensitive Data.
- What the audit log records: and why secret fields are left out.
Didn’t find your answer?Write to support with your project name and what you see on screen.
info@smartestqa.ai