How do I run the same test with many sets of data?
In SmartestQA you keep the data as a dataset in Data Source › Datasets and write the test once, with {{excel:name}}_column in place of fixed values. In Schedule, the pipeline's Data & loop card then runs the test case once per row of that dataset.
In short
On this page you can
- Run one test case once per row of an Excel or CSV dataset.
- Take all rows, the first N, all rows in random order, or N random rows.
- Keep one dataset per environment with a
_DEV,_TESTor_PRPname. - Run the whole pipeline once per row of a table you type in.
- Open each row's result as its own iteration in the report.
How to use it
- In Data Source › Datasets, click Upload dataset (Excel / CSV) and save.
- In the test step, open the data picker and pick a dataset column:
{{excel:users}}_email. - Add the test case to a pipeline and open Pipeline Parameters.
- In Data & loop, set Loop mode to Loop and choose Which rows.
- Click Save and run the pipeline.
The short answer#
You write the test case once and take its values from a dataset. A dataset is a table in Data Source › Datasets, uploaded from XLSX, XLS or CSV or typed in. A step reads a column as {{excel:name}}_column, for example {{excel:users}}_email. In Schedule, the pipeline’s Data & loop card with Loop mode Loop runs every test case that reads the dataset once per row. Each row is a full run of the test case with a fresh browser.
The loop belongs to the pipeline, not to the test case. A Dry Run from Automate does not iterate: it uses the first row only.
The problem this solves#
Many flows must work for many inputs: different kinds of customer, amounts, currencies, invalid values. By hand, a tester tries a few of them and hopes the rest behave the same. In automation, teams often copy the test case once per data set, or write a loop in a script that reads a file.
Both ways have a cost. With copies, a change in the flow must be made in every copy. With a loop in a script, all rows often end up in one log, and it takes time to see which row failed and what the page looked like for that row. What is missing is one test case, a table of data next to it, and a result per row.
An example#
This is an illustration, not a customer story. A team tests the transfer form of a banking app’s web client, and a tester has a spreadsheet with 12 transfers to try.
- In Data Source › Datasets the tester clicks Upload dataset (Excel / CSV), picks
transfers.xlsx, ticks Header and clicks Save: “Dataset created”. - In Automate the tester opens the recorded transfer test and uses the picker to replace the fixed values with
{{excel:transfers}}_ibanand{{excel:transfers}}_amount. A Dry Run passes. It used the first row only. - In Schedule the tester adds the test case to a pipeline, opens Pipeline Parameters and sets Loop mode to Loop in Data & loop. Under Which rows the tester takes all rows in order and clicks Save. The card shows “12 case runs”.
- The pipeline runs. Each row is a full run of the test case with a fresh browser.
- In the report the tester opens the test case’s tab and picks the iterations under Run. Iteration 7 is red. Its steps show what happened for that row.
- For the TEST environment the tester uploads a second dataset named
transfers_TEST. The steps keep the nametransfers, and a run in TEST uses the copy.
The team keeps one test case and one table, and a new case to try is a new row.
By hand, with a loop in your own script, and with SmartestQA#
| Question | By hand | A loop in your own script | SmartestQA |
|---|---|---|---|
| Where does the data live? | In a spreadsheet the tester reads from | In a file the script reads, which works well when developers own the data | In a dataset, uploaded from XLSX, XLS or CSV or typed in |
| How many copies of the test are there? | None; the tester repeats the flow | One, if the loop is written well | One; the steps hold references such as {{excel:users}}_email |
| Who decides which rows run? | The tester, by time available | The code | Which rows on the pipeline: all, the first N, all in random order, or N random rows |
| Does each row start clean? | If the tester logs out and in again | If the script resets the browser | Each row is a full run of the test case with a fresh browser |
| How do I see which row failed? | From the tester’s notes | From the log, if the script prints the row | The Run selector in the report opens each iteration |
| Can each environment have its own data? | Yes, with one sheet per environment | Yes, with one file per environment and code that picks it | Yes: a run in TEST that reads users uses users_TEST when that dataset exists |
| Can I try the loop before a pipeline exists? | Not applicable | Yes, by running the script | No: a Dry Run does not iterate, it uses the first row only |
Steps#
Upload the dataset
Go to Data Source › Datasets and click Upload dataset (Excel / CSV). Pick the file (“Supported: XLSX, XLS, CSV”); for a workbook, only the first sheet is read. Tick Header when the first row holds column names. Check the Name, because steps use it, and click Save.
“Dataset created”. The dataset appears in the list with its size and its columns.
Read the columns in the test case
In Automate, select the step and click the picker icon next to the value (“Data Sources - Global Parameters - Datasets”). On the Datasets tab, pick a column. The value becomes, for example,
{{excel:users}}_email. Click Save.Check it with a Dry Run
A Dry Run uses the first row of each dataset. It does not iterate.
Open the pipeline’s parameters
In Schedule, add the test case to a pipeline and open Pipeline Parameters. Data & loop sits at the top. You can also link a value to a dataset here: click the table icon next to a value and pick the column header, not one cell. A fixed cell is always the same row.
Choose Loop
Set Loop mode to Loop. The card shows the dataset, “n cases loop over it” and “u of t columns used”, with Edit data.
Choose which rows
Under Which rows, take the rows in order (all, or the first N), all rows in random order, or N random rows. Random rows “are picked again on every run (scheduled runs too).”
Set the other cases and save
Test cases without a dataset run 1 or more times “with the same values (constants and variables as they are)”. Click the pipeline’s Save.
A line such as “15 case runs” shows the total.
Run and read the rows
Run the pipeline. In the report, the test case’s tab has a Run selector: pick iteration 1, 2, 3… to see the steps of each row.
The three loop modes#
| Loop mode | What runs | Use it when |
|---|---|---|
| None | “The pipeline runs once; each case runs once.” | The test needs one set of data. |
| Loop | “Cases that read a dataset run once per row; the others run once” or the number of times you set. | One or more test cases should repeat for every row of an uploaded dataset. |
| Table (manual) | “Rows you type here; each row is a separate run of the whole pipeline.” The columns are the pipeline’s inputs; an empty cell uses the pipeline’s value. | The whole flow, not one test case, should repeat with other values. |
In Table (manual), Run table queues every enabled row (“5 rows queued for running”) and Run remaining only runs the rows whose last result is not a success. Rows run one after another, and “If a row fails, continue with the next” is on by default. The table needs the pipeline saved first.
What to write, and which rows run#
| You write | Where | The run gets |
|---|---|---|
{{excel:users}}_email | A step value | Column email of the current row in a loop; the first row when the test does not loop. |
{{excel:users}}_# | A step value | The number of rows. |
{{excel:users}}_# | Execute count | All rows, in order. |
{{excel:users}}_#5 | Execute count | The first 5 rows. |
{{excel:users}}_? | Execute count | All rows, in random order. |
{{excel:users}}_?5 | Execute count | 5 random rows. |
If you ask for more runs than there are rows, the rows repeat from the start. The total on the card is counted per dataset: rows taken per run × the test cases that read it, plus the repeat count of the other test cases. A dataset of 12 rows, the first 5 taken, three test cases that read it and two that run once give 5 × 3 + 2 = 17 case runs.
What you get#
- One test case, many data sets. The steps hold references such as
{{excel:users}}_email, not fixed values. - Three file types. XLSX, XLS and CSV, first sheet only, or an empty table that you fill in the panel. The table shows 50 rows per page.
- Safe column names. With Header ticked, “Login Email” becomes
loginEmail. Without it, columns are calledcolumn1,column2… - The same rows for every looping case. All looping cases read the same rows in the same order in one run.
- Data per environment. A run in TEST that reads
usersusesusers_TESTwhen that dataset exists, otherwiseusers. - A result per row. The Reports list counts each iteration as its own attempt, and the test case tab lets you open each one.
- Only Test rows repeat. Setup, Before, After and Teardown run as their phase says; sub-pipelines use their own settings.
Questions and answers#
Can I use an Excel or CSV file as test data?Customers
Yes. SmartestQA takes XLSX, XLS and CSV files in Data Source › Datasets. For a workbook only the first sheet is read, so upload one dataset per sheet. See Upload a dataset.
Why did my Dry Run use only one row?Customers
A Dry Run does not iterate. It uses the first row of each dataset and the values stored in the steps. Iterating over rows is a pipeline feature, set in Data & loop. See Run a test case once per data row.
Can I run only some of the rows, or pick rows at random?Customers
Yes. Which rows in the pipeline’s Data & loop card offers rows in order (all, or the first N), all rows in random order, or N random rows. Random rows are picked again on every run, scheduled runs too. See Loop over a dataset.
What happens to the test cases in the pipeline that do not read the dataset?Customers
They run once, or the number of times you set on the card, with the same values. Only the test cases that read the dataset run once per row. See Data & loop.
Can I repeat the whole pipeline with different values, not just one test case?Customers
Yes, with Loop mode Table (manual). You type one row per run; the columns are the pipeline’s inputs. Run table queues every enabled row, and each scheduled start runs every enabled row too. See Run the whole pipeline once per row.
Can I use different data on DEV, TEST and PRP?Customers
Yes. Name the copies with the suffix _DEV, _TEST or _PRP, upper case, and keep the plain name in the steps. A run in that environment uses its copy when it exists, and the card says so (“environment copy of customers”). See One dataset per environment.
How do I see which row failed?Customers
Open the test case’s tab in the report. The Run selector lists the iterations 1, 2, 3…, and the execute count chip shows the number of rows and the data source name. If a row was retried, Retry lets you pick First Attempt or a retry. See Iterations, retries and execute count.
Can I keep the test users’ passwords in the dataset?Customers
You should not. Every project member can open a dataset and see all its cells. Keep only user names in the table and put passwords in encrypted keys. See Encrypt a value.
Can a step always read one fixed cell instead of the current row?Customers
Yes. Create a key in Variables with Value comes from set to Dataset cell and click the cell. The key always returns that cell, also inside a loop. See Read a dataset cell.
What if the data arrives over time, not as one file?Customers
Use a queue. For records that arrive from a webhook, an e-mail or an Excel upload, a queue runs its processing pipeline once per item. See Queues.
Read more#
- Datasets: Excel and CSV tables: upload, column names, editing rows.
- Data & loop: the loop modes and the table of manual rows.
- Variables and test data: the
{{…}}syntax and where a value comes from. - Reference syntax: keys, datasets and test case variables side by side.
- Data & loop count: how “n case runs” is counted.
- Data Source: keys, datasets and connections in one place.
- How do I build, run and schedule a regression suite?: the pipeline that runs these tests on a schedule.
Didn’t find your answer?Write to support with your project name and what you see on screen.
info@smartestqa.ai