Editing run configuration

This section walks through editing a single, simple config field - Cycles. The same principles apply to every parameter we'll encounter later in this tutorial, including more complex ones like matrices.

Editing config parameter

Selecting Test vs Test Run

First of all, let's look into the execution tree and select the test node itself, not the test run instance. The difference is that the test node (Hello World) is the test's definition, shared by every run, while the test run instance (Test Run: Hello World) is one specific execution of it, with its own status and results.

Changing a parameter

Let's focus on the configuration tab, next to the log. Currently, you can see only one config parameter, Cycles. If you remember, we didn't define it in the test's code - that's because it's a built-in parameter that will always be shown for any test (number_of_cycles in code - see Cycles).

Let's change the number of cycles to 2:

As you can see, the Cycles field is now purple - meaning it changed from its default - and an orange dot appears next to it, meaning the configuration is now different from the plan. Notice that Update Plan and Reset To Defaults both became available. We won't press either of those for now.

Also notice that Total Cases is now 2 as well. We'll talk about matrices and cases later on, since they also affect this number - for now, just get familiar with the formula, no need to remember it yet; see the recap in the chapter's summary below.

Running unplanned

Let's click Run:

As you can see, another test run instance appeared, with a Plan Mismatch badge - indicating that something changed since the plan was last updated (maybe by mistake, or just to try something out), so this run isn't included in the statistics. The previous run is dimmed because it was superseded by this new run on the same endpoint configuration (which is just localhost here).

Updating the plan

Now let's click Update Plan, and click Yes on the confirmation dialog:

Now we see Cycles is blue, meaning this is the new plan, different from the default. Pay attention that the Plan Mismatch badge went to the previous run, since the last run now matches the plan (2 cycles) and previous one doesn't (1 cycle). Also notice that the statistics columns now show 2 in each column - the planned run is now counted.

Note on statistics

It's worth emphasizing that the statistics don't simply sum up every test run. Instead, they select which runs to count based on criteria such as: whether the run is planned or unplanned, whether it was superseded by a later run, and whether the endpoints configuration matches (we'll cover endpoints configuration later in this tutorial).

Resetting to defaults

Now let's click Reset To Defaults:

Now you can see that Cycles went back from blue to white, meaning it's back to its default value - and the orange dot appears again, because we changed the plan from the last update (from 2 cycles back to 1).

Finally, let's click Update Plan once more. This brings us back to the default, planned state we started with:

Rerunning test runs

Rerunning a planned test

Let's click on the first planned test run instance we ran. Its configuration tab now shows the exact configuration it was run with, and we can rerun it with that very same configuration:

Let's click Rerun and see another planned instance appear, while the two previous ones are now superseded:

Rerunning an unplanned test

Let's click on the Plan Mismatch test run instance instead. Its configuration tab shows Cycles set to 2, different from the current plan (1) - this run was unplanned, but its configuration is still preserved and can be rerun just the same:

Also notice that Reset To Defaults is enabled now, since 2 isn't the default value (1 is).

Resetting to Run Settings

Now let's say we change this previous run's config to 3 cycles:

You can see that it now appears as different from both the default and the plan, and that Reset To Run Settings is enabled too. Clicking it takes us back to this run's original configuration.

It's worth emphasizing that each test run instance carries its own configuration, independent of the test's current plan. Whether a given run shows up as planned or unplanned depends solely on whether its own configuration happens to match the plan at the time - not on which one you're looking at.

Chapter's summary

Key takeaways

• The test node holds the current plan; each test run instance is a separate execution with its own, independent configuration.
• A field's color shows how its value compares to the default and to the plan (see the legend below).
• Update Plan makes the current configuration the new plan.
• Reset To Defaults and Reset To Run Settings revert a field to its default or to the run's original value, respectively.
• Rerun runs a test run instance again with its own preserved configuration, whether planned or not.
• Statistics only count runs that meet certain criteria - planned vs. unplanned, superseded or not, and matching endpoints configuration.

Field and badge legend

Total number of cases formula

Total number of planned cases
=
Number of test cycles
×
Number of cases in test
×
Number of cycles per each case

Test cycles and case cycles are both fields of the test config - number_of_cycles and case_number_of_cycles - and the order in which they repeat the cases is described in Cycles. The number of cases in a test comes from its matrix.