Controlled Endpoints Disconnection

Some tests have to take the machine they run on out of service. A driver installation needs a reboot before it takes effect, a service under test has to be restarted with a new configuration, a resilience test wants to see what a device does after a power cycle. On an ordinary test runner that is the end of the run: the process that was executing the test is gone with the machine. Orcraft separates the two. The test runs on the controller, and the endpoint application on the machine only executes the steps sent to it, so an endpoint can go away in the middle of a flow and the flow simply waits for it to come back.

The framework gives a flow two calls for this. restart_endpoint asks the endpoint application to restart itself and waits until it is serving again. wait_for_endpoint_state waits until the endpoint reaches a given state: Offline when it no longer answers, Online when it answers but cannot take work yet, Ready when it can. With them a flow can trigger whatever disruption the test needs, wait for the endpoint to go down and come back, and then run more steps on it to prove it recovered. The Endpoints group has two tests built this way, sharing one flow: Endpoint Restart restarts the endpoint application, and Controlled Endpoint Disconnection reboots the whole machine.

The Endpoint Lifecycle flow

Both tests hand their case to Flow: Endpoint Lifecycle. The flow reads like any other flow from Structure and Lifecycle, with the disruption in the middle:

Delete Existing Maze File and Generate / Save run on the endpoint and leave a maze file on its disk. This is the state that has to survive.
The disruption. In restart mode the flow calls restart_endpoint. In disconnect mode it runs a Trigger Reboot step that issues the operating system's shutdown command, then waits for the endpoint to go Offline and then Ready again.
Load / Solve runs on the endpoint again, opens the maze saved before the disruption and solves it. A checkpoint reports whether the endpoint recovered.

The flow does not know or care what happened to the machine in between. It waits for a state, with the timeouts the test exposes, and either continues or fails with a message naming the state that never came.

Endpoint Restart

The Endpoint Restart test refreshes the endpoint application on a machine without rebooting it, which is what a test wants after installing a new version of something the application loads at start. Its endpoint field is a fan-out, so one run restarts several machines at once. Select the test, pick GMG-TEST-1 and GMG-TEST-3, set the Maze Executable Folder and leave the timeouts at their defaults:

Click Update Plan and Run, and watch the Endpoints panel. The two runs start together, and each card is Busy, in purple, while its run holds the endpoint:

Then the restart comes. The endpoint application stops answering for a moment and starts again. The card stays Busy through it, because the run keeps its claim on the endpoint while it waits for it to come back. The log of the GMG-TEST-1 run tells the restart from the flow's side. The restart request itself is answered by a disconnection, which is the expected outcome of asking a process to restart, and the flow then waits for Ready, takes its lock on the endpoint again and carries on:

            [FlowEndpointLifecycle] Requesting endpoint restart (GMG-TEST-1:50051)...
            [FlowEndpointLifecycle] Endpoint disconnected while sending restart command; treating as expected restart behavior.

            [FlowEndpointLifecycle] Waiting for endpoint to go Ready (GMG-TEST-1:50051), timeout 5 m...
            [FlowEndpointLifecycle] Endpoint is Ready (GMG-TEST-1:50051), re-acquiring lock...
            [FlowEndpointLifecycle] Lock re-acquired for GMG-TEST-1:50051
            [FlowEndpointLifecycle] Endpoint restarted successfully (GMG-TEST-1:50051)
                Path Found   | True
                Path Length  | 24

                 Checkpoint Pass   Endpoint recovered — maze solved in 24 step(s).

Both runs pass, one per machine, with the checkpoint confirming that the maze saved before the restart was solved after it:

A restarted endpoint application comes back as the same endpoint: the card keeps its place, its version and its security setting. Only the process changed.

Controlled Endpoint Disconnection

The Controlled Endpoint Disconnection test covers the disruptions the framework does not trigger itself: a reboot of the machine, or anything done to the endpoint by hand. Its panel adds the disruption's own controls to the same fields:

Enable Reboot chooses who disrupts the endpoint. Off, the default, the test waits for you to do it; on, the Trigger Reboot step runs the operating system's restart command on the machine, after Reboot Delay Sec.
Disconnect Timeout Sec bounds the first wait: for the endpoint to go Offline when it reboots, or to come back Online when you restart it by hand.
Connect Timeout Sec bounds the wait for Ready. It has to cover the whole boot, the endpoint application's start and the time it takes to become reachable again; five minutes is the default.

The manual exercise

Run the test with the box unchecked, on GMG-TEST-3 here. The flow saves its maze, the Trigger Reboot step reports that it did nothing, and the flow starts waiting for the endpoint to come back Online:

                                                  Step: Trigger Reboot | Configuration                                 

                Start Time               | Target Endpoint                           | Enable Reboot | Reboot Delay Sec
                -------------------------------------------------------------------------------------------------------
                2026-09-12  22:43:30.061 | GMG-TEST-3:50051 [Secure] (v0.1.0.dev197) | False         | 0

                Reboot skipped (enable_reboot=False). Endpoint will not be rebooted.

                                     Step: Trigger Reboot | Result                     

                Name                 | Status | Finish Time              | Elapsed Time
                -----------------------------------------------------------------------
                Step: Trigger Reboot | PASS   | 2026-09-12  22:43:30.062 | 0.000 s

            [FlowEndpointLifecycle] Waiting for endpoint to go Online (GMG-TEST-3:50051), timeout 1 m...

This is your cue. Go to the machine, close the endpoint application and start it again, within the minute the Disconnect Timeout allows. The card in the Endpoints panel goes Offline, in grey, while the application is down, Online while it starts, and Ready when it is serving again, and the flow follows the same states: it sees Online, waits for Ready, takes its lock on the endpoint again and runs Load / Solve on the saved maze. Leave the application closed for longer than the timeout and the run ends in an error naming the state that never came, which is the same message a machine that failed to come back would produce.

The reboot

With Enable Reboot on, nothing is left to you. The Trigger Reboot step issues the restart command, the flow waits for Offline and then for Ready, and the rest is as above. What makes the round trip possible is the endpoint's auto-start registration from the setup guide: the endpoint application comes back by itself after the machine's logon, so nobody has to start anything. Run it on a machine of your own that is free to reboot and set to log on automatically, and watch the card go Offline, in grey, and return through Online to Ready.

The next page, Handling Asynchronous Steps, stays on the endpoint but keeps it running: it starts work there and comes back for the result later.