Orcraft vs NI TestStand
Both run sequences of test steps against hardware and software, decide pass or fail and report the results. They differ in where the sequence lives, in what the steps are written, and on which machines a run takes place.
See the note on other products
Side by side
yes · no · words where a tool does it in its own way
| Orcraft | NI TestStand | |
|---|---|---|
| What it is | Test framework, controller and endpoint application | Test sequencer |
| Test is code in | Python | Sequence file |
| Writing a structured test suite with AI agents | The whole suite, from instructions written for AI agents | Code modules; the sequence is built in the editor |
| Running tests with AI agents across the lab | On the machines the request names | On the test station |
| Code first | yes | no |
| Code once | yes | Limits and properties loaded from files |
| Code only | yes | no |
| GUI | Built from the test code | Sequence Editor, operator interfaces |
| Remote machines in a run | yes | Windows computers with TestStand |
| Machine chosen per step | yes | Per called sequence |
| Test package upload | yes | Files on a shared path |
| Endpoint software update | yes | no |
| Live logs | From every machine, in the GUI | Execution view |
| Results tree | At every level, from the session down to the step, with the configuration beside the results and a filter by status | Report, per step |
| Results export | XLSX, session files, text for AI agents | HTML, XML, ATML, database |
| Machines with roles | yes | no |
| Fan-out: one run per machine | yes | Per unit under test, parallel and batch models |
| Machines shared by the team | yes | no |
| Machine pools | yes | no |
| Case matrix | From dimensions: randomized, fetched at run time, with exclusion rules, filtered in the GUI | Loops in the sequence |
| Reboot inside a test | yes | no |
| Run in a CI pipeline | A snapshot of the run, exported from the GUI | Through the command line or the API |
| Past runs loaded into the GUI | Runs from the command line, CI or another machine, with their configuration, to change and rerun | Saved reports, to read |
| OS | Windows, Linux, macOS | Windows |
The sequence and the steps
In NI TestStand the sequence and the step code are two things. The sequence is built in the Sequence Editor and saved as a sequence file in the tool's own format; the code of each step lives in a separate module that the sequence calls, written in LabVIEW, LabWindows/CVI, C and C++, .NET or Python.
In Orcraft the sequence and the steps are one Python suite. There is no editor to learn and no sequence file to keep in step with the code.
This is a flow and its step from the tutorial's Hello Endpoint test. The flow runs one step, and the step runs a command on the machine chosen for the run:
class FlowHelloEndpoint(OrcraftTestFlow):
result_type = FlowHelloEndpointResult
def __init__(self, config: FlowHelloEndpointConfig) -> None:
super().__init__(config)
def _run_flow(self) -> None:
# No endpoint is named here: the step goes to the test's Test Endpoint.
step_result = self.run_step(StepHostname)
if self.handle_errors(step_result, self.result):
return
self.result.hostname = step_result.hostname
class StepHostname(OrcraftTestStep):
def __init__(self, config: StepHostnameConfig | None = None) -> None:
super().__init__(config or StepHostnameConfig())
def _run_step(self) -> OrcraftTestStepResult:
self.result = StepHostnameResult()
# The step's body runs on the endpoint it was sent to, so the command
# answers with that machine's name.
completed = subprocess.run(["hostname"], capture_output=True, text=True)
hostname = completed.stdout.strip()
self.checkpoint(
completed.returncode == 0 and bool(hostname),
f"hostname answered: {hostname}",
f"hostname failed with exit code {completed.returncode}: {completed.stderr.strip()}",
)
Logger.log(f"Hello from {hostname}", color=Logger.Color.BLUE)
self.result.hostname = hostname
return self.result
The controller reads the suite and builds the GUI from it: the tree of tests, the fields that configure a run, the plan and the results. The Framework section describes each level of a suite.
Machines and operating systems
NI TestStand runs on Windows, on the test station. A sequence call can run a sequence on another Windows computer that has TestStand installed.
An Orcraft run covers the controller's own machine and any number of endpoints: Windows, Linux and macOS machines with the endpoint application installed. The flow decides which step goes to which machine, so one test can act on several machines in order: one pulls a cable, another checks what it saw. A test can also reboot its machine and continue when it comes back. The live logs and the results of all the machines are in the controller.
AI agents
A TestStand sequence is built in the Sequence Editor and saved in the tool's own format, so an AI agent can write the code modules behind a sequence and not the sequence itself. An Orcraft suite is plain Python in a defined structure, the sequence included: an AI agent writes all of it, configures a run, runs it on the lab's machines and reads the results. Every part of the work can be done by a person or by an AI agent. The AI Agents section shows how.
CI pipelines
A run configured in the controller's GUI is exported as a snapshot, and a CI pipeline runs that snapshot from the command line: the same suite, the same plan.
Existing step code
A step is ordinary Python code. Code that already exists as an executable, a command-line tool or a Python library can be called from an Orcraft step as it is.
All other product names are trademarks of their respective owners and are used for identification and comparison purposes only. Orcraft is not affiliated with, sponsored by or endorsed by any of them. The descriptions of other products on this page compare features and functionality to help you decide whether Orcraft fits your needs; they are not intended to disparage any product or its owner. They are summaries written by us, reflect our understanding at the time of writing, and may be inaccurate or out of date. For an accurate description of any product, visit its own site. If you believe a description is wrong, tell us and we will correct it.