Orcraft vs OpenTAP
Both run a sequence of test steps and report the results, from a GUI and from the command line. They differ in where the sequence lives, in what the steps are written, and on which machines a run takes place.
See the note on other products
Side by side
yes · no · words where a tool does it in its own way
| Orcraft | OpenTAP | |
|---|---|---|
| What it is | Test framework, controller and endpoint application | Open-source test sequencer on .NET |
| Test is code in | Python | XML test plan |
| Writing a structured test suite with AI agents | The whole suite, from instructions written for AI agents | Plugins as code, the plan as XML |
| Running tests with AI agents across the lab | On the machines the request names | On its own machine |
| Code first | yes | no |
| Code once | yes | Plan parameters and settings |
| Code only | yes | no |
| GUI | Built from the test code | Editor |
| Remote machines in a run | yes | no |
| Machine chosen per step | yes | no |
| Test package upload | yes | no |
| Endpoint software update | yes | no |
| Live logs | From every machine, in the GUI | Log of the run |
| Results tree | At every level, from the session down to the step, with the configuration beside the results and a filter by status | Per step, through result listeners |
| Results export | XLSX, session files, text for AI agents | CSV and databases, through result listeners |
| Machines with roles | yes | no |
| Fan-out: one run per machine | yes | no |
| Machines shared by the team | yes | no |
| Machine pools | yes | no |
| Case matrix | From dimensions: randomized, fetched at run time, with exclusion rules, filtered in the GUI | Sweep steps |
| Reboot inside a test | yes | no |
| Run in a CI pipeline | A snapshot of the run, exported from the GUI | The same command |
| Past runs loaded into the GUI | Runs from the command line, CI or another machine, with their configuration, to change and rerun | Results viewer, to read from a database |
| OS | Windows, Linux, macOS | Windows, Linux, macOS |
The plan and the plugins
OpenTAP is Keysight's open-source test sequencer. Its engine is C# on .NET. A test plan is a sequence of
steps saved as a .TapPlan XML file, built in the tool's editor or its text user interface and
run with the tap run command, and the steps come from plugins written in C# or Python. That is
two layers to keep aligned: the plan file and the plugins behind it.
In Orcraft the plan and the steps are one Python suite, with no plugin toolchain and no .NET.
This is a flow and its step from the tutorial's Hello Endpoint test. The flow runs one step, and the step runs a command on the machine chosen for the run:
class FlowHelloEndpoint(OrcraftTestFlow):
result_type = FlowHelloEndpointResult
def __init__(self, config: FlowHelloEndpointConfig) -> None:
super().__init__(config)
def _run_flow(self) -> None:
# No endpoint is named here: the step goes to the test's Test Endpoint.
step_result = self.run_step(StepHostname)
if self.handle_errors(step_result, self.result):
return
self.result.hostname = step_result.hostname
class StepHostname(OrcraftTestStep):
def __init__(self, config: StepHostnameConfig | None = None) -> None:
super().__init__(config or StepHostnameConfig())
def _run_step(self) -> OrcraftTestStepResult:
self.result = StepHostnameResult()
# The step's body runs on the endpoint it was sent to, so the command
# answers with that machine's name.
completed = subprocess.run(["hostname"], capture_output=True, text=True)
hostname = completed.stdout.strip()
self.checkpoint(
completed.returncode == 0 and bool(hostname),
f"hostname answered: {hostname}",
f"hostname failed with exit code {completed.returncode}: {completed.stderr.strip()}",
)
Logger.log(f"Hello from {hostname}", color=Logger.Color.BLUE)
self.result.hostname = hostname
return self.result
The controller reads the suite and builds the GUI from it: the tree of tests, the fields that configure a run, the plan and the results. The Framework section describes each level of a suite.
Machines and operating systems
An OpenTAP test plan runs on the machine where it is started, on Windows, Linux or macOS.
An Orcraft run covers the controller's own machine and any number of endpoints, with the three operating systems mixed in one run. Every step names the machine it runs on, so one test can act on several machines in order, and a test can reboot its machine and continue when it comes back. The live logs and the results of all the machines are in the controller.
AI agents
An OpenTAP test plan is an XML file and its steps come from plugins written in C# or Python, so an AI agent has two things to write and keep aligned. In Orcraft the plan and the steps are one Python suite in a defined structure: an AI agent writes it, configures a run, runs it on the lab's machines and reads the results. Every part of the work can be done by a person or by an AI agent. The AI Agents section shows how.
All other product names are trademarks of their respective owners and are used for identification and comparison purposes only. Orcraft is not affiliated with, sponsored by or endorsed by any of them. The descriptions of other products on this page compare features and functionality to help you decide whether Orcraft fits your needs; they are not intended to disparage any product or its owner. They are summaries written by us, reflect our understanding at the time of writing, and may be inaccurate or out of date. For an accurate description of any product, visit its own site. If you believe a description is wrong, tell us and we will correct it.