Orcraft vs pytest
Both run tests written in Python. pytest runs test functions on one machine and reports pass or fail, and so does unittest. Orcraft covers what happens around that: the structure of a suite, the GUI of a run, the machines each step runs on, the combinations a test covers and the results of every run.
See the note on other products
Side by side
yes · no · words where a tool does it in its own way
| Orcraft | pytest | |
|---|---|---|
| What it is | Test framework, controller and endpoint application | Python test framework |
| Test is code in | Python | Python |
| Writing a structured test suite with AI agents | The whole suite, from instructions written for AI agents | Test functions, as Python |
| Running tests with AI agents across the lab | On the machines the request names | On its own machine |
| Code first | yes | yes |
| Code once | yes | Options the test code adds |
| Code only | yes | yes |
| GUI | Built from the test code | no |
| Remote machines in a run | yes | no |
| Machine chosen per step | yes | no |
| Test package upload | yes | no |
| Endpoint software update | yes | no |
| Live logs | From every machine, in the GUI | Terminal output |
| Results tree | At every level, from the session down to the step, with the configuration beside the results and a filter by status | Per test |
| Results export | XLSX, session files, text for AI agents | JUnit XML; more through plugins |
| Machines with roles | yes | no |
| Fan-out: one run per machine | yes | no |
| Machines shared by the team | yes | no |
| Machine pools | yes | no |
| Case matrix | From dimensions: randomized, fetched at run time, with exclusion rules, filtered in the GUI | Parametrized tests |
| Reboot inside a test | yes | no |
| Run in a CI pipeline | A snapshot of the run, exported from the GUI | The same command |
| Past runs loaded into the GUI | Runs from the command line, CI or another machine, with their configuration, to change and rerun | no |
| OS | Windows, Linux, macOS | Windows, Linux, macOS |
The structure of a suite
A pytest test is a function, and a project arranges its functions, fixtures and plugins as it sees fit.
An Orcraft suite has one structure: groups, tests, cases, flows and steps. Because every suite has the same structure, the controller reads it and builds the GUI from it: the tree of tests, the fields that configure a run, the plan and the results. The Framework section describes each level.
This is a flow and its step from the tutorial's Hello Endpoint test. The flow runs one step, and the step runs a command on the machine chosen for the run:
class FlowHelloEndpoint(OrcraftTestFlow):
result_type = FlowHelloEndpointResult
def __init__(self, config: FlowHelloEndpointConfig) -> None:
super().__init__(config)
def _run_flow(self) -> None:
# No endpoint is named here: the step goes to the test's Test Endpoint.
step_result = self.run_step(StepHostname)
if self.handle_errors(step_result, self.result):
return
self.result.hostname = step_result.hostname
class StepHostname(OrcraftTestStep):
def __init__(self, config: StepHostnameConfig | None = None) -> None:
super().__init__(config or StepHostnameConfig())
def _run_step(self) -> OrcraftTestStepResult:
self.result = StepHostnameResult()
# The step's body runs on the endpoint it was sent to, so the command
# answers with that machine's name.
completed = subprocess.run(["hostname"], capture_output=True, text=True)
hostname = completed.stdout.strip()
self.checkpoint(
completed.returncode == 0 and bool(hostname),
f"hostname answered: {hostname}",
f"hostname failed with exit code {completed.returncode}: {completed.stderr.strip()}",
)
Logger.log(f"Hello from {hostname}", color=Logger.Color.BLUE)
self.result.hostname = hostname
return self.result
Machines
pytest runs on the machine where it is started. Reaching another machine is code the test writes itself: an SSH session, a socket, a vendor's tool.
In Orcraft the other machine is an endpoint: a Windows, Linux or macOS machine with the endpoint application installed. Every step names the machine it runs on, so one test can act on several machines in order, and a test can reboot its machine and continue when it comes back. Nothing is copied to the machines beforehand: the controller uploads the test package to each endpoint when the run starts.
Case matrices
pytest parametrizes a test function with the parameters listed on it. In Orcraft, dimensions declared on a test expand it into a planned matrix of cases. The cases are filtered in the GUI and run in a defined or a randomized order, and the same test can fan out to several endpoints, one independent run each.
AI agents
An AI agent can write a pytest file as easily as an Orcraft test. With Orcraft, what it writes has the same structure, the same configuration fields and the same results as every other suite, and it runs on the lab's machines. Every part of the work can be done by a person or by an AI agent: writing the tests, configuring a run, running it and reading the results. The AI Agents section shows how.
Existing pytest tests
A step is ordinary Python code, so existing Python test logic can be called from an Orcraft step. Nothing has to be thrown away.
All other product names are trademarks of their respective owners and are used for identification and comparison purposes only. Orcraft is not affiliated with, sponsored by or endorsed by any of them. The descriptions of other products on this page compare features and functionality to help you decide whether Orcraft fits your needs; they are not intended to disparage any product or its owner. They are summaries written by us, reflect our understanding at the time of writing, and may be inaccurate or out of date. For an accurate description of any product, visit its own site. If you believe a description is wrong, tell us and we will correct it.