Orcraft and other test tools
Orcraft is the platform for test development and orchestration that lets you and your AI agents run the entire testing operation. You and your AI agents write the test logic, in plain Python, and nothing else. The Orcraft framework takes care of the rest: the GUI of a run, the machines each step runs on, the logs and the results, and the snapshot that repeats a run in a CI pipeline.
The table sets Orcraft beside the test tools engineers already know, one characteristic to a row. Each name opens a page that sets Orcraft beside that tool alone.
The tools were picked to cover the kinds of tool a lab chooses between: a commercial test sequencer (NI TestStand), an open-source one (OpenTAP), a keyword-driven framework (Robot Framework), the Python test framework (pytest), a hardware test library (OpenHTF) and two systems for boards in a lab (labgrid and LAVA). If the tool you use is missing, tell us.
See the note on other products
Side by side
yes · no · words where a tool does it in its own way
| Orcraft | NI TestStand | Robot Framework | pytest | OpenTAP | OpenHTF | labgrid | LAVA | |
|---|---|---|---|---|---|---|---|---|
| What it is | Test framework, controller and endpoint application | Test sequencer | Keyword-driven test framework | Python test framework | Open-source test sequencer on .NET | Open-source Python library for hardware tests | Python library for embedded boards in a lab | Continuous integration system for boards |
| Test is code in | Python | Sequence file | Keywords | Python | XML test plan | Python | Python | YAML job |
| Writing a structured test suite with AI agents | The whole suite, from instructions written for AI agents | Code modules; the sequence is built in the editor | Keyword files, as text | Test functions, as Python | Plugins as code, the plan as XML | Phases, as Python | pytest tests, as Python | Job files, as YAML |
| Running tests with AI agents across the lab | On the machines the request names | On the test station | On its own machine | On its own machine | On its own machine | On the test station | On boards, from the command line | Jobs submitted from the command line |
| Code first | yes | no | no | yes | no | yes | yes | no |
| Code once | yes | Limits and properties loaded from files | Variables on the command line | Options the test code adds | Plan parameters and settings | Configuration files | Environment file | A job file per device type |
| Code only | yes | no | no | yes | no | yes | no | no |
| GUI | Built from the test code | Sequence Editor, operator interfaces | Separate editors, such as RIDE | no | Editor | Web view of the test station | no | Web interface of the server |
| Remote machines in a run | yes | Windows computers with TestStand | Through a remote library | no | no | no | Boards, driven from the test host | Boards |
| Machine chosen per step | yes | Per called sequence | Per keyword library | no | no | no | Per command, a board | Per job |
| Test package upload | yes | Files on a shared path | no | no | no | no | Files synced to the lab host | Images and test definitions, by the job |
| Endpoint software update | yes | no | no | no | no | no | no | no |
| Live logs | From every machine, in the GUI | Execution view | Console progress; the log after the run | Terminal output | Log of the run | Web view | Terminal output | Job log in the web interface |
| Results tree | At every level, from the session down to the step, with the configuration beside the results and a filter by status | Report, per step | Log file, down to the keyword | Per test | Per step, through result listeners | Per phase and measurement | Per test | Per test case |
| Results export | XLSX, session files, text for AI agents | HTML, XML, ATML, database | HTML, XML, xUnit | JUnit XML; more through plugins | CSV and databases, through result listeners | JSON, output callbacks | As pytest | CSV, YAML, through the API |
| Machines with roles | yes | no | no | no | no | no | Named targets in the environment file | Roles in a multi-device job |
| Fan-out: one run per machine | yes | Per unit under test, parallel and batch models | no | no | no | no | no | A job per device |
| Machines shared by the team | yes | no | no | no | no | no | Places, through the coordinator | Devices, through the scheduler |
| Machine pools | yes | no | no | no | no | no | Reservations by tags | By device type and tags |
| Case matrix | From dimensions: randomized, fetched at run time, with exclusion rules, filtered in the GUI | Loops in the sequence | Test templates | Parametrized tests | Sweep steps | no | As pytest | no |
| Reboot inside a test | yes | no | no | no | no | no | Power cycles the board | Boots the board in the job |
| Run in a CI pipeline | A snapshot of the run, exported from the GUI | Through the command line or the API | The same command | The same command | The same command | The test script | As pytest | It is the CI system for boards |
| Past runs loaded into the GUI | Runs from the command line, CI or another machine, with their configuration, to change and rerun | Saved reports, to read | Log and report files, to read in a browser | no | Results viewer, to read from a database | Past tests of the station, to read in the web view | no | Every job, in the web interface; a job can be resubmitted |
| OS | Windows, Linux, macOS | Windows | Windows, Linux, macOS | Windows, Linux, macOS | Windows, Linux, macOS | Windows, Linux, macOS | Linux | Linux |
Beside the tools around a run
Two kinds of tools do a different job, and Orcraft works together with them:
- Test management tools such as TestRail, Xray and Zephyr: they track cases, plans and results; Orcraft runs the tests.
- CI servers such as Jenkins, GitHub Actions and GitLab CI: they trigger the run; Orcraft carries it out on the lab's machines.
All other product names are trademarks of their respective owners and are used for identification and comparison purposes only. Orcraft is not affiliated with, sponsored by or endorsed by any of them. The descriptions of other products on this page compare features and functionality to help you decide whether Orcraft fits your needs; they are not intended to disparage any product or its owner. They are summaries written by us, reflect our understanding at the time of writing, and may be inaccurate or out of date. For an accurate description of any product, visit its own site. If you believe a description is wrong, tell us and we will correct it.