Orcraft vs OpenHTF
Both are Python frameworks for testing hardware, and in both a test is plain Python. They differ in how far a test reaches: OpenHTF runs a test on the station it was written for, and Orcraft runs the steps of a test on the machines of the lab.
See the note on other products
Side by side
yes · no · words where a tool does it in its own way
| Orcraft | OpenHTF | |
|---|---|---|
| What it is | Test framework, controller and endpoint application | Open-source Python library for hardware tests |
| Test is code in | Python | Python |
| Writing a structured test suite with AI agents | The whole suite, from instructions written for AI agents | Phases, as Python |
| Running tests with AI agents across the lab | On the machines the request names | On the test station |
| Code first | yes | yes |
| Code once | yes | Configuration files |
| Code only | yes | yes |
| GUI | Built from the test code | Web view of the test station |
| Remote machines in a run | yes | no |
| Machine chosen per step | yes | no |
| Test package upload | yes | no |
| Endpoint software update | yes | no |
| Live logs | From every machine, in the GUI | Web view |
| Results tree | At every level, from the session down to the step, with the configuration beside the results and a filter by status | Per phase and measurement |
| Results export | XLSX, session files, text for AI agents | JSON, output callbacks |
| Machines with roles | yes | no |
| Fan-out: one run per machine | yes | no |
| Machines shared by the team | yes | no |
| Machine pools | yes | no |
| Case matrix | From dimensions: randomized, fetched at run time, with exclusion rules, filtered in the GUI | no |
| Reboot inside a test | yes | no |
| Run in a CI pipeline | A snapshot of the run, exported from the GUI | The test script |
| Past runs loaded into the GUI | Runs from the command line, CI or another machine, with their configuration, to change and rerun | Past tests of the station, to read in the web view |
| OS | Windows, Linux, macOS | Windows, Linux, macOS |
One station and a lab of machines
OpenHTF is Google's open-source framework for hardware testing on a test station. A test is a sequence of phases, plain Python functions; plugs wrap the instruments and the device under test, measurements with validators decide the outcome, and an operator web interface shows the station. The test runs on the station it was written for. Running it elsewhere, configuring it and collecting results across stations are left to the team.
In Orcraft the controller runs the steps on the machines registered as endpoints: Windows, Linux and macOS machines with the endpoint application installed. An endpoint is any machine that can reach the hardware: a lab PC wired to a device under test, a server on the same network as the instruments, a machine with the vendor's tools installed. The flow decides which step goes to which machine, so one test can act on several machines in order.
Configuring and running
The fields that configure a run are declared in the suite, and the controller builds the run's GUI from them. A run configured there is exported as a snapshot, and a CI pipeline runs that snapshot from the command line. Dimensions expand one test into a matrix of cases.
AI agents
An AI agent can write OpenHTF phases as easily as an Orcraft test: both are Python. With Orcraft, what it writes has the same structure, the same configuration fields and the same results as every other suite, and the AI agent runs it on the lab's machines, beyond one test station. Every part of the work can be done by a person or by an AI agent. The AI Agents section shows how.
Existing Python code
A step is ordinary Python code. Python code that already drives an instrument or a device can be called from an Orcraft step as it is.
All other product names are trademarks of their respective owners and are used for identification and comparison purposes only. Orcraft is not affiliated with, sponsored by or endorsed by any of them. The descriptions of other products on this page compare features and functionality to help you decide whether Orcraft fits your needs; they are not intended to disparage any product or its owner. They are summaries written by us, reflect our understanding at the time of writing, and may be inaccurate or out of date. For an accurate description of any product, visit its own site. If you believe a description is wrong, tell us and we will correct it.