All of the test runners have their issues. I've had as much problems with Jenkins as I've had with Github Runners and similar solutions.
The best way out of this is to subdivide responsibilities:
- Put all build/publish/test logic in Makefiles or scripts in the repo. This means that devs can run it locally as well. The only interaction between a test runner and the codebase should be running `make <target>`.
- Put all permissions and code checkout and artifact publishing credentials in the test runner, but no logic. At most you would put processing of test output like making junit/coverage/tap more web-readable.
That's it. Split the efforts cleanly, and things fall into place. Also, you can switch runners easily - less needs to be reimplemented in whatever runner config DSL is picked.
The best way out of this is to subdivide responsibilities:
- Put all build/publish/test logic in Makefiles or scripts in the repo. This means that devs can run it locally as well. The only interaction between a test runner and the codebase should be running `make <target>`.
- Put all permissions and code checkout and artifact publishing credentials in the test runner, but no logic. At most you would put processing of test output like making junit/coverage/tap more web-readable.
That's it. Split the efforts cleanly, and things fall into place. Also, you can switch runners easily - less needs to be reimplemented in whatever runner config DSL is picked.