appinsights-surefire-action: Notable Alternatives

In my previous post I discussed the jUnit reporting feature I’m used to using in Jenkins, and the lack of an satisfactory alternative when implementing build pipelines with GitHub Actions.

GitHub Actions and Jenkins are far from the only options for implementing build pipelines, but before I continue I wanted to highlight two notable alternatives.

Azure DevOps

When working with GitHub and Azure it seems natural to take a quick look at what is available through Azure DevOps. When it comes to complete comparisons A Cloud Guru published a comparison of Azure DevOps and GitHub recently. My gut feeling when reading discussions such as this one on reddit, and the extent to which Actions are used on open source projectes is that GitHub Actions is a better investment of my time than building up experience in Azure DevOps.

That said, Azure DevOps does have great test result integration out of the box. Following the standard example for performing a maven build, surefire and failsafe results get published to Azure DevOps. In the build pipeline this looks like the following screenshot.

This is a great overview as it shows easy to interpret graphs of pass/failure status in addition to information about whether the test is newly failing or an existing failure, in addition to total run duration compared to previous run. That’s a lot of information in a small, yet easily digestable format.

In addition the test run is published to the Azure DevOps test management interface where the test run is presented as follows.

Again, easy to digest graphs, although I must admit all of the four graphs appear to be showing the same information. You can also drill down into details for each individual test case including it’s history.

This looks like some very nice tooling if you have a dedicated tester on the team, and the overview graphs would be nice for change management reporting.

GitLab

The other tool I wanted to call out specifically is GitLab. This is my tool of choice for my hobby projects unless there are technical reasons to have them on GitHub.

Similar to Azure DevOps, reporting of test results comes out of the box as a supported feature on GitLab.

Less graphs but still a good, easy to understand, overview of the status on a build result.

It’s possible to drill down in the list at the bottom to see the breakdown of tests

When running as part of a merge request (GitLab language for pull request) the test results are compared to the latest run in the base branch providing an indication of whether a test failure is new or if it is the result of a flaky test.

This feature would have been worth it’s weight in gold on a previous project where we had an extensive suite of end-to-end GUI driven tests that we notoriously unstable at times.

While I’m not specifically setting a goal of reproducing the functionality provided by Azure DevOps or GitLab in my little hobby project here I did want to hightight them as providing inspiration and as clear documentation that GitHub seems to be lacking something that I would consider a basic feature.