Codecov has a status called codecov/project which succeeds if the entire project's test coverage on the staging branch is greater than the entire project's test coverage on the master branch, and fails otherwise.
I have a project (https://github.com/bcbi/ModelSanitizer.jl) that has 100% coverage on the master branch. Therefore, the codecov/project status will fail on the staging branch if the coverage on the staging branch falls below 100%. This is good - I want to enforce the policy that our coverage must never fall below 100% on the master branch.
Therefore, I have required the codecov/project status in bors.toml, i.e. my bors.toml file includes the following lines:
status = [
"Travis CI - Branch",
"codecov/project",
]
My test suite is divided into multiple Travis jobs. Each Travis job submits a coverage report to Codecov, and Codecov merges all of the reports together.
The coverage on each of the Travis jobs is less than 100%, but after all of the reports are merged together, the coverage is 100%.
Unfortunately, the Travis jobs do not all complete at the same time. Therefore, the coverage reports are not all submitted to CodeCov at the same time. An example timeline may look like this:
-
Bors pushes to the
stagingbranch, and the Travis jobs start. For the sake of this example, suppose that there are two Travis jobs.- Current state of GitHub status checks:
-
Travis CI - Branch: pending -
codecov/project: this GitHub status does not yet exist
-
- Current state of GitHub status checks:
-
Five minutes later, Travis job #1 finishes and submits a coverage report to Codecov with 80% coverage. Since this is the only report, Codecov has nothing else to merge. Codecov creates the
codecov/projectGitHub status check and marks it as failing because the coverage of 80% is less than 100%.- Current state of GitHub status checks:
-
Travis CI - Branch: pending -
codecov/project: failure (coverage decreased from 100% to 80%)
-
- Current state of GitHub status checks:
-
Ten minutes later, Travis job #2 finishes and submits a coverage report to CodeCov with 60% coverage. Codecov now has two reports total, so it merges the two reports and finds that the total coverage is now 100%. Codecov updates the existing
codecov/projectGitHub status check and changes it from failing to passing because the coverage is 100%.- Current state of GitHub status checks:
-
Travis CI - Branch: success -
codecov/project: success (coverage 100% unchanged from previous)
-
- Current state of GitHub status checks:
Unfortunately, at step 2 above, Bors sees that the codecov/project status check has failed, and thus Bors fails the build.
However, the state of the codecov/project status at the end of step 2 is not accurate, because we will not know the true project coverage until all of the Travis builds have finished.
This is a bug because at the end of step 3, all of the status checks have succeeded, which means that Bors should have merged. However, because Bors looked at the codecov/project status at the end of step 2, Bors incorrectly failed the build.
Workaround
There is a workaround, but it's not great. You remove "codecov/project" from the [status] section of bors.toml, and you add codecov/project as a required status check in the GitHub protected branches setting for the master branch:
With this workaround, if the total coverage on the staging branch is less than 100%, GitHub won't allow bors to fast-forward the master branch. This causes Bors to crash with the following error:
Crash batch
{{:badmatch,
{:error, :push, 422,
"{\"message\":\"Required status check \\\"codecov/project\\\" is failing.\",\"documentation_url\":\"https://help.github.com/articles/about-protected-branches\"}"}},
[
{BorsNG.Worker.Batcher, :complete_batch, 3,
[file: 'lib/worker/batcher.ex', line: 390]},
{BorsNG.Worker.Batcher, :maybe_complete_batch, 1,
[file: 'lib/worker/batcher.ex', line: 376]},
{BorsNG.Worker.Batcher, :poll_, 1,
[file: 'lib/worker/batcher.ex', line: 202]},
{BorsNG.Worker.Batcher, :handle_info, 2,
[file: 'lib/worker/batcher.ex', line: 184]},
{:gen_server, :try_dispatch, 4,
[file: 'gen_server.erl', line: 637]},
{:gen_server, :handle_msg, 6,
[file: 'gen_server.erl', line: 711]},
{:proc_lib, :init_p_do_apply, 3,
[file: 'proc_lib.erl', line: 249]}
]}
This is good because Bors was not allowed to fast-foward master to a commit with less than 100% code coverage!
This is bad because instead of posting a comment on the pull request indicating that the push failed, Bors crashes. Therefore, I receive no notification that the build failed.