I'm considering using Bors for a somewhat peculiar set of requirements. I have a project that has very slow CI. CI is so slow that it currently runs on a daily fixed schedule. I would like to apply a "merge queue" style strategy to avoid having a broken
master branch as with a once-a-day cadence missing even a couple of runs can be a substantial delay for updates. Bors obviously sounds perfect for this project as it makes a merge queue setup easy and supports batching.
However I am concerned that if we experience a failure that bisection will take a long time to complete. nixpkgs gets about 100 commits a day so a simple bisection would take 6-7 runs to identify the culprit (assuming that there is one). In order to avoid blocking up for a week upon every failure it will be necessary to implement a smarter search than running CI on each commit.
: This might actually be much quicker due to hopefully failing quickly however there is nothing to guarantee that.
One option to transfer some information from the original failure to the new build. For example a list of failing tests. In that case we could run those tests first which should significantly reduce the time to identify if this build is failing.
However this isn't perfect either. Imagine that the previous build took 12h and the last commit was the problematic one. We would still need to do 7 runs and take 3 days to resolve the issue.
This is basically building on Ordering by only running the failed tests on the first pass. For example we could do an "optimistic bisection" with just the relevant tests, then once we believe we have found the culprit we can run a full build at a commit that is expected to pass.
In this case imagine that we can run the failed tests in 1h. We would need 8 runs, but the first 7 should take 1h each. The final run would be very slow but is expected to succeed.
I would need to think about the details a bit more. Especially if there is more than one failure present and how the behaviour changes with fix commits thrown into the mix. However it seems like this should make the project feasible if implemented.