Pre-RFC: Creating a formal decision-making process for bors-ng

process-rfc

#1

Summary: Adopt a process for proposing, approving, and tracking major changes to bors-ng.

Motivation

Some of you might remember way back when @khodzha opened a pull request for an E-easy issue, but @Macarse, who had never even realized the issue existed, wasn’t sure if it was a good idea or not. The project leadership (@notriddle) overruled their concerns, mostly because he figured it would be kind of mean to have a newbie open a pull request just to have it rejected with “actually, we didn’t want that anyway”. This essentially indicates a mismatch of expectations: @khodzha and @notriddle thought that the decision had already been made, while @Macarse still thought it was up for debate.

This is not the only time the project leadership ended up making judgement without very much discussion in response to an issue report, pull request, or support thread. Even when there is feedback, it isn’t very organized, and the design decisions that are made there get mixed with support questions, bug reports, and announcements. We need a statement of record about what has been decided and who decided it, and it needs to strike the reasonable balance between “set in stone” and “overridden on the Benevolent Dictator Pro Tempore’s daily whims”. It also needs to maintain the balance between “maintaining a coherent features set and overall vision” and “getting community involvement in decision making”.

Description

In a blatant rip-off of other projects, I propose that we (and I do mean WE, the whole purpose of this exercise is community participation) adopt a lightweight RFC process for making design calls and distinguishing between a proposed change and a planned one.

The project leadership

Throughout this RFC, the project leadership will be a role given a number of responsibilities. In particular, project leadership approves and rejects RFC’s.

Currently, the project leadership is Michael Howell, also known as @notriddle, the Benevolent Dictator Pro Tempore. Future RFC’s are expected to replace him with a team structure, who will act in consensus as the project leadership.

Scope and the Types of RFCs

After this process is put into effect, major changes and events will require RFC’s. These types of RFC’s can be classed as:

  • Proposal RFC’s, which include:
    • All breaking changes, such as removing or renaming commands, bors.toml configuration options, environment configuration options, changing the name or packaging of the OTP application (which would break people’s deployment scripts), or bumping the Erlang or Elixir version requirement (we will probably make an RFC putting breaking dependency bumps onto some kind of schedule)
    • Adding user-visible features, because they need to be tested and maintained to avoid breakage when users relied on them
  • Informational RFC’s, which include:
    • Starting a talk, meetup, or social networking account that will be expected to officially “represent bors-ng”
    • Documenting design issues, deciding to never implement a feature, proposing an experiment, or recording a proof-generated insight
  • Process RFC’s, which include:
    • Changing the RFC process, the organization of the issue tracker or the support forum, or other community infrastructure
    • Amending the Code of Conduct

Major changes do not include:

  • Most moderation actions
  • Bumping a hex.pm dependency, since the Distillery release process automatically pulls those in anyway
  • Refactoring the codebase, even if it’s a refactor that requires lots of churn
  • Making a blog post
  • Fixing something that is clearly a bug, such as breaking the Not Rocket Science Rule under some edge-case circumstance, crashing, or displaying wrong information
  • Minor look and feel improvements, or even minor feature additions to the dashboard that aren’t likely to translate to a we-will-not-break-this commitment

You’re allowed to open an RFC even if it seems like it matches an item on this do-not-include list if there’s some external circumstance that makes it seem important. Worst case scenario we just close it and tell you to “go ahead and implement it”.

Process

The stages

  • Pre-RFC: this “pseudo-stage” is not governed by the RFC process, but it is recommended that would-be champions go through one before actually opening an RFC.
    • Good ideas to do this include opening a wiki, opening a thread in the Development section of the forum, or discussing it with their coworkers on a corporate chatroom or other non-affiliated space.
    • An RFC should have one idea. The project leadership may reject an RFC if it appears too broad.
    • Would-be champions should search existing RFC’s before opening a new one.
  • RFC Discussion: this is the first actual stage in the process. The responsibilities of all participants are spelled out below this list
    • Final comment period: an optional part of the discussion stage, where the discussion is timeboxed at two weeks
  • Postponed: project leadership can lock a discussion, either time-boxed or until-further-notice
  • Approved: if an RFC is approved, it is assigned a number and its implementation is tracked
  • Rejected or Withdrawn: if an RFC is rejected, it does not get a number, but its discussion is preserved

Responsibilities of the champion

The individual (or, I guess, an organization would be allowed to act as a champion as long as they can agree on everything out-of-band before acting together as a hive mind) who opens an RFC up will act as the RFC’s “champion”. They are responsible for defending or editing the RFC based on criticism that comes up during the discussion, acting as the RFC’s advocate.

If the champion no longer wishes to champion the RFC, they can hand it over to someone else (obviously, the new champion and the old champion both have to publicly agree to this) or withdraw the RFC. If the RFC is withdrawn, it will be closed.

For any discussions that happened before the RFC was opened that the champion thinks are relevant, the champion should copy any knowledge they created into the RFC or the discussion thread. They should also copy any necessary knowledge from the RFC’s own discussion thread into the RFC, so that the RFC’s text can stand alone. Linking to external discussions is allowed, but they should copy or summarize it in case the third-party discussion host goes away.

For an RFC to be accepted it must meet certain minimum criteria. It must not be too broad. It must be a clear and complete description of the proposal. The enhancement must represent a net improvement. The RFC must be proven possible, either by having already implemented it, simulated it, or by including analyses or an argument. The RFC should also summarize the rationale behind the change.

Responsibilities of the project leadership

While anyone can participate in the RFC discussion, the final decision about approving or rejecting an RFC lies with the project leadership (and the mod team, if they decide that an “RFC” is pure spam they can just reject it, but the mod team can’t just accept an RFC). They do not have to put it up to a community vote, though they will need to come to an internal consensus, and they may ask for a non-binding poll if they think it would help.

The project leadership is expected to voice their concerns, allowing the champion to edit or defend the RFC. As described in the Code of Conduct, the project leadership shall make every effort to maintain a professional atmosphere, and to make the champion feel that they are being helped, not hindered. The win condition is one where everyone involved feels that they made the best possible decision with what they have.

Project leadership must put an RFC into a “Final Comment Period” if they wish to accept it, and may (but need not) do so if they wish to reject it. Final Comment Period lasts two weeks, to make sure that anyone who receives RFC’s in weekly digests has a chance to respond. After FCP, all non-staff commenting is disabled and project leadership must make their final decision, moving the RFC from the discussion phase on to the accepted or rejected category, and assigning it a number if they accepted it.

Responsibilities for all discussion participants

The most important rule is to keep tabs on the discussion, and avoid relitigating a point that has already been made. As always, follow the Code of Conduct.

It is highly recommended that third-party participants avoid acting as the devil’s advocate. Instead, please represent what your actually needs are in an honest way; the personal experience of individuals who are not the champion or a member of the project is the most valuable contribution that third-party participants can make.

Also, participants should send any mechanical corrections (spelling, grammar, formatting) to the champion as a Private Message, not as a public post in the thread. The project leadership appreciates mechanical fixes, and the champion probably does too, but once they’ve been fixed any post that corrects them is just clutter, not something that people who want to understand what happened should have to read through years later.

How the discussion system should work

The process overview given below is intentionally a God’s-Eye-View, with no consideration for how it will be implemented from a technical perspective. Before converting this pre-RFC into the final RFC-0001, this section “How the discussion system should work” should be replaced with a section “How to open an RFC”, documenting how the chosen system works. Anyone who proposes to build a new system from scratch must make a credible commitment to write and maintain the system themselves.

The discussion system for RFC’s must support the following:

  • Separate discussion threads for each RFC
  • Tagging, to mark an RFC with its stage
  • Thread locking
    • The “final comment period” mechanic, where a thread is automatically locked after two weeks
    • The time-boxed locking would be nice, for postponing an RFC until a set time, but is not mandatory
  • Some sort of editable document for the RFC’s text, with change tracking
  • Public readability
  • Accessibility with a web browser
  • A documented API, so that further conveniences can be added over time
  • Persistence
  • Google must be able to index the RFC text

While it’s not required for the MVP version of the RFC process, we would very much benefit from a Kanban Board of all the RFCs. I’m imagining something that looks like this:

Discussion Final Comment Period Approved Implemented Rejected/Withdrawn Postponed

Write Status to a GitHub Check

Minimally-Viable GitLab Support

New-Style Emoji

Non-Batching Mode

Batch-Size Limit

Localization Support for the Web Site

GitLab Worker Protocol

Basic REST API

Try Cancellation Command (bors try-)

Vape spam

Turn bors into a SPA

Full-blown issue tracking

These examples are not intended to say anything about whether the example text is appropriate or not. Except the vape spam; obviously when anything like that shows up in the RFC discussion system, it’ll be rejected.

If it doesn’t have built-in support for that Kanban board, which it probably doesn’t if it’s not custom-built, it should be possible to build it on top of the API.

Drawbacks

So far, I’ve basically copied what Rust does, with a couple of tweaks to fit Bors’s slower development pace and a few notes and bits pulled from Swift Evolution, BEP, and PEP.

This means we’re already very familiar with the major drawbacks:

  • Public RFC’s have been known to attract a lot of attention, sometimes creating a discussion that is so long that it’s impossible to read it all, which compounds the problem because it causes people to post the same concern twice.
  • Public RFC’s also attract concern trolls, since they have so much attention paid to them, making the length problem even worse.
  • Settling on a single canonical discussion medium requires one to make final decisions about a bunch of trade-offs, such as ease of archive (the contents of a git repository excel at this one), moderation tools (Discourse excels at this one), familiarity (GitHub Issues excel at this one), ease of entry (chat rooms excel at this one), sophisticated review features (reviewable.io and Gerrit excel at this one), and others that I can’t think of right off the bat. The disorganized status quo allows participants to make an in-the-moment best call.

What happens now?

This is a proposed text for RFC-0001, minus the parts that will eventually be added that are platform-specific. We need to discuss the RFC process first, then decide on what the best platform is for it, then we can finally put the RFC process into effect.


Rebase a pull request
#2

@notriddle excellent idea, +1 from me. Actually we use this same RFC process at work which we also adopted from rust, emberjs and my previous experience when serving the mapserver project steering committee.

My only question is about the approval process. Usually that happens by means of a vote, so in the case of bors-ng is voting open to everyone, restricted to “contributors”, or something else?

Thanks for bors-ng, the great write-up and your excellent stewardship of this project :slight_smile:


Adopt a process for proposing, approving, and tracking major changes to bors-ng
#3

Good question. Looking at the listed inspiration projects, none of them open voting up to the general public. I can think of a couple reasons for this: online voting is hard to secure against double-votes, and it doesn’t make sense to allow people with no stake in the matter to approve Boaty McBoatface joke petitions (there aren’t very many Bors users; the trolls would outvote us).

You should probably read PEP-8002: https://www.python.org/dev/peps/pep-8002/ Heck, read all of the Python post-Guido governance docs. They’re dry, but they articulate a lot of thought put into this stuff.

As a matter of principle, having a council of contributors would be better than just one dictator. But since bors-ng really only has one principle contributor, there isn’t a pool of people to choose the initial council from. Thus, the principle contributor is the Benevolent Dictator by default. That’s true for most FOSS projects, honestly.

So, my plan is basically to follow the same path that Python did, except that I’ll actually plan for stepping down instead of burning out and retiring out of nowhere and leaving an ill-equipped foundation to scramble around looking for alternatives.


Edited the pre-RFC:

  • Added notes about postponement
  • Mentioned the role of project leadership under it’s header

#4

That sounds like a sensible plan.

I glanced at the pep you linked and I saw it covers a lot of projects (including jupyter which I’m working on atm as free-time project), wow! Will try to read it as soon as I can :slight_smile:


#5

How I’d like to implement the RFC system: using Discourse categories, tags, and mod levels

Guide-level explanation

The RFC process itself

Following the pre-RFC discussion, the proposal should be submitted as a draft in #draft-rfcs. The draft should follow the RFC template as described below, or it will fail review immediately (though minor errors may be corrected by a shepherd or reviewer).

The standard RFC workflow is:

  • You, the RFC author, open a new topic in the #draft-rfc category on https://forum.bors.tech that contains the content of the RFC.

    • The title should not contain the word “RFC” unless it’s a change to the RFC process itself. Being in the correct category and tag is enough indication that it’s an RFC draft.
    • The RFC should be tagged with #process-rfc, #feature-rfc, or #informational-rfc.
    • Fill in all of the fields in the RFC template. A copy of the RFC template is given below, though it should already be present by default when opening a topic in that category.
  • The RFC is reviewed by the community:

    • Mechanical fixes to your English or your Markdown will be submitted via User-to-User Messaging, unless the RFC is impossible to read, in which case it may simply be closed.
    • Discussion of the technical contents of the RFC will occur on the RFC’s comment thread.
  • The project leadership will set a timer to lock the RFC after two weeks (14 days) and state the disposition accept, reject, or postpone, putting it into Final Comment Period. This is done when enough of the tradeoffs have been discussed to make a decision. It does not require a consensus among everyone involved in the discussion thread, since that’s often impossible, but there’s shouldn’t be a unified consensus against it.

  • Once approved, your RFC will be moved to the #approved-rfc category, and comments on it will remain locked. It will also be assigned a number and copied to the archive on GitHub.

An RFC will usually not be rejected without a FCP except in a few cases: duplication of effort, being technically unsound, not providing proper motivation, or not being in line with the goals of the Bors project.

As updates are necessary, the RFC author or a project member with TL4 or higher privileges may edit the RFC topic’s post. The author may not edit an RFC after it has been approved, and it should be rare that TL4 members have to do so.

An RFC topic may also be moved to #postponed-rfc. A TL4-level project member may assign the RFC this status when no progress is being made, and we neither want to think about evaluating the proposal nor about implementing it until later. Once an RFC is postponed, a TL4-tier project member may move it back to draft status. We don’t really have a process for doing so; just flag it or ask someone somewhere else.

After an RFC is accepted

Accepting an RFC is not a promise to implement it in any particular timeframe. It is mostly just an acknowledgement that there is no objection to it. The author of an RFC is not obligated to implement it. Of course, the RFC author (like any other developer) is welcome to post an implementation for review after the RFC has been accepted.

That being said, once an RFC is accepted, its implementation will be tracked on a GitHub issue. If you are interested in working on the implementation for an “active” RFC, but cannot determine if someone else is already working on it, feel free to ask (e.g. by leaving a comment on the associated issue).

RFCs can also be superseded by a different RFC, rendering the original obsolete. This is intended for Informational RFCs, where version 2 of an API can replace version 1.

Detailed explanation

This “sub-pre-RFC” (it’ll be merged into the main one if there’s no strong consensus against it) proposes to structure the RFC process into Discourse categories. Specifically, four of them, all sub-categories of Development:

All of these settings are possible in Discourse. I checked.

  • #draft-rfcs has the permissions “Everyone” can “Create/Reply/See”. This is where everyone is pointed at to create RFCs. It also has a setting requiring a minimum of 1 tag, and has a topic template associated with it. It also only allows the tags #informational-rfc, #feature-rfc, and #process-rfc to be used. All posts created in this topic are Wiki-ed by default, since otherwise the RFC author would be unable to edit the post after a day; I would probably have to raise the minimum trust level to edit Wiki posts to enable this.

  • #fcp-rfcs allows Everyone to “Reply/See”, but only allows TL4 to “Create”. It is also configured to auto-close topics after 14 days. Post must also be de-Wiki-ed when moved to FCP, though that’ll probably just have to be done by hand initially.

  • #approved-rfcs allows Everyone to “See” only. It requires TL4 to “Create/Reply/See”.

  • #rejected-rfcs allows Everyone to “See” only. It requires TL4 to “Create/Reply/See”.

  • #postponed-rfcs allows Everyone to “See” only. It requires TL4 to “Create/Reply/See”.

Note that these categories should NOT have Discourse Solved enabled on them. I think I’d have to turn Solved off on the Development category, but that’s fine. We really weren’t using that anyway.

RFC’s should rarely be edited after they have been accepted or rejected. They should be superseded instead. A TL4 user might edit an RFC to add a note when it has been superseded by a future one, to help anyone who browses the documents later.

Future work

If the RFC process starts to have trouble being tracked in one thread, it should be possible to split it into more than one thread. To do so:

  • Allow anyone to create topics in the main #draft-rfcs category, but only allow TL4 to make direct replies to those “tracking threads.”

  • Move the main discussion into a separate category, and have sub-threads link to the main topic (so that Discourse backlinking allows the sub-threads to be seen listed below the main thread).

The main problem here is ergonomics; Discourse collapses the list of links when shown below the topic, which I’m worried would make it harder to find sub-threads than it ought to be. I’m also concerned about redundancy.

The main reason I don’t worry about it yet, though, is that it’s obviously overkill.


#6

Appendix: RFC template

When copied to GitHub, the Discourse-level metadata (tags, especially) should be converted to Jekyll-style YAML top matter.


Summary: a one-paragraph description of what the RFC is supposed to be.

I allow this RFC document to be modified and redistributed under the terms of the Apache-2.0 license.

Motivation

The motivation is critical for feature RFCs that want to add more surface area to maintain, or that want to make breaking changes to it. It should clearly explain why the existing feature set is inadequate to address the problem that the RFC solves. RFC submissions without sufficient motivation may be rejected outright.

Guide-level explanation

Describe the feature you want to add as if you were documenting an already implemented version of it. What does it do? Why might a user want to exercise it. How do they access it? You should especially focus on examples and use cases. A common tactic is to show an example comment exchange with bors. For example,

If bors.toml contains a like gizmo = true, then the user can trigger it with the bors gizmo command:

You: bors gizmo

bors: Adding gizmo commit…

* bors committed xxxxx [Gizmo commit]

If this RFC is going to deprecate, or break, an existing feature, announce it here.

Reference-level explanation

While the guide-level documentation should go in “Getting Started” or a blog post, this section should be written with the deep-dive user in mind. It should describe:

  • Exact schemas and syntaxes for any feature-related entry points. Write things like bors gizmo [-k] <commit-hash> to describe a command syntax, not just examples.

  • How the feature interacts with other features.

  • Corner cases and gotchas are dissected by example. If this RFC introduces a breaking change that you think is minor enough to not require an announcement, you can mention this here, along with a justification for why you think we can deploy it silently.

Drawbacks

Why should we not do this?

Rationale and alternatives

  • Why is this design the best in the space of possible designs?
  • What other designs have been considered and what is the rationale for not choosing them?
  • What is the impact of not doing this?

Prior art

  • For feature RFC’s: What other CI/CD systems implement anything similar to this one? What do they do well? What do they do poorly? If this proposal is similar to what one of them already does, did you change anything, and why or why not? Why would the user pick bors over just using that other thing instead?
  • Also for feature RFC’s: How does this feature you’re proposing complement the software development lifecycle of real teams? Compare what you’re implementing to things that they’re doing by hand, or using other tools. Please make sure that this feature is likely to be used by more than one project.
  • For process RFC’s: What do other major open-source projects do? Focus on ones that make their environment welcome to marginalized groups. If they have a reputation for being mean, we don’t want to copy them.
  • What lessons can we learn from what other communities have done here? Why do they do what they do? Why is it applicable to bors, or why isn’t it?
  • Papers: Are there any published papers or great posts that discuss this? If you have some relevant papers to refer to, this can serve as a more detailed theoretical background.

Unresolved questions

  • What parts of the design do you expect to resolve through the RFC process before this gets accepted?
  • What parts of the design do you expect to resolve through the implementation of this feature before deployment?
  • What related issues do you consider out of scope for this RFC that could be addressed in the future independently of the solution that comes out of this RFC?

Future possibilities

Think about what the natural extension and evolution of your proposal would be and how it would affect the product and project as a whole in a holistic way. Try to use this section as a tool to more fully consider all possible interactions with the existing feature set in your proposal. Also consider how the this all fits into the roadmap.

This is also a good place to “dump ideas”, if they are out of scope for the RFC you are writing but otherwise related.

If you have tried and cannot think of any future possibilities, you may simply state that you cannot think of anything.

Note that having something written down in the future-possibilities section is not a reason to accept the current or a future RFC; such notes should be in the section on motivation or rationale in this or subsequent RFCs. The section merely provides additional information.


closed #7