Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] What: Cloud Native Guide Posts Issue #961 #997

Closed
wants to merge 10 commits into from

Conversation

TheFoxAtWork
Copy link
Contributor

Why:

  • Capture milestones from successful projects
  • provide guidance and recommendations to projects seeking graduation
  • Provide criteria to guide TOC on determining when sandbox projects should depart from sandbox

This change addresses the need by:

  • Modifying archiving.md to reflect Sandbox Departure link
  • Modify sandbox.md to set Sandbox Departure criteria
  • Modify README.md to capture Milestones link
  • create project milestones for community review, improvement, and feedback

Signed-off-by: Emily Fox [email protected]

Why:

* Capture milestones from successful projects
* provide guidance and recommendations to projects seeking graduation
* Provide criteria to guide TOC on determining when sandbox projects should depart from sandbox

This change addresses the need by:

* Modifying archiving.md to reflect Sandbox Departure link
* Modify sandbox.md to set Sandbox Departure criteria
* Modify README.md to capture Milestones link
* create project milestones for community review, improvement, and feedback

Signed-off-by: Emily Fox <[email protected]>
process/archiving.md Outdated Show resolved Hide resolved
process/sandbox.md Outdated Show resolved Hide resolved
* A roadmap clearly defines initiatives that align with releases
* Releases begin to have establish a pattern of regularity
* Project has engaged a Technical Advisory Group for feedback on some portion of the project relevant to the domain
* Project has a clearly discoverable governance doc that covers the basics of decision making to include the commit bit
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* Project has a clearly discoverable governance doc that covers the basics of decision making to include the commit bit
* Project has a clearly discoverable governance doc that covers the basics of decision making and how to earn the commit bit

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"the commit bit" is probably not a well understood term. I don't know if the CNCF has a defined term ("maintainer" is definitely overloaded), but I think "how to earn permission to approve pull requests" or something similar would help.

(I assume the SCM for CNCF projects does not have to include GitHub and a more inclusive terms than Pull Requests might be appropriate)

Copy link
Contributor

@lizrice lizrice Feb 13, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've been thinking about Roadmaps. In the past I've been in favour of them as a requirement but I think I'm changing my mind because I'm not convinced they are as valuable as we'd like to think.

Background / food for thought: the Linux kernel does not have a published roadmap. Everyone involved knows what's going through mailing list discussions.

When we say things like a roadmap that's aligned with releases, combined with a regular cadence of releases, this feels dangerously close to requiring open source projects to make commitments that end users might come to think they can rely on.

And it doesn't seem very aligned with the ethos of open source to require developers to sign up to fixed commitments. Sure, in our community most people are being paid to work on projects but not everyone is. And if someone from Company A makes a commitment to deliver a particular feature in a certain release, but then Company A decides they should spend their time elsewhere, who from the project is supposed to pick up the slack to make sure the commitment is reached?

It might work for very large projects, but the more I think about it, the more it seems like a recipe for burnout amongst contributors, and disappointment amongst end users.

Projects generally use issues to communicate what features and fixes people are working on. A feature that meets some set of criteria (up to the project) makes it into the next release. Isn't that sufficient?

And if we require more, isn't that rather overstepping the idea that we don't impose any particular governance on projects?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+100 - we really need to stop turning CNCF into a giant bureaucracy that treats all projects the same. This kills innovation stone dead and will put so many people off. Also, please stop imposing requirements on project maintainers that add a lot of work and will lead to burnout, without a very strong compensating reason and balance

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lizrice Thanks for bringing this up, I dont disagree. We currently have roadmaps integrated through various portions of the lifecycle so we'll need to evaluate what the value is of these in both understanding the project's longer goals as well as the direction they are heading, however they are always point in time for evaluations along the lifecycle.

Your point on perceived commitment is spot on as well, and I'm not confident "messaging" here will resolve this disconnect. We'll need to consider more about how we can view the direction of the project at an evaluation, without dictating how the project expresses that.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@monadic Thank you for the feedback. This PR on milestones does not establish new or additional requirements on projects, it is designed to serve as guiding points along their [project's] journey — from sandbox to beyond graduation which they may leverage. However, as @craigbox mentioned later, these may be seen as requirements. This is something will need to rework in this PR to avoid.
Regarding burnout, agreed. The CNCF lifecycle is structured to evaluate projects for stability, maturity, and adoption potential (to varying degrees) over the years, as we've learned more about project success, we're beginning to distill steps projects have taken along their path to success. Not as requirements, more as a catalog of what has and hasnt worked and more context around why.

Would providing more reasoning/clarity/what-it-does assist with the milestones? Such that as a project maintainer looking to move to incubation from sandbox (for instance) has ample information about how these may be beneficial to decide if documenting a roadmap is worthwhile for their project, or perhaps including a vision/direction, or neither, works for their needs?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've suggested different language for this point.

@lizrice
Copy link
Contributor

lizrice commented Jan 27, 2023

Preparing for graduation, we (Cilium) have been getting recommendations and requirements to do a lot of security-related audits and assessments:

They’re all good things on their own, but taken together they’re repetitive and overlapping, and are causing quite a lot of frustration ("we're drowning in checklists and paperwork".) It would be really helpful to have a clear steer on which of these are actually required.

@TheFoxAtWork
Copy link
Contributor Author

There's a lot here we can get better clarity on.

The TAG Security Review/Assessment process was heavily reworked a few years ago to structure it such that projects may complete portions of the overall "Security Package" aligned with where they are at in the CNCF "lifecycle". This revision was designed to prevent the security review (which often doesn't have enough reviewers and as a result does take longer) from becoming a bottleneck for a project pursuing graduation. If projects completed the "self-assessment" during Sandbox, then optional "joint-review" during Incubation (well before considering graduation) these two security documents form a security package that can be leveraged by Security Auditors to assist in the formal audit. The joint-review builds on top of the self-assessment, but both documents do NOT replace a Security Audit, while joint-reviews do have some areas of overlap with Security Audits, the Security Audits go more in depth in other areas. They should be viewed as cumulatively complementary to one another, however a joint-review is NOT a requirement for graduation (nor is the self-assessment for incubation).

I am not comfortable placing TAG Security and the joint-reviews in a position where all projects come to them b/c they are now the executors of a graduation requirement. It's not feasible with the amount of projects in the landscape and the amount of security reviewers we have.

There as a been a lot of evolution in open source and supply chain security these past few years, with increasing maturity of various assessments that look at different portions of a project - from its architecture to its repository configuration and release processes. We've not yet revisited how these changes should be weighed or considered in the CNCF lifecycle and I believe this is worth pursuing by community through the Security TAG.

@tomkerkhove
Copy link
Contributor

Preparing for graduation, we (Cilium) have been getting recommendations and requirements to do a lot of security-related audits and assessments:

They’re all good things on their own, but taken together they’re repetitive and overlapping, and are causing quite a lot of frustration ("we're drowning in checklists and paperwork".) It would be really helpful to have a clear steer on which of these are actually required.

Agreed

@tomkerkhove
Copy link
Contributor

Some practical things that I think would be nice to include are:

  • The fact that new proposals should end up in CNCF-based Google workspace (see Provide CNCF-hosted Google workspace for DD docs #980)
  • If we should use issues vs pull requests to open proposals. If we stick with pull requests - What should be in them? Because typically people include a whole proposal in the PR which then gets ignored/must be ported over to the DD doc which costs a lot of time
    • Been through this twice to write a full proposal (which takes a lot of time) only to see it get ignored and need to create a Google doc afterwards

@TheFoxAtWork
Copy link
Contributor Author

I had an excellent discussion with a community member yesterday about the amount of security asks of projects and the current throughput of Security TAG in reviews, specifically regarding the joint-reviews.
It was recommended that we extract the threat modeling process (when its ready - this is still in development) from the joint review and focus on this for projects immediately after moving to incubation. The threat modeling portion is designed to take up 2-3 hours of a projects time and is likely useful for projects immediately following incubation.

What do others think about this as a milestone?

@dims
Copy link
Member

dims commented Feb 1, 2023

+1 @TheFoxAtWork

process/README.md Outdated Show resolved Hide resolved
@craigbox
Copy link
Contributor

I would caution that even though it might not be the stated intention, these may well end up interpreted as "unwritten criteria" for moving between maturity levels, especially in light of #992.

@TheFoxAtWork
Copy link
Contributor Author

TheFoxAtWork commented Feb 13, 2023

The fact that new proposals should end up in CNCF-based Google workspace (see #980)

@tomkerkhove this would be best in the graduation criteria/process documentation as this was a recent change. @amye is this something you can ensure is integrated?

Been through this twice to write a full proposal (which takes a lot of time) only to see it get ignored and need to create a Google doc afterwards

Completely agree this is time consuming and inconsistent. @amye whatever updates we make for this should reduce duplicative effort. We the template for graduation with a lot of information in it, but perhaps its worth figuring out where this goes in a DD or a DD refresh versus initiating a process.

Why:

* Many community members have added or clarified milestones and intent.

This change addresses the need by:

* Inclusion of @xmulligan 's suggestions
* tweaks to wording based on @monadic 's feedback
* adjustments per @lizrice 's comments
* Adjustments per @tomkerkhove 's comments
* adding milestones per @raravena80 's feedback
* include @craigbox 's suggestions

Signed-off-by: Emily Fox <[email protected]>
Why:

* Roadmap isnt a commitment by maintainers to do work, its subject to change

This change addresses the need by:

* providing explanation/examples on 'long term planning'

Signed-off-by: Emily Fox <[email protected]>
@TheFoxAtWork
Copy link
Contributor Author

Per discussion on today's TOC/TAG Call here is a google doc of the primary changes for better discussion/facilitation of this PR: https://docs.google.com/document/d/1aL0mxInyz-qtRT2QM7kdr9riM36uihILtgM4TR35cPQ/edit?usp=sharing

process/project_milestones.md Outdated Show resolved Hide resolved
process/project_milestones.md Outdated Show resolved Hide resolved
process/project_milestones.md Outdated Show resolved Hide resolved
process/project_milestones.md Outdated Show resolved Hide resolved
process/project_milestones.md Outdated Show resolved Hide resolved
process/sandbox.md Outdated Show resolved Hide resolved
Why:

* received more community feedback

This change addresses the need by:

* resolves @leecalcote & @jberkus comments

Signed-off-by: Emily Fox <[email protected]>
process/project_milestones.md Outdated Show resolved Hide resolved
process/project_milestones.md Outdated Show resolved Hide resolved
Why:

* additional milestones for incubation suggested

This change addresses the need by:

* including suggested milestones from @raravena80 (with adjustment)

Signed-off-by: Emily Fox <[email protected]>
Why:

* removed per feedback and now in separate PR

This change addresses the need by:

* removed second sentence from line 38 in sandbox.md

Signed-off-by: Emily Fox <[email protected]>
@@ -57,6 +55,9 @@ Incubation milestones are guiding points to highlight success areas a maturing p
* Project has engaged with a few Technical Advisory Groups to ensure they are considering best practices across multiple technical and governance facets
* Project has defined a process for managing the conduct of its community, either establishing its own group to do this, leveraging the existing leadership, or deferring to the CNCF
* Project maintains a dependency graph so adopters fully understand the complexity and risk of use
* Project has 3 or more adopters using the project in production to support completion of Due Diligence and adopter interviews
* Project plans activities to increase project adoption trajectory 3 adopters
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* Project plans activities to increase project adoption trajectory 3 adopters
* Project plans activities to increase project adoption

@@ -57,6 +55,9 @@ Incubation milestones are guiding points to highlight success areas a maturing p
* Project has engaged with a few Technical Advisory Groups to ensure they are considering best practices across multiple technical and governance facets
* Project has defined a process for managing the conduct of its community, either establishing its own group to do this, leveraging the existing leadership, or deferring to the CNCF
* Project maintains a dependency graph so adopters fully understand the complexity and risk of use
* Project has 3 or more adopters using the project in production to support completion of Due Diligence and adopter interviews
* Project plans activities to increase project adoption trajectory 3 adopters
* Project has more than 5 regular contributors, more than half of which are from different organizations
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one worries me A LOT. It clearly incentivizes projects and companies to do the wrong thing, in order to meet these "milestones".

For example, fearing that they are being seen to have "too many" contributions going into the open source project, the major backer of a project decides to hold their engineering efforts back into closed-source add-ons. Or use outsourcing companies and partners rather than hire people to work full-time on the project. Or decide not to contribute a project to the CNCF because it's going to be such an uphill struggle for the project to see itself as successful according to these criteria.

None of these outcomes are good for anyone in our community.

(Also, why arbitrarily 5? And what does "regular" mean? We could spend time debating these points - but that is time that could be better spent on actually progressing / supporting projects.)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense, I suggested a slightly different variation of this here: https://docs.google.com/document/d/1aL0mxInyz-qtRT2QM7kdr9riM36uihILtgM4TR35cPQ. I think it's hard to determine a fixed number that can work for all the variety of different projects. IMO a more 'open' or 'flexible' language would work better.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are good counterpoints @lizrice . We also don't want to have a single entity (company or organization or individual) be the majority decision maker of a projects technical direction, if that entity is acquired or disappears, the project can be left out to dry, and in some case unable to recover because the primary pool of contribution has evaporated.

@raravena80 @lizrice - How do we strike a balance in these milestones — as guiding points projects could leverage to in order to become more resilient, mature, and widely adopted versions — that could help projects avoid the scenarios you present as well as a increase the likelihood of project's survivability with primary entity's complete absence? What actionable goals/ideals/structure/governance could be provided for projects to increase their resilience and survival likelihood in the event a core contributing entity ceases to be or continue?

Ricardo mentioned open and flexible language here, would the following adjustments meet the intent?

Suggested change
* Project has more than 5 regular contributors, more than half of which are from different organizations
* Project has multiple contributors who regularly participate in the growth and development of the project in accordance with the projects synchronous and asynchronous cadence
* Project contribution governance defines timeliness and methods for recovering roles and permissions of contributors that drop from regular participation as _defined by the project_ (roles such as issue triage, community management, PR reviewer, maintainer, release lead, etc. and regularity such as participation in two of three releases, facilitation of 4 community meetups annually, etc.)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't say I'm keen on removing specifics from this. Folks in the community manager role for a project like having clear targets. Once a project gets to applying Graduated, if they don't have at least five regular contributors, the application isn't likely to go well.

Note that these are milestones, not updates to the specific requirements.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 I believe all these can be covered in the March 21st TOC meeting.

@monadic
Copy link
Contributor

monadic commented Mar 17, 2023

Hi, I say this with care and respect for the intentions of the people on this thread. If the CNCF wants to continue to be a home for innovative and leading projects, then it needs to listen to people who work on them and make them successful. In my view, the kind of rules being proposed here will greatly narrow what sorts of projects can thrive in CNCF and make it ultimately unappealing. Please can we arrange a TOC call with appropriate representation to discuss.

-alexis

@TheFoxAtWork
Copy link
Contributor Author

@monadic thank you for engaging, we can certainly schedule this for a discussion on a TOC call! It is apparent that the guiding points identified here are still being viewed as rules, requirements, or other forms of constraint which is not what they should be used for.

I would love to hear more about your experience to see which, if any, of these describe the path for moving levels of the projects you've been involved in. Scaling that experience and making it approachable for other projects — if they would like to leverage it — would be beneficial for new projects and maybe even existing projects seeking to grow and move levels. When we have a date, I'll update here. Thank you again!

@monadic
Copy link
Contributor

monadic commented Mar 17, 2023

thanks @TheFoxAtWork

this is something we spent a lot of time thinking about when creating CNCF - what makes a successful OSS project? the answer is that they come from different places but a consistent theme is small companies innovating ** and sustaining ** them. this has numerous consequences. I love multipolar projects too.

@TheFoxAtWork
Copy link
Contributor Author

Looks like this actually already on the Agenda for the meeting next week (march 21st) @monadic (apologies for missing it earlier)

@monadic
Copy link
Contributor

monadic commented Mar 17, 2023

8am PT? please can you advertise on toc & alums email lists

@craigbox
Copy link
Contributor

It is apparent that the guiding points identified here are still being viewed as rules, requirements, or other forms of constraint which is not what they should be used for.

The points under discussion today are being discussed as a criteria for graduation in #992. I've commented at length there.

(I would love to be involved in an in-person discussion, but not enough to get up at 4am I'm afraid.)

@lizrice
Copy link
Contributor

lizrice commented Mar 18, 2023

I have some concrete ideas for this milestones document, to de-couple the advice on growing a project from the TOC’s assessment of its maturity.

  • Make this document about guidance to projects on how to grow. Some indication of what success looks like could be helpful, but be careful to make it applicable to all projects (so, no arbitrary numbers.)
  • Remove all mention of the sandbox / incubation / graduation levels so that people aren’t tempted to draw their own conclusions about whether they should be considered by the TOC when moving levels.
  • Remove the points in this document that appear as guidance to the TOC on how to assess projects. Anything the TOC requires from all projects at a particular maturity level should be documented in the criteria, and anything else should be a matter of judgement.

I’m aiming to be there for the discussion coming up in this week’s TOC meeting - looking forward to it.

Folks in the community manager role for a project like having clear targets.

As someone with experience on both the TOC and in community management for Cilium, I can tell you that no-one wants yet more checklists where it’s unclear how important they are - our experience with the myriad security assessments advised to us during the course of our graduation application is an example. If something looks like it might be considered, projects feel obliged to document them in their DD just in case it affects their application. And (sadly) people with some axe to grind will bring up the lack of some specific “milestone” during the public comment period. Neither of these outcomes are remotely helpful to the project or the user community - they just create additional work for the projects and for the TOC.

Once a project gets to applying Graduated, if they don't have at least five regular contributors, the application isn't likely to go well.

Most Sandbox projects have at least five regular contributors, so this number isn’t helpful as a target for graduation. What’s the right number? For massive (Kubernetes-esque) projects it’s probably one or two orders higher than this; for a small but critical security library, five might be totally appropriate.

There is no “one size fits all” set of measurements that is appropriate for the needs of all projects. That’s why levelling assessments can’t be boiled down to a set of checkboxes, and it’s why previous incarnations of the TOC have deliberately resisted setting arbitrary thresholds, in favour of exercising judgement on a per-project basis.

@JustinCappos
Copy link
Contributor

I'll also mention a few more boundary cases so they are on everyone's radar. Specifically, that it's hard to assess projects which are more specification focused given guidelines which are more often to assess an implementation.

For example, when considering a specification project, should you count maintainers / approvers for the specification as those controlling entities? Should this also include the implementation maintainers?

A healthy rate of churn for a successful and active specification may only involve a few changes a year.

What if there are different compliant implementations from different groups that individually do not have a high degree of maintainer diversity? Would this be considered reasonable diversity, given one can always pick which group's implementation to use?

Why:

* Community provided tangible improvements that have high usability for projects

This change addresses the need by:

* Changing milestones to guide posts
* removing references to moving levels
* removing indicators of TOC consideration

Signed-off-by: Emily Fox <[email protected]>
@TheFoxAtWork TheFoxAtWork changed the title What: Cloud Native Milestones Issue #961 What: Cloud Native Guide Posts Issue #961 May 22, 2023
@TheFoxAtWork
Copy link
Contributor Author

Checking in on this PR to determine if any further changes suggested have not yet been addressed. Please take a few moments to review and determine if I've incorporated your changes. Thank you all in advance

Copy link
Member

@nikhita nikhita left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR has been open for a while. LGTM, let's :shipit:

Having guide posts will help with evaluating sandbox annual reviews and providing a structured way to help sandbox projects figure out a growth path.

@TheFoxAtWork TheFoxAtWork changed the title What: Cloud Native Guide Posts Issue #961 WIP What: Cloud Native Guide Posts Issue #961 Oct 10, 2023
@TheFoxAtWork TheFoxAtWork self-assigned this Apr 16, 2024
@TheFoxAtWork TheFoxAtWork changed the title WIP What: Cloud Native Guide Posts Issue #961 [WIP] What: Cloud Native Guide Posts Issue #961 Apr 16, 2024
@TheFoxAtWork
Copy link
Contributor Author

PR needs refreshed to align with recent changes in the TOC repo. Will update later.

@riaankleinhans riaankleinhans added the process-documentation Doc changes for process and procedures label Apr 29, 2024
TheFoxAtWork added a commit to TheFoxAtWork/toc that referenced this pull request May 10, 2024
Why:
* Capture milestones from successful projects
provide guidance and recommendations to projects seeking to move levels
* previous PR (cncf#997) was very out of date and had unresolvable conflicts due to repo changes

Signed-off-by: Emily Fox <[email protected]>
@TheFoxAtWork TheFoxAtWork deleted the 961-cn-milestones branch May 10, 2024 17:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement process-documentation Doc changes for process and procedures
Projects
None yet
Development

Successfully merging this pull request may close these issues.