Plugin vs Tool Versioning Strategy

[mickmcgrath13]

How should we handle the plugin repo versions compared to the plugin's tool version.
Terraform for example might have 0.12.*

Options

• the plugin version matches the tool version
• the plugin version is independent of the tool version (allowing for multiple TF version in a single plugin version -might not work with other tools like ansible)
• include both: 1.0.0-tf0.12.0

[mickmcgrath13]

Only implication for not deciding this now is if there's a conflict with our version convention for plugins and a released tool version.

[arm4b]

Per #307 the official BitOps images will be packaging and providing the latest tools by default.

However, user should be able to pin the tool version (ansible==1.2.3, terraform=3.4.5). This might be achievable when building their custom BitOps Docker image.
https://bitovi.github.io/bitops/plugins/#creating-your-own-bitops-image

The version parameter could be provided via config like:

bitops:
  plugins:
    terraform:
      source: https://github.com/bitops-plugins/terraform
      version: 1.2.3 # <-- new setting

and so 1.2.3 will be passed to the install.sh to install the needed tool version:
https://github.com/bitops-plugins/terraform/blob/b189c2dd7cb38e7e883837f0f3204e5b7e8a6e6b/install.sh#L13-L14

Here tool version = plugin version, where version is just a parameter and not a repo git tag.
That would fit customized use cases and environments which is the main purpose of building the custom bitops image with the chosen plugins.

[arm4b]

WDYT?

@mickmcgrath13 I think users are caring about the tool version at the end of the day.
What could be other cases when we need a plugin version (aka repository version) separated from the tool version?

I'm skeptical about packaging multiple versions of the tool in one BitOps image at least at this moment, as it doesn't scale, increase the image size, is inconsistent, and might need more complex workarounds, especially with the increased amount of plugins. It may lead to a situation when every other user requests a new specific version to be bundled in the omnibus image.

[mickmcgrath13]

I see 3 core options for this:

1. versioned plugins with pinned tool versions (i.e. 1.0.0-tf1.2.3)
2. versioned plugins with sets of available pinned tools (i.e. 1.0.0 has tf 1.2.0, 1.2.2, 1.2.3)
3. no versioned plugins, and users create their own bitops image to pin their TF version with a version flag

Each has pros and cons. Here are some considerations:

• Who has to manage the overhead?
  1. Bitovi (BitOps support)
  2. Users
• What do we do with the Omnibus Image?

I'll attempt to outline each of our core options with regard to the considerations and outline the pros and cons:

---

1. versioned plugins with pinned tool versions (i.e. 1.0.0-tf1.2.3)

Who manages the overhead

The effort here would primarily be on the BitOps support team, though there would be some overhead to the user to find the appropriate omnibus image.

What do we do with the omnibus image

We would have new versions of the omnibus image for each permutation of plugin version and tool version (i.e. number of tools, number of supported version for each tool, and number of versions for each plugin)

Pros

• users have the lowest amount of overhead in this scenario. They can reference the omnibus list to find their desired set of tool versions and use that one.
• having plugin versions feels nice, conceptually

Cons

• This could result in a massive amount of images (i.e. doesn't scale well)
• We'd need to figure out how to version the omnibus images (i.e. something like 2.0.0-1.2.3 wherein 2.0.0 represents the bitops base and 1.2.3 represents sets of plugins each with a set of tools). This could be confusing to users.

Alternatives

1.1 User-requested plugin versions

We could suggest that users request new versions of tools and only create new plugin+tool versions upon request. This would lower the overhead

Additional effort for this option

• We'd want to make the process for creating new omnibus images as easy as possible. Perhaps an automation where when we create a new git tag/release, it will trigger a new omnibus image to be created automatically (with release notes for which tool versions are installed into it)
• We'd want an omnibus list somewhere that outlines all of the permutations of omnibus images

Thoughts

I think I agree that this option is not good at scale given the overhead.

2. versioned plugins with sets of available pinned tools (i.e. 1.0.0 has tf 1.2.0, 1.2.2, 1.2.3)

Who manages the overhead

BitOps support would still manage the overhead, though it'd be slightly less than the previous option because we could pre-empt user requests to provide a set of images.

What do we do with the omnibus image

We'd have versioned sets of omnibus images for each set of plugin tool versions.

Pros

• We could respect the ${tool}.options.version flag in bitops.config.yaml in ops repos given that the requested version is in the set of installed versions.
• Lowest user overhead for getting started

Cons

• We'd need to figure out how to version the omnibus images (i.e. something like 2.0.0-1.2.3 wherein 2.0.0 represents the bitops base and 1.2.3 represents sets of plugins each with a set of tools). This could be confusing to users.
• This approach might not work with some tools (can you install multiple versions of Ansible, for example?)
• The omnibus image(s) would be somewhat bloated (though this is mitigated by the fact that the goal for the omnibus image is to allow users to start using BitOps quickly, and the recommendation would be to go with their own custom bitops image, anyway)

Additional effort for this option

• We'd want to make the process for creating new omnibus images as easy as possible. Perhaps an automation where when we create a new git tag/release, it will trigger a new omnibus image to be created automatically (with release notes for which tool versions are installed into it)
• We'd want an omnibus list somewhere that outlines all of the permutations of omnibus images

Thoughts

I think this is worth considering because it is the lowest overhead for new users. There's probably a set of users that are stuck with older versions of tools but can't create their own custom bitops image (or don't have time/resources to). To be fair, I don't know how big the set of people with that condition would be. It might be non-zero, but it's probably low enough to say if someone isn't able to use one of the omnibus images, we could step in and help them create their own bitops.

3. no versioned plugins, and users create their own bitops image to pin their TF version with a version flag

(this is what was proposed in the comment above)

Who manages the overhead

Primarily the user because they will need to create their own image if they are not using latest (or look up a previous version of the omnibus image which has the risk of missing other updates to the bitops base code).
This is the least amount of overhead for BitOps maintainers.

What do we do with the omnibus image

Nothing beyond providing a list of omnibus images and which tools versions are installed (maybe release notes and docs pointing to it)

Pros

• lowest overhead for bitops maintainers
• very scalable (i.e. no build-up of image permutations)

Cons

• writers of the plugins need to support the bitops.plugins.${tool}.version option
• there is a slight risk of someone needing a previous version (than latest) of a given tool and missing new base functionality features if an upgrade to the base coincides with a new tool version being released.

Alternatives

3.1 on-demand plugin versions

I do think there's value in creating a mechanism which allows installation of a specific tool version at deploy time (i.e. in the ops repo's bitops.config.yaml for a given tool, support the ${tool}.options.version flag). That way, if users want to specify a version of a tool, it'll work (though it'll take longer). Then, when they are comfortable with things, they can create their own bitops image with pinned versions. This was originally suggested here (for the TF plugin): #307 (comment). We could probably even leverage the same install script for each plugin to accomplish this (given they respect the bitops.plugins.${tool}.version option).

Additional effort for this option

• we need documentation for plugin creation which should include the need to support the bitops.plugins.${tool}.version option
• We need to document which tool versions are installed in each omnibus image (release notes with docs pointing to them here somewhere?)

Thoughts

This seems like a good first approach. It's relatively simple to implement and maintain, and it doesn't force us into a corner if we need to adjust later.

Conclusion

I think we should go with option 3 as well as option 3.1.
(i.e. what was proposed in the comment above)

Additional Effort in any case (i.e. new tickets)

The following are in addition to the "Additional effort for this option" for the selected option

• We should have a place where we maintain our versioned list of omnibus images and which tool versions are included in each
• We should provide a set of guides on how to create bitops images for various use cases (github actions -> docker hub, github actions -> github artifacts, gitlab -> docker hub, gitlab/github -> ECR, etc)

[arm4b]

That's a very detailed breakdown @mickmcgrath13, nice stuff 👍

That way, if users want to specify a version of a tool, it'll work (though it'll take longer). Then, when they are comfortable with things, they can create their own bitops image with pinned versions.

For 3.1, installing tools during the runtime, I like the way of thinking, eg. giving users a try with the other tool version first before pre-packaging it. That's a good perspective, especially for beginners.

The pros:

• ease of use
• user-friendly
• no need to re-package everything for an isolated version try

The cons:

• new tool install can fail anytime during the runtime, making the entire pipeline more error-prone
  • download/networking issues
  • compatibility/install version-specific errors
  • can lead to frustration (blinking OK/FAIL deployments) when used in a bigger production (expands the blast radius of failure)
    • been there, when the entire pipeline is a big process with many moving pieces. With hundreds of external downloads involved, even a 1% error rate will fail every build
    • add it to environments with hundreds of builds and tens of deployments per day
    • we can WARN via logging that it's not for prod, recommending users to pre-build their own image
• comes against the docker immutability approach
  • not the best practice
  • is slow
  • not reproducible builds
• won't work in isolated private net environments
  • when the original omnibus docker image is pre-fetched/cached, but no other egress internet traffic is allowed
• bad security
  • having a dynamic ability for the image to switch the exec binary on fly might be a no-go for some demanding users/environments in terms of security
• can lead to some negativity
  • from more experienced engineers RE best practices
  • from the users abusing it and having issues

From the implementation side, this means doing the same plugin install during the deploy stage. Additionally, for all the plugins we'll need adding new logic that will handle installing the second tool version somewhere sideways (might be harder for non-binary installs like ansible, etc).

I think overall 3 vs 3.1 are consistent with each other in a way that we allow setting the version in bitops config. The concern about 3.1 that it could be dangerous long term. While the usability benefit is clear, the risk is allowing users to hit themselves in their feet with no good practices might boomerang to us back. If implementing 3.1, we should add a logging warning that it's not recommended for production to download the version tool on a runtime/deploy time, encouraging and reminding users to pre-package their custom docker image.

[arm4b]

@mickmcgrath13 Based on conversations, the 2 feature requests were created for 3 and 3.1:

• Set the plugin version on custom image build #315
• Dynamically install plugin version on runtime (deploy) #316

[mickmcgrath13]

Thanks @armab ! I commented on the dynamic install ticket #316 .
I think many of the cons can be mitigated quite well with 1) a flag so users need to opt-in to the auto-download functionality, 2) a warning message at deploy time when the flag is set, and 3) a warning message on the flag stating that it should not be used in production.

Additionally, I agree with what you said that our warnings should clearly state that this is for testing/development purposes only and should NOT be used for a production environment and encourage users to pre-package their custom image.

[mickmcgrath13]

with #315 and #316, I think we can lock this conversation. Thoughts?

[arm4b]

Just added a tag STATUS:DECIDED for this one so the plan is clear for others 👀.
https://github.com/bitovi/bitops/discussions?discussions_q=label%3Astatus%3ADECIDED

Let's still leave a room for the future feedback by keeping the issue unlocked.

[mickmcgrath13]

@armab can we also get tickets for the following?

Additional Effort in any case (i.e. new tickets)

The following are in addition to the "Additional effort for this option" for the selected option

• We should have a place where we maintain our versioned list of omnibus images and which tool versions are included in each
• We should provide a set of guides on how to create bitops images for various use cases (github actions -> docker hub, github actions -> github artifacts, gitlab -> docker hub, gitlab/github -> ECR, etc)

[arm4b]

Good call. Added:

• Document Latest Tool Versions packaged in the Omnibus image #317, TBD in this release