Release 4.x plan

[markpeek]

To better keep up with changes coming from the AWS CloudFormation team, there was a need to support auto-generation. The main branch has a set of changes which:

1. The core files are auto-generated
2. Validators have been refactored into separate files
3. Change to support (mainly) backward compatibility

Given early work on troposphere was all hand crafter, there have been some changes that are not backward compatible. Also, there are changes that bring things up-to-date with the current resource specification.

Here's the plan:

• Finish converting existing resources to be auto-generated (I believe ec2.py is the last one)
• Release version 4.0.0b1 (major version bump due to the number of changes made and beta to get feedback)
• Address feedback from the community
• Wait some period of time before releasing 4.0.0

Hoping to release 4.0.0b1 in a matter of days but wanted to get input.

Thoughts?

[davidjdixon]

I'm generally fully in favour of this approach. I would like to see more comprehensive detail on the actual changes that would be visible to developers using the library to ensure we can start thinking / working on any changes needed to make this solution compatible.

[markpeek]

@davidjdixon In the future we may want to remove some of the backward compatibility changes I put into this new release. This is mainly ensuring the legacy troposphere object naming was kept rather than the AWS naming and accepting the same types). There was separating some resources into their own files (OpsWorksCM and Route53Resolver) along with other minor changes. I anticipate only minor and straightforward changes will be needed for existing scripts.

[markpeek]

BTW, here's part of the reason to make the code auto-generated. For this week the release history shows a single change to WAFv2 for 2022-02-17. However, if you look at the this commit (bfa31c4) based on the resource spec file 57.0.0 (changes from the last week), you will see 5 other changes that were not listed in the release history.

[JohnPreston]

@markpeek I have a watch in librairies.io on this (as a dependency to ecs_composex and other projects).
When you do push 4.0.0b1/rc1 I should see it published (and in case, if you give this thread a nudge should see it) to test it out. Will be able to test Custom resources too.

Hey @davidjdixon - long time no see since TVX :) Hope you're well

[markpeek]

Release 4.0.0-beta.0 is now available

[JohnPreston]

This one change is particularly painful 🤔 (considering they are identical)
AWS::ElasticLoadBalancingV2::ListenerRule Action was renamed ListenerRuleAction due to conflict with Listener Action AuthenticateOidcConfig

[markpeek]

Agreed, it was due to the AuthenticateOidcConfig properties being different that had a ripple effect for one of the Action properties to change to handle the rename.

[markpeek]

@JohnPreston besides your above comment about ElasticLoadBalancingV2, how easy/hard was the 3.x to 4.x conversion?

[JohnPreston]

@JohnPreston besides your above comment about ElasticLoadBalancingV2, how easy/hard was the 3.x to 4.x conversion?

So far, everything else is working without change.
Of course I haven't tested every resource but I haven't had to change anything else

PS @markpeek I only meant "annnoying" as very sneaky considering the identical side of the properties.
Having everything else work as intended without change in the source code is quite amazing. Outstanding!

[JohnPreston]

@markpeek is the plan then to consistently have the classes of resources and properties generated and the validators separately ? what will the guideline be on implementing validate functions for the resources and properties ?

[markpeek]

@JohnPreston glad the updated worked well for you. It took longer to maintain backward compatibility but sets things up well for adoption and future improvements.

Yes, the resources/properties will be autogenerated into troposphere/*.py and the validators have implementations in troposphere/validators/*.py. Previously I had added some comments to the CONTRIBUTING doc but need to update it to reflect the 4.x changes. As the validators are implemented through tags in their docstrings, the corresponding file just needs to be autogenerated to update the resource/property.

For general resource specification changes, the flow is:

# pull down an updated resource specification
make spec
# regenerate the files
make regen
# verify what changes were made
git diff

This has worked out well for the last 5 versions of the spec file. The 54.0.0 change did need a json patch to remove an override needed due to a bug in one of the earlier resource specs.

[JohnPreston]

Thanks @markpeek . I have implemented so validation on the compose-x side which I think are low-level enough (troposphere level) for these to be backported into troposphere. I did see the validators folder, I was just unsure how the imports for specific types would work.

[michael-k]

I've just ported our core infrastructure repository to v4. I didn't have to change anything apart from Action → ListenerRuleAction and the tests say that the generated CloudFormation templates (567 templates with a total of 213147 lines) did not change. 👍

[markpeek]

@michael-k thank you for that feedback. Great to hear it was that compatible with your code base.

[JohnPreston]

@markpeek
Should we get another beta release with the changes that went in since ? 4.0.0-beta1 ?

[markpeek]

@JohnPreston yes, I will be preparing a 4.0.0-beta.1 later this week which will include:

• Changes merged to the main branch so far
• Likely a few breaking changes to DataBrew, ImageBuilder, and SageMaker to fix some property conflicts (objects with the same name but have different property sets)
• Changes from spec updates (note: the 60.0.0 spec had some EC2 removals that I hope are backed out in the next version, or at least documented as removed)

If all looks good after a week or two I hope to release 4.0.0.

[JohnPreston]

Amazing @markpeek
Quite the ride thinking that just 2y ago we were in the 2.x versions and now the code generated from Specs 🚀

[markpeek]

Released 4.0.0-beta.1

[JohnPreston]

Happy to report that for all the resources that I use in compose-x, everything is working totally fine.
Need to update and confirm for custom resources too

[markpeek]

Released 4.0.0