[2019 02 08] Log: Initial improvements' discussions

UPDATE: This has effectively been processed into the main SaltStack Formulas project.

---

This is a (temporary) page to capture the initial discussions had about improvements that can be made to SaltStack Formulas. The idea is for this content to be processed into separate (permanent) pages for planning and decision-making purposes.

---

GitHub

https://github.com/saltstack-formulas/template-formula/pull/19:

Person	Time	Message
Imran Iqbal	2019-02-07 08:55	I don’t intend to hijack this discussion but this PR has reminded me of a more general situation in respect to these SaltStack formulas. With a number of active contributors involved in this PR discussion, hopefully someone can suggest a better place to talk about the points I mention below: In discussions on the SaltStack Community Slack workspace, I’ve found some resistance to these SaltStack formulas. Main issues appear to be that these formulas are too heavily reliant on using pillars (leading to excess load on the master) alongside general quality issues. I’m not expecting anywhere near a complete solution to this. However, I believe it would be useful to compare ideas, to identify any (small) improvements that realistically could be made and maintained.
Niels Abspoel	19:43	@myii I agree hijacking a discussion for the greater good…​:) I would love to see these salt-formula’s become something like a community maintenance project. something allong the line of vox pupuli https://voxpupuli.org/ but then for salt
Imran Iqbal	20:05	@aboe76 I just had a quick look at https://voxpupuli.org/ and I’m impressed by the effort they have put in over the years. That far surpasses what I was envisaging! I’m intrigued by the idea — but I don’t think it’s right to continue off-topic for this PR. Is there a more appropriate place for this discussion to be continued?

---

Slack

Slack SaltStack Community #formulas:

Person	Time	Message
Imran Iqbal	20:23	So how can saltstack-formulas improve?
Niels Abspoel	20:24	having a stable salt sls language guide…​. at the moment everything is possible…​
Imran Iqbal	20:25	What do you mean by "language guide"?
Niels Abspoel	20:26	having one way to use salt states like the file.managed long format instead of the short version
Imran Iqbal	20:27	Ah OK, one way I’ve done this for myself is that I’ve used vim-snippets based on the SaltStack documentation. So whenever I type any state or module, I have the whole template in front of me in a consistent order, and I can strip it down to what’s required for that particular state. But nothing is ever moved from the base position, which is in the same order as the SaltStack documentation. Using ultisnips, I even made many dynamic entries so making a particular selection would enable or disable certain sections of the snippet. But that became a bit bothersome, so I’m happy with the basic structure and then pruning as require
Niels Abspoel	20:32	but having a document like docs.saltstack.com having more extensive examples on how something can be done would be better. some people use vim, others use some different $EDITOR…​.:)
Imran Iqbal	20:33	Of course, but the idea is the same. The snippets can be expansive as necessary, to ensure clear and consistent use of each state/module/etc. Or are you suggesting that docs.saltstack.com needs more examples as well?
Niels Abspoel	20:35	yes because a new user of saltstack or formula writer would look at the docs first. https://docs.saltstack.com/en/latest/topics/development/conventions/formulas.html#writing-formulas
Imran Iqbal	20:37	Improving the main documentation is definitely a useful task — but that will take time. Are there any improvements that can be attained in the shorter-term?
Niels Abspoel	20:38	thinking about the map.jinja part, I haven’t tested it but I think the new map.jinja from salt-formula makes the formula faster…​because less jinja. and it makes it easier to add distro’s or options via osmap.yaml and osfamilymap.yaml files
Imran Iqbal	20:42	Those changes feel like definite improvements to me as well; if only from the readability angle. Easier to track down where the mapping is not being constructed properly. I was thinking that the same structure should be moved to the template-formula as well.
Niels Abspoel	20:43	yes and I like the idea of files switch, I have used that on different configuration to ease people in to saltstack and then coach them into more pillar driven options…​
Imran Iqbal	20:46	I haven’t had enough time to familiarise myself with it but I get the idea of how useful it can be. In terms of pillar-driven formulas, then I’m aware that it may not be a recommended practice. Is there a better way that can be adopted?
Niels Abspoel	20:47	the fun part of saltstack, is there a recommended way?
Imran Iqbal	20:48	I would suppose that larger-scale deployments would suggest what works well and what doesn’t so much.
Javier Bértoli	20:48	joined #formulas.
Niels Abspoel	20:49	hi @Javier Bértoli
Javier Bértoli	20:49	aloha, @aboe! Nice to talk to you in a more immediate way that github messages 😛
Niels Abspoel	20:50	yeah but merges are faster ✨
Imran Iqbal	20:50	Definitely, direct messaging FTW. @Javier Bértoli Any thoughts about some of the points we’ve discussed just above?
Javier Bértoli	20:59	yes…​ I agree that I like the way the map.jinja changed, is clearer to read. I think that proposing some 'guides' would be a good thing, obviously. And if they reflect in the formula-template that’d be great. From the top of my head, I’d add a test scaffold in the formula, with kitchen-ci and inspec examples for the basic tasks (add/remove package, check service, etc) ?
Niels Abspoel	21:01	What about something like the what’s proposed about the files_switch stuff, I think it is reasonable to have it available as an option, to be able to insert a file instead of heavy pillar yaml files is a blessing for some people And don’t forget about @noelmcloughlin remove state…​
Imran Iqbal	21:02	Tests, definitely.
Javier Bértoli	21:02	agreed on the file_switch thing too
Imran Iqbal	21:03	How about making the template-formula functional, showing common types of processes that can be run/tested? (edited)
Niels Abspoel	21:03	I like the idea. and hopefully it will mitigate those formula:ng states…​
Javier Bértoli	21:03	oh, please!
Niels Abspoel	21:04	I think with the files_switch option most of those can be integrated and removed and only keep one formula
Javier Bértoli	21:07	yes. I see two 'big' tasks at hand, mostly: 1st., write these proposals in docs and the template-formula, for 'future generations' and code 😛 2. refactor existing formulas to be consistent with these ideas. Ie, move all the existing 'map.jinja' messes → *.yaml files, add tests (at least basic scaffolds) to existing ones, update existing old serverspec tests to inspec or testinfra, etc…​ I think a consistent image on formulas would help people coming to salt, wouldn’t it?
Imran Iqbal	21:08	Where’s the best place to collect these plans? The available wiki pages on the template-formula?
Niels Abspoel	21:09	yes make the template-formula als sticky…​
Javier Bértoli	21:09	the github repos have a wiki available to them, downloadble via git, perhaps there?
Niels Abspoel	21:09	most of the github formulas in salt-formulas have the wiki and projects enabled.
Javier Bértoli	21:09	https://github.com/saltstack-formulas/template-formula/wiki
Imran Iqbal	21:10	So make template-formula wiki the central location for planning and organisation? As well as the template-formula itself being the best representation of how to construct formulas in general, right? (edited)
Javier Bértoli	21:12	makes sense to me, yes. As you guys mentioned earlier, a newcomer will go to the docs pages and then to the template, right?
Niels Abspoel	21:12	hopefully if we can pin the template-formula on top within the saltstack-formulas organisation then it will always be visible.
Imran Iqbal	21:12	The template-formula doesn’t stand out so far. I think that could be addressed over time. Yes, pinning it would help. But perhaps after making these initial improvements?
Javier Bértoli	21:13	👍 I’m have to leave now, will catch up later. see you guys!
Niels Abspoel	21:14	@Javier Bértoli nice evening!
Imran Iqbal	21:15	Thanks for that feedback. I think we’ve got a fair bit to start with.
Niels Abspoel	21:15	can we integrate saltstack with riot.im and saltstack irc? because @baby-gnu didn’t participate
Imran Iqbal	21:15	Yes, definitely. I’ve been doing a bit of work with Matrix recently so I’m confident of that. Does SaltStack have any presence on Matrix so far?
Niels Abspoel	21:26	irc only
Imran Iqbal	21:34	OK, for the time being, I’m going to grab this discussion and put it in a temporary page in the wiki, so that it can be processed later into proper pages. There’s only so much history that is retained here when on Slack’s free plan.
Noel MacLoughlin	21:52	Okay. I understand this discussion is exploring ideas around improving formulas. Thats a good thing. First question coming to mind - Is writing formulas the correct way to use salt? - is there an alternative to using formulas with salt and where is that alternative documented? Secondly a template-formula should be template for doing common things in common way - but I’d like to see a solution-formula too (i.e. take a case study like openstack or OPNFV or way I am experiementing with OpenSDS) If I had a solution-formula (template) I’d look at building solutions for other community projects like edgeXfoundry I have a backlog of about 20 formulas (DevOps stuff); Thirdly - some kind of cloud-native template formula. Its not clear to me who to use salt to managed distributed microservice architecture, etc. Unfortunately my salt knowledge is limited enough.
Imran Iqbal	22:00	Thanks for those points. Some very interesting ones there, let me offer my take on those.
Noel MacLoughlin	22:03	Forth - template-formula is too focussed on PKG. There are many was to distribute packages. All the Cloud native projects seem to distribute (1) Tarball releases (i.e. AMD64), (2|) github repo and three source tarball (3). So template-formula should support different ways of absorbing software and be a template for this.
Imran Iqbal	22:03	Correct way or not so correct, there’s a large corpus of formulas that have been built up over the years. It would be premature at this moment to make a decision about these. However, there are a number of improvements that could be made, even if only to gain better consistency and quality across all of the formulas.
Noel MacLoughlin	22:04	Fifth - A macros.jinja with common macros that "Any formulae would find useful" .. repeatable patterns deserve repeatable code But if I wanted to automate with salt without formulas. We have states v.s. modules?
Imran Iqbal	22:06	I haven’t given much thought to the solution-formula concept. It would be good to get a better idea of that. I believe a significant improvement in the template-formula can help towards that goal, in any case.
Noel MacLoughlin	22:07	Sure - a solution-formula is just a showcase of how to manage multiple things (A, B, C).
Imran Iqbal	22:08	Definitely agree with the cloud formula idea. How about sections of the template-formula for each of the common actions done throughout all of the formulas? That would spread the content to more than just pkg. We’ve definitely started improving macros.jinja across some of the more popular formulas. That work can be distilled into the repeatable patterns that you’ve mentioned.
Noel MacLoughlin	22:13	A solution-formula should showcase how you might model system containing A, B, C. C. Everyone is trying to solve this problem. I was looking at https://www.airshipit.org/ and its same problem - how to model cloud-native systems in YAML and deploy. airshipit.org Home | Airship Airship So template (single formula is this pattern). template-formula pattern templateA: package: xx version: nn section: solution formula yaml probably needs to be driven by jinja2 looping - I’ve been experimenting with this pattern in OpenSDS formula. solution-formula Domain: ids: - A - B - C Collapse The directory structure is solution/A, solution/B, solution/C .. each subdomain is entire formula …​ There is one map.jinja but YAML for each subdomain (i.e. subdirectory ) If you could build a template how to deploy A, B, C in good way - its called model-driven design - then that would be interesting. Nobody seems to be doing that (puppet, ansible, etc) Salt probably has stuff to do this out of the box - but nobody knows how to build distributed solution formula in a fast repeatable and consistent way (correction: I dont)
Imran Iqbal	22:27	Do you think that these are within the scope of what can be achieved by saltstack-formulas in the short/medium-term? My concern is that there are more fundamental failings that are getting in the way at the moment. E.g. A lack of testing within the formulas, affecting the quality.
Noel MacLoughlin	22:31	Yes I do. But starting with basics - a good template-formula (for A) would be start. My idea of solution-formula is simply a repo containing three template-formulas (A, B, C). This would need a macros.jinja with shared macros. Some formulas need improvement (osmap, map.jinja, defaults, pattern) (edited)
Imran Iqbal	22:33	Standardisation across formulas?
Noel MacLoughlin	22:33	Yes!! but based on agreed template
Imran Iqbal	22:33	Are our formulas trying to do too much?
Noel MacLoughlin	22:34	Thats my first question - is formulas correct way to go?
Imran Iqbal	22:34	Should they be more directed, more specific, rather than trying to cater for all situations from one formula?
Noel MacLoughlin	22:34	every state should do one thing, and do it well. pkg/install, pkg/clean, repo/install, repo/clean, etc, etc granularity is important for flexibility. monolithic state files are NOT easy to scale. (edited) Sorry to clarify .. pkg/install.sls, pkg/clean.sls, repo/install.sls, repo/clean.sls, etc, etc
Imran Iqbal	22:39	But then comes the question of inter-dependence / cohesion between formulas. I understand that this has been avoided as much as possible and there are good reasons for that. But that has probably affected the way each formula has grown. Breaking it down as you mentioned above has helped, where I’ve seen it implemented.
Noel MacLoughlin	22:41	template-formula must stand alone. solution-formula combines (say) three standalones
Imran Iqbal	22:41	In your experience, are all of the formulas standalone so far?
Noel MacLoughlin	22:42	Mostly - but since they evolved separately - tons of duplication.
Imran Iqbal	22:42	Can you remember any examples of the top of your head, which aren’t?
Noel MacLoughlin	22:43	There was some formula which depended on vim formula I think - Cannot remember which - it seemed wrong design.
Imran Iqbal	22:44	It isn’t a good idea. Not reliable enough over time.
Noel MacLoughlin	22:47	In my view saltstack-formulas is not bad - much better than ansible galaxy mess - but the lack of consistent patterns causes pain for everyone, reduces quality too. that would be place to start EXCEPT what is the approved template guiding that effort?
Daniel Wallace	22:48	someone should build a repo frontend for the saltstack-formulas, that allows building new spm packages on tags in the formulas repositories
Imran Iqbal	22:48	As was discussed above, it makes sense to collect that into template-formula and its wiki for the time being.
Noel MacLoughlin	22:48	+1
Imran Iqbal	22:49	@noelmcloughlin Have you seen much tagging in the various formula repos?
Noel MacLoughlin	22:50	tagging of what? sorry for being pedantic but you know.
Imran Iqbal	22:51	I’m assuming releases — fixed points for each formula that can be relied upon when pulling.
Noel MacLoughlin	22:51	What constitutes a release - thats the problem!!
Daniel Wallace	22:52	each merged pr would potentially be a release
Imran Iqbal	22:52	And that’s part of the problem that needs to be resolved. There’s no clear goals set for each formula repo.
Noel MacLoughlin	22:52	typically a major release container features. point release contains bug fixes. --- as long as release criteria is agreed then tagging is way to go.
Imran Iqbal	22:53	Each merged PR is an easy way to do it for the time being, for sure.
Daniel Wallace	22:53	each pr automatically bumps the patch version and then when you make breaking changes, bump the major version
Noel MacLoughlin	22:53	+1 interesting idea.
Imran Iqbal	22:54	Yes, that makes a lot of sense.
Noel MacLoughlin	22:54	Could a cross-repo tag work. i.e. salt-formulas 2019.5
Daniel Wallace	22:54	then we could finally say whether or not formulas are stable because people could pin to a major version for spm and then they know that their stuff will work at least until a major version change which is something that can’t be done now
Imran Iqbal	22:55	Which makes it easier than having to maintain your own forks.
Daniel Wallace	22:55	which is why imho formulas are not super useful, cause you don’t know when someone is going to make a major change to them exactly
Noel MacLoughlin	22:55	@gtmanfred what is alternative to formulas?
Imran Iqbal	22:56	There’s no reason why this can’t be actioned in the very short term. This doesn’t require any major planning.
Daniel Wallace	22:56	not formulas? all custom states? which a lot of people do i also think that the formulas should move away from focusing on using pillars because it is not scalable
Imran Iqbal	22:56	To be specific, move to what instead?
Daniel Wallace	22:57	template files in the fileroot or grains pillars do not scale you should only be putting secret data in pillars if you are putting extra system configuration stuff in pillars, you are going to end up with too many pillars and your system being slow you could easily just have templating files in the fileroot that could be added seperataly to the state
Imran Iqbal	22:58	Yep — and this will require some planning.
Daniel Wallace	22:58	see https://github.com/saltstack-formulas/systemd-formula/blob/master/TOFS_pattern.md TOFS_pattern.md # TOFS: A pattern for using Saltstack All that follows is a proposal based on my experience with [Saltstack](http://www.saltstack.com/). The good thing of a piece of software like this is that you can "bend it" to suit your needs in many possible ways, and this is one of them. All the recommendations and thoughts are given "as it is" with no warranty of any type. ## Usage of values in pillar vs templates in file_roots Show more saltstack-formulas/systemd-formulaAdded by GitHub that basically tells you how to do it ^^
Imran Iqbal	22:58	We’ve just got a PR for this in the template-formula as well, so we’re already starting this process — thanks for the example.
Daniel Wallace	22:59	awesome
Imran Iqbal	22:59	@gtmanfred Appreciate the feedback, it’s been very useful.
Noel MacLoughlin	23:09	The reason using pillar data is attractive is because you can model your use case in that same pillar data. The TOFS approach seems interesting - need to look into it.
Imran Iqbal	23:11	I’m having a think about the tagging across all of the formula repos. I’ve got a few ideas bubbling but I think I’ll collect them alongside these discussions, so that we can make some initial resolutions.
Noel MacLoughlin	23:12	Be wary of doing thing for the sake of it. needs some thoughts.
Imran Iqbal	23:13	No, there’s nothing wrong with tagging itself. As long as the semantic versioning is consistent, it will be very useful all round. At a very simple level, when one is looking at the git history for a particular repo, it’s impossible to differentiate between major and minor changes (PR merges). If each PR is evaluated by the member performing the merge and then tagged accordingly, it will help everyone else get a much clearer idea of the level of changes to expect when pulling in the latest upstream changes.
Daniel Wallace	23:29	the problem with using pillar is that rendering pillars for all minions is by far the most expensive thing to do, so to scale you have to figure out what is more worth it to you, the simplicity of pillars or the simplicity of not having to scale horizontally with syndics not really something you have to worry about if you are under 200 minions though… usually unless you have an insane number of pillars and you regularly trigger all minions to refresh semi frequently (edited)
Imran Iqbal	23:30	It’s going to take a while but we’ve started. The TOFS_pattern PR has just been merged into the template-formula, so the revolution has begun!