Skip to content

[2019 02 11] Log: Ongoing improvements' discussions

Jump to bottom Edit New page
Imran Iqbal edited this page Feb 12, 2019 · 2 revisions

UPDATE: This has effectively been processed into the main SaltStack Formulas project.

This is a (temporary) page to capture ongoing discussions had about improvements that can be made to SaltStack Formulas. The idea is for this content to be processed into separate (permanent) pages for planning and decision-making purposes.

Slack

Person Time Message

Imran Iqbal

06:53

@Javier Bértoli I didn’t get a chance to respond to this yesterday: "…​ but will provide a single place to see all’s pending in the formulas, which is a good thing to me" — then this is already available via. the search:

• PRs: https://github.com/pulls?utf8=%E2%9C%93&q=is%3Aopen+is%3Apr+org%3Asaltstack-formulas+archived%3Afalse+
• Issues: https://github.com/issues?utf8=%E2%9C%93&q=is%3Aopen+is%3Aissue+org%3Asaltstack-formulas+archived%3Afalse+

Imran Iqbal

07:33

@aboe @Javier Bértoli So I’ve updated https://github.com/orgs/saltstack-formulas/projects/1 using all of our discussions so far on both GitHub and Slack. Please feel free to modify/update as you see fit.

Ryan Walder

15:29

How/where do you manage custom files in a TOFS style formula?

Imran Iqbal

15:30

+ Daniel Wallace
see https://github.com/saltstack-formulas/systemd-formula/blob/master/TOFS_pattern.md
Posted in #formulasFeb 8thView message

Ryan Walder

15:31

Right now the customisation/templating is stored in the pillar which is easy to manage (stick it in git) with TOFS I’m not clear how I manage formulas and the custom configs/overides
ok, but how do I manage nfs/files/mycompany/file.ext?

Imran Iqbal

15:32

This is something newly being implemented in the formulas and I personally haven’t had much experience with it. It may be worth pinging Daniel about it, or the PR authors.
https://github.com/saltstack-formulas/template-formula/pull/19
baby-gnu
#19 Feature/template override and files switch
This is the TOFS pattern from @moreda with the modifications I did in systemd-formula.
Comments
20
Reviewers
@javierbertoli
saltstack-formulas/template-formulaFeb 6thAdded by GitHub
The PR author isn’t available on Slack but is responsive to comments on GitHub.

Bradley Hieber

15:35

@ryanwalder generally you would have a file in

<formula>/files/default/<service>/<config>.conf
<formula>/files/<osfamily>/<host>/<config>.conf
Or you could remove osfamily entirely
then in your file.managed, you use multiple sources, with the default config being at the bottom.

Ryan Walder

15:35

Yeah but in’t the point of the TOFS method to remove general config from pillar?
So I would want a company custom config for base settings in ntp for example

Bradley Hieber

15:36

none of that is in pillar
it is in your file tree

Ryan Walder

15:39

Maybe I’m explaining badly. Using the ntp example:

0. I load the nfs-formula via gitfs
1. Where do I put nfs/files/comapny/blah.ext?

Do I fork and add to the fork?
Do I have a completely seperate repo which also contains nfs/files/company/blah.ext, if so how do I modify the formula to reference the mycompany file?

Bradley Hieber

15:40

fork and add to fork
or add to .gitignore
but then you would need some way of tracking changes to blah (edited)

Ryan Walder

15:40

Doesn’t this mix config (settings) and code (applying those settings)?

Bradley Hieber

15:41

infrastructure as code 😉

Ryan Walder

15:41

I can see the benefits due to reduced rendering on the master just trying to see how it could be used effectively without having to maintain a fork of every single formula

Bradley Hieber

15:42

The logical answer is, if you are working on the formula and are going to contribute to it, you will need a fork of it anyway

Ryan Walder

15:43

Which is fine for people who contribute but not for people who consume formulas
who are the vast majority I reckon

Imran Iqbal

15:43

https://docs.saltstack.com/en/latest/topics/development/conventions/formulas.html#adding-a-formula-as-a-gitfs-remote:

>*We strongly recommend forking a formula repository* into your own GitHub account to avoid unexpected changes to your infrastructure.
>
> Many Salt Formulas are highly active repositories so pull new changes with care. Plus any additions you make to your fork can be easily sent back upstream with a quick pull request!
(edited)

Ryan Walder

15:44

Forking makes sense due to the complete lack of versioning in formulas, forking then modifying makes a lot less sense
That makes updating a lot harder

Imran Iqbal

15:45

Do you mean version numbers (release tagging)? If so, that’s something we’re starting to remedy.

Ryan Walder

15:45

I mean modifying upstream seems like a Bad Idea in any circumstance
especially just to load a custom config
It seems to make a lot of management overhead for some efficency
I could of course be wrong and missing something blatantly obvious

Imran Iqbal

15:48

As I said, I’m not familiar — but what I understand is that custom config is stored in YAML files that can modified as required for your needs.

Ryan Walder

15:51

Cheers for your help, will take a deeper look when I get home.

Imran Iqbal

15:52

You’re welcome.

Shawn Welter

17:06

I have one suggestion for formulas; make them portable in the fiile sturcure. Many assume the formual lives in the root of salt:// Moving the formula to another location is not possible due to paths to map.jinja and defaults.yaml are absolute and not relative to the formula location. Some like the mysql formula have switch to using from tpldir ~
The Template formula is not portable or movable

Imran Iqbal

17:26

@redundancyd Excellent point. As you’ve noted, some of the formulas have started using that. I’ll add that to the list of tasks that we’re working on to overhaul the formulas in general.

Max Arnold

17:31

Quite soon it will be possible to use relative Jinja imports: https://github.com/saltstack/salt/pull/47490
plastikos
#47490 Support Jinja imports with relative paths. Issue #13889
Jinja imports will be interpreted as originating from the top of each
of the directories in the searchpath when the template name does not
begin with './' or '../'. When a template name begins with './' or
'../' then the import will be relative to the importing file.

Prior practices required the following construct:

{% from tpldir ~ '/foo' import bar %}

This patch supports a more "natural" construct:

{% from './foo' import bar %}
Show more
Comments
6
saltstack/saltMay 5th, 2018Added by GitHub

Imran Iqbal

17:32

@redundancyd I just remembered that there is one problem with this, tpldir can’t be used in template files. See https://github.com/saltstack/salt/issues/41195.
hatifnatt
#41195 Can’t use tpldir or slspath in template file
Can’t use tpldir or slspath in template file

I want to load defaults with construction like this

{%- import_yaml slspath ~ "/defaults.yaml" as defaults %}


Which work flawless in sls state file. But fails in template file. On #salt IRC channel I received a recommendation to use 'tpldir' variable instead, but it doesn’t work as well.

Setup

init.sls

{%- import_yaml slspath ~ "/defaults.yaml" as defaults %}
config:
  file.managed:
Show more
Labels
Bug, Core, High Severity, P3
Comments
11
saltstack/saltMay 11th, 2017Added by GitHub
@marnold That will be extremely useful. Thanks for the heads-up.

Imran Iqbal

5 hours ago

> Daniel Wallace [Feb 8th at 22:54]
> because people could pin to a major version for spm
> 55 replies
@gtmanfred Looking into this further and I’ve come across a contradiction.

https://docs.saltstack.com/en/latest/topics/spm/spm_formula.html#version

> …​ it is suggested that this version is specified in a YYYYMM format.
> …​ If multiple releases are made in a month, the release field should be used.

https://docs.saltstack.com/en/latest/topics/development/conventions/formulas.html#versioning

> Formula are versioned according to Semantic Versioning, http://semver.org/.

I favour the latter since it is easier to highlight the differences between major/minor/patch changes. @aboe Has already applied this to the template-formula. Are there any reasons to use the suggested SPM format instead?
Semantic Versioning
Semantic Versioning 2.0.0
Semantic Versioning spec and website

Daniel Wallace

5 hours ago

meh
i like the second one
date based versioning is a mistake most of the time
but it might make sense for salt formulas

Imran Iqbal

5 hours ago

SPM works OK with semver?

Daniel Wallace

5 hours ago

i don’t know
it should
it should just use LooseVersion to determine the version comparisons

Imran Iqbal

5 hours ago

OK, this is something that needs to be tested before finalising the format to apply. Thanks for your replies.

Wayne Werner

3 hours ago

I like date based versions when there’s not a clear API, personally.

Daniel Wallace

3 hours ago

yes, but then you just release with a random date, and no one knows if you broke any backwards compatibility

Imran Iqbal

3 hours ago

@Wayne Werner How can the method be used to differentiate between versions that aren’t backwards-compatible?

Daniel Wallace

3 hours ago

it offers nothing to the user other than how old it is

Imran Iqbal

3 hours ago

Another negative point is that it can be associated with the version of Salt it is "supposed" to work with.

Daniel Wallace

3 hours ago

that should be in the FORMULA file

MINIMUM_VERSION
Minimum recommended version of Salt to use this formula. Not currently enforced.

Imran Iqbal

3 hours ago

I’m referring to the version number itself.
If we had a formula that was recently updated, say 201902 — the version number suggests it has been tested for 2019.2. (edited)

Wayne Werner

3 hours ago

Well, if you don’t have a clear API, nobody knows if you have any kinds of compatibility anyway :rolling_on_the_floor_laughing:

Daniel Wallace

3 hours ago

@myii if you go with semver, you should def cut a 0 ver release before making the change to semver, so that people that are using them as is don’t have to quick pivot everything to TOFS (edited)

Imran Iqbal

3 hours ago

Sorry, is there a difference between symver and semver, or just a typo?

Daniel Wallace

3 hours ago

typo
@myii if you wanted to work on getting a web api/frontend for updating and managing spm packages, i would be interested in writing/helping to write that

Imran Iqbal

3 hours ago

OK, @aboe has started the process using v0.x.x, so once we start applying TOFS, we’ll do that.

Daniel Wallace

3 hours ago

awesome

Imran Iqbal

3 hours ago

That would be incredibly helpful and hopefully a motivating factor. Something to rally around, with actual results available immediately.

Daniel Wallace

3 hours ago

yeah

Imran Iqbal

3 hours ago

Still at the stage of initial planning, at the moment. I’ve collected all of the points made so far under https://github.com/orgs/saltstack-formulas/projects/1?fullscreen=true.

Daniel Wallace

3 hours ago

i will see what I can put together
awesome

Imran Iqbal

3 hours ago

Need a few more SaltStack Formula members to get involved, to start clearing through the tasks.
Thanks for your support.

Daniel Wallace

3 hours ago

of course

Imran Iqbal

3 hours ago

By the way, what stack would you be proposing to use for the web API/front-end?
@gtmanfred ^?

Daniel Wallace

3 hours ago

flask + sqlalchemy + postgres
nothing crazy
for a frontend, i don’t know, that would be something we could do later, just really need an api to upload packages

Wayne Werner

3 hours ago

Yeah, if all you need is an API, Flask is pretty great for that. Responder is an interesting new package if you can use Python 3.6+, I think.
(supports asyncio, so, there’s that)

Daniel Wallace

3 hours ago

or, i could use the thing that is most used by the world of developers according to the jetbrains survey

Imran Iqbal

3 hours ago

OK, I haven’t used flask and sqlalchemy but not too far removed from Django’s ORM as far as I’m aware…​ oh.

Daniel Wallace

3 hours ago

It is pretty easy, and would be very fast to template and get started with flask, to just have a poc to build and do the basics
Then we could store all the spm files in s3

Imran Iqbal

3 hours ago

OK, sounds good.

Wayne Werner

3 hours ago

 

from flask import Flask

api = Flask(__name__)

@api.route('/upload', methods=['POST'])
 def upload():
     return 'You uploaded something or another!'

api.run(debug=True)

Not much more than that to get started 👍

Daniel Wallace

3 hours ago

The biggest thing I am going to have to do is pick apart how spm builds the manifest, cause i am going to use the s3 api to pull down the file list, and build it remotely without access to the files.

Imran Iqbal

2 hours ago

Are there any examples of these manifests?

Daniel Wallace

2 hours ago

yeah, just use spm create_repo to build one for a repository (edited)

Imran Iqbal

1 hour ago

Thanks, I’ll have a look at that and spm in general.
Add a custom footer
Add a custom sidebar
Clone this wiki locally