[CT-2140] [Feature] Parse versions with specific version standards

[emmyoop]

Is this your first time submitting a feature request?

• I have read the expectations for open source contributors
• I have searched the existing issues, and I could not find an existing issue for this feature
• I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

Currently, in semver.py we have a single function to parse both dbt version (PEP 440 compliant) and the package versions (SemVer 2.0 compliant). It's both confusing and broken.

The only versions that are complant for both standards do not contain any pre-release or dev info:

• 1.5.2
• 10.5.22

We also allow strictly PEP440 compliant versions:

• 1.5.0rc1
• 1.5.0rc1.dev1234

As well as SemVer 2.0 compliant versions

• 1.5.0-rc1

Some PEP440 compliant but SemVer 2.0 uncomplianty versions fail

• 1.5.0.dev1232

And some versions that are not compliant by either standard pass

• 1.5.0-rc1b2

The first thing that should happen is to pull apart checking bopth PEP440 compliance and SemVer 2.0 compliance in the same place. Be explicit about which is expected and audit for it specifically. This will involve looking into everywhere this logic is called to be sure the right functionality get invoked.

Instead of using our own homegrown solution here, we should use packaging.version.Version (same thing we're using in the parse-semver action) to check the version of dbt projects against PEP440. Possibly use the semvver package for checking package versions against SemVer 2.0.

Note: Adapters also do some version parsing in setup.py. It's worth exploring if this solution could work for them as well to remove as much manual regex we have to maintain as possible.

Describe alternatives you've considered

Tweaking the current regex to work for all version patterns but this is undesirable since we're already in the code.

Who will this benefit?

Maintainers for ease of readability of what's happening as well as package maintainers and adapter maintainers as it would allow the use of full versioning instead of a restricted subset.

Are you interested in contributing this feature?

No response

Anything else?

Would allowing packages fuller versioning have an effect on the package hub?

[dbeatty10]

Relevant SemVer 2.0 RegEx for dbt packages

Here's some regex that can be used for the SemVer portion of things:

• https://github.com/dbt-labs/hubcap/blob/b4e2634ca7bfc7126dbdbc7f628407648b971403/tests/test_version_regex.py
• [CT-589] [Bug] dbt deps does not precisely match the regex for SemVer #5201

Especially note the examples here and here.

Some related issues and PRs

• CT 2000 fix semver prerelease comparisons #6838
• feature/deps-regex-semver #5370
  • [CT-589] [Bug] dbt deps does not precisely match the regex for SemVer #5201
• Make hyphen optional for prerelease suffix hubcap#130
  • Support prerelease identifiers without hyphens hubcap#129
• [CT-253] Standardize Pre-release Version Format #4741

[jtcohen6]

Currently, in semver.py we have a single function to parse both dbt version (PEP 440 compliant) and the package versions (SemVer 2.0 compliant). It's both confusing and broken.

Agree that this is super confusing, and that this code is hand-rolled and really ought to be replaced with an off-the-shelf library. That said, do we still have any PEP 440-compliant logic in semver.py, now that we've removed packaging from that code? (#6838)

The only other place inside our codebase that we use packaging (PEP 440-compliant) is to detect the installed version of git and see if it supports subdirectory sparse checkouts:

dbt-core/core/dbt/clients/git.py

Lines 37 to 39 in eb200b4

	out, _ = run_cmd(cwd, ["git", "--version"], env={"LC_ALL": "C"})
	git_version = version.parse(re.search(r"\d+\.\d+\.\d+", out.decode("utf-8")).group(0))
	if not git_version >= version.parse("2.25.0"):

But git isn't a Python package, so... shouldn't we be using semver for this...?

$ git --version
git version 2.37.1 (Apple Git-137.1)

Obviously, we need to keep packaging / PEP 440 code around for all our release code, because dbt-core is, after all, a Python package :)

---

Possibly use the semvver package for checking package versions against SemVer 2.0.

No question that we try to do should do this, and delete a ton of our hand-rolled code in semver.py. We originally wrote this code in Feb 2018; it's possible that the libraries were less stable then, or we were trying to limit unneeded dependencies. At this point, these packages seem pretty stable; we just shouldn't pin them too tightly, or at all :)

@colin-rogers-dbt apparently likes semantic-version.