Installing dbt Core: saying goodbye to brew and hello to “bundles”

[jtcohen6]

[tl;dr] We are not planning to add Homebrew formulae for new dbt Core versions, starting with v1.6 (released this week). We will maintain existing formuale for supported versions on a best-effort basis, until they reach end of life.

---

Historically, we've supported a few different mechanisms for installing dbt: https://docs.getdbt.com/docs/core/installation

Each installation mechanism serves a slightly different motivation and persona. To paraphrase @colin-rogers-dbt :

• PyPI: “I want to install specific dbt packages.”
• Docker: “I want to run dbt in a remote environment.”
• Homebrew: “I just want dbt to work. Even if that means also installing Python, Rust, Postgres, the kitchen sink — I’m willing to wait (a while) for brew install to handle it all.”

brew core no more

In the early days of dbt Core, brew was a good shorthand for setting up everything that someone needed to use it. I really mean everything: Python, Postgres, whatever is needed. This made a big difference when we were trying to convince and cajole people who had never before used a terminal to try it out for the first time.

Unfortunately, over the past several years, brew install has not been a reliable way to “just make dbt work.” If anything, the reverse has held true: Homebrew installation issues have been the source of the majority of Core team’s incidents over the last year.

• Often the problem was with dependencies. brew's default behavior resolves differently from pip, opting to build from source instead of using prebuilt “wheels.” Some adapters can’t be released with Homebrew due to issues with platform-specific dependencies.
• Sometimes it was changes to Homebrew itself. Because we've been maintaining our own tap (in order to support a longer history and array of formula versions), we only find out after the fact, or from nightly builds with lots of noisy failures.
• There have been several times where we had to regenerate all of our formulae, meaning every single patch version for every major/minor version, as the result of an incident.

We take these incidents seriously. If you cannot install dbt, you cannot use it! But they've also interrupted the Core engineering team, and taken us away from focusing on the thing that we should be spending most of our time on: building dbt.

Over the past few years, given these difficulties, Homebrew installation has fallen to secondary status:

• Homebrew releases are harder to automate, given the number of things that can go wrong, which is why these releases tend to lag so far behind releases to PyPI.
• Overall, Homebrew is only used by a small slice of the dbt install base—a few hundred out of tens of thousands. And while it's preferable for our install methods to be OS-agnostic, brew is primarily for MacOS users, plus some on Linux, with no support for Windows.

In all, we no longer believe that the benefits of Homebrew for installing dbt-core & adapters outweigh the costs. We would rather commit to supporting fewer, better mechanisms for installation.

So: We are not planning to add Homebrew formulae for new dbt Core versions, starting with v1.6 (released this week). We will maintain existing formuale for supported versions on a best-effort basis, until they reach end of life.

All of that said: Homebrew remains a great mechanism for installing lighter-weight applications, such as a thin CLI written in a compiled language that can compile down to a binary executable, without any need for additional dependencies. Don’t be surprised when you see us suggesting brew install again very soon for something fitting that bill.

Hello “bundles”

When it comes to managing third-party dependencies, there are two competing interests:

• Avoid unneeded dependency conflicts with other projects, which arise from overly aggressive pinning within each project.
• Prevent breaking changes that inevitably arise in minor and patch releases—which, despite everyone’s best efforts & intentions, happens frequently.

This is a dilemma that confronts the entire Python ecosystem. If we were to tightly pin within our own project, we can guarantee a stable experience for dbt users. But if everyone did it, no one could use any projects together.

It’s been a subject of active discussion over the course of the year. Bit by bit, we’ve been reviewing, consolidating, and loosening pins on our dependencies:

• Revise our approach to pinning dependencies #6495
• Reorganize, annotate, and revise dependency pins #7368

We’ve also been standardizing our policies across our OSS and source-available libraries. With the relaunch of the Semantic Layer, and the inclusion of dbt-semantic-interfaces in dbt Core v1.6, we made sure to relax dependencies wherever possible:

• Relax Dependencies for Better Compatibility with Other Projects dbt-semantic-interfaces#112

Still, this isn’t perfect. Even though patch releases in third-party dependencies shouldn’t introduce unexpected changes or regressions, they can and do. What can we do about it?

---

Enter stage left: dbt-core-bundles. We've been developing "bundles" over the course of this year, and while currently in beta, we're already mettle-testing them with our own production workflows in dbt Cloud.

How do bundles work?

Each week, if not more frequently, we will create a new bundle for each supported dbt version. They contain dbt-core, all verified adapters, and all of their collective dependencies, at specific versions, ready to install all together. This is everything you need for a “bare install” in a new or containerized environment. Most importantly, we validate that this collection of versioned packages can be installed together in CI before the bundle is released. In the near term, we are also planning to add functional and performance testing for each bundle.

Who are bundles for?

Bundles are designed to support repeatable installations of dbt core: a given bundle can be guaranteed to install and perform the exact same way every time.

Intended use cases:

• Ensuring every member of a team is developing on the same version of dbt packages
• Avoiding CI/CD issues with dependency conflicts
• Consistent production runs of dbt

Who aren't bundles for?

If you need to install dbt-core into the same Python virtual environment as other projects, it’s very likely that you won’t be able to use bundles. That’s okay: The standard and most common way to install dbt-core and adapters will remain pip install from PyPI. (Always, always, always into a virtual environment!) As we gradually loosen the dependency pins within dbt-core and adapters, this should get easier. However, you’re also taking a more active role in managing your own installation. In the event that a third-party dependency has an unexpected breaking change (in a minor or patch release), you will need to pin that dependency yourself, until we’re able to identify root cause and turn around our own fix (or temporary pin) in a patch release.

How to install from a bundle

The full set of instructions are available in the README of the repository I’ve linked above, and we’ll be adding more information to our docs.

To use a bundle you download the relevant archive (containing all dependencies), unzip it, and pass the resulting directory to pip. You can use the install_bundle.sh script to do this for you. For example, to install the latest bundle of dbt Core v1.6 (1.6.3) using Python 3.9 on a mac:

curl -s https://raw.githubusercontent.com/dbt-labs/dbt-core-snapshots/main/install_bundle.sh | bash -s -- 1.6.3 3.9 mac

[katieclaiborne]

Will it be possible to use bundles to install dbt-core and only a specified adapter (or adapters)?

I tried out the command above to great success, but noticed that I had also installed additional adapters that I didn't really care about.

[colin-rogers-dbt]

Great call out and should be a fairly low lift, opened an issue to track: dbt-labs/dbt-core-bundles#68

[katieclaiborne]

Thanks, Colin!

[Spince]

Will docker images be available for each bundle? I'd love to just pull an image instead of run the install_bundle.sh script.

[Spince]

maybe I am missing the point here but I assume this is related to issues like what happened recently with sqlparse and the bigquery adapter.. if I was referencing an immutable docker image tag that wouldn't have been an issue.

[doublenns]

What is mettle testing?

[emmyoop]

As previously communicated, dbt Core 1.5 reached end of life on April 27, 2024. Consequently, we no longer support Homebrew as an installation method for any version of dbt Core.

Effective today, we have removed Homebrew as an option for installing dbt Core. If you attempt to brew install you will get the following error:

Error: dbt-snowflake has been disabled because it is no longer a supported installation method.  See https://docs.getdbt.com/docs/core/installation-overview#install-dbt-core for other options!

We encourage you to explore alternative installation methods outlined in our documentation here.