diff --git a/CHANGELOG.md b/CHANGELOG.md index c223dd6..8f8658e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,9 +1,32 @@ -## New features -- `generate_model_yaml` with `upstream_descriptions=True` now reads from upstream sources in addition to models. +# dbt-codegen v0.12.0-b1 + +## What's Changed + +### Features + +- `generate_model_yaml` with `upstream_descriptions=True` now reads from upstream sources in addition to models.(#112)[https://github.com/dbt-labs/dbt-codegen/issues/112] +- `generate_source` now has options for case sensitivity in all fields ([#112](https://github.com/dbt-labs/dbt-codegen/issues/112) + +### Fixes -## Fixes - Column `description` fields are now correctly escaped in `generate_model_yaml` ([#142](https://github.com/dbt-labs/dbt-codegen/issues/142)) -- Fix `generate_source` behavior of applying a lowercase function to all object names ([#112](https://github.com/dbt-labs/dbt-codegen/issues/112)) + +### Docs + +- Fixed `generate_source` documentation +- Rewrote the contributor README at `integration_tests/README.md` + +## New Contributors + +- @wircho made their first contribution in https://github.com/dbt-labs/dbt-codegen/pull/159 +- @yatsky made their first contribution in https://github.com/dbt-labs/dbt-codegen/pull/164 +- @pnadolny made their first contribution in https://github.com/dbt-labs/dbt-codegen/pull/168 +- @esegal made their first contribution in https://github.com/dbt-labs/dbt-codegen/pull/154 +- @gwenwindflower made their first contribution in https://github.com/dbt-labs/dbt-codegen/pull/163 + +**Full Changelog**: https://github.com/dbt-labs/dbt-codegen/compare/0.11.0...0.12.0-b1 + +## New features # dbt-codegen v0.11.0 @@ -13,6 +36,7 @@ and are lowercase to align with the dbt style guide. Scale & precision are **not** included. Previous logic for `generate_source` defaulted to `false` and the resulting data types were uppercase and included scale & precision ([#122](https://github.com/dbt-labs/dbt-codegen/pull/122)). [Dispatch](https://docs.getdbt.com/reference/dbt-jinja-functions/dispatch) can be used to utilize the column data type formatting of previous versions. Namely, by adding this macro to your project: + ```sql {% macro default__data_type_format_source(column) %} {{ return(column.data_type | upper) }} @@ -20,43 +44,48 @@ and are lowercase to align with the dbt style guide. Scale & precision are **not ``` And then adding this within `dbt_project.yml`: + ```yaml dispatch: - macro_namespace: codegen - search_order: ['my_project', 'codegen'] + search_order: ["my_project", "codegen"] ``` ## What's Changed -* GitHub Action to add/remove triage labels as-needed by @dbeatty10 in https://github.com/dbt-labs/dbt-codegen/pull/133 -* GitHub Action to close issues as stale as-needed by @dbeatty10 in https://github.com/dbt-labs/dbt-codegen/pull/134 -* Update README.md by @cohms in https://github.com/dbt-labs/dbt-codegen/pull/129 -* Remove hard-coded values for database and schema by @dbeatty10 in https://github.com/dbt-labs/dbt-codegen/pull/139 -* Instructions for the release process by @dbeatty10 in https://github.com/dbt-labs/dbt-codegen/pull/137 -* Add `include_data_types` argument to `generate_model_yaml` macro by @linbug in https://github.com/dbt-labs/dbt-codegen/pull/122 + +- GitHub Action to add/remove triage labels as-needed by @dbeatty10 in https://github.com/dbt-labs/dbt-codegen/pull/133 +- GitHub Action to close issues as stale as-needed by @dbeatty10 in https://github.com/dbt-labs/dbt-codegen/pull/134 +- Update README.md by @cohms in https://github.com/dbt-labs/dbt-codegen/pull/129 +- Remove hard-coded values for database and schema by @dbeatty10 in https://github.com/dbt-labs/dbt-codegen/pull/139 +- Instructions for the release process by @dbeatty10 in https://github.com/dbt-labs/dbt-codegen/pull/137 +- Add `include_data_types` argument to `generate_model_yaml` macro by @linbug in https://github.com/dbt-labs/dbt-codegen/pull/122 ## New Contributors -* @cohms made their first contribution in https://github.com/dbt-labs/dbt-codegen/pull/129 -* @linbug made their first contribution in https://github.com/dbt-labs/dbt-codegen/pull/122 + +- @cohms made their first contribution in https://github.com/dbt-labs/dbt-codegen/pull/129 +- @linbug made their first contribution in https://github.com/dbt-labs/dbt-codegen/pull/122 **Full Changelog**: https://github.com/dbt-labs/dbt-codegen/compare/0.10.0...v0.10.0 # dbt-codegen v0.10.0 ## What's Changed -* added comments to verbose regex in generate_model_import_ctes by @graciegoheen in https://github.com/dbt-labs/dbt-codegen/pull/93 -* Feature/hackathon model generator by @fivetran-joemarkiewicz in https://github.com/dbt-labs/dbt-codegen/pull/83 -* Suggestion to include packages.yml example in README.md by @Maayan-s in https://github.com/dbt-labs/dbt-codegen/pull/77 -* Add include_data_types flag to generate_source macro by @GSokol in https://github.com/dbt-labs/dbt-codegen/pull/76 -* Expected result of nested struct in BigQuery by @dbeatty10 in https://github.com/dbt-labs/dbt-codegen/pull/105 -* issue106/get_models helper macro by @erkanncelen in https://github.com/dbt-labs/dbt-codegen/pull/115 -* Feat/generate sources add database and schema by @jeremyholtzman in https://github.com/dbt-labs/dbt-codegen/pull/124 + +- added comments to verbose regex in generate_model_import_ctes by @graciegoheen in https://github.com/dbt-labs/dbt-codegen/pull/93 +- Feature/hackathon model generator by @fivetran-joemarkiewicz in https://github.com/dbt-labs/dbt-codegen/pull/83 +- Suggestion to include packages.yml example in README.md by @Maayan-s in https://github.com/dbt-labs/dbt-codegen/pull/77 +- Add include_data_types flag to generate_source macro by @GSokol in https://github.com/dbt-labs/dbt-codegen/pull/76 +- Expected result of nested struct in BigQuery by @dbeatty10 in https://github.com/dbt-labs/dbt-codegen/pull/105 +- issue106/get_models helper macro by @erkanncelen in https://github.com/dbt-labs/dbt-codegen/pull/115 +- Feat/generate sources add database and schema by @jeremyholtzman in https://github.com/dbt-labs/dbt-codegen/pull/124 ## New Contributors -* @fivetran-joemarkiewicz made their first contribution in https://github.com/dbt-labs/dbt-codegen/pull/83 -* @Maayan-s made their first contribution in https://github.com/dbt-labs/dbt-codegen/pull/77 -* @GSokol made their first contribution in https://github.com/dbt-labs/dbt-codegen/pull/76 -* @erkanncelen made their first contribution in https://github.com/dbt-labs/dbt-codegen/pull/115 -* @jeremyholtzman made their first contribution in https://github.com/dbt-labs/dbt-codegen/pull/124 + +- @fivetran-joemarkiewicz made their first contribution in https://github.com/dbt-labs/dbt-codegen/pull/83 +- @Maayan-s made their first contribution in https://github.com/dbt-labs/dbt-codegen/pull/77 +- @GSokol made their first contribution in https://github.com/dbt-labs/dbt-codegen/pull/76 +- @erkanncelen made their first contribution in https://github.com/dbt-labs/dbt-codegen/pull/115 +- @jeremyholtzman made their first contribution in https://github.com/dbt-labs/dbt-codegen/pull/124 **Full Changelog**: https://github.com/dbt-labs/dbt-codegen/compare/0.9.0...0.10.0 @@ -67,30 +96,40 @@ dispatch: # dbt-codegen v0.8.0 # Unreleased + ## Breaking changes + ## New features + ## Quality of life + - Now uses `print` instead of `log` to output the generated text into the console. This enables you to invoke dbt with the `--quiet` flag and directly pipe the codegen output into a new file, ending up with valid yaml ## Under the hood + ## Contributors: + - [@JorgenG](https://github.com/JorgenG) (#86) # dbt-codegen v0.7.0 ## 🚨 Breaking change + - Add support for including description placeholders for the source and table, which changes the behavior of `generate_source` when `include_descriptions` is set to `True`. Previous logic only created description placeholders for the columns ([#64](https://github.com/dbt-labs/dbt-codegen/issues/64), [#66](https://github.com/dbt-labs/dbt-codegen/pull/66)) ## New features + - Add optional `table_names` arg to `generate_source` ([#50](https://github.com/dbt-labs/dbt-codegen/issues/50), [#51](https://github.com/dbt-labs/dbt-codegen/pull/51)) - Add support for importing descriptions from columns with the same names in upstream models. It is available by setting the parameter `upstream_descriptions` to `True` in `generate_model_yaml` ([#61](https://github.com/dbt-labs/dbt-codegen/pull/61)) - Added `case_sensitive_cols` argument to `generate_base_model` macro ([#63](https://github.com/dbt-labs/dbt-codegen/pull/63)) - Add optional `name` arg to `generate_source` ([#64](https://github.com/dbt-labs/dbt-codegen/issues/64), [#66](https://github.com/dbt-labs/dbt-codegen/pull/66)) ## Fixes + - `generate_model_yaml` now correctly handles nested `STRUCT` fields in BigQuery ([#27](https://github.com/dbt-labs/dbt-codegen/issues/27), [#54](https://github.com/dbt-labs/dbt-codegen/pull/54)) ## Contributors: + - [@rahulj51](https://github.com/rahulj51) (#51) - [@bodschut](https://github.com/bodschut) (#54) - [@b-per](https://github.com/b-per) (#61) @@ -102,6 +141,7 @@ dispatch: This release creates breaking changes to the `generate_source.sql` macro. ## Features + - add optional `table_pattern` argument to `generate_source.sql` macro. Default value is '%' to pull all tables in the raw data schema to preserve existing behavior if the `table_pattern` argument is not specified by the user. # dbt-codegen v0.5.0 @@ -109,6 +149,7 @@ This release creates breaking changes to the `generate_source.sql` macro. This release supports any version (minor and patch) of v1, which means far less need for compatibility releases in the future. ## Under the hood + - Change `require-dbt-version` to `[">=1.0.0", "<2.0.0"]` - Bump dbt-utils dependency - Replace `source-paths` and `data-paths` with `model-paths` and `seed-paths` respectively @@ -116,22 +157,27 @@ This release supports any version (minor and patch) of v1, which means far less - Replace `dbt_modules` with `dbt_packages` in `clean-targets` # dbt-codegen v0.4.1 + 🚨 This is a compatibility release in preparation for `dbt-core` v1.0.0 (🎉). Projects using this version with `dbt-core` v1.0.x can expect to see a deprecation warning. This will be resolved in the next minor release. # dbt-codegen v0.4.0 ## Breaking changes + - Requires `dbt>=0.20.0` and `dbt-utils>=0.7.0` - Depends on `dbt-labs/dbt_utils` (instead of `fishtown-analytics/dbt_utils`) ## Features + - Add optional `leading_commas` arg to `generate_base_model` (#41 @jaypeedevlin) - Add optional `include_descriptions` arg to `generate_source` (#40 @djbelknapdbs) ## Fixes + - In the `generate_source` macro, use `dbt_utils.get_relations_by_pattern` instead of `get_relations_by_prefix`, since the latter will be deprecated in the future (#42) ## Under the hood + - Use new adapter.dispatch syntax (#44) # dbt-codegen v0.3.2 @@ -139,57 +185,73 @@ This release supports any version (minor and patch) of v1, which means far less This is a quality of life release ## Other -* Fix rendering issues on hub.getdbt.com -* Fix integration tests due to python version compatibility + +- Fix rendering issues on hub.getdbt.com +- Fix integration tests due to python version compatibility # dbt-codegen v0.3.1 + This is a bugfix release ## Fixes + - Use latest version of dbt-utils (0.6.2) to ensure generate_source_yaml works for non-target schemata (#34) # dbt-codegen v0.3.0 -## 🚨 Breaking change + +## 🚨 Breaking change + This release requires dbt v0.18.0, and dbt-utils v0.6.1. If you're not ready to upgrade, consider using a previous release of this package. ## Quality of life + - Use dbt v0.18.0 (#31) -- Fix README rendering on hub (#32 @calvingiles) +- Fix README rendering on hub (#32 @calvingiles) # dbt-codegen v0.2.0 + ## 🚨 Breaking change + The lower bound of `dbt-utils` is now `0.4.0`. This won't affect most users, since you're likely already using version of dbt-utils higher than this to achieve 0.17.0 compatibility. ## Quality of life: + - Change dbt-utils dependencies to `[>=0.4.0, <0.6.0]` (#29) - Fix tests (#29) # dbt-codegen v0.1.0 + ## 🚨 Breaking change! This package now requires dbt v0.17.x! ## Features: -* Add `generate_model_yaml` (#18 @jtalmi) +- Add `generate_model_yaml` (#18 @jtalmi) ## Under the hood: -* Update to v0.17.0, including `dbt_project.yml` version 2 syntax (#23) -* Add GitHub templates and installation instructions (#23) + +- Update to v0.17.0, including `dbt_project.yml` version 2 syntax (#23) +- Add GitHub templates and installation instructions (#23) ## Acknowledgements + @marzaccaro made a PR for `generate_model_yaml`, and, although I had reviewed it, I let the PR go stale and somehow completely forgot about it when merging PR #18 — this is completely my bad! So equal credit to @marzaccaro and @jtalmi for their work :clap: # dbt-codegen v0.0.4 + This is a bugfix release to improve compatibility with Snowflake # dbt-codegen v0.0.3 + Bump utils version range # dbt-codegen v0.0.2 + Small quality of life improvements # dbt-codegen v0.0.1 + Initial release diff --git a/README.md b/README.md index 29066e4..ae1d8e0 100644 --- a/README.md +++ b/README.md @@ -70,14 +70,13 @@ which you can then paste into a schema file. - `include_schema` (optional, default=False): Whether you want to add the schema to your source definition - `case_sensitive_databases` (optional, default=False): Whether you want database names to be -in lowercase, or to match the case in the source table + in lowercase, or to match the case in the source table — not compatible with Redshift - `case_sensitive_schemas` (optional, default=False): Whether you want schema names to be -in lowercase, or to match the case in the source table + in lowercase, or to match the case in the source table — not compatible with Redshift - `case_sensitive_tables` (optional, default=False): Whether you want table names to be -in lowercase, or to match the case in the source table + in lowercase, or to match the case in the source table — not compatible with Redshift - `case_sensitive_cols` (optional, default=False): Whether you want column names to be -in lowercase, or to match the case in the source table - + in lowercase, or to match the case in the source table ### Outputting to a file diff --git a/integration_tests/dbt_project.yml b/integration_tests/dbt_project.yml index 166f04e..6f194d0 100644 --- a/integration_tests/dbt_project.yml +++ b/integration_tests/dbt_project.yml @@ -18,13 +18,9 @@ clean-targets: seeds: +schema: raw_data +quote_columns: false - codegen_integration_tests: - data__Case_Sensitive: - +schema: Raw_Data_Case_Sensitive - quote_columns: true + vars: my_table_reference: table_c models: +bind: false - diff --git a/integration_tests/seeds/data__Case_Sensitive.csv b/integration_tests/seeds/data__Case_Sensitive.csv deleted file mode 100644 index d0f2e97..0000000 --- a/integration_tests/seeds/data__Case_Sensitive.csv +++ /dev/null @@ -1,3 +0,0 @@ -Col_A,Col_B -1,a -2,b diff --git a/integration_tests/tests/test_generate_source_case_sensitive.sql b/integration_tests/tests/test_generate_source_case_sensitive.sql deleted file mode 100644 index 48f3d69..0000000 --- a/integration_tests/tests/test_generate_source_case_sensitive.sql +++ /dev/null @@ -1,35 +0,0 @@ - -{% set raw_schema = generate_schema_name('Raw_Data_Case_Sensitive') %} - --- test default args -{% set actual_source_yaml = codegen.generate_source( - schema_name=raw_schema, - database_name=target.database, - generate_columns=True, - name=raw_schema, - include_database=True, - include_schema=True, - case_sensitive_databases=True, - case_sensitive_schemas=True, - case_sensitive_tables=True, - case_sensitive_cols=True -) %} - -{% set expected_source_yaml %} -version: 2 - -sources: - - name: codegen_integration_tests_postgres_raw_data_case_sensitive - database: circle_test - schema: codegen_integration_tests_postgres_Raw_Data_Case_Sensitive - tables: - - name: data__Case_Sensitive - columns: - - name: Col_A - data_type: integer - - name: Col_B - data_type: text -{% endset %} - - -{{ assert_equal (actual_source_yaml | trim, expected_source_yaml | trim) }} diff --git a/macros/generate_source.sql b/macros/generate_source.sql index f769911..b3790f2 100644 --- a/macros/generate_source.sql +++ b/macros/generate_source.sql @@ -1,5 +1,5 @@ {% macro get_tables_in_schema(schema_name, database_name=target.database, table_pattern='%', exclude='') %} - + {% set tables=dbt_utils.get_relations_by_pattern( schema_pattern=schema_name, database=database_name,