Merge pull request #127 from fivetran/release/v0.15.0

Release/v0.15.0
fivetran · Nov 15, 2023 · baf5228 · baf5228
2 parents 6b1900d + 40089fb
commit baf5228
Show file tree

Hide file tree

Showing 12 changed files with 101 additions and 11 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,3 +1,28 @@
+# dbt_hubspot v0.15.0
+
+[PR #127](https://github.com/fivetran/dbt_hubspot/pull/127) includes the following updates:
+
+## Bug fixes
+- Updated variables used to determine if engagements are enabled in `hubspot__contacts` to also check variable `hubspot_engagement_contact_enabled`.
+
+## Features
+- The following changes stem from changes made in the source package. See the [source package v0.14.0 CHANGELOG entry](https://github.com/fivetran/dbt_hubspot_source/blob/main/CHANGELOG.md#dbt_hubspot_source-v0140) for more details. 
+
+- Added the following staging models, along with documentation and tests:
+  - `stg_hubspot__property`
+  - `stg_hubspot__property_option`
+  - These tables can be disabled by setting `hubspot_property_enabled: False` in your dbt_project.yml vars. See [Step 4 of the README](https://github.com/fivetran/dbt_hubspot#step-4-disable-models-for-non-existent-sources) for more details. 
+
+- When including a passthrough `property_hs_*` column, you now have the option to include the corresponding, human-readable labels. 
+  - The above-mentioned `property` tables are required for this feature. If you do not have them and have to disable them, unfortunately you will not be able to use this feature.
+  - See the [Adding property label section](https://github.com/fivetran/dbt_hubspot#adding-property-label) of the README for more details on how to enable this feature! 
+  - We recommend being selective with the label columns you add. As you add more label columns, your run time will increase due to the underlying logic requirements.
+  - This update applies to models:
+    - `hubspot__companies`
+    - `hubspot__contacts`
+    - `hubspot__deals`
+    - `hubspot__tickets` 
+
 # dbt_hubspot v0.14.0
 
 ## 🚨 Breaking Changes 🚨

diff --git a/README.md b/README.md
@@ -73,7 +73,7 @@ Include the following hubspot package version in your `packages.yml` file:
 ```yaml
 packages:
   - package: fivetran/hubspot
-    version: [">=0.14.0", "<0.15.0"] # we recommend using ranges to capture non-breaking changes automatically
+    version: [">=0.15.0", "<0.16.0"] # we recommend using ranges to capture non-breaking changes automatically
 
 ```
 Do **NOT** include the `hubspot_source` package in this file. The transformation package itself has a dependency on it and will install the source package as well.
@@ -144,6 +144,7 @@ vars:
   hubspot_engagement_note_enabled: false
   hubspot_engagement_task_enabled: false
   hubspot_owner_enabled: false
+  hubspot_property_enabled: false                         # Disables property and property_option tables
   
   # Service
   hubspot_service_enabled: true                           # Enables all service/ticket models. Default = false
@@ -198,6 +199,33 @@ vars:
 vars:
   hubspot__pass_through_all_columns: true # default is false
 ```
+### Adding property label
+For `property_hs_*` columns, you can enable the corresponding, human-readable `property_option`.`label` to be included in the staging models. 
+
+#### Important! 
+- You must have sources `property` and `property_option` enabled to enable labels. By default, these sources are enabled.  
+- You CANNOT enable labels if using `hubspot__pass_through_all_columns: true`.
+- We recommend being selective with the label columns you add. As you add more label columns, your run time will increase due to the underlying logic requirements.
+
+To enable labels for a given property, set the property attribute `add_property_label: true`, using the below format.
+
+```yml
+vars:
+  hubspot__ticket_pass_through_columns:
+    - name: "property_hs_fieldname"
+      alias: "fieldname"
+      add_property_label: true
+```
+  Alternatively, you can enable labels for all passthrough properties by using variable `hubspot__enable_all_property_labels: true`, formatted like the below example. 
+
+```yml
+vars:
+  hubspot__enable_all_property_labels: true
+  hubspot__ticket_pass_through_columns:
+    - name: "property_hs_fieldname1"
+    - name: "property_hs_fieldname2"
+```
+
 ### Including calculated fields
 This package also provides the ability to pass calculated fields through to the `company`, `contact`, `deal`, and `ticket` staging models. If you would like to add a calculated field to any of the mentioned staging models, you may configure the respective `hubspot__[table_name]_calculated_fields` variables with the `name` of the field you would like to create, and the `transform_sql` which will be the actual calculation that will make up the calculated field.
 ```yml
@@ -286,7 +314,7 @@ This dbt package is dependent on the following dbt packages. Please be aware tha
 ```yml
 packages:
     - package: fivetran/hubspot_source
-      version: [">=0.13.0", "<0.14.0"]
+      version: [">=0.14.0", "<0.15.0"]
 
     - package: fivetran/fivetran_utils
       version: [">=0.4.0", "<0.5.0"]

diff --git a/dbt_project.yml b/dbt_project.yml
@@ -1,5 +1,5 @@
 name: 'hubspot'
-version: '0.14.0'
+version: '0.15.0'
 
 config-version: 2
 require-dbt-version: [">=1.3.0", "<2.0.0"]

diff --git a/docs/catalog.json b/docs/catalog.json
diff --git a/docs/manifest.json b/docs/manifest.json
diff --git a/docs/run_results.json b/docs/run_results.json
diff --git a/integration_tests/dbt_project.yml b/integration_tests/dbt_project.yml
@@ -1,12 +1,16 @@
 name: 'hubspot_integration_tests'
-version: '0.14.0'
+version: '0.15.0'
 
 profile: 'integration_tests'
 config-version: 2
 vars:
   hubspot_schema: hubspot_integration_tests_8
   hubspot_source:
     hubspot_service_enabled: true
+    # hubspot_sales_enabled: true # enable when generating docs
+    # hubspot_marketing_enabled: true # enable when generating docs
+    # hubspot_contact_merge_audit_enabled: true # enable when generating docs
+    # hubspot_using_all_email_events: true # enable when generating docs
     hubspot_company_property_history_identifier: "company_property_history_data"
     hubspot_company_identifier: "company_data"
     hubspot_contact_identifier: "contact_data"
@@ -41,6 +45,8 @@ vars:
     hubspot_engagement_note_identifier: "engagement_note_data"
     hubspot_engagement_task_identifier: "engagement_task_data"
     hubspot_owner_identifier: "owner_data"
+    hubspot_property_identifier: "property_data"
+    hubspot_property_option_identifier: "property_option_data"
     hubspot_ticket_company_identifier: "ticket_company_data"
     hubspot_ticket_contact_identifier: "ticket_contact_data"
     hubspot_ticket_deal_identifier: "ticket_deal_data"

diff --git a/integration_tests/requirements.txt b/integration_tests/requirements.txt
@@ -4,5 +4,4 @@ dbt-redshift>=1.3.0,<2.0.0
 dbt-postgres>=1.3.0,<2.0.0
 dbt-spark>=1.3.0,<2.0.0
 dbt-spark[PyHive]>=1.3.0,<2.0.0
-dbt-databricks>=1.3.0,<2.0.0
-oscrypto @ git+https://github.com/wbond/oscrypto.git@d5f3437
+dbt-databricks>=1.3.0,<2.0.0
diff --git a/integration_tests/seeds/property_data.csv b/integration_tests/seeds/property_data.csv
@@ -0,0 +1,11 @@
+_fivetran_id,_fivetran_synced,calculated,created_at,description,field_type,group_name,hubspot_defined,hubspot_object,label,name,show_currency_symbol,type,updated_at
++mlGrLvmij8zluyeRByh7DPbXpM=,2023-11-11 08:18:16.519000 UTC,false,2023-05-03 08:51:08.199000 UTC,M0xKTEL9t51+vD5ztRfm+A==,date,fVQYOcf70/KsHU/KFoXQfQ==,,contact,test_pro,test_pro,false,date,2023-05-03 08:51:08.199000 UTC
+6fnvV+pkZR3di3USRclW2NZMNTA=,2023-11-11 08:18:15.151000 UTC,false,2019-08-06 02:41:52.364000 UTC,k+VMRhjE/W4QB470Jor1yA==,date,j8mgV9/MYzXvxQKD3c0AHw==,true,deal,Close Date,closedate,,datetime,2023-06-20 22:32:28.368000 UTC
+5yTEFwFlLPTKlzXc3PviX5DMdJ4=,2023-11-11 08:18:16.569000 UTC,false,2019-08-06 02:41:09.071000 UTC,nRGOJjIZrs4d0WJLkC15fg==,date,fVQYOcf70/KsHU/KFoXQfQ==,true,contact,Close Date,closedate,,datetime,2022-05-27 20:35:26.640000 UTC
+u0bk1tHdJJir2SmRkwo1r22t2PM=,2023-11-11 08:18:15.837000 UTC,false,2019-08-06 02:41:44.665000 UTC,HRIf2XFKVmDQxMzI2w3mcA==,date,XArNQdwSO+DY7119vj+JEw==,true,company,Close Date,closedate,,datetime,2022-05-27 22:49:16.893000 UTC
+RTJ5KTwQe/zMYl8uKwNFlOeNWUY=,2023-11-11 08:18:15.792000 UTC,false,2022-11-30 10:52:48.277000 UTC,1B2M2Y8AsgTpgAmY7PhCfg==,date,XArNQdwSO+DY7119vj+JEw==,,company,test_date,test_date,false,date,2022-11-30 10:52:48.277000 UTC
+sv5Aa2A1+wJwoKRX4Zfe3CTEHEE=,2023-11-11 08:18:16.575000 UTC,false,2019-08-06 02:41:09.148000 UTC,jjSyQlM9DMhpVol12tp4fg==,date,fVQYOcf70/KsHU/KFoXQfQ==,true,contact,Create Date,createdate,,datetime,2023-05-01 20:02:33.973000 UTC
+GjTSotYsbt+HoRY5e6bHRgNJYFI=,2023-11-11 08:18:36.368000 UTC,false,2019-08-06 02:42:48.261000 UTC,nBljQWslMNrEDwXeYmRYyg==,date,0NoL94xOqMRlkFF+GAX6QQ==,true,product,Create Date,createdate,,datetime,2022-05-28 01:26:21.260000 UTC
+6iFNLSeY0Fz5DgwskGjrMoW3JIg=,2023-11-11 08:18:13.861000 UTC,false,2019-08-06 02:42:38.781000 UTC,lMRanmvFHjNC0pQdw79H5Q==,date,kdJdJeqzIqBPlKTSRZoyGQ==,true,line_item,Create Date,createdate,,datetime,2022-05-28 01:31:47.067000 UTC
+onbRYvHwxXS3NwSP2zaoD+Mv/us=,2023-11-11 08:18:36.567000 UTC,false,2019-08-06 02:43:00.503000 UTC,MD3INB3ab4k2LqaTom0eEA==,date,CfZkcOOMcc/dg5a9ofwkUg==,true,ticket,Create date,createdate,,datetime,2022-08-18 12:55:35.731000 UTC
+XhKQ/eq5pqJEB6smEKCPSSyfUH4=,2023-11-11 08:18:15.836000 UTC,false,2019-08-06 02:41:44.631000 UTC,wyvwVnWdqsdNRapzG8Y/RA==,date,XArNQdwSO+DY7119vj+JEw==,true,company,Create Date,createdate,,datetime,2023-05-01 19:33:57.873000 UTC
diff --git a/integration_tests/seeds/property_option_data.csv b/integration_tests/seeds/property_option_data.csv
@@ -0,0 +1,21 @@
+label,property_id,_fivetran_synced,display_order,hidden,value
+psScnliy99RZj01H2CO0Bg==,QbL3/uGeJqHy1PwdKiAryA==,2023-07-06 05:18:21.205000 UTC,0,false,Rk7Ftpi8GPQtaS7MlroMGQ==
+pSz+WBU1TmUa4LuUa1rgkQ==,QbL3/uGeJqHy1PwdKiAryA==,2023-07-06 05:18:21.205000 UTC,1,false,LA3JfM9iZ4SdcrsahteEeA==
+AvzvMEDzIxXmOQHpmJkR2Q==,QbL3/uGeJqHy1PwdKiAryA==,2023-07-06 05:18:21.205000 UTC,2,false,T9S3rtCg0J+FcavCHGDYtw==
+82p3OmCpiuf6fXiV9CNUGg==,stMBaoy9Irlfo81ajZF/Ww==,2023-07-06 05:18:20.949000 UTC,54,false,MpeVBohL7jkWOlWD1n2MEQ==
+iXyRZw2q18uicoYZGNbgag==,stMBaoy9Irlfo81ajZF/Ww==,2023-07-06 05:18:20.949000 UTC,69,false,ag2ORssJEO3Ix9VleEtdQg==
+kNCldBZipThyNGTWrrfg2A==,stMBaoy9Irlfo81ajZF/Ww==,2023-07-06 05:18:20.949000 UTC,61,false,DCbZyLGAXmlPv5n4SfYx9Q==
+nXBeZz/aTEAYfzniBd/75g==,stMBaoy9Irlfo81ajZF/Ww==,2023-07-06 05:18:20.949000 UTC,53,false,isnpiTWMqdJGp0MdQexQlA==
+Y6UJ4WGHacm+BqSY+9LIWw==,stMBaoy9Irlfo81ajZF/Ww==,2023-07-06 05:18:20.949000 UTC,68,false,NxElfDRGA3UVvLxOm2zBuA==
+9/yByDltisVJ/ItQKfpDug==,stMBaoy9Irlfo81ajZF/Ww==,2023-07-06 05:18:20.949000 UTC,58,false,vIGBU9BcEMJOtKde0VQiqw==
+olS9+/n99uTy3AIPobhBFQ==,stMBaoy9Irlfo81ajZF/Ww==,2023-07-06 05:18:20.949000 UTC,73,false,0TcZBF4uKrmsLk3qXWb/Hg==
+B1Y77Uvfwx7eFghmY0fMnA==,stMBaoy9Irlfo81ajZF/Ww==,2023-07-06 05:18:20.949000 UTC,64,false,/cjAina18SVT3IObiSTw3w==
+K9C6Ej/GCH0SeJeYNHAilQ==,stMBaoy9Irlfo81ajZF/Ww==,2023-07-06 05:18:20.949000 UTC,55,false,1m8HK+BNhT7R+gsuVesdxw==
+pywmmY+CL/LxU1GOsa6inw==,stMBaoy9Irlfo81ajZF/Ww==,2023-07-06 05:18:20.949000 UTC,63,false,kqfrguMbRr20bZ8iN/xUww==
+0ZCS71MB2VSJEfh6gX+MRA==,stMBaoy9Irlfo81ajZF/Ww==,2023-07-06 05:18:20.949000 UTC,59,false,/6WH59+ebbdk/66q3ur5Cw==
+o9hmABcYPHTCe5qrVu1lIQ==,stMBaoy9Irlfo81ajZF/Ww==,2023-07-06 05:18:20.949000 UTC,75,false,JUbNYa3Fg/uz4J33ml2QoQ==
+du4+7EMJLifwRGQjGBJNbA==,stMBaoy9Irlfo81ajZF/Ww==,2023-07-06 05:18:20.949000 UTC,62,false,oGDrsJou/ln6KJivl+ruWQ==
+pTcKx6PCI9FNH+BrqtoqDg==,stMBaoy9Irlfo81ajZF/Ww==,2023-07-06 05:18:20.949000 UTC,60,false,+rN2xb29RGurpWYBfnHC5g==
+OHV/B1A6v1fd9MipdPIUKQ==,stMBaoy9Irlfo81ajZF/Ww==,2023-07-06 05:18:20.949000 UTC,70,false,jDwQNnk3/oWaB7pqsOXBLA==
+rrQoVUoKzGzFA6Lk2mrmGg==,stMBaoy9Irlfo81ajZF/Ww==,2023-07-06 05:18:20.949000 UTC,76,false,IrC/vHIdyaGW7KTyFPL+LA==
+6v8b3yT8/+DhTimhv/UaEg==,stMBaoy9Irlfo81ajZF/Ww==,2023-07-06 05:18:20.949000 UTC,65,false,Z/qUdomTCUc+d/5MgyA10g==
diff --git a/models/marketing/hubspot__contacts.sql b/models/marketing/hubspot__contacts.sql
@@ -1,6 +1,6 @@
 {{ config(enabled=fivetran_utils.enabled_vars(['hubspot_marketing_enabled', 'hubspot_contact_enabled'])) }}
 {% set emails_enabled = fivetran_utils.enabled_vars(['hubspot_marketing_enabled', 'hubspot_email_event_enabled']) %}
-{% set engagements_enabled = fivetran_utils.enabled_vars(['hubspot_sales_enabled', 'hubspot_engagement_enabled']) %}
+{% set engagements_enabled = fivetran_utils.enabled_vars(['hubspot_sales_enabled', 'hubspot_engagement_enabled','hubspot_engagement_contact_enabled']) %}
 
 with contacts as (
 

diff --git a/packages.yml b/packages.yml
@@ -1,3 +1,3 @@
 packages:
   - package: fivetran/hubspot_source
-    version: [">=0.13.0", "<0.14.0"]
+    version: [">=0.14.0", "<0.15.0"]