Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Behavior for external path #823

Merged
merged 7 commits into from
Oct 11, 2024
Merged

Behavior for external path #823

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
4d09cca
add option to scope path
benc-db Oct 9, 2024
04ecadf
try more in tests
benc-db Oct 10, 2024
90d5fd5
remove unneeded line
benc-db Oct 11, 2024
b2393d2
fix unit tests
benc-db Oct 11, 2024
0fdd3c6
fix linting
benc-db Oct 11, 2024
887bf96
changelog
benc-db Oct 11, 2024
7c8d5b7
Merge branch '1.9.latest' into behavior_for_external_path
benc-db Oct 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions dbt/adapters/databricks/impl.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,7 @@ class DatabricksConfig(AdapterConfig):
file_format: str = "delta"
table_format: TableFormat = TableFormat.DEFAULT
location_root: Optional[str] = None
include_full_name_in_path: bool = False
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can it be moved to the last argument to be backward compatible if someone calls w/o named arguments

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Normally I would agree with you, however...

Configs are never invoked with named arguments, as they are not actually exposed to users. Instead, users provide config via yaml, which dbt bundles together into a generic config object, which gets cast for type-checking purposes. This process is generic, i.e. dbt is also not invoking DatabricksConfig with named arguments. The primary reason to add fields here is to get code completion support, since basically all config provided by user (through yaml file) ends up available on the config object, and if you add the field here, then your IDE knows it will be on the object. Given that, it is more valuable to keep config fields that are related close together than to worry about ordering.

partition_by: Optional[Union[List[str], str]] = None
clustered_by: Optional[Union[List[str], str]] = None
liquid_clustered_by: Optional[Union[List[str], str]] = None
Expand Down Expand Up @@ -206,6 +207,25 @@ def update_tblproperties_for_iceberg(
result["delta.universalFormat.enabledFormats"] = "iceberg"
return result

@available.parse(lambda *a, **k: 0)
def compute_external_path(
self, config: BaseConfig, model: BaseConfig, is_incremental: bool = False
) -> str:
location_root = config.get("location_root")
database = model.get("database", "hive_metastore")
schema = model.get("schema", "default")
identifier = model.get("alias")
if location_root is None:
raise DbtConfigError("location_root is required for external tables.")
include_full_name_in_path = config.get("include_full_name_in_path", False)
if include_full_name_in_path:
path = os.path.join(location_root, database, schema, identifier)
else:
path = os.path.join(location_root, identifier)
if is_incremental:
path = path + "_tmp"
return path

# override/overload
def acquire_connection(
self, name: Optional[str] = None, query_header_context: Any = None
Expand Down
7 changes: 2 additions & 5 deletions dbt/include/databricks/macros/adapters/python.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,11 +60,8 @@ writer.saveAsTable("{{ target_relation }}")
{%- set buckets = config.get('buckets', validator=validation.any[int]) -%}
.format("{{ file_format }}")
{%- if location_root is not none %}
{%- set identifier = model['alias'] %}
{%- if is_incremental() %}
{%- set identifier = identifier + '__dbt_tmp' %}
{%- endif %}
.option("path", "{{ location_root }}/{{ identifier }}")
{%- set model_path = adapter.compute_external_path(config, model, is_incremental()) %}
.option("path", "{{ model_path }}")
{%- endif -%}
{%- if partition_by is not none -%}
{%- if partition_by is string -%}
Expand Down
3 changes: 2 additions & 1 deletion dbt/include/databricks/macros/relations/location.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,8 @@
{%- set file_format = config.get('file_format', default='delta') -%}
{%- set identifier = model['alias'] -%}
{%- if location_root is not none %}
location '{{ location_root }}/{{ identifier }}'
{%- set model_path = adapter.compute_external_path(config, model, is_incremental()) %}
location '{{ model_path }}'
{%- elif (not relation.is_hive_metastore()) and file_format != 'delta' -%}
{{ exceptions.raise_compiler_error(
'Incompatible configuration: `location_root` must be set when using a non-delta file format with Unity Catalog'
Expand Down
2 changes: 2 additions & 0 deletions tests/functional/adapter/basic/test_incremental.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ def project_config_update(self):
"models": {
"+file_format": "parquet",
"+location_root": f"{location_root}/parquet",
"+include_full_name_in_path": "true",
"+incremental_strategy": "append",
},
}
Expand Down Expand Up @@ -61,6 +62,7 @@ def project_config_update(self):
"models": {
"+file_format": "csv",
"+location_root": f"{location_root}/csv",
"+include_full_name_in_path": "true",
"+incremental_strategy": "append",
},
}
Expand Down
3 changes: 3 additions & 0 deletions tests/functional/adapter/incremental/test_incremental_strategies.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ def project_config_update(self):
"models": {
"+file_format": "parquet",
"+location_root": f"{location_root}/parquet_append",
"+include_full_name_in_path": "true",
"+incremental_strategy": "append",
},
}
Expand Down Expand Up @@ -129,6 +130,7 @@ def project_config_update(self):
"models": {
"+file_format": "parquet",
"+location_root": f"{location_root}/parquet_insert_overwrite",
"+include_full_name_in_path": "true",
"+incremental_strategy": "insert_overwrite",
},
}
Expand All @@ -144,6 +146,7 @@ def project_config_update(self):
"models": {
"+file_format": "parquet",
"+location_root": f"{location_root}/parquet_insert_overwrite_partitions",
"+include_full_name_in_path": "true",
"+incremental_strategy": "insert_overwrite",
"+partition_by": "id",
},
Expand Down
2 changes: 2 additions & 0 deletions tests/functional/adapter/persist_docs/fixtures.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
description: 'A seed description'
config:
location_root: '{{ env_var("DBT_DATABRICKS_LOCATION_ROOT") }}'
include_full_name_in_path: true
persist_docs:
relation: True
columns: True
Expand All @@ -22,6 +23,7 @@
description: 'A seed description'
config:
location_root: '/mnt/dbt_databricks/seeds'
include_full_name_in_path: true
persist_docs:
relation: True
columns: True
Expand Down
3 changes: 2 additions & 1 deletion tests/functional/adapter/python_model/fixtures.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,8 @@ def model(dbt, spark):
marterialized: table
tags: ["python"]
create_notebook: true
location_root: "{root}/{schema}"
include_full_name_in_path: true
location_root: "{{ env_var('DBT_DATABRICKS_LOCATION_ROOT') }}"
columns:
- name: date
tests:
Expand Down
9 changes: 0 additions & 9 deletions tests/functional/adapter/python_model/test_python_model.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,15 +119,6 @@ def project_config_update(self):
}

def test_expected_handling_of_complex_config(self, project):
unformatted_schema_yml = util.read_file("models", "schema.yml")
util.write_file(
unformatted_schema_yml.replace(
"root", os.environ["DBT_DATABRICKS_LOCATION_ROOT"]
).replace("{schema}", project.test_schema),
"models",
"schema.yml",
)

util.run_dbt(["seed"])
util.run_dbt(["build", "-s", "complex_config"])
util.run_dbt(["build", "-s", "complex_config"])
Expand Down
Loading