Skip to content

Commit

Permalink
Fix tests
Browse files Browse the repository at this point in the history
1. Fix redundant space for spark <3.3
2. Fix uncompativility of Poetry 1.6 and Python 3.7

 Changes to be committed:
	modified:   .github/workflows/ci.yml
	modified:   README.md
	modified:   tests/test_dataframe_validator.py
  • Loading branch information
SemyonSinchenko committed Aug 31, 2023
1 parent 9ae44d1 commit 41a3707
Show file tree
Hide file tree
Showing 3 changed files with 13 additions and 4 deletions.
10 changes: 10 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,18 @@ jobs:
with:
python-version: ${{ steps.python_version.outputs.value }}

- name: Get supported Poetry version
uses: haya14busa/action-cond@v1
id: poetry_version
with:
cond: ${{ startsWith(matrix.pyspark-version, '2.') }}
if_true: '1.5' # latest supported version for PySpark 2.x
if_false: '1.6' # PySpark 3+

- name: Install Poetry
uses: snok/install-poetry@v1
with:
version: ${{ steps.poetry_version.outputs.value }}

- name: Cache Poetry virtualenv
uses: actions/cache@v1
Expand Down
3 changes: 1 addition & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
# Quinn

![![image](https://github.com/MrPowers/quinn/workflows/build/badge.svg)](https://github.com/MrPowers/quinn/actions/workflows/ci.yml/badge.svg)
<!-- ![![image](https://github.com/MrPowers/mack/workflows/build/badge.svg)](https://github.com/MrPowers/quinn/actions/workflows/flake8.yml/badge.svg) -->
[![Linter: Ruff](https://img.shields.io/badge/Linter-Ruff-brightgreen?style=flat-square)](https://github.com/MrPowers/quinn/actions/workflows/ruff.yml/badge.svg)
[![image](https://github.com/MrPowers/quinn/workflows/build/badge.svg)](https://github.com/MrPowers/quinn/actions/workflows/ruff.yml/badge.svg)
![PyPI - Downloads](https://img.shields.io/pypi/dm/quinn)
[![PyPI version](https://badge.fury.io/py/quinn.svg)](https://badge.fury.io/py/quinn)

Expand Down
4 changes: 2 additions & 2 deletions tests/test_dataframe_validator.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,10 +37,10 @@ def it_raises_when_struct_field_is_missing1(spark):
with pytest.raises(quinn.DataFrameMissingStructFieldError) as excinfo:
quinn.validate_schema(source_df, required_schema)

if semver.compare(spark.version, "3.2.0") >= 0: # Spark 3.3+
if semver.compare(spark.version, "3.3.0") >= 0: # Spark 3.3+
expected_error_message = "The [StructField('city', StringType(), True)] StructFields are not included in the DataFrame with the following StructFields StructType([StructField('name', StringType(), True), StructField('age', LongType(), True)])" # noqa
else:
expected_error_message = "The [StructField(city,StringType,true)] StructFields are not included in the DataFrame with the following StructFields StructType(List(StructField(name,StringType,true), StructField(age,LongType,true)))" # noqa
expected_error_message = "The [StructField(city,StringType,true)] StructFields are not included in the DataFrame with the following StructFields StructType(List(StructField(name,StringType,true),StructField(age,LongType,true)))" # noqa
assert excinfo.value.args[0] == expected_error_message

def it_does_nothing_when_the_schema_matches(spark):
Expand Down

0 comments on commit 41a3707

Please sign in to comment.