feat(datasets): Improved compatibility, functionality and testing for SnowflakeTableDataset

[tdhooghe]

Description

• Enable saving a Pandas DataFrame directly into a SnowflakeTable, i.e. to ingest a .csv directly into Snowflake within Kedro
• Update tests to use Snowpark's local testing framework
• Updated dependencies

Development notes

Skips pytest for Python version >3.11

Bump of cloudpickle required to allow for snowflake-snowpark-python >= 1.23

Checklist

• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added a description of this change in the relevant RELEASE.md file
• Added tests to cover my changes

[merelcht]

Left some minor comments, but otherwise this looks great! ⭐ Thank you so much @tdhooghe for getting this dataset in better shapes and adding tests that can now run as part of the test suite for Python versions < 3.12!

[merelcht]

I did some more testing, and it looks like you can replace this with snowpark = pytest.importorskip("snowpark") and it will pass the builds.

[tdhooghe]

No way, that's awesome! Let me test :)

[tdhooghe]

Doesn't help unfortunately, as it still includes the tests in the coverage..

[merelcht]

Yeah unfortunately coverage still needs to be skipped, but now at least the tests themselves pass. Before the build were showing:

__________ ERROR collecting tests/snowflake/test_snowpark_dataset.py ___________
ImportError while importing test module '/home/runner/work/kedro-plugins/kedro-plugins/kedro-datasets/tests/snowflake/test_snowpark_dataset.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/opt/hostedtoolcache/Python/3.12.7/x64/lib/python3.12/importlib/__init__.py:90: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/snowflake/test_snowpark_dataset.py:15: in <module>
    from snowflake.snowpark import DataFrame, Session
E   ModuleNotFoundError: No module named 'snowflake'
=========================== short test summary info ============================

[DimedS]

Many thanks for the PR, @tdhooghe! I really appreciate your proposal to allow saving Pandas DataFrames in the _save() method - it's a very useful feature. I implemented a similar approach in a custom version of the Snowpark dataset for an ETL project (https://github.com/DimedS/kedro-pypi-to-snowflake/tree/main), so I’m glad to see it being incorporated into the official dataset now! I leaved a one small proposal.

[DimedS]

Maybe we should add an additional check and raise an error here, like this:

# Check if the input is a Pandas DataFrame and convert it to Snowpark DataFrame
if isinstance(data, pd.DataFrame):
    # Convert the Pandas DataFrame to a Snowpark DataFrame
    snowpark_df = self._session.create_dataframe(data)
elif isinstance(data, DataFrame):
    # If it's already a Snowpark DataFrame, use it as is
    snowpark_df = data
else:
    raise DatasetError(f"Data of type {type(data)} is not supported for saving.")

This ensures we handle different types of DataFrames appropriately, as we're currently allowing not only Snowpark DataFrames but other DataFrame types as well.

[tdhooghe]

Thank you for your kind words @DimedS! 🙂

I like your proposal, I will include it!

[merelcht]

Left some final comments, but otherwise let's get this merged!