Integrate Spark-Connect Tests & Update the Documentation

[nijanthanvijayakumar]

Proposed changes

1. Updates to the ci.yml:
   Invoke the tests using Spark-Connect & make

2. Updates to the CONTRIBUTING.md document.
   Brings the changes in from the main branch describing the pre-commit installation & setup, and auto-assigning the issues.

3. Refactor the following *.py files to work with the Spark-Connect tests and also update the tests accordingly
   a) dataframe_helpers.py
   b) functions.py
   c) transformations.py
   d) test_functions.py
   e) test_transformations.py

4. [Highlight]: Handling unsupported functions on Spark-Connect less than 3.5.2
   a) The functions array_choice and the sort_columns don't work properly on the Spark-Connect v3.5.2 and below. So, raising an exception if the Spark version is < 3.5.2 and if SPARK_CONNECT_MODE is enabled.
   b) Create a wrapper within the test_transformations.py, as the sort_columns function/method is being used widely across 10+ test cases. Using a wrapper would help reduce the duplicate lines of code.

Types of changes

What types of changes does your code introduce to Quinn?
Put an x in the boxes that apply

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation Update (if none of the other choices apply)

Further comments

N/A