fix(datasets): Fix spark tests

[felipemonroy]

Description

Spark tests are failing randomly. This could be due that spark sessions are not properly closed, so some tests are using the one that do not correspond. See #755

Development notes

I replaced all SparkSession.builder.getOrCreate() calls for the fixture spark session, so we can control which spark session is used. Additionally, in the tear down of the spark session fixtures I added spark._instantiatedContext = None, which allegedly ensure that another spark session is created after (see here).

I am not 100% sure that this is going to fix the issue. Maybe we can rerun the actions a few time just to be sure.

Checklist

• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added a description of this change in the relevant RELEASE.md file
• Added tests to cover my changes

[ankatiyar]

Hey @felipemonroy, first of all, thank you so much for taking this on! 🙏🏾
I re-ran the unit tests and it failed with the same errors for the spark tests https://github.com/kedro-org/kedro-plugins/actions/runs/10023089947/job/27738930808
If you need any help, let us know

[astrojuanlu]

Should we add .enableHiveSupport(), like it's done in another fixture?