Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide users with an easy setup experience in a variety of execution contexts #19

Open
MrPowers opened this issue Jan 22, 2023 · 1 comment

Comments

@MrPowers
Copy link
Collaborator

The jodie project will be used in a variety of different ways. Some users will like to include jodie as a dependency in their build.sbt file and others will want to attach the JAR file directly to a cluster. Here are some cases we'd like to handle:

  • user adds jodie to their build.sbt file that specifies specific Delta Lake / Spark versions
  • user attaches a JAR file to an existing cluster
  • user adds jodie to their build.sbt file that specified a specific PySpark version, but doesn't have Delta Lake. The right Delta Lake version should be pulled in as a transitive dependency.

We should probably build an automated process that builds JAR files and attaches them to releases whenever a new release is created.

@brayanjuls
Copy link
Collaborator

brayanjuls commented Feb 2, 2023

user adds jodie to their build.sbt file that specified a specific PySpark version, but doesn't have Delta Lake. The right Delta Lake version should be pulled in as a transitive dependency.

Do you have a reference repo/doc where this is done?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants