Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added spark 3.4 to the project #346

Merged
merged 8 commits into from
May 16, 2023
Merged

Added spark 3.4 to the project #346

merged 8 commits into from
May 16, 2023

Conversation

alfonsorr
Copy link
Member

@alfonsorr alfonsorr commented Apr 17, 2023 •
edited by eruizalo
Loading

SPARK 3.4, what else

Description

This pr adds spark 3.4.0 to the CI and adds all needed changes to the schemas to use them in the project and tests.
To avoid problems with the new error messages in unfunded columns I've implemented a way to make it simple using ColumnNotFoundError builder

Related Issue and dependencies

How Has This Been Tested?

  • This pull request contains appropriate tests?:
    • YES

@github-actions github-actions bot added spark_3.0 PR changes to spark 3.0 spark_3.1 PR changes to spark 3.1 spark_3.2 PR changes to spark 3.2 spark_3.3 PR changes to spark 3.3 labels Apr 17, 2023
@alfonsorr
Copy link
Member Author

Missing documentation to make it final

@github-actions
Copy link

github-actions bot commented Apr 17, 2023 •
edited
Loading

:octocat: This is an auto-generated comment created by:

  • Date : 2023-05-16 08:21:02 +0000 (UTC)
  • Workflow : PR comment
  • Job name : create_test_summary_report
  • Run : 4989553426
  • Commit : a803eed Documentation error
Actor Triggering actor Sender
alfonsorr
alfonsorr 		alfonsorr
alfonsorr 		alfonsorr
alfonsorr
Triggered by:

Test summary report 📊

Spark version testing
2.4.1 627 passed, 2 skipped
2.4.2 627 passed, 2 skipped
2.4.3 627 passed, 2 skipped
2.4.4 627 passed, 2 skipped
2.4.5 627 passed, 2 skipped
2.4.6 628 passed, 2 skipped
2.4.7 628 passed, 2 skipped
2.4 628 passed, 2 skipped
3.0.0 678 passed, 2 skipped
3.0.1 678 passed, 2 skipped
3.0.2 678 passed, 2 skipped
3.0 678 passed, 2 skipped
3.1.0 706 passed, 2 skipped
3.1.1 706 passed, 2 skipped
3.1.2 706 passed, 2 skipped
3.1 706 passed, 2 skipped
3.2.0 710 passed, 2 skipped
3.2.1 710 passed, 2 skipped
3.2.2 710 passed, 2 skipped
3.2 710 passed, 2 skipped
3.3.0 710 passed, 2 skipped
3.3.1 710 passed, 2 skipped
3.3 710 passed, 2 skipped
3.4 710 passed, 2 skipped

@codecov
Copy link

codecov bot commented Apr 18, 2023 •
edited
Loading

Codecov Report

Merging #346 (a803eed) into main (619e228) will not change coverage.
The diff coverage is n/a.

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##             main     #346   +/-   ##
=======================================
  Coverage   97.61%   97.61%           
=======================================
  Files          62       62           
  Lines        1253     1253           
  Branches       15       15           
=======================================
  Hits         1223     1223           
  Misses         30       30
Flag Coverage Δ
spark-2.4.x 94.68% <ø> (ø)
spark-3.0.x 96.75% <ø> (ø)
spark-3.1.x 97.49% <ø> (ø)
spark-3.2.x 97.71% <ø> (ø)
spark-3.3.x 97.71% <ø> (ø)
spark-3.4.x 97.71% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...ore/src/main/scala/doric/syntax/ArrayColumns.scala 100.00% <ø> (ø)
....2_3.3_3.4/scala/doric/syntax/ArrayColumns3x.scala 100.00% <ø> (ø)
...2_3.3_3.4/scala/doric/syntax/CommonColumns3x.scala 100.00% <ø> (ø)
....1_3.2_3.3_3.4/scala/doric/syntax/DStructs3x.scala 100.00% <ø> (ø)
..._3.2_3.3_3.4/scala/doric/syntax/MapColumns3x.scala 100.00% <ø> (ø)
....2_3.3_3.4/scala/doric/syntax/StringColumn3x.scala 100.00% <ø> (ø)
..._3.4/scala/doric/syntax/AggregationColumns31.scala 100.00% <ø> (ø)
..._3.3_3.4/scala/doric/syntax/BooleanColumns31.scala 100.00% <ø> (ø)
..._3.3_3.4/scala/doric/syntax/NumericColumns31.scala 100.00% <ø> (ø)
...2_3.3_3.4/scala/doric/syntax/StringColumns31.scala 100.00% <ø> (ø)
... and 4 more

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 619e228...a803eed. Read the comment docs.

@github-actions github-actions bot added the CI/CD Continuous integration and continuous delivery label Apr 18, 2023
eruizalo
eruizalo requested changes Apr 18, 2023
View reviewed changes
build.sbt Show resolved Hide resolved
.github/labeler.yml Outdated Show resolved Hide resolved
.github/workflows/matrix_includes.json Show resolved Hide resolved
docs/docs/docs/quickstart.md Show resolved Hide resolved
@alfonsorr alfonsorr mentioned this pull request Apr 20, 2023
Closed
@alfonsorr alfonsorr requested a review from eruizalo April 26, 2023 08:26
@eruizalo eruizalo mentioned this pull request Apr 29, 2023
Open
17 tasks
@github-actions github-actions bot added the spark_2.4 PR changes to spark 2.4 label May 16, 2023
@eruizalo eruizalo marked this pull request as ready for review May 16, 2023 09:18
@eruizalo eruizalo requested a review from a team as a code owner May 16, 2023 09:18
@eruizalo eruizalo merged commit de16314 into hablapps:main May 16, 2023
@alfonsorr alfonsorr deleted the feature/spark_3.4 branch May 16, 2023 09:20
@eruizalo eruizalo added the spark_3.4 PR changes to spark 3.4 label May 16, 2023
@eruizalo eruizalo added the enhancement New feature or request label May 16, 2023
alfonsorr added a commit to alfonsorr/doric that referenced this pull request May 16, 2023
@alfonsorr @eruizalo
Added spark 3.4 to the project (hablapps#346) (issue hablapps#351)
8211ba5 
* Added spark 3.4 to the project

* Documentation and CI

* Update .github/labeler.yml

Co-authored-by: Eduardo Ruiz Alonso <[email protected]>

* Documentation and CI

* Latest rebase

* All "column not found" errors with builders

* Default spark version setted to 3.4

* Documentation error

---------

Co-authored-by: Eduardo Ruiz Alonso <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@eruizalo eruizalo eruizalo approved these changes

Assignees
No one assigned
Labels
CI/CD Continuous integration and continuous delivery enhancement New feature or request spark_2.4 PR changes to spark 2.4 spark_3.0 PR changes to spark 3.0 spark_3.1 PR changes to spark 3.1 spark_3.2 PR changes to spark 3.2 spark_3.3 PR changes to spark 3.3 spark_3.4 PR changes to spark 3.4
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Feature request]: Spark version 3.4.0
2 participants
@alfonsorr @eruizalo