Skip to content

Custom spark 3.5.0 image built with spark connect giving serialization error with rapids libs #10611

Answered by tgravescs
kkdatta asked this question in Q&A
Discussion options

You must be logged in to vote

we haven't tested with Spark connect but we have seen some a similar issue with our jar that has multiple shims have classloader issues.

I would suggest rebuilding spark-rapids for just the single spark release (3.5.0) that you are using to see if it fixes this issue:
https://github.com/NVIDIA/spark-rapids/blob/branch-24.02/CONTRIBUTING.md#building-a-distribution-for-a-single-spark-release

Please let us know if this works for you.

Replies: 2 comments 9 replies

Comment options

You must be logged in to vote
3 replies
@kkdatta
Comment options

@kkdatta
Comment options

@tgravescs
Comment options

Answer selected by kkdatta
Comment options

You must be logged in to vote
6 replies
@KevinLu233
Comment options

@jlowe
Comment options

@jlowe
Comment options

@KevinLu233
Comment options

@KevinLu233
Comment options

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
Q&A
Labels
None yet
5 participants