update rapids version for 24.10 release #1248
Conversation
Signed-off-by: liyuan <[email protected]>
|
@viadea please help review. CC @jayadeep-jayaraman @cjac |
|
Oh hey, thanks for the ping. I'll check it out. |
|
/gcbrun |
There was a problem hiding this comment.
Let's get the docs updated to reflect the versions of Dataproc supported in Q4 2024
Our current supported versions follow:
2.2-debian12
2.1-debian11
2.0-debian10
2.2-ubuntu22
2.2-ubuntu20
2.0-ubuntu18
2.2-rocky9
2.1-rocky8
2.0-rocky8
And if all of these platforms are not supported by spark-rapids, then we'll need to merge in my changes to dask-rapids.
spark-rapids/README.md
Outdated
| * NCCL 2.11.4+ | ||
| * Ubuntu 18.04, Ubuntu 20.04 or Rocky Linux 7, Rocky Linux8, Debian 10, Debian 11 | ||
| * Ubuntu 20.04, Ubuntu 22.04, CentOS 7, or Rocky Linux 8, Debian 10, Debian 11 |
There was a problem hiding this comment.
Do you have support for CentOS 7, stil, really? I commend you. Do the tests exercise that platform?
There was a problem hiding this comment.
I would also add Rocky Linux 9.
If you do not have support for rocky linux 9 yet, we can merge my work from spark-dask. I'll continue my review.
There was a problem hiding this comment.
Thanks for pointing it out, we don't support centos7 now, update a link in the doc for the soft/hardware requirements.
There was a problem hiding this comment.
File related pr for the doc issue.
spark-rapids/README.md
Outdated
| @@ -63,7 +63,7 @@ export CUDA_VER=11.5 | |||
|
|
|||
| gcloud dataproc clusters create $CLUSTER_NAME \ | |||
| --region $REGION \ | |||
| --image-version=2.0-ubuntu18 \ | |||
| --image-version=2.1-ubuntu20 \ | |||
There was a problem hiding this comment.
Please recommend 2.2 where possible. If 2.1 doesn't have other representation, then this is fine, but we should be emphasizing that 2.2 is the better choice.
There was a problem hiding this comment.
update to 2.2 image
Signed-off-by: liyuan <[email protected]>
Signed-off-by: liyuan <[email protected]>
This pr is to update the spark-rapids script version to 24.10.0 and update the readme doc