Ray on EKS - Import Error due to outdated KubeRay Version

[Mageswaran1989]

Description

Please provide a clear and concise description of the issue you are encountering, and a reproduction of your configuration.

If your request is for a new feature, please use the Feature request template.

• ✋ I have searched the open/closed issues and my issue is not listed.

⚠️ Note

Before you submit an issue, please perform the following for Terraform examples:

1. Remove the local .terraform directory (! ONLY if state is stored remotely, which hopefully you are following that best practice!): rm -rf .terraform/
2. Re-initialize the project root to pull down modules: terraform init
3. Re-attempt your terraform plan or apply and check if the issue still persists

Versions

• Module version [Required]: v1.0.3

• Terraform version: v1.9.1

• Provider version(s): As dedfined in the repo : https://github.com/awslabs/data-on-eks/tree/main/ai-ml/ray/terraform

Reproduction Code [Required]

Follow the steps mentioned in offcial documentation: https://awslabs.github.io/data-on-eks/docs/blueprints/ai-ml/ray#ray-on-kubernetes

Steps to reproduce the behavior:

Follow the steps mentioned in Deploy Ray Clusters and Workloads -> XGBoost
Follow the steps mentioned in Deploy Ray Clusters and Workloads -> PyTorch

Expected behavior

Ray XGBoost example should run to its completion without any error.

Ray PyTorch Benchmark should run to its completion without any error.

Actual behavior

XGBoost: The provided example uses non existing xgboost benchmark example which is not present in current version of Ray

PyTorch: Getting following error on the ray driver, preventing the example from running:

1Cloning into 'ray'...2Updating files:  90% (6870/7605)
Updating files:  91% (6921/7605)
Updating files:  92% (6997/7605)
Updating files:  93% (7073/7605)
Updating files:  94% (7149/7605)
Updating files:  95% (7225/7605)
Updating files:  96% (7301/7605)
Updating files:  97% (7377/7605)
Updating files:  98% (7453/7605)
Updating files:  99% (7529/7605)
Updating files: 100% (7605/7605)
Updating files: 100% (7605/7605), done.3Traceback (most recent call last):4  File "ray/release/air_tests/air_benchmarks/workloads/tune_torch_benchmark.py", line 11, in <module>5    from ray.train import ScalingConfig6ImportError: cannot import name 'ScalingConfig' from 'ray.train' (/home/ray/anaconda3/lib/python3.7/site-packages/ray/train/__init__.py)7

Terminal Output Screenshot(s)

Additional context

[Mageswaran1989]

It worked by redeploying with updated Kuberay operator version to 1.1.1 @

• https://github.com/awslabs/data-on-eks/blob/main/ai-ml/ray/terraform/main.tf#L316
• https://github.com/awslabs/data-on-eks/blob/main/ai-ml/ray/terraform/modules/ray-cluster/variables.tf#L19

And rayproject/ray-ml to 2.31.0.deprecated-py311 from 2.3.0 @ https://github.com/awslabs/data-on-eks/blob/main/ai-ml/ray/terraform/examples/pytorch/main.tf#L53