Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[OCI] Support OCI Object Storage #4501

Open
wants to merge 17 commits into
base: master
Choose a base branch
from
Open

Conversation

HysunHe
Copy link
Contributor

@HysunHe HysunHe commented Dec 21, 2024

This is the implementation of the OCI Object Storage store (Re-submit).

It is an usually solid requirement for those using SkyPilot with OCI, otherwise user has to enable other supported cloud (with buckets) to leverage its cloud storage for data sync, for example, sky serve require cloud storage to sync workdir.

Actually I have implemented and used the OCI Object Storage with SkyPilot quite a long time (mainly for POC projects), now submit the PR to merge the code to the repo because there are requirements from other users.

For documentation to add OCI store, we can do it later after this PR.

sky launch -y -c smoke_test --cloud oci --num-nodes 2 examples/oci/oci-mounts.yaml

Tested (run the relevant ones):

  • Code formatting: bash format.sh
  • Any manual or new tests for this PR (please specify below)
  • All smoke tests: pytest tests/test_smoke.py
  • Relevant individual smoke tests: pytest tests/test_smoke.py::test_oci_mounts
  • Backward compatibility tests: conda deactivate; bash -i tests/backward_compatibility_tests.sh

@HysunHe
Copy link
Contributor Author

HysunHe commented Dec 22, 2024

@Michaelvll @cblmemo Hi Zhanghao & Tianxia, Ask for your support to review/someone else to review the PR. Thanks.

@Michaelvll Michaelvll requested a review from cblmemo December 22, 2024 06:51
Copy link
Collaborator

@cblmemo cblmemo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this amazing work @HysunHe ! Left some discussions :))

examples/oci/dataset-mount.yaml Outdated Show resolved Hide resolved
examples/oci/dataset-mount.yaml Show resolved Hide resolved
examples/oci/dataset-mount.yaml Outdated Show resolved Hide resolved
examples/oci/dataset-upload-and-mount.yaml Outdated Show resolved Hide resolved
sky/adaptors/oci.py Outdated Show resolved Hide resolved
sky/data/storage.py Outdated Show resolved Hide resolved
sky/data/storage.py Outdated Show resolved Hide resolved
sky/data/storage.py Outdated Show resolved Hide resolved
sky/data/storage.py Outdated Show resolved Hide resolved
sky/data/storage.py Show resolved Hide resolved
@HysunHe
Copy link
Contributor Author

HysunHe commented Dec 24, 2024

@cblmemo Thanks for the prompt review. All points are addressed.

@HysunHe HysunHe requested a review from cblmemo December 24, 2024 09:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants