-
Notifications
You must be signed in to change notification settings - Fork 693
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Added test for create-pytorchjob.ipynb python notebook #2274
Open
saileshd1402
wants to merge
18
commits into
kubeflow:master
Choose a base branch
from
saileshd1402:pytorch-job-notebook-test
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
18 commits
Select commit
Hold shift + click to select a range
0256146
Added test for create-pytorchjob.ipynb
saileshd1402 222bf0e
fix yaml syntax
saileshd1402 34be848
Fix uses path
saileshd1402 107e5ed
Add actions/checkout
saileshd1402 ec0f2d3
Add bash to action.yaml
saileshd1402 bb00e73
Install pip dependencies step
saileshd1402 21c32ee
Add quotes for args
saileshd1402 a7a15aa
Add jupyter
saileshd1402 95422f7
Add nbformat_minor: 5 to fix invalid format error
saileshd1402 2632690
Fix job name
saileshd1402 67a6b83
test papermill-args-yaml
saileshd1402 22ceaa4
testing multi line args
saileshd1402 32a4160
testing multi line args1
saileshd1402 1042ea2
testing multi line args2
saileshd1402 9e78f0e
testing multi line args3
saileshd1402 51b3f09
Parameterize sdk install
saileshd1402 7188ccc
Remove unnecessary output
saileshd1402 50e9ade
nbformat normailze
saileshd1402 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,70 @@ | ||
name: Notebook test template | ||
description: A composite action to setup and run example notebooks using Papermill | ||
|
||
inputs: | ||
kubernetes-version: | ||
required: true | ||
description: kubernetes version | ||
python-version: | ||
required: false | ||
description: Python version | ||
# Most latest supporting version | ||
default: "3.10" | ||
papermill-args-yaml: | ||
description: 'Additional arguments to pass to Papermill in yaml format' | ||
required: false | ||
default: "" | ||
notebook-input: | ||
description: 'Path to the input notebook' | ||
required: true | ||
notebook-output: | ||
description: 'Path to save the output notebook' | ||
required: true | ||
|
||
runs: | ||
using: composite | ||
steps: | ||
- name: Free-Up Disk Space | ||
uses: ./.github/workflows/free-up-disk-space | ||
|
||
- name: Setup Python | ||
uses: actions/setup-python@v5 | ||
with: | ||
python-version: ${{ inputs.python-version }} | ||
|
||
- name: Install Python dependencies | ||
shell: bash | ||
run: | | ||
pip install jupyter ipykernel papermill==2.2.0 | ||
|
||
- name: Create k8s Kind Cluster | ||
uses: helm/[email protected] | ||
with: | ||
node_image: kindest/node:${{ inputs.kubernetes-version }} | ||
cluster_name: training-operator-cluster | ||
kubectl_version: ${{ inputs.kubernetes-version }} | ||
|
||
- name: Build training-operator | ||
shell: bash | ||
run: | | ||
./scripts/gha/build-image.sh | ||
env: | ||
TRAINING_CI_IMAGE: kubeflowtraining/training-operator:test | ||
|
||
- name: Deploy training operator | ||
shell: bash | ||
run: | | ||
./scripts/gha/setup-training-operator.sh | ||
docker system prune -a -f | ||
docker system df | ||
df -h | ||
env: | ||
KIND_CLUSTER: training-operator-cluster | ||
TRAINING_CI_IMAGE: kubeflowtraining/training-operator:test | ||
GANG_SCHEDULER_NAME: "none" | ||
KUBERNETES_VERSION: ${{ inputs.kubernetes-version }} | ||
|
||
- name: Run Jupyter Notebook with Papermill | ||
shell: bash | ||
run: | | ||
papermill ${{ inputs.notebook-input }} ${{ inputs.notebook-output }} -p kubeflow_python_sdk "./sdk/python" --parameters_yaml "${{ inputs.papermill-args-yaml }}" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,31 @@ | ||
name: Test example notebooks | ||
|
||
on: | ||
- pull_request | ||
|
||
concurrency: | ||
group: ${{ github.workflow }}-${{ github.ref }} | ||
cancel-in-progress: true | ||
|
||
jobs: | ||
create-pytorchjob-notebook-test: | ||
runs-on: ubuntu-latest | ||
timeout-minutes: 30 | ||
strategy: | ||
fail-fast: false | ||
matrix: | ||
kubernetes-version: ["v1.28.7"] | ||
python-version: ["3.9", "3.10", "3.11"] | ||
steps: | ||
- name: Checkout | ||
uses: actions/checkout@v4 | ||
|
||
- name: Test Notebook | ||
uses: ./.github/workflows/template-notebook-test | ||
with: | ||
kubernetes-version: ${{ matrix.kubernetes-version }} | ||
python-version: ${{ matrix.python-version }} | ||
notebook-input: ./examples/pytorch/image-classification/create-pytorchjob.ipynb | ||
notebook-output: ./examples/pytorch/image-classification/create-pytorchjob-output.ipynb | ||
papermill-args-yaml: | | ||
namespace: default |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For papermill, all the parameters should be in one cell. But this seems to be affecting the readability of the user. How should we improve this?
cc: @andreyvelich @tenzen-y @Electronic-Waste
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@akshaychitneni @shravan-achar Any thoughts on papermill usage ?
We can pass parameters using
parameters_yaml
only to the single Notebook cell ?