Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

nodeSelector set by default requires tpu provisioner #619

Open
samos123 opened this issue Aug 2, 2024 · 0 comments · May be fixed by #620
Open

nodeSelector set by default requires tpu provisioner #619

samos123 opened this issue Aug 2, 2024 · 0 comments · May be fixed by #620

Comments

@samos123
Copy link
Contributor

samos123 commented Aug 2, 2024

These are the nodeSelectors that got added:

Node-Selectors:              cloud.google.com/gke-accelerator-count=4
                             cloud.google.com/gke-spot=true
                             cloud.google.com/gke-tpu-accelerator=tpu-v5-lite-podslice
                             cloud.google.com/gke-tpu-topology=16x16
                             provisioner-nodepool-id=stoelinga-8733bd

This was my launch job:

export BASTION_TIER=1
axlearn gcp gke start --instance_type=tpu-v5litepod-256 --num_replicas=1 \
        --cluster=v5e-256-bodaborg-us-west4 --bundler_spec=allow_dirty=True \
        --bundler_type=artifactregistry --bundler_spec=image=tpu \
        --bundler_spec=dockerfile=Dockerfile --bundler_spec=target=tpu \
        -- python3 -c "'import jax; print(jax.devices())'"

Expectation:
The job should not have this selector provisioner-nodepool-id=stoelinga-8733bd since that assumes the tpu provisioner is always used. This may not be the case for external users.
samos123 added a commit to samos123/axlearn that referenced this issue Aug 2, 2024
@samos123
fix apple#619 remove nodeSelector provisioner-nodepool
8a3ff39
samos123 added a commit to samos123/axlearn that referenced this issue Aug 2, 2024
@samos123
fix apple#619 remove nodeSelector provisioner-nodepool
b81e427
@samos123 samos123 linked a pull request Aug 3, 2024 that will close this issue
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

remove nodeSelector provisioner-nodepool samos123/axlearn
1 participant
@samos123