Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

generate_custom_image script doesn't work #77

Open
JSPromisel opened this issue Jan 31, 2023 · 0 comments
Open

generate_custom_image script doesn't work #77

JSPromisel opened this issue Jan 31, 2023 · 0 comments

Comments

@JSPromisel
Copy link

I tried running the following command:

python generate_custom_image.py --image-name some-name --dataproc-version 2.0.36-ubuntu18 --zone us-east1-c --disk-size 100 --metadata somekey=somevalue --customization-script /tmp/script.sh --gcs-bucket some-bucket

And got the following error:

INFO:custom_image_utils.args_inferer:Getting Dataproc base image name...
Traceback (most recent call last):
  File "generate_custom_image.py", line 95, in <module>
    main()
  File "generate_custom_image.py", line 86, in main
    args = parse_args(sys.argv[1:])
  File "generate_custom_image.py", line 57, in parse_args
    args_inferer.infer_args(args)
  File "/home/jakepromisel/Github/custom-images/custom_image_utils/args_inferer.py", line 225, in infer_args
    _infer_base_image(args)
  File "/home/jakepromisel/Github/custom-images/custom_image_utils/args_inferer.py", line 191, in _infer_base_image
    args.dataproc_version)
  File "/home/jakepromisel/Github/custom-images/custom_image_utils/args_inferer.py", line 175, in _get_dataproc_image_path_by_version
    "Cannot find dataproc base image with dataproc-version=%s." % version)
RuntimeError: Cannot find dataproc base image with dataproc-version=2.0.36-ubuntu18.

But I believe this version does exist. Looking into this further, I tried recreating the gcloud command run to verify the dataproc version:

gcloud compute images list --project cloud-dataproc --filter "labels.goog-dataproc-version = 2-0-36 AND NOT name ~ -eap$ AND status = READY" --format "csv[no-heading=true](name,labels.goog-dataproc-version)" --sort-by=~creationTimestamp

And I get the following warning:

WARNING: --filter : operator evaluation is changing for consistency across Google APIs.  labels.goog-dataproc-version=2-0-36 currently does not match but will match in the near future.  Run `gcloud topic filters` for details.

If this repository is no longer maintained and shouldn't be used, can someone point me to how I should go about creating a custom image? From what I can tell, the docs still say to use this script. I even tried on a vm where I installed the recommended GCloud version (181.0.0) which is from 2017 and it still didn't work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant