Release 0.4.4

This release includes:

• dsub
  • Implement --block-external-network to support network-sandboxed user action.
  • setup.py: Update dsub dependent libraries to pick up newer versions.
  • Check for python or python3 executable in local runner script. Also fix error handling for the first "write_event".
  • Add zones for Seoul, Jakarta, Salt Lake City, and Las Vegas
• Documentation
  • Rename references to the master branch to the main branch.

Release 0.4.3

This release includes:

• dsub

  • Update parsing of worker assigned event text to account for changes in Pipelines API.
  • Hide the --nvidia-driver-version flag as it is now ignored by the Pipelines API (since Sept 2020).

• Documentation

  • Fix link to accelerator API doc in README.
  • Fix documentation that assumed TMPDIR is /tmp.
  • Mention SSH from the browser in troubleshooting docs.

• Tests

  • Add e2e test that confirms GPU is installed when --accelerator-type and --accelerator-count is used.
  • Removed infrequently used environment variables, CHECK_RESULTS_ONLY and ALLOW_DIRTY_TESTS, from dsub tests.

Release 0.4.2

Release 0.4.2 of dsub ends support Python 3.5, which reached its "end of life" at the end of September, 2020.

The last version of dsub that supports Python 3.5 is 0.4.1.
Please use Python 3.6 or greater.

This release includes:

• Python code health
  • Remove uses of future from dsub
  • Remove six and its usages from dsub
  • Explicitly support Python 3.6 and up.
• Feature updates
  • Improvements to dstat output
  • Use "tenacity" library instead of "retrying" library for API retries.
  • Add a get_credentials function that Python clients of dsub, dstat, ddel can override for non-standard runtime environments.
• google-cls-v2 provider updates:
  • Use batch endpoint in google-cls-v2 provider for job deletion (ddel).
  • google-cls-v2: Use the batch endpoint only for --location us-central1.

Release 0.4.1

This release includes one change to support users of Python 3.5:

• Remove a use of f-strings, which was introduced in Python 3.6.

Release 0.4.0

Release 0.4.0 completes the sunsetting of Python 2 support for dsub.
The last version of dsub that supports Python 2 is 0.3.10.

This release also adds a WARNING when the --image flag is omitted from a call to dsub.
The default image is available as a getting started convenience, but for ongoing reproducible workflows, the image should be specified by the caller.
The current default is ubuntu:14.04 which reached End Of Life in April 2019. The default image will change in future releases and it is likely to be changed on a semi-regular basis, as popular base Docker linux images change.

---

This release includes:

• dsub
  • Update setup.py in dsub to be Python 3 only.
  • Lint dsub source files as Python3 only. Fix a few lint warnings.
  • Emit warning if default image is used.
  • Print full path of exceptions that are retried.
  • Print retry errors for socket timeout error.
  • Add socket.timeout exceptions to the retry list.
  • Fix markdown formatting in dsub README

Release 0.3.10

This release includes:

• dsub
  • Update Makefile to use Python3 venv
  • Add documentation around Compute Engine Quotas.
  • Have dsub output job-name and user-id in addition to job-id prior to launching job.
  • Fix for --users '*'
  • Update httplib2 for dsub to 0.18.1
  • Retry transient http error codes when checking GCS
  • Fix for yaml 5.3 where timestamps are already loaded as timezone aware.
  • Improve performance of GCS output file checks for --skip

Release 0.3.9

This release includes:

• dsub
  • Update version of cloudsdk docker image and revert workaround for gsutil/gcloud auth token bug on GCE, which should now be fixed in the updated image.
  • Update google-auth to 1.18.0 and pin google-api-core to 1.21.0
  • Remove leading characters that are not a letter, number, or underscore when auto-creating a job-name from a command string.
  • Move google_v2 arguments to google_common in the --help text.
  • Add a section in the documentation for Google provider-specific command-line flags.
  • Remove support/tests for legacy local provider job metadata file
• Testing updates:
  • Re-enable the test e2e_errors.sh for all providers.
  • Add unit test for retrying BrokenPipeError
  • Fix ResourceWarning when running python unit tests.

Release 0.3.8

This release includes:

• dsub
  • Remove the google provider and its documentation.
  • Document the existence of the google-cls-v2 provider
  • Document using venv for installation
  • Add a flag --credentials-file to pass service account credentials to the provider.
• google-v2 provider updates:
  • Add --ssh to dstat.
• google-cls-v2 provider updates:
  • Add default locations for google-cls-v2 in dstat and ddel.

Release 0.3.7

This release includes:

• dsub
  • (New) (Experimental) Implemented google-cls-v2 provider that passes all tests.
  • Pin all dsub dependencies to a max version
  • Fix broken urls in dsub docs.
  • Add dsub --summary output in wait_and_retry loop.
• google-v2 provider updates:
  • Enable a shared PID namespace when --ssh is specified.
  • Also retry broken pipe errors
  • Setting --preemptible 0 should not cause an error.
• Testing
  • Change Travis Python3 version from 3.8 to 3.7.
  • Remove sorting_util_test.

Release 0.3.6

This release includes:

• dsub
  • Add periodic status update in output (via --summary flag).
  • Update help text to clarify that timeout has a default of 7 days.
  • Replace apiclient with googleapiclient.
  • Emit a message to make it more clear that the dsub process must continue running for retries.
  • Add missing quotes in documentation in example for --mount.
• google-v2 provider updates:
  • Update gsutil rsync warning message to include command being run.
  • Add workaround for gsutil/gcloud bug that prevents re-authentication.