Merge pull request #220 from DataBiosphere/dev

PR for 0.4.4 release

README.md

@@ -1,5 +1,5 @@
 # dsub: simple batch jobs with Docker
-[![License](https://img.shields.io/badge/license-Apache%202.0-brightgreen.svg)](https://github.com/DataBiosphere/dsub/blob/master/LICENSE)
+[![License](https://img.shields.io/badge/license-Apache%202.0-brightgreen.svg)](https://github.com/DataBiosphere/dsub/blob/main/LICENSE)
 
 ## Overview
 
@@ -52,7 +52,7 @@ To deactivate the virtual environment in your shell, run the command:
 
 Alternatively, a set of convenience scripts are provided that activate the
 virutalenv before calling `dsub`, `dstat`, and `ddel`. They are in the
-[bin](https://github.com/DataBiosphere/dsub/tree/master/bin) directory. You can
+[bin](https://github.com/DataBiosphere/dsub/tree/main/bin) directory. You can
 use these scripts if you don't want to activate the virtualenv explicitly in
 your shell.
 
@@ -102,7 +102,7 @@ Choose one of the following:
 ### Makefile
 
 After cloning the dsub repo, you can also use the
-[Makefile](https://github.com/DataBiosphere/dsub/blob/master/Makefile)
+[Makefile](https://github.com/DataBiosphere/dsub/blob/main/Makefile)
 by running:
 
         make
@@ -243,7 +243,7 @@ implements a consistent runtime environment. The current providers are:
 - google-cls-v2 (*new*)
 
 More details on the runtime environment implemented by the backend providers
-can be found in [dsub backend providers](https://github.com/DataBiosphere/dsub/blob/master/docs/providers/README.md).
+can be found in [dsub backend providers](https://github.com/DataBiosphere/dsub/blob/main/docs/providers/README.md).
 
 ### Differences between `google-v2` and `google-cls-v2`
 
@@ -315,7 +315,7 @@ Note: your `--image` must include the
 
 For more information on using the
 `--image` flag, see the
-[image section in Scripts, Commands, and Docker](https://github.com/DataBiosphere/dsub/blob/master/docs/code.md#--image-docker-image)
+[image section in Scripts, Commands, and Docker](https://github.com/DataBiosphere/dsub/blob/main/docs/code.md#--image-docker-image)
 
 ### Passing parameters to your script
 
@@ -335,7 +335,7 @@ environment variable, as `${MESSAGE}`.
 **Be sure to enclose your command string in single quotes and not double
 quotes. If you use double quotes, the command will be expanded in your local
 shell before being passed to dsub. For more information on using the
-`--command` flag, see [Scripts, Commands, and Docker](https://github.com/DataBiosphere/dsub/blob/master/docs/code.md)**
+`--command` flag, see [Scripts, Commands, and Docker](https://github.com/DataBiosphere/dsub/blob/main/docs/code.md)**
 
 To set multiple environment variables, you can repeat the flag:
 
@@ -356,7 +356,7 @@ the cloud storage bucket path. Paths can be:
 * folder paths like `gs://my-bucket/my-folder`
 * wildcard paths like `gs://my-bucket/my-folder/*`
 
-See the [inputs and outputs](https://github.com/DataBiosphere/dsub/blob/master/docs/input_output.md)
+See the [inputs and outputs](https://github.com/DataBiosphere/dsub/blob/main/docs/input_output.md)
 documentation for more details.
 
 ### Transferring input files to a Google Cloud Storage bucket.
@@ -484,7 +484,7 @@ your local machine.
 `dsub` tasks run using the `google`, `google-v2`, or `google-cls-v2` providers can take advantage
 of a wide range of CPU, RAM, disk, and hardware accelerator (eg. GPU) options.
 
-See the [Compute Resources](https://github.com/DataBiosphere/dsub/blob/master/docs/compute_resources.md)
+See the [Compute Resources](https://github.com/DataBiosphere/dsub/blob/main/docs/compute_resources.md)
 documentation for details.
 
 ### Submitting a batch job
@@ -549,17 +549,17 @@ The task range values can take any of the following forms:
 ### Logging
 
 The `--logging` flag points to a location for `dsub` task log files. For details
-on how to specify your logging path, see [Logging](https://github.com/DataBiosphere/dsub/blob/master/docs/logging.md).
+on how to specify your logging path, see [Logging](https://github.com/DataBiosphere/dsub/blob/main/docs/logging.md).
 
 ### Job control
 
 It's possible to wait for a job to complete before starting another.
-For details, see [job control with dsub](https://github.com/DataBiosphere/dsub/blob/master/docs/job_control.md).
+For details, see [job control with dsub](https://github.com/DataBiosphere/dsub/blob/main/docs/job_control.md).
 
 ### Retries
 
 It is possible for `dsub` to automatically retry failed tasks.
-For details, see [retries with dsub](https://github.com/DataBiosphere/dsub/blob/master/docs/retries.md).
+For details, see [retries with dsub](https://github.com/DataBiosphere/dsub/blob/main/docs/retries.md).
 
 ### Labeling jobs and tasks
 
@@ -568,7 +568,7 @@ cancel tasks using your own identifiers. In addition, with the Google
 providers, labeling a task will label associated compute resources such as
 virtual machines and disks.
 
-For more details, see [Checking Status and Troubleshooting Jobs](https://github.com/DataBiosphere/dsub/blob/master/docs/troubleshooting.md)
+For more details, see [Checking Status and Troubleshooting Jobs](https://github.com/DataBiosphere/dsub/blob/main/docs/troubleshooting.md)
 
 ### Viewing job status
 
@@ -599,12 +599,12 @@ each job includes:
     gets a sequential value of the form "task-*n*" where *n* is 1-based.
 
 Note that the job metadata values will be modified to conform with the "Label
-Restrictions" listed in the [Checking Status and Troubleshooting Jobs](https://github.com/DataBiosphere/dsub/blob/master/docs/troubleshooting.md)
+Restrictions" listed in the [Checking Status and Troubleshooting Jobs](https://github.com/DataBiosphere/dsub/blob/main/docs/troubleshooting.md)
 guide.
 
 Metadata can be used to cancel a job or individual tasks within a batch job.
 
-For more details, see [Checking Status and Troubleshooting Jobs](https://github.com/DataBiosphere/dsub/blob/master/docs/troubleshooting.md)
+For more details, see [Checking Status and Troubleshooting Jobs](https://github.com/DataBiosphere/dsub/blob/main/docs/troubleshooting.md)
 
 #### Summarizing job status
 
@@ -667,8 +667,7 @@ The image below illustrates this:
 
 ![Pipelines Runner Architecture](./docs/images/pipelines_runner_architecture.png)
 
-By default, `dsub` will use the [default Compute Engine service account]
-(https://cloud.google.com/compute/docs/access/service-accounts#default_service_account)
+By default, `dsub` will use the [default Compute Engine service account](https://cloud.google.com/compute/docs/access/service-accounts#default_service_account)
 as the authorized service account on the VM instance. You can choose to specify
 the email address of another service acount using `--service-account`.
 
@@ -716,19 +715,19 @@ of the service account will be `[email protected]`.
 
 *   See the examples:
 
-    *   [Custom scripts](https://github.com/DataBiosphere/dsub/tree/master/examples/custom_scripts)
-    *   [Decompress files](https://github.com/DataBiosphere/dsub/tree/master/examples/decompress)
-    *   [FastQC](https://github.com/DataBiosphere/dsub/tree/master/examples/fastqc)
-    *   [Samtools index](https://github.com/DataBiosphere/dsub/tree/master/examples/samtools)
+    *   [Custom scripts](https://github.com/DataBiosphere/dsub/tree/main/examples/custom_scripts)
+    *   [Decompress files](https://github.com/DataBiosphere/dsub/tree/main/examples/decompress)
+    *   [FastQC](https://github.com/DataBiosphere/dsub/tree/main/examples/fastqc)
+    *   [Samtools index](https://github.com/DataBiosphere/dsub/tree/main/examples/samtools)
 
 *   See more documentation for:
 
-    *   [Scripts, Commands, and Docker](https://github.com/DataBiosphere/dsub/blob/master/docs/code.md)
-    *   [Input and Output File Handling](https://github.com/DataBiosphere/dsub/blob/master/docs/input_output.md)
-    *   [Logging](https://github.com/DataBiosphere/dsub/blob/master/docs/logging.md)
-    *   [Compute Resources](https://github.com/DataBiosphere/dsub/blob/master/docs/compute_resources.md)
-    *   [Compute Quotas](https://github.com/DataBiosphere/dsub/blob/master/docs/compute_quotas.md)
-    *   [Job Control](https://github.com/DataBiosphere/dsub/blob/master/docs/job_control.md)
-    *   [Retries](https://github.com/DataBiosphere/dsub/blob/master/docs/retries.md)
-    *   [Checking Status and Troubleshooting Jobs](https://github.com/DataBiosphere/dsub/blob/master/docs/troubleshooting.md)
-    *   [Backend providers](https://github.com/DataBiosphere/dsub/blob/master/docs/providers/README.md)
+    *   [Scripts, Commands, and Docker](https://github.com/DataBiosphere/dsub/blob/main/docs/code.md)
+    *   [Input and Output File Handling](https://github.com/DataBiosphere/dsub/blob/main/docs/input_output.md)
+    *   [Logging](https://github.com/DataBiosphere/dsub/blob/main/docs/logging.md)
+    *   [Compute Resources](https://github.com/DataBiosphere/dsub/blob/main/docs/compute_resources.md)
+    *   [Compute Quotas](https://github.com/DataBiosphere/dsub/blob/main/docs/compute_quotas.md)
+    *   [Job Control](https://github.com/DataBiosphere/dsub/blob/main/docs/job_control.md)
+    *   [Retries](https://github.com/DataBiosphere/dsub/blob/main/docs/retries.md)
+    *   [Checking Status and Troubleshooting Jobs](https://github.com/DataBiosphere/dsub/blob/main/docs/troubleshooting.md)
+    *   [Backend providers](https://github.com/DataBiosphere/dsub/blob/main/docs/providers/README.md)

docs/code.md

@@ -3,7 +3,7 @@
 `dsub` provides a few choices for how to get your code into the Docker
 container in order to run:
 
-* --command "shell string"
+* --command 'shell string'
 * --script "script file (Bash, Python, etc.)"
 * --image "Docker image"
 * --input "path to file in cloud storage"

docs/compute_quotas.md

@@ -126,7 +126,7 @@ Most commonly, the quotas relevant for `dsub` tasks are:
 > **_NOTE:_** To eliminate dependence on the `In-use IP addresses` quota,
 > the Google providers support the `--use-private-address` flag.
 > See the `Public IP addresses` section of
-> [Compute Resources](https://github.com/DataBiosphere/dsub/blob/master/docs/compute_resources.md).
+> [Compute Resources](https://github.com/DataBiosphere/dsub/blob/main/docs/compute_resources.md).
 
 
 ## Troubleshooting Quota issues

docs/compute_resources.md

@@ -92,7 +92,7 @@ For more information on Compute Engine IP addresses, see:
 Running the job on VMs without a public IP address has the advantage that it
 does not consume `In-use IP addresses` quota, which can otherwise limit your
 ability to scale up your concurrently running tasks.
-See the [Compute Quotas](https://github.com/DataBiosphere/dsub/blob/master/docs/compute_quotas.md)
+See the [Compute Quotas](https://github.com/DataBiosphere/dsub/blob/main/docs/compute_quotas.md)
 documentation for more details.
 
 Running jobs on VMs without a public IP address requires the following:

docs/providers/README.md

@@ -341,6 +341,9 @@ The following `dsub` parameters are specific to the `google-v2` and
     - `--use-private-address`:
       - If set to true, do not attach a public IP address to the VM.
         (default: False)
+    - `--block-external-network`:
+      - If set to true, prevents the container for the user's script/command
+        from accessing the external network. (default: False)
 
 - Per-task compute resources
     - `--boot-disk-size`:

docs/troubleshooting.md

@@ -348,7 +348,7 @@ $ dstat \
 ## Viewing logs
 
 Each `dsub` task produces log files whose destination is determined by the `--logging` flag.
-See [Logging](https://github.com/DataBiosphere/dsub/blob/master/docs/logging.md)
+See [Logging](https://github.com/DataBiosphere/dsub/blob/main/docs/logging.md)
 for more information.
 
 ## SSH to the VM

dsub/_dsub_version.py

@@ -26,4 +26,4 @@
   0.1.3.dev0 -> 0.1.3 -> 0.1.4.dev0 -> ...
 """
 
-DSUB_VERSION = '0.4.3'
+DSUB_VERSION = '0.4.4'

dsub/commands/dsub.py

@@ -564,6 +564,13 @@ def _parse_arguments(prog, argv):
       action='store_true',
       help="""If set to true, enables Stackdriver monitoring on the VM.
               (default: False)""")
+  google_common.add_argument(
+      '--block-external-network',
+      default=False,
+      action='store_true',
+      help="""If set to true, prevents the container for the user's
+          script/command from accessing the external network.
+          (default: False)""")
 
   google_cls_v2 = parser.add_argument_group(
       title='"google-cls-v2" provider options',
@@ -633,7 +640,8 @@ def _get_job_resources(args):
       ssh=args.ssh,
       enable_stackdriver_monitoring=args.enable_stackdriver_monitoring,
       max_retries=args.retries,
-      max_preemptible_attempts=args.preemptible)
+      max_preemptible_attempts=args.preemptible,
+      block_external_network=args.block_external_network)
 
 
 def _get_job_metadata(provider, user_id, job_name, script, task_ids,

dsub/lib/job_model.py

@@ -436,6 +436,7 @@ class Resources(
         'enable_stackdriver_monitoring',
         'max_retries',
         'max_preemptible_attempts',
+        'block_external_network',
     ])):
   """Job resource parameters related to CPUs, memory, and disk.
 
@@ -473,6 +474,8 @@ class Resources(
     max_preemptible_attempts (param_util.PreemptibleParam): Int representing
       maximum allowed number of attempts on a preemptible machine, or boolean
       representing always preemtible.
+    block_external_network (bool): Prevents the containers from accessing the
+      external network.
   """
   __slots__ = ()
 
@@ -503,7 +506,8 @@ def __new__(cls,
               ssh=None,
               enable_stackdriver_monitoring=None,
               max_retries=None,
-              max_preemptible_attempts=None):
+              max_preemptible_attempts=None,
+              block_external_network=None):
     return super(Resources,
                  cls).__new__(cls, min_cores, min_ram, machine_type, disk_size,
                               disk_type, boot_disk_size, preemptible, image,
@@ -512,7 +516,8 @@ def __new__(cls,
                               subnetwork, use_private_address, accelerator_type,
                               accelerator_count, nvidia_driver_version, timeout,
                               log_interval, ssh, enable_stackdriver_monitoring,
-                              max_retries, max_preemptible_attempts)
+                              max_retries, max_preemptible_attempts,
+                              block_external_network)
 
 
 def ensure_job_params_are_complete(job_params):

dsub/providers/google_base.py

@@ -21,16 +21,15 @@
 import re
 import warnings
 
+import google.auth
+from google.oauth2 import service_account
 import googleapiclient.discovery
 import googleapiclient.errors
 from ..lib import job_model
 from ..lib import retry_util
 import pytz
 import tenacity
 
-import google.auth
-from google.oauth2 import service_account
-
 
 # The google v1 provider directly added the bigquery scope, but the v1alpha2
 # API automatically added:
@@ -86,12 +85,18 @@
     'asia-northeast2-a',
     'asia-northeast2-b',
     'asia-northeast2-c',
+    'asia-northeast3-a',
+    'asia-northeast3-b',
+    'asia-northeast3-c',
     'asia-south1-a',
     'asia-south1-b',
     'asia-south1-c',
     'asia-southeast1-a',
     'asia-southeast1-b',
     'asia-southeast1-c',
+    'asia-southeast2-a',
+    'asia-southeast2-b',
+    'asia-southeast2-c',
     'australia-southeast1-a',
     'australia-southeast1-b',
     'australia-southeast1-c',
@@ -135,6 +140,12 @@
     'us-west2-a',
     'us-west2-b',
     'us-west2-c',
+    'us-west3-a',
+    'us-west3-b',
+    'us-west3-c',
+    'us-west4-a',
+    'us-west4-b',
+    'us-west4-c',
 ]
 
 

dsub/providers/google_v2_base.py

@@ -824,6 +824,7 @@ def _build_pipeline_request(self, task_view):
         google_v2_pipelines.build_action(
             name='user-command',
             pid_namespace=pid_namespace,
+            block_external_network=job_resources.block_external_network,
             image_uri=job_resources.image,
             mounts=[mnt_datadisk] + persistent_disk_mounts,
             environment=user_environment,
@@ -1510,6 +1511,10 @@ def get_field(self, field, default=None):
       # The ssh flag is determined by if an action named 'ssh' exists.
       value['ssh'] = self._is_ssh_enabled(self._op)
 
+      value[
+          'block-external-network'] = google_v2_operations.external_network_blocked(
+              self._op)
+
       # The VM instance name and zone can be found in the WorkerAssignedEvent.
       # For a given operation, this may have occurred multiple times, so be
       # sure to grab the most recent.

dsub/providers/google_v2_operations.py

@@ -158,6 +158,21 @@ def get_last_event(op):
   return None
 
 
+def external_network_blocked(op):
+  """Retun True if the blockExternalNetwork flag is set for the user action."""
+  user_action = get_action_by_name(op, 'user-command')
+  if user_action:
+    if _API_VERSION == google_v2_versions.V2ALPHA1:
+      flags = user_action.get('flags')
+      if flags:
+        return 'BLOCK_EXTERNAL_NETWORK' in flags
+    elif _API_VERSION == google_v2_versions.V2BETA:
+      return user_action.get('blockExternalNetwork')
+    else:
+      assert False, 'Unexpected version: {}'.format(_API_VERSION)
+  return False
+
+
 def is_unexpected_exit_status_event(e):
   """Retun True if the event is for an unexpected exit status."""
   if _API_VERSION == google_v2_versions.V2ALPHA1:

dsub/providers/google_v2_pipelines.py

@@ -166,7 +166,8 @@ def build_action(name=None,
                  labels=None,
                  always_run=None,
                  enable_fuse=None,
-                 run_in_background=None):
+                 run_in_background=None,
+                 block_external_network=None):
   """Build an Action object for a Pipeline request.
 
   Args:
@@ -183,6 +184,8 @@ def build_action(name=None,
     always_run (bool): Action must run even if pipeline has already failed.
     enable_fuse (bool): Enable access to the FUSE device for this action.
     run_in_background (bool): Allow the action to run in the background.
+    block_external_network (bool): Prevents the container from accessing the
+      external network.
 
   Returns:
     An object representing an Action resource.
@@ -210,6 +213,8 @@ def build_action(name=None,
       flags.append('ENABLE_FUSE')
     if run_in_background:
       flags.append('RUN_IN_BACKGROUND')
+    if block_external_network:
+      flags.append('BLOCK_EXTERNAL_NETWORK')
 
     if flags:
       action['flags'] = flags
@@ -221,6 +226,7 @@ def build_action(name=None,
     action['alwaysRun'] = always_run
     action['enableFuse'] = enable_fuse
     action['runInBackground'] = run_in_background
+    action['blockExternalNetwork'] = block_external_network
 
   else:
     assert False, 'Unexpected version: {}'.format(_API_VERSION)