sdk/python/jobs/single-step/debug-and-monitor/debug-and-monitor.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Debug & monitor your training jobs\n",
    "\n",
    "**Requirements** - In order to benefit from this tutorial, you will need:\n",
    "- A basic understanding of Machine Learning\n",
    "- An Azure account with an active subscription. [Create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F)\n",
    "- An Azure ML workspace  - [Configure workspace](../../../configuration.ipynb) \n",
    "\n",
    "- A python environment\n",
    "- Installed Azure Machine Learning Python SDK v2 - [install instructions](../../../../README.md) - check the getting started section\n",
    "\n",
    "**Learning Objectives** - By the end of this tutorial, you should be able to:\n",
    "- Connect to your AML workspace from the Python SDK\n",
    "- Create and run a `Command` which executes a Python command\n",
    "- Use a local file as an `input` to the Command\n",
    "- Enable live debugging & monitoring by specifying `services` to the Command job\n",
    "\n",
    "**Motivations** - This notebook explains how to setup and run a Command. The Command is a fundamental construct of Azure Machine Learning. It can be used to run a task on a specified compute (either local or on the cloud). The Command accepts `environment` and `compute` to setup required infrastructure. Alternatively, [serverless compute (preview)](https://learn.microsoft.com/azure/machine-learning/how-to-use-serverless-compute?view=azureml-api-2&tabs=python) can also be used to run the training job. You can define a `command` to run on this infrastructure with `inputs`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Connect to Azure Machine Learning Workspace\n",
    "\n",
    "The [workspace](https://docs.microsoft.com/en-us/azure/machine-learning/concept-workspace) is the top-level resource for Azure Machine Learning, providing a centralized place to work with all the artifacts you create when you use Azure Machine Learning. In this section we will connect to the workspace in which the job will be run.\n",
    "\n",
    "## 1.1. Import the required libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1634854439863
    }
   },
   "outputs": [],
   "source": [
    "# import required libraries\n",
    "from azure.ai.ml import MLClient\n",
    "from azure.ai.ml import command\n",
    "from azure.identity import DefaultAzureCredential\n",
    "from azure.ai.ml.entities import JobService"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1634854439994
    }
   },
   "outputs": [],
   "source": [
    "# Enter details of your AML workspace\n",
    "subscription_id = \"<SUBSCRIPTION_ID>\"\n",
    "resource_group = \"<RESOURCE_GROUP>\"\n",
    "workspace = \"<AML_WORKSPACE_NAME>\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2. Configure workspace details and get a handle to the workspace\n",
    "\n",
    "To connect to a workspace, we need identifier parameters - a subscription, resource group and workspace name. We will use these details in the `MLClient` from `azure.ai.ml` to get a handle to the required Azure Machine Learning workspace. We use the default [default azure authentication](https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity.defaultazurecredential?view=azure-python) for this tutorial. Check the [configuration notebook](../../../configuration.ipynb) for more details on how to configure credentials and connect to a workspace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1634854440154
    }
   },
   "outputs": [],
   "source": [
    "# get a handle to the workspace\n",
    "ml_client = MLClient(\n",
    "    DefaultAzureCredential(), subscription_id, resource_group, workspace\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Configure and run the Command\n",
    "In this section we will configure and run a standalone job using the `command` class. The `command` class can be used to run standalone jobs and can also be used as a function inside pipelines.\n",
    "\n",
    "## 2.1 Configure the Command\n",
    "The `command` allows user to configure the following key aspects.\n",
    "- `code` - This is the path where the code to run the command is located\n",
    "- `command` - This is the command that needs to be run. If you need to reserve your cluster for debugging or monitoring purposes, you can use the `sleep` command.\n",
    "- `environment` - This is the environment needed for the command to run. Curated or custom environments from the workspace can be used. Or a custom environment can be created and used as well. Check out the [environment](../../../../assets/environment/environment.ipynb) notebook for more examples.\n",
    "- `compute` - The compute on which the command will run. In this example we are using [serverless compute (preview)](https://learn.microsoft.com/azure/machine-learning/how-to-use-serverless-compute?view=azureml-api-2&tabs=python) so there is no need to specify any compute. You can also replace serverless with any other compute in the workspace. You can run it on the local machine by using `local` for the compute. This will run the command on the local machine and all the run details and output of the job will be uploaded to the Azure ML workspace.\n",
    "- `display_name` - The display name of the Job\n",
    "- `description` - The description of the experiment\n",
    "- `services` - Specify the applications (or SSH) that you need to interact with the live running job. You can specify `vs_code`, `tensor_board` (needs log directory), `jupyter_lab` or `SSH` (needs public key). For distributed jobs, you can specify the specific compute `nodes` index you would like to interact with. If `nodes` are not specified, interactive services are enabled only on the head node by default."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1634854440714
    }
   },
   "outputs": [],
   "source": [
    "from azure.ai.ml.entities import (\n",
    "    JobService,\n",
    "    SshJobService,\n",
    "    VsCodeJobService,\n",
    "    TensorBoardJobService,\n",
    "    JupyterLabJobService,\n",
    ")\n",
    "\n",
    "# create the command\n",
    "job = command(\n",
    "    code=\"./src\",  # local path where the code is stored\n",
    "    command=\"python tfevents.py && sleep 2000\",  # the sleep command allows you to reserve your compute -- recommended if you are using interactive services\n",
    "    environment=\"AzureML-tensorflow-2.12-cuda11@latest\",\n",
    "    display_name=\"debug-and-monitor-example\",\n",
    "    services={\n",
    "        \"My_jupyterlab\": JupyterLabJobService(),\n",
    "        \"My_vscode\": VsCodeJobService(),\n",
    "        \"My_tensorboard\": TensorBoardJobService(\n",
    "            properties={\n",
    "                \"logDir\": \"outputs/tblogs\"  # relative path of Tensorboard logs (same as in your training script)\n",
    "            },\n",
    "        ),\n",
    "        # \"My_ssh\": SshJobService(\n",
    "        #     nodes=\"all\",  # For distributed jobs, use the `nodes` property to pick which node you want to enable interactive services on. If `nodes` are not selected, by default, interactive applications are only enabled on the head node. Values are \"all\", or compute node index (for ex. \"0\", \"1\" etc.)\n",
    "        #     ssh_public_keys=\"<add-public-key>\",\n",
    "        # ),\n",
    "    }\n",
    "    # experiment_name: debug-and-monitor-example\n",
    "    # description: Enable live debugging & monitoring for your training jobs\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2 Show running applications\n",
    "Using the `MLClient` created earlier, we will now run this Command as a job in the workspaces and view endpoints for the interactive services specified during job submission."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1634854446905
    }
   },
   "outputs": [],
   "source": [
    "# submit the command\n",
    "returned_job = ml_client.create_or_update(job)\n",
    "\n",
    "# view interactive service endpoints -- wait for job to be running before calling the below function\n",
    "# ml_client.jobs.show_services(returned_job.name, 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Next Steps\n",
    "You can see further examples of running a job [here](../../../single-step/). To learn more about debugging & monitoring your jobs, check out our [documentation](https://aka.ms/interactive_debugging)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "description": {
   "description": "Run a Command to train a basic neural network with TensorFlow on the MNIST dataset"
  },
  "kernel_info": {
   "name": "python3-azureml"
  },
  "kernelspec": {
   "display_name": "Python 3.10 - SDK V2",
   "language": "python",
   "name": "python310-sdkv2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  },
  "nteract": {
   "version": "nteract-front-end@1.0.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}