Platform_Introduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/platform_introduction_native_app_header.png\">\n",
    "\n",
    "## Requirements\n",
    "- Install the [Globus Python SDK](https://globus-sdk-python.readthedocs.io/en/stable/installation.html).\n",
    "- Join the [Tutorial Users Group](https://app.globus.org/groups/50b6a29c-63ac-11e4-8062-22000ab68755) in order to experiment with sharing/guest collections.\n",
    "\n",
    "## Globus SDK References\n",
    "- Source code: https://github.com/globus/globus-sdk-python\n",
    "- Documentation: https://globus-sdk-python.readthedocs.io/en/stable/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json  # just so we can pretty-print response data\n",
    "import globus_sdk\n",
    "\n",
    "# Feel free to replace the collection UUIDs below with those of your own collections\n",
    "tutorial_collection_1 = \"6c54cade-bde5-45c1-bdea-f4bd71dba2cc\"  # collection \"Globus Tutorial Collection 1\"\n",
    "tutorial_collection_2 = \"31ce9ba0-176d-45a5-add3-f37d233ba47d\"  # collection \"Globus Tutorial Collection 2\"\n",
    "tutorial_users_group = \"50b6a29c-63ac-11e4-8062-22000ab68755\"  # group \"Tutorial Users\"\n",
    "\n",
    "# This is the Tutorial GCS endpoint, which will be used for creating new Guest Collections\n",
    "tutorial_endpoint_1 = \"261692a4-7e49-4def-b59e-cbfc65e3907d\"\n",
    "\n",
    "CLIENT_ID = \"3b1925c0-a87b-452b-a492-2c9921d3bd14\"  # client ID of the Jupyter Demo App in Globus Auth\n",
    "native_auth_client = globus_sdk.NativeAppAuthClient(CLIENT_ID)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Globus Scopes\n",
    "\n",
    "Globus scopes are how applications request access to services. Scopes are used in OAuth2 Grants to obtain Access Tokens. This tutorial will use two scopes:\n",
    "\n",
    "* Transfer Scope -- For access to the Globus Transfer API\n",
    "* GCS Endpoint Scope -- For managing Guest Collections on Globus Tutorial Collection 1\n",
    "\n",
    "Both of these scopes require access to data on collections, and so require the \"data_access\" scope dependency."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get GCS Data Access scopes for both mapped collections. These are needed to access any data on\n",
    "# a mapped collection\n",
    "tc1_data_access = globus_sdk.scopes.GCSCollectionScopeBuilder(tutorial_collection_1).data_access\n",
    "tc2_data_access = globus_sdk.scopes.GCSCollectionScopeBuilder(tutorial_collection_2).data_access\n",
    "\n",
    "# Add scopes as dependencies for the transfer scope, allowing transfer operations to access data\n",
    "# on both tutorial mapped collections.\n",
    "transfer_scope = globus_sdk.scopes.TransferScopes.make_mutable(\"all\")\n",
    "transfer_scope.add_dependency(tc1_data_access)\n",
    "transfer_scope.add_dependency(tc2_data_access)\n",
    "\n",
    "# This scope will be used to manage the Endpoint on Tutorial 1, allowing us to Create/Update/Delete\n",
    "# Guest Collections on this endpoint.\n",
    "tutorial_ep1_manage_collections = globus_sdk.scopes.GCSEndpointScopeBuilder(tutorial_endpoint_1).make_mutable(\"manage_collections\")\n",
    "tutorial_ep1_manage_collections.add_dependency(tc1_data_access)\n",
    "\n",
    "requested_scopes = [transfer_scope, tutorial_ep1_manage_collections]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Globus Authorization Code Grant via JupyterHub\n",
    "\n",
    "When you clicked \"Login\" when visiting jupyter.demo.globus.org, you were already asked to login with the scopes above. The tokens were then passed to this notebook. We can simply use those tokens instead of doing a Native Login Flow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import base64\n",
    "import pickle\n",
    "\n",
    "USE_HUB_TOKENS = True\n",
    "# The tokens are passed into the environment base64 encoded pickled Python dictionary \n",
    "# assigned to the GLOBUS_DATA variable. We'll grab the variable and unpack it.\n",
    "globus_env_data = os.getenv('GLOBUS_DATA')  # get the raw base64 encoded data\n",
    "\n",
    "if globus_env_data and USE_HUB_TOKENS is True:\n",
    "    pickled_tokens = base64.b64decode(globus_env_data)  # get the pickled tokens\n",
    "    # Unpickle and get the dictionary\n",
    "    tokens = pickle.loads(pickled_tokens)[\"tokens\"]\n",
    "    print(json.dumps(tokens, indent=2))\n",
    "else:\n",
    "    print(\"Doing Native Auth instead of using Hub Tokens\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Globus Native App Grant\n",
    "\n",
    "The Native App Grant OAuth2 flow will do the following:\n",
    "1. Log into Globus using a one-time, generated URL\n",
    "2. Consent to allow this Jupyter Notebook to access Globus Transfer on our behalf\n",
    "3. Return to the notebook with an Authorization Code (the result of step 2)\n",
    "4. Exchange the Authorization Code for an Access Token\n",
    "5. Create a TransferClient object using that Access Token as its authorization method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if not globus_env_data or USE_HUB_TOKENS is False:\n",
    "    # Explicitly start the flow (some clients may support multiple flows)\n",
    "    native_auth_client.oauth2_start_flow(requested_scopes=requested_scopes, refresh_tokens=False)\n",
    "    print(f\"Login Here:\\n\\n{native_auth_client.oauth2_get_authorize_url()}\")\n",
    "    print(\"\\nIMPORTANT NOTE: the link above can only be used once!\")\n",
    "    print(\"If login or a later step in the flow fails, you must execute this cell again to generate a new link.\")\n",
    "    auth_code = input(\"PASTE YOUR CODE HERE> \")\n",
    "    tokens = native_auth_client.oauth2_exchange_code_for_tokens(auth_code).by_resource_server\n",
    "    print(\"Tokens Received!\")\n",
    "else:\n",
    "    print(\"Using 'Hub' Tokens instead.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract the access token for the Globus Transfer service, known as \"transfer.api.globus.org\"\n",
    "transfer_access_token = tokens['transfer.api.globus.org']['access_token']\n",
    "tutorial_ep1_access_token = tokens[tutorial_endpoint_1]['access_token']\n",
    "\n",
    "# Wrap the token in an object that implements the globus_sdk.GlobusAuthorizer interface\n",
    "# In this case, an AccessTokenAuthorizer, which takes an access token and produces Bearer Auth headers\n",
    "transfer_authorizer = globus_sdk.AccessTokenAuthorizer(transfer_access_token)\n",
    "tutorial_ep1_authorizer = globus_sdk.AccessTokenAuthorizer(tutorial_ep1_access_token)\n",
    "\n",
    "# Create a TransferClient object which Authorizes its calls using that GlobusAuthorizer\n",
    "tc = globus_sdk.TransferClient(authorizer=transfer_authorizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Help\n",
    "\n",
    "Test that our transfer client works by requesting help on the `get_endpoint` method. You can use this to get help on any method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "help(tc.get_endpoint)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using the client\n",
    "\n",
    "The transfer client makes REST resources available via easy to use methods. The response from these methods wraps the HTTP response status, content type, text and JSON response body. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = tc.get_endpoint(tutorial_collection_1)\n",
    "print(f\"HTTP Status Code: {response.http_status}\")\n",
    "print(f\"Content Type: {response.content_type}\")\n",
    "print(f\"Collection Display Name: {response['display_name']}\")  # shortcut for response.data['display_name']\n",
    "print(f\"Data: {response}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Helper methods for APIs that returns lists have iterable responses, and automatically take care of paging where required:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = tc.endpoint_search(filter_scope=\"my-endpoints\")\n",
    "for ep in response:\n",
    "    print(f\"{ep['display_name'] or ep['canonical_name']} ({ep['id']})\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If a helper method is not yet available for the desired API call, or for more low level control, the low-level interface can be used. Note that the low level interface does not do automatic paging:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = tc.get(\"/endpoint_search\", query_params=dict(filter_fulltext=\"Globus Tutorial Collection\", limit=1))\n",
    "if response['DATA']:\n",
    "    print(f\"Endpoint ID: {response['DATA'][0]['id']} (Owner: {response['DATA'][0]['owner_string']})\")\n",
    "    print(f\"More matches? {response['has_next_page']}\")\n",
    "else:\n",
    "    print(\"No results\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Handling errors\n",
    "\n",
    "If the API returns an error response (HTTP status code 4xx or 5xx), it will be translated to a Python exception and raised:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    response = tc.get_endpoint(\"dcb2e10e-de27-4b99-8722-1a69aa3fc467\")\n",
    "except globus_sdk.GlobusAPIError as error:\n",
    "    print(f\"HTTP Status Code: {error.http_status}\")\n",
    "    print(f\"Error Code      : {error.code}\")\n",
    "    print(f\"Error Message   : {error.message}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are five basic classes of errors:\n",
    "\n",
    "1. Bad request - there is something wrong with the request from the client, like a mispelled parameter name or missing required data. These errors have a code that starts with ``BadRequest`` or ``ClientError.BadRequest``.\n",
    "2. State conflict - this is a very broad category, and covers all the errors that can happen during normal operation, and neither the client nor the server could have anticipated and avoided the error. Examples: local filesystem permissions not allowing the requested path on a remote GridFTP endpoint, endpoint not found (could have been deleted concurrently by another client). This also includes network errors communicating with GridFTP endpoints and other external services. These errors typically have a code containing ``PermissionDenied``, ``Conflict``, or ``ExternalError``.\n",
    "3. Network error - network failure between the REST client and the REST API server. These errors will result in a ``globus_sdk.NetworkError`` being raised by the SDK.\n",
    "4. Planned downtime - code ``ServiceUnavailable``.\n",
    "5. Server error - caused by a bug in the REST API server (code ``ServerError.InternalError``). We log such errors and incorperate fixes into our next release, but developers are still encouraged to submit details to the mailing list when they encounter these errors. Note that sometimes these errors are actually a sign of a bad request type error, i.e. the bug in the server is that it's not anticipating the exact type of bad data, so it's not reporting the correct error code, but the problem can still be resolved by a change to the client."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Collection Management"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Collection search\n",
    "\n",
    "Globus has tens of thousands of registered collections. To find collections of interest you can access powerful search capabilities via the SDK. For example, to search for a given string across the descriptive fields of collections (names, description, keywords):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "search_str = \"Globus Tutorial Collection\"\n",
    "collections = tc.endpoint_search(search_str)\n",
    "print(f\"==== Displaying collections that match '{search_str}' ===\")\n",
    "for col in collections:\n",
    "    print(f\"{col['display_name'] or col['canonical_name']} ({col['id']})\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Restricting search scope with filters\n",
    "\n",
    "There are also a number of default filters to restrict the search for 'my-endpoints', 'my-gcp-endpoints',     'recently-used', 'in-use', 'shared-by-me','shared-with-me') "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "search_str = None\n",
    "collections = tc.endpoint_search(filter_fulltext=search_str, filter_scope=\"recently-used\")\n",
    "for col in collections:\n",
    "    print(f\"{col['display_name'] or col['canonical_name']} ({col['id']})\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Collection details\n",
    "\n",
    "You can also retrieve complete information about an collection, including name, owner, and configuration details. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "collection = tc.get_endpoint(tutorial_collection_1)\n",
    "print(f\"Display Name: {collection['display_name']}\")\n",
    "print(f\"Owner       : {collection['owner_string']}\")\n",
    "print(f\"ID          : {collection['id']}\")\n",
    "print(f\"Network Use - Concurrency = {collection['preferred_concurrency']}\")\n",
    "print(f\"Network Use - Parallelism = {collection['preferred_parallelism']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# File operations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get a directory listing\n",
    "\n",
    "Having activated an collection, you can now perform operations on it. For example, performing an ls command to retrieve directory contents. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# help(tc.operation_ls)\n",
    "collection_id = tutorial_collection_1\n",
    "collection_path = \"/home/share/godata/\"\n",
    "response = tc.operation_ls(collection_id, path=collection_path)\n",
    "print(f\"==== 'ls' for {collection_path} on collection {collection_id} ====\")\n",
    "for item in response:\n",
    "    print(f\"{item['type']}: {item['name']} [{item['size']}]\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Make directory\n",
    "\n",
    "You can create a new directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# help(tc.operation_mkdir)\n",
    "try:\n",
    "    new_path = \"/~/tutorial_dir\"\n",
    "    mkdir_result = tc.operation_mkdir(collection_id, path=new_path)\n",
    "    print(mkdir_result['message'])\n",
    "except globus_sdk.GlobusAPIError as error:\n",
    "    print(f\"Error code: {error.code}\\nError message: {error.message}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Rename\n",
    "\n",
    "You can rename files and directories on your collections. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# help(tc.operation_rename)\n",
    "try:\n",
    "    response = tc.operation_rename(collection_id, oldpath=\"/~/tutorial_dir\", newpath=\"/~/tutorial_dir_renamed\")\n",
    "    print(response['message'])\n",
    "except globus_sdk.GlobusAPIError as error:\n",
    "    print(f\"Error code: {error.code}\\nError message: {error.message}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Task submission and management\n",
    "\n",
    "The Globus task interface allows you to create and manage asynchronous file transfer and deletion tasks. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transfer\n",
    "\n",
    "Creating a transfer is a two stage process. First you must create a description of the data you want to transfer (which also creates a unique submission_id), and then you can submit the request to Globus to transfer that data. \n",
    "\n",
    "If the submit_transfer fails, you can safely resubmit the same transfer_data again. The submission_id will ensure that this transfer request will be submitted once and only once."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# help(tc.submit_transfer)\n",
    "source_collection_id = tutorial_collection_1\n",
    "source_path = \"/home/share/godata/\"\n",
    "\n",
    "dest_collection_id = tutorial_collection_2\n",
    "dest_path = \"/~/\"\n",
    "\n",
    "label = \"My tutorial transfer\"\n",
    "\n",
    "# TransferData() automatically gets a submission_id for once-and-only-once submission\n",
    "tdata = globus_sdk.TransferData(tc, source_collection_id, dest_collection_id, label=label)\n",
    "\n",
    "## Recursively transfer source path contents\n",
    "tdata.add_item(source_path, dest_path, recursive=True)\n",
    "\n",
    "## Alternatively, transfer a specific file\n",
    "# tdata.add_item(\"/source/path/file.txt\", \"/dest/path/file.txt\"))\n",
    "\n",
    "submit_result = tc.submit_transfer(tdata)\n",
    "print(f\"Task ID: {submit_result['task_id']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get Task By ID\n",
    "\n",
    "While the task is running, or after completion, you can get information that describes the transfer task. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = tc.get_task(submit_result['task_id'])\n",
    "print(f\"Label: {response['label']}\")\n",
    "print(f\"Status: {response['status']}\")\n",
    "print(f\"Transfer: {response['source_endpoint_display_name']} -> {response['destination_endpoint_display_name']}\")\n",
    "    \n",
    "if response.data[\"status\"] == \"SUCCEEDED\":\n",
    "    print(f\"Bytes transferred: {response['bytes_transferred']}\")\n",
    "    print(f\"Files transferred: {response['files_transferred']}\")\n",
    "    print(f\"Transfer rate: {response['effective_bytes_per_second']} Bps\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Check destination collection\n",
    "\n",
    "After the transfer has finished you can list the contents of the destination collection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls_iter = tc.operation_ls(dest_collection_id, path=dest_path)\n",
    "print(f\"==== 'ls' for {dest_path} on collection {dest_collection_id} ====\")\n",
    "for item in ls_iter:\n",
    "    print(f\"{item['type']}: {item['name']} [{item['size']}]\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get task list\n",
    "\n",
    "You can get a list of past or current tasks with the following call. Note that only `TRANSFER` tasks are returned by default - the type filter is necessary to get `DELETE` tasks as well. This is a remnant of a legacy backward compatibility concern, and will likely be changed in the future to both being included by default."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# help(tc.task_list)\n",
    "response = tc.task_list(limit=10, filter=\"type:TRANSFER,DELETE\")\n",
    "for i, item in enumerate(response):\n",
    "    print(item['status'],\n",
    "          item['task_id'], \n",
    "          item['type'],\n",
    "          item['source_endpoint_display_name'],\n",
    "          item['destination_endpoint_display_name'],\n",
    "          item['label'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Filter task list\n",
    "\n",
    "Retrieve only active tasks:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = tc.task_list(limit=10, filter=\"type:TRANSFER,DELETE/status:ACTIVE\")\n",
    "for i, item in enumerate(response):\n",
    "    print(item['status'],\n",
    "          item['task_id'], \n",
    "          item['type'],\n",
    "          item['source_endpoint_display_name'],\n",
    "          item['destination_endpoint_display_name'],\n",
    "          item['label'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See [Common Query Parameters](https://docs.globus.org/api/transfer/task/#common_query_parameters) for a description of the `filter` parameter, and [Task List filters](https://docs.globus.org/api/transfer/task/#filter_and_order_by_options) for details of what is supported by task list."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cancel task\n",
    "\n",
    "You can also cancel a running task. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# help(tc.cancel_task)\n",
    "response = tc.cancel_task(submit_result['task_id'])\n",
    "print(f\"{response['code']}: {response['message']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get event list for task\n",
    "\n",
    "Every task stores periodic event markers (e.g., errors, performance markers, etc.). You can retrieve and filter this list as follows. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# help(tc.task_event_list)\n",
    "response = tc.task_event_list(submit_result['task_id'], limit=10)\n",
    "for event in response:\n",
    "    print(event['time'], event['code'], event['is_error'], event['details'])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Delete files task\n",
    "\n",
    "File deletion is also an asynchronous task, that is submitted and monitored similar to a transfer task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# help(tc.submit_delete)\n",
    "# Create a folder, delete it, wait for completion\n",
    "collection_id = tutorial_collection_1\n",
    "path = \"/~/tutorial_delete_example\"\n",
    "try:\n",
    "    tc.operation_mkdir(collection_id, path=path)\n",
    "except globus_sdk.GlobusAPIError as error:\n",
    "    if \"Exists\" in error.code:\n",
    "        print(\"Directory already exists, ignoring error\")\n",
    "    else:\n",
    "        raise\n",
    "\n",
    "label = \"My tutorial delete\"\n",
    "\n",
    "# DeleteData() automatically gets a submission_id for once-and-only-once submission\n",
    "# Note that recursive is a top level option for delete, not a per-path option like\n",
    "# it is for transfers.\n",
    "ddata = globus_sdk.DeleteData(tc, collection_id, label=label, recursive=True)\n",
    "\n",
    "## Recursively delete path contents (because of recursive flag set above)\n",
    "ddata.add_item(path)\n",
    "\n",
    "submit_result = tc.submit_delete(ddata)\n",
    "print(f\"Task ID: {submit_result['task_id']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wait for task to complete\n",
    "\n",
    "Transfer and delete tasks are asynchronous operations, and depending on their size may take a long time to complete. If you wish to wait for a task to complete, the TransferClient provides a task_wait helper method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Wait for a task to finish for 10 minutes, polling every 15 seconds.\n",
    "completed = tc.task_wait(submit_result['task_id'], timeout=600, polling_interval=15)\n",
    "if completed:\n",
    "    print(\"Task finished!\")\n",
    "else:\n",
    "    print(\"Task still running after timeout reached.\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bookmarks\n",
    "\n",
    "Bookmarks allow you to keep a list of frequently used collections and paths. Full management capabilities (create, retrieve, update, delete) are supported on bookmarks. Note that the REST API itself does not directly support bookmarks when performing operations. It is the responsibility of the client to allow the users to choose bookmarks, and then translate them to collection ids to perform ls operations and submit transfers. In particular, the www.globus.org website has full support for bookmarks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a Bookmark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bookmark_name = \"My Tutorial Bookmark\"\n",
    "collection_id = tutorial_collection_1\n",
    "collection_path = \"/home/share/godata/\"\n",
    "response = tc.create_bookmark({\"endpoint_id\": collection_id, \"path\": collection_path,\"name\": bookmark_name})\n",
    "bookmark_id = response['id']\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get a list of bookmarks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = tc.bookmark_list()\n",
    "for b in response:\n",
    "    print (b['name'], b['path'], b['id'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Update a bookmark\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bookmark_data = {\n",
    "    'name': 'My Updated Tutorial Bookmark'\n",
    "}\n",
    "response = tc.update_bookmark(bookmark_id, bookmark_data)\n",
    "print (response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Delete a Bookmark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = tc.delete_bookmark(bookmark_id)\n",
    "print (response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Guest Collections\n",
    "\n",
    "Guest Collections are virtual collections that refer to a particular \"host guest collection\" and path, which allows Globus to manage access to that mapped collection. Folders on the guest collection can be easily shared with other Globus users and groups via access control rules."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a guest collection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fetch the URL for the tutorial endpoint, so we can Query the GCS Manager\n",
    "tutorial_collection_info = tc.get_endpoint(tutorial_endpoint_1)\n",
    "gcs_manager_url = tutorial_collection_info[\"gcs_manager_url\"]\n",
    "\n",
    "# Fetch the local credential information from the GCS Manager\n",
    "gc = globus_sdk.GCSClient(tutorial_collection_info[\"gcs_manager_url\"], authorizer=tutorial_ep1_authorizer)\n",
    "local_ep1_user_credential = gc.get_user_credential_list()[\"data\"][0]\n",
    "print(f\"Found local user credential with username {local_ep1_user_credential['username']}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a dir to share\n",
    "host_collection_id = tutorial_collection_1\n",
    "host_collection_path = \"/~/shared_dir2\"\n",
    "try:\n",
    "    response = tc.operation_mkdir(host_collection_id, path=host_collection_path)\n",
    "except globus_sdk.GlobusAPIError as error:\n",
    "    # Ignore the error if the directory already exists, otherwise raise\n",
    "    if \"Exists\" not in error.code:\n",
    "        raise\n",
    "\n",
    "# Define the guest collection\n",
    "guest_collection = {\n",
    "        \"DATA_TYPE\": \"collection#1.4.0\",\n",
    "        \"collection_type\": \"guest\",\n",
    "        \"display_name\": \"My Tutorial Guest Collection2\",\n",
    "        \"description\": \"Test creating a share from globus-jupyter-notebook\",\n",
    "        \"identity_id\": local_ep1_user_credential[\"identity_id\"],\n",
    "        \"storage_gateway_id\": local_ep1_user_credential[\"storage_gateway_id\"],\n",
    "        \"collection_base_path\": host_collection_path,\n",
    "        \"public\": True,\n",
    "        \"user_credential_id\": local_ep1_user_credential[\"id\"],\n",
    "        \"mapped_collection_id\": host_collection_id,\n",
    "}\n",
    "\n",
    "response = gc.create_collection(guest_collection)\n",
    "guest_collection_id = response[\"id\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get collection information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = tc.get_endpoint(guest_collection_id)\n",
    "print(f\"Display name: {response['display_name']}\")\n",
    "print(f\"Owner: {response['owner_string']}\")\n",
    "print(f\"Host Endpoint ID: {response['host_endpoint_id']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get a list of guest collections"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "endpoints = tc.endpoint_search(filter_scope=\"shared-by-me\")\n",
    "print(\"==== Displaying guest collections ===\")\n",
    "for ep in endpoints:\n",
    "    print(f\"{ep['display_name']} ({ep['id']})\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Add a new access control rule\n",
    "\n",
    "You can share access to different paths within your guest collection with users, groups, or publicly. The principal_type can be one of 'identity', 'group', 'all_authenticated_users', or 'anonymous'.  Each access rule is given a unique access_rule_id, which can be used to manage that access rule.\n",
    "\n",
    "Here is an example of sharing with the tutorial users group. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rule_data = {\n",
    "    'DATA_TYPE': 'access',\n",
    "    'permissions': 'rw',\n",
    "    'principal' : tutorial_users_group,  # use this if sharing with a group of users\n",
    "    'principal_type' : 'group',  # use this if sharing with a group of users\n",
    "    #'principal': 'IDENTITY_ID',  # use this if sharing with a single user (identity)\n",
    "    #'principal_type': 'identity',  # use this if sharing with a single user (identity)\n",
    "    'path': '/'\n",
    "}\n",
    "\n",
    "try:\n",
    "    response = tc.add_endpoint_acl_rule(guest_collection_id, rule_data)\n",
    "    access_rule_id = response['access_id']\n",
    "    print (response)\n",
    "except globus_sdk.GlobusAPIError as error:\n",
    "    if \"Exists\" in error.code:\n",
    "        print(\"ACL already exists, ignoring error\")\n",
    "    else:\n",
    "        raise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get list of access rules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = tc.endpoint_acl_list(guest_collection_id)\n",
    "for rule in response:\n",
    "    print (rule['id'], rule['principal_type'], rule['principal'], rule['permissions'], rule['path'])\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get access rule by id\n",
    "\n",
    "Get the access rule details using its access_rule_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = tc.get_endpoint_acl_rule(guest_collection_id, access_rule_id)\n",
    "print (response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Update access rule\n",
    "\n",
    "Update an access rule using its access_rule_id."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rule_update = {\n",
    "    'DATA_TYPE': 'access',\n",
    "    'permissions': 'r',\n",
    "}\n",
    "response = tc.update_endpoint_acl_rule(guest_collection_id, access_rule_id, rule_update)\n",
    "print (response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Delete access rule\n",
    "\n",
    "Delete an access rule using its access_rule_id."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = tc.delete_endpoint_acl_rule(guest_collection_id, access_rule_id)\n",
    "print (response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Low-level SDK interface\n",
    "\n",
    "The helper methods are all built on top of the low level interface. If a helper method is not yet available for the API resource you wish to use, the low level interface can be used directly.\n",
    "\n",
    "Note that the examples in this section use the endpoint management API resources, which DO have helper methods, but they still serve as good examples for how to use the low level interface."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## POST request\n",
    "\n",
    "Create an endpoint using the low level API:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "guest_collection = {\n",
    "        \"DATA_TYPE\": \"collection#1.4.0\",\n",
    "        \"collection_type\": \"guest\",\n",
    "        \"display_name\": \"Tutorial Create Example\",\n",
    "        \"description\": \"Test creating a share from globus-jupyter-notebook\",\n",
    "        \"identity_id\": local_ep1_user_credential[\"identity_id\"],\n",
    "        \"storage_gateway_id\": local_ep1_user_credential[\"storage_gateway_id\"],\n",
    "        \"collection_base_path\": host_collection_path,\n",
    "        \"public\": True,\n",
    "        \"user_credential_id\": local_ep1_user_credential[\"id\"],\n",
    "        \"mapped_collection_id\": host_collection_id,\n",
    "}\n",
    "\n",
    "response = gc.post(\"/collections/\", data=guest_collection)\n",
    "print(response)\n",
    "guest_collection_id = response[\"data\"][0][\"id\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GET request\n",
    "\n",
    "Do a GET on the newly create endpoint:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = gc.get(f\"/collections/{guest_collection_id}\",\n",
    "           query_params=dict(fields=\"id,display_name,description\"))\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## PUT request\n",
    "\n",
    "Update the description on the newly created endpoint:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "endpoint_update = {\n",
    "    \"DATA_TYPE\": \"collection#1.4.0\",\n",
    "    \"public\": True,\n",
    "    \"collection_type\": \"guest\",\n",
    "    \"display_name\": \"Tutorial Update Example\",\n",
    "    \"collection_base_path\": host_collection_path,\n",
    "    \"description\": \"Test updating description using low level API\",\n",
    "    \"identity_id\": local_ep1_user_credential[\"identity_id\"],\n",
    "    \"storage_gateway_id\": local_ep1_user_credential[\"storage_gateway_id\"],\n",
    "    \"user_credential_id\": local_ep1_user_credential[\"id\"],\n",
    "    \"mapped_collection_id\": host_collection_id,\n",
    "}\n",
    "response = gc.put(f\"/collections/{guest_collection_id}\", data=endpoint_update)\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DELETE request\n",
    "\n",
    "Now delete the endpoint:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = gc.delete(f\"/collections/{guest_collection_id}\")\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More exercises\n",
    "\n",
    "You can practice working with the SDK and the Globus Transfer service with [the exercises in this notebook](Transfer_API_Exercises.ipynb)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}