sdk/python/using-mltable/from-paths-example/from-paths-example.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 🛣️ Creating a Table from paths\n",
    "\n",
    "You can create a table containing the paths on cloud storage. In this example, there are some dog and cat images stored in cloud storage in the following folder structure:\n",
    "\n",
    "```\n",
    "/pet-images\n",
    "  /cat\n",
    "    0.jpeg\n",
    "    1.jpeg\n",
    "    ...\n",
    "  /dog\n",
    "    0.jpeg\n",
    "    1.jpeg\n",
    "```\n",
    "\n",
    "MLTable can extract the storage URIs of these images and the useful folder names for labelling purposes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 📦 Install dependencies\n",
    "\n",
    "Ensure you have the latest MLTable library and dependencies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -r ../mltable-requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 🐍 Create an MLTable using the Python SDK\n",
    "\n",
    "Here you build your data loading steps using the `mltable` Python SDK. The `show()` method allows you to see the effect of the data loading transformation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import mltable\n",
    "\n",
    "# create paths to the data files\n",
    "paths = [\n",
    "    {\n",
    "        \"pattern\": \"wasbs://data@azuremlexampledata.blob.core.windows.net/pet-images/**/*.jpg\"\n",
    "    }\n",
    "]\n",
    "\n",
    "# create the mltable\n",
    "tbl = mltable.from_paths(paths)\n",
    "\n",
    "# extract useful information from the path\n",
    "tbl = tbl.extract_columns_from_partition_format(\n",
    "    \"{account}/{container}/{folder}/{label}\"\n",
    ")\n",
    "\n",
    "tbl = tbl.drop_columns([\"account\", \"container\", \"folder\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🐼 Load into a Pandas data frame\n",
    "\n",
    "You can load your Azure ML Table into Pandas using:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = tbl.to_pandas_dataframe()\n",
    "df.head(5)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 📉 Plot the images\n",
    "The Path column is `StreamInfo` type, which means we can open and read the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plot images on a grid. Note this takes ~1min to execute.\n",
    "import matplotlib.pyplot as plt\n",
    "from PIL import Image\n",
    "\n",
    "fig = plt.figure(figsize=(20, 20))\n",
    "columns = 4\n",
    "rows = 5\n",
    "for i in range(1, columns * rows + 1):\n",
    "    with df.Path[i].open() as f:\n",
    "        img = Image.open(f)\n",
    "        fig.add_subplot(rows, columns, i)\n",
    "        plt.imshow(img)\n",
    "        plt.title(df.label[i])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 💾 Save data loading steps \n",
    "Next, you'll save all your data loading steps into an `MLTable` file. This allows you to *reproduce* your Pandas data frame at a later point in time without having to redefine the data loading steps in your code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save the data loading steps in an MLTable file\n",
    "tbl.save(\"./pets\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 🔍 View the saved file\n",
    "\n",
    "In the next code cell, we show you the `MLTable` file so you can understand how the data loading steps are serialized into a file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"./pets/MLTable\", \"r\") as f:\n",
    "    print(f.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ♻️ Reproduce data loading steps\n",
    "\n",
    "Now that the data loading steps have been serialized into a file, you can reproduce them at any point in time using the `load()` method. This means you do not need to redefine your data loading steps in code and makes it easier to share with others."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import mltable\n",
    "\n",
    "# load the previously saved MLTable file\n",
    "tbl = mltable.load(\"./pets/\")\n",
    "df = tbl.to_pandas_dataframe()\n",
    "df.head(5)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🤝 Create a data asset to aid sharing and reproducibility\n",
    "\n",
    "You'll now create a data asset, which will automatically upload the `MLTable` to cloud storage (the default AzureML datastore) so that others can use it easily."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "subscription_id = \"<SUBSCRIPTION_ID>\"\n",
    "resource_group = \"<RESOURCE_GROUP>\"\n",
    "workspace = \"<AML_WORKSPACE_NAME>\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "from azure.ai.ml import MLClient\n",
    "from azure.ai.ml.entities import Data\n",
    "from azure.ai.ml.constants import AssetTypes\n",
    "from azure.identity import DefaultAzureCredential\n",
    "\n",
    "# set the version number of the data asset to the current UTC time\n",
    "VERSION = time.strftime(\"%Y.%m.%d.%H%M%S\", time.gmtime())\n",
    "\n",
    "# connect to the AzureML workspace\n",
    "ml_client = MLClient(\n",
    "    DefaultAzureCredential(), subscription_id, resource_group, workspace\n",
    ")\n",
    "\n",
    "my_data = Data(\n",
    "    path=\"./pets\",\n",
    "    type=AssetTypes.MLTABLE,\n",
    "    description=\"A sample of cat and dog images\",\n",
    "    name=\"pets-mltable-example\",\n",
    "    version=VERSION,\n",
    ")\n",
    "\n",
    "ml_client.data.create_or_update(my_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 📖 Read the data asset in an interactive session\n",
    "\n",
    "Now you have your MLTable stored in the cloud, you and Team members can access it using a friendly name in an interactive session (for example, a notebook)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import mltable\n",
    "from azure.ai.ml import MLClient\n",
    "from azure.identity import DefaultAzureCredential\n",
    "\n",
    "# connect to the AzureML workspace\n",
    "ml_client = MLClient(\n",
    "    DefaultAzureCredential(), subscription_id, resource_group, workspace\n",
    ")\n",
    "\n",
    "# get the latest version of the data asset\n",
    "# Note: the variable VERSION is set in the previous code code\n",
    "data_asset = ml_client.data.get(name=\"pets-mltable-example\", version=VERSION)\n",
    "\n",
    "# the table from the data asset id\n",
    "tbl = mltable.load(f\"azureml:/{data_asset.id}\")\n",
    "\n",
    "# load into pandas\n",
    "df = tbl.to_pandas_dataframe()\n",
    "df.head()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10 - SDK V2",
   "language": "python",
   "name": "python310-sdkv2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}