greengrass_ml_inference.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9ee3c677",
   "metadata": {},
   "source": [
    "## 概要\n",
    "Greengrass を用いた機械学習アプリの開発と配布を行うハンズオンです。\n",
    "\n",
    "* EC2 をエッジデバイスと見立てて、エッジ側で ML 推論するアプリを開発・配布します。\n",
    "* 実際の開発を模して、開発機(1台)と、ステージング機(1台)、本番機(2台)を作成します。\n",
    "* 開発機でアプリ開発、ステージング機で配布と動作のチェック、本番機でアプリを一括配布することを体験します。\n",
    "* 使用する ML モデルは [build_mnist_dcgan_and_classifier.ipynb](./build_mnist_dcgan_and_classifier.ipynb) で作成したものを利用します(すでに配置済)\n",
    "* このノートブックを実行するコンピューティングリソースには、下記ポリシーがアタッチされている前提です。\n",
    "    * `AmazonEC2FullAccess`\n",
    "    * `AmazonSSMFullAccess`\n",
    "    * `AmazonS3FullAccess`\n",
    "    * `IAMFullAccess`\n",
    "    * `AWSGreengrassFullAccess`\n",
    "    * `AWSIoTFullAccess`\n",
    "    * `AmazonEC2ContainerRegistryFullAccess`\n",
    "\n",
    "## 構成\n",
    "* ハンズオンの流れ \n",
    "  1. エッジデバイスを模した EC2 のセットアップと Greengrass CLI をインストール\n",
    "  2. 開発機 でローカル開発＆ローカルデプロイ\n",
    "  3. クラウド側でコンポーネント作成\n",
    "  4. ステージング機 にデプロイ & 確認\n",
    "  5. 本番機 2台にデプロイ\n",
    "* 開発するもの\n",
    "  1. Publisherで撮像して画像ファイルのパスをIPCでパブリッシュ(実態はGANで画像生成）\n",
    "  2. Subscriber で画像に写った数字が奇数か偶数か判定\n",
    "  3. 1と2の機能をまとめて MQTT でパブリッシュするコンテナアプリ\n",
    "  4. 3 のモデルのアップデートしたコンテナアプリ\n",
    "  \n",
    "ハンズオンが完了するとこのような構成が出来上がります\n",
    "![ハンズオンの最終形](./image/image01.png)\n",
    "\n",
    "## 手順\n",
    "1. [エッジデバイスの事前セットアップ](#エッジデバイスの事前セットアップ)\n",
    "    1. [作成する Greengrass デバイスの定義](#作成する-Greengrass-デバイスの定義)\n",
    "    2. [EC2 で利用するキーペアを作成](#EC2-で利用するキーペアを作成)\n",
    "    3. [EC2 のロールを作成](#EC2-のロールを作成)\n",
    "    4. [EC2 起動](#EC2-起動)\n",
    "    5. [Greengrass Artifact 用 S3 バケットを作成](#Greengrass-Artifact-用-S3-バケットを作成)\n",
    "    6. [Greengrass core が作成・使用するロールとポリシーが存在する場合の事前削除](#Greengrass-core-が作成・使用するロールとポリシーが存在する場合の事前削除)\n",
    "    7. [Greengrass core が使用するロール作成](#Greengrass-core-が使用するロール作成)\n",
    "    8. [RoleAlias の作成](#RoleAlias-の作成)\n",
    "2. [Greengrass CLI をエッジ (EC2) にインストール](#Greengrass-CLI-をエッジ-(EC2)-にインストール)\n",
    "    1. [既存 thing と group を削除](#既存-thing-と-group-を削除)\n",
    "    2. [EC2 に Greengrass CLI をインストールするコマンドの準備](#EC2-に-Greengrass-CLI-をインストールするコマンドの準備)\n",
    "    3. [インストールコマンドを実行](#インストールコマンドを実行)\n",
    "3. [Publisher / Subscriber の開発](#Publisher-/-Subscriber-の開発)\n",
    "    1. [Publisher のローカルデプロイ](#Publisher-のローカルデプロイ)\n",
    "        1. [開発機で Publisher のローカルコンポーネント作成とローカルデプロイ](#開発機で-Publisher-のローカルコンポーネント作成とローカルデプロイ)\n",
    "        2. [Publisher のデプロイ結果を確認](#Publisher-のデプロイ結果を確認)\n",
    "        3. [デプロイした Publisher コンポーネントの動作を確認](#デプロイした-Publisher-コンポーネントの動作を確認)\n",
    "    2. [Subscriber のローカルデプロイ](#Subscriber-のローカルデプロイ)\n",
    "        1. [開発機で Subscriber のローカルコンポーネント作成とローカルデプロイ](#開発機で-Subscriber-のローカルコンポーネント作成とローカルデプロイ)\n",
    "        2. [Subscriber のデプロイ結果を確認](#Subscriber-のデプロイ結果を確認)\n",
    "        3. [デプロイした Subscriber コンポーネントの動作を確認](#デプロイした-Subscriber-コンポーネントの動作を確認)\n",
    "    3. [Greengrass クラウドからデプロイ](#Greengrass-クラウドからデプロイ)\n",
    "        1. [既存コンポーネントが存在する場合の削除](#既存コンポーネントが存在する場合の削除)\n",
    "        2. [Publisher のコンポーネントを作成](#Publisher-のコンポーネントを作成)\n",
    "        3. [Subscriber のコンポーネントを作成](#Subscriber-のコンポーネントを作成)\n",
    "        4. [Publisher / Subscriber のクラウドデプロイ](#Publisher-/-Subscriber-のクラウドデプロイ)\n",
    "            1. [ステージング機へのデプロイ](#ステージング機へのデプロイ)\n",
    "            2. [ステージング機へのデプロイ結果確認](#ステージング機へのデプロイ結果確認)\n",
    "            3. [本番機へのデプロイ](#本番機へのデプロイ)\n",
    "            4. [本番機へのデプロイ結果確認](#本番機へのデプロイ結果確認)\n",
    "4. [コンテナを利用したコンポーネントの開発](#コンテナを利用したコンポーネントの開発)\n",
    "    1. [コンテナイメージのビルド](#コンテナイメージのビルド)\n",
    "    2. [ECR へコンテナイメージをプッシュ](#ECR-へコンテナイメージをプッシュ)\n",
    "    3. [コンテナを使ったコンポーネントを開発機にローカルデプロイ](#コンテナを使ったコンポーネントを開発機にローカルデプロイ)\n",
    "        1. [コンテナローカルデプロイ用 Recipe を作成](#コンテナローカルデプロイ用-Recipe-を作成)\n",
    "        2. [開発機でコンテナコンポーネントのローカルデプロイ](#開発機でコンテナコンポーネントのローカルデプロイ)\n",
    "        3. [ローカルデプロイした IoTPublisher が MQTTで パブリッシュした内容を確認する](#ローカルデプロイした-IoTPublisher-が-MQTT-でパブリッシュした内容を確認する)\n",
    "    4. [コンテナを使ったコンポーネントをステージング機にクラウドデプロイ](#コンテナを使ったコンポーネントをステージング機にクラウドデプロイ)\n",
    "        1. [既存コンテナコンポーネントが存在する場合の削除](#既存コンテナコンポーネントが存在する場合の削除)\n",
    "        2. [コンテナを用いたコンポーネントの作成](#コンテナを用いたコンポーネントの作成)\n",
    "        3. [ステージング機のデプロイの改定](#ステージング機のデプロイの改定)\n",
    "        4. [ステージング機にクラウドデプロイした IoTPublisher が MQTT でパブリッシュした内容を確認する](#ステージング機にクラウドデプロイした-IoTPublisher-が-MQTT-でパブリッシュした内容を確認する)\n",
    "5. [コンテナを利用したコンポーネントの更新](#コンテナを利用したコンポーネントの更新)\n",
    "    1. [コンテナイメージの更新](#コンテナイメージの更新)\n",
    "    2. [ECR へ更新したコンテナイメージをプッシュ](#ECR-へ更新したコンテナイメージをプッシュ)\n",
    "    3. [更新したコンテナを使ったコンポーネントを開発機にローカルデプロイ](#更新したコンテナを使ったコンポーネントを開発機にローカルデプロイ)\n",
    "        1. [更新したコンテナのローカルデプロイ用 Recipe を作成](#更新したコンテナのローカルデプロイ用-Recipe-を作成)\n",
    "        2. [開発機で更新したコンテナのローカルデプロイ](#開発機で更新したコンテナのローカルデプロイ)\n",
    "        3. [更新したコンテナのローカルデプロイ結果を確認（開発機）](#更新したコンテナのローカルデプロイ結果を確認（開発機）)\n",
    "    4. [更新したコンテナを使ったコンポーネントをクラウドデプロイ](#更新したコンテナを使ったコンポーネントをステージング機にクラウドデプロイ)\n",
    "        1. [更新したコンテナを用いたコンポーネントの作成](#更新したコンテナを用いたコンポーネントの作成)\n",
    "        2. [更新したコンテナを用いたステージング機のデプロイの改定](#更新したコンテナを用いたステージング機のデプロイの改定)\n",
    "        3. [ステージング機で更新したコンテナの MQTT 動作確認](#ステージング機で更新したコンテナの-MQTT-動作確認)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60de6282",
   "metadata": {},
   "outputs": [],
   "source": [
    "!date"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee795146",
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3, json, yaml, os\n",
    "from time import sleep"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2bbf7507",
   "metadata": {},
   "outputs": [],
   "source": [
    "# region 取得\n",
    "region = boto3.session.Session().region_name\n",
    "\n",
    "# 使用するサービスのクライアント生成\n",
    "ec2_client = boto3.client('ec2', region_name=region)\n",
    "ssm_client = boto3.client('ssm', region_name=region)\n",
    "s3_client = boto3.client('s3', region_name=region)\n",
    "iam_client = boto3.client('iam')\n",
    "ggv2_client = boto3.client('greengrassv2', region_name=region)\n",
    "iot_client = boto3.client('iot', region_name=region)\n",
    "ecr_client = boto3.client('ecr', region_name=region)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2362b038",
   "metadata": {},
   "source": [
    "## エッジデバイスの事前セットアップ\n",
    "EC2 をエッジデバイスとして扱うためのセットアップ\n",
    "\n",
    "### 作成する Greengrass デバイスの定義 \n",
    "* 本ハンズオンではEC2 4 台をエッジデバイスとして扱うため、AWS IoT Core 上のグループ、名前を設定する\n",
    "* クラウドでの名前の登録自体は Greengrass CLI をインストールするときに行われるが、ここでは変数にそれぞれの名前を格納する"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "737d4538",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 開発環境\n",
    "develop_group = {\n",
    "    \"name\": \"gg-ml-iot-thing-development-group\",\n",
    "    \"thing_names\":['gg-ml-iot-thing-development']\n",
    "}\n",
    "# ステージング環境\n",
    "staging_group = {\n",
    "    \"name\": \"gg-ml-iot-thing-staging-group\",\n",
    "    \"thing_names\":['gg-ml-iot-thing-staging']\n",
    "}\n",
    "# 本番環境\n",
    "production_group = {\n",
    "    \"name\": \"gg-ml-iot-thing-production-group\",\n",
    "    \"thing_names\":[\n",
    "        'gg-ml-iot-thing-production1',\n",
    "        'gg-ml-iot-thing-production2'\n",
    "    ]\n",
    "}\n",
    "all_groups = [develop_group, staging_group, production_group]\n",
    "device_name_list=[y for x in all_groups for y in x[\"thing_names\"]]\n",
    "print(device_name_list)\n",
    "print(f\"作成するデバイスの数: {len(device_name_list)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2faf1f85",
   "metadata": {},
   "source": [
    "### EC2 で利用するキーペアを作成\n",
    "* 本ハンズオンの本筋とは関係なく、EC2特有の操作\n",
    "* sshでログインできるようにするための「key-pair-for-greengrass-on-ec2」というsshキーを作成する\n",
    "* 同名のキーがあった場合は削除して作り直す"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9702a22e",
   "metadata": {},
   "outputs": [],
   "source": [
    "key_pair_name = 'key-pair-for-greengrass-on-ec2'\n",
    "key_pairs = ec2_client.describe_key_pairs()\n",
    "key_names = list(map(lambda x : x['KeyName'], key_pairs['KeyPairs']))\n",
    "\n",
    "if key_pair_name in key_names:\n",
    "    ec2_client.delete_key_pair(KeyName=key_pair_name)\n",
    "\n",
    "ec2_key_pair = ec2_client.create_key_pair(\n",
    "    KeyName=key_pair_name,\n",
    ")\n",
    "\n",
    "key_pair = str(ec2_key_pair['KeyMaterial'])\n",
    "with open('ec2-key-pair.pem','w') as f:\n",
    "    f.write(key_pair)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "620f425e",
   "metadata": {},
   "source": [
    "### Greengrass Artifact 用 S3 バケットを作成\n",
    "* Greengrass の Artifact(Greengrassで動かすアプリのファイル群) を保存する S3 Bucket を作成\n",
    "* 他にも EC2 へ SSM 経由で流し込んだ出力結果を保存するのにも使用\n",
    "* Greengrass が使用するロールにこのバケットへのアクセス権限を与える必要があるため、この段階でバケット名を決めて作成する\n",
    "\n",
    "![バケット作成](./image/image02.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c17c4692",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# バケットが存在している場合は事前に削除\n",
    "bucket = 'type your great bucket name' # <- グローバルでユニークなS3バケット名を入力\n",
    "!aws s3 rb s3://{bucket} --force\n",
    "# バケット作成\n",
    "location = {'LocationConstraint': region}\n",
    "if region == 'us-east-1':\n",
    "    response = s3_client.create_bucket(\n",
    "        Bucket = bucket\n",
    "    )\n",
    "else:\n",
    "    response = s3_client.create_bucket(\n",
    "        Bucket = bucket,\n",
    "        CreateBucketConfiguration = location\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1cc44377",
   "metadata": {},
   "source": [
    "### EC2 のロール及びインラインポリシーを作成\n",
    "* 本ハンズオンの本筋とは関係なく、EC2特有の操作\n",
    "* EC2にシェルコマンドを送り込むのにsession_managerというサービスを使うため、それらの権限が付与されたロールを作成\n",
    "* Greengrass CLI をインストールする際に必要なポリシーもあるので(IAMFullAccess,AWSIoTFullAccess)、別途そのポリシーを有したIAMユーザやロールを発行して、ACCESS_KEYを設定するか、[IoTクライアント証明書を利用](https://docs.aws.amazon.com/ja_jp/iot/latest/developerguide/device-certs-create.html)するなどをしておく"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b5d22473",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 定数定義\n",
    "ec2_role_name = 'EC2GreengrassRole'\n",
    "ec2_inline_policy_name = 'GreengrassEc2InlinePolicy'\n",
    "ec2_profile=ec2_role_name + \"Profile\"\n",
    "\n",
    "ec2_role_attach_policy_arn_list = [\n",
    "    \"arn:aws:iam::aws:policy/AmazonEC2FullAccess\",\n",
    "    \"arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore\",\n",
    "    \"arn:aws:iam::aws:policy/IAMFullAccess\",\n",
    "    \"arn:aws:iam::aws:policy/AWSIoTFullAccess\",\n",
    "    \"arn:aws:iam::aws:policy/AWSGreengrassFullAccess\",\n",
    "    'arn:aws:iam::aws:policy/service-role/AmazonEC2RoleforSSM',\n",
    "    'arn:aws:iam::aws:policy/AmazonSSMDirectoryServiceAccess',\n",
    "    'arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy',\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d30f9064",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# 既存ポリシーのデタッチ、ポリシーバージョンの削除、ポリシーの削除\n",
    "# 存在しない場合はexceptでスキップされる\n",
    "\n",
    "# インラインポリシーの削除\n",
    "policies = iam_client.list_policies()\n",
    "for policy in policies['Policies']:\n",
    "    policy_arn = policy['Arn'] if policy['PolicyName'] in ec2_inline_policy_name else None\n",
    "    if policy_arn:\n",
    "        # Policy をデタッチ\n",
    "        try:\n",
    "            response = iam_client.detach_role_policy(\n",
    "                RoleName=ec2_role_name,\n",
    "                PolicyArn=policy_arn\n",
    "            )\n",
    "            print(f'detach {ec2_role_name} {policy_arn}')\n",
    "            print(respoonse)\n",
    "        except:\n",
    "            print('did not detach role policy')\n",
    "        # Policy バージョンの削除\n",
    "        try:\n",
    "            for policy_version in iam_client.list_policy_versions(PolicyArn=policy_arn)['Versions']:\n",
    "                response = iam.delete_policy_version(PolicyArn=policy_arn,VersionId=policy_version['VersionId'])\n",
    "                print(f'delete policy version {policy_arn} {policy_version[\"VersionId\"]}')\n",
    "                print(response)\n",
    "        except:\n",
    "            print('did not delete policy version')\n",
    "        # Policy を削除\n",
    "        try:\n",
    "            response = iam_client.delete_policy(PolicyArn=policy_arn)\n",
    "            print(f'delete policy {policy_arn}')\n",
    "            print(response)\n",
    "        except:\n",
    "            print('did not delete policy')\n",
    "\n",
    "# その他ポリシーの削除\n",
    "for policy_arn in ec2_role_attach_policy_arn_list:                      \n",
    "    try:\n",
    "        response = iam_client.detach_role_policy(RoleName=ec2_role_name,PolicyArn=policy_arn)\n",
    "        print(f'detach ec2 role {policy_arn}')\n",
    "        print(json.dumps(response, indent=2))\n",
    "    except:\n",
    "        print(f'could not detach {policy_arn}')\n",
    "                      \n",
    "\n",
    "\n",
    "# インスタンスプロファイルを削除\n",
    "try:\n",
    "    response = iam_client.remove_role_from_instance_profile(\n",
    "        InstanceProfileName=ec2_profile,\n",
    "        RoleName=ec2_role_name\n",
    "    )\n",
    "    print('delete ec2 role')\n",
    "    print(json.dumps(response, indent=2))\n",
    "except:\n",
    "    print('nothing to remove role from instance profile')\n",
    "\n",
    "# ロールの削除\n",
    "try:\n",
    "    response = iam_client.delete_role(RoleName = ec2_role_name)\n",
    "    print('delete ec2 role')\n",
    "    print(json.dumps(response, indent=2))\n",
    "except:\n",
    "    print('nothing to delete role')\n",
    "try:\n",
    "    response = iam_client.delete_instance_profile(InstanceProfileName=ec2_profile)\n",
    "    print('delete instance profile')\n",
    "    print(json.dumps(response, indent=2))\n",
    "except:\n",
    "    print('nothing to delete instance profile')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "365c71d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "sleep(30)\n",
    "# インラインポリシーの作成\n",
    "s3_get_from_ec2_policy_doc = {\n",
    "    'Version': '2012-10-17',\n",
    "    'Statement': [\n",
    "        {\n",
    "            'Effect': 'Allow',\n",
    "            'Action': [\n",
    "                's3:GetObject',\n",
    "                's3:PutObject',\n",
    "                's3:PutObjectAcl',\n",
    "            ],\n",
    "            'Resource': [\n",
    "                f'arn:aws:s3:::{bucket}/*',\n",
    "                f'arn:aws:s3:::aws-ssm-{region}/*',\n",
    "                f'arn:aws:s3:::aws-windows-downloads-{region}/*',\n",
    "                f'arn:aws:s3:::amazon-ssm-{region}/*',\n",
    "                f'arn:aws:s3:::amazon-ssm-packages-{region}/*',\n",
    "                f'arn:aws:s3:::{region}-birdwatcher-prod/*',\n",
    "                f'arn:aws:s3:::aws-ssm-distributor-file-{region}/*',\n",
    "                f'arn:aws:s3:::aws-ssm-document-attachments-{region}/*',\n",
    "                f'arn:aws:s3:::patch-baseline-snapshot-{region}/*'\n",
    "            ]\n",
    "        },\n",
    "        {\n",
    "            \"Effect\": \"Allow\",\n",
    "            \"Action\": [\n",
    "                \"ssmmessages:CreateControlChannel\",\n",
    "                \"ssmmessages:CreateDataChannel\",\n",
    "                \"ssmmessages:OpenControlChannel\",\n",
    "                \"ssmmessages:OpenDataChannel\"\n",
    "            ],\n",
    "            \"Resource\": \"*\"\n",
    "        },\n",
    "        {\n",
    "            \"Effect\": \"Allow\",\n",
    "            \"Action\": [\n",
    "                \"s3:GetEncryptionConfiguration\"\n",
    "            ],\n",
    "            \"Resource\": \"*\"\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "response = iam_client.create_policy(\n",
    "    PolicyName=ec2_inline_policy_name,\n",
    "    PolicyDocument=json.dumps(s3_get_from_ec2_policy_doc),\n",
    "    Description='s3 get from ec2 policy',\n",
    ")\n",
    "ec2_inline_policy_arn = response['Policy']['Arn']\n",
    "ec2_inline_policy_name = response['Policy']['PolicyName']\n",
    "print(f'作成した policy の\\narn は \"{ec2_inline_policy_arn}\"')\n",
    "print(f'名前は \"{ec2_inline_policy_name}\"\\nです')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5be48b79",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ロールの作成\n",
    "assume_role_policy_document = {\n",
    "  \"Version\": \"2012-10-17\",\n",
    "  \"Statement\": [{\"Sid\": \"\",\"Effect\": \"Allow\",\"Principal\": {\"Service\": \"ec2.amazonaws.com\"},\"Action\": \"sts:AssumeRole\"}]\n",
    "}\n",
    "response = iam_client.create_role(\n",
    "    RoleName = ec2_role_name,\n",
    "    AssumeRolePolicyDocument = json.dumps(assume_role_policy_document),\n",
    "    Description='edge equivalent ec2 role',\n",
    "    MaxSessionDuration=3600*12 # 12時間\n",
    ")\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a3f2b16",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# ポリシーをロールにアタッチ\n",
    "# inline policy\n",
    "response = iam_client.attach_role_policy(\n",
    "    RoleName=ec2_role_name,\n",
    "    PolicyArn=ec2_inline_policy_arn\n",
    ")\n",
    "print(json.dumps(response, indent=2))\n",
    "\n",
    "# managed policy\n",
    "for policy_arn in ec2_role_attach_policy_arn_list:\n",
    "    response = iam_client.attach_role_policy(\n",
    "        RoleName=ec2_role_name,\n",
    "        PolicyArn=policy_arn\n",
    "    )\n",
    "    print(f'attach policy_arn')\n",
    "    print(json.dumps(response, indent=2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8adb6b1",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "response = iam_client.create_instance_profile(InstanceProfileName=ec2_profile)\n",
    "response = iam_client.add_role_to_instance_profile(\n",
    "    RoleName=ec2_role_name,\n",
    "    InstanceProfileName=ec2_profile\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4545bea4",
   "metadata": {},
   "source": [
    "### EC2 起動\n",
    "* 本ハンズオンの本筋とは関係ない\n",
    "* user_dataで流しんでいるコマンドは下記を実現するためのものなので、エッジデバイス側で予め設定すること\n",
    "  * python3 コマンドで起動する Python のバージョンが 3.8 にする\n",
    "  * awscli, unzip, zip, docker, pip3 のインストール\n",
    "\n",
    "![EC2の起動](./image/image03.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1cd7fa8a",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "work_dir = '/root/work'\n",
    "exec_log_file = os.path.join(work_dir,'1_ec2_userdata.log')\n",
    "detail_log_file = os.path.join(work_dir,'2_ec2_detail_userdata.log')\n",
    "user_data = f\"\"\"#!/bin/bash\n",
    "export work_dir={work_dir}\n",
    "export exec_log_file={exec_log_file}\n",
    "export detail_log_file={detail_log_file}\n",
    "whoami >> $exec_log_file\n",
    "mkdir -p $work_dir\n",
    "cd $work_dir\n",
    "pwd >> $exec_log_file\n",
    "\n",
    "date >> $exec_log_file\n",
    "echo \"apt update\" >> $exec_log_file\n",
    "apt update -y >> $detail_log_file\n",
    "\n",
    "date >> $exec_log_file\n",
    "echo \"apt install\" >> $exec_log_file\n",
    "apt install python3-pip unzip awscli zip docker docker.io -y >> $detail_log_file\n",
    "\n",
    "date >> $exec_log_file\n",
    "echo \"pip3 install\" >> $exec_log_file\n",
    "pip3 install awscli >> $detail_log_file\n",
    "\n",
    "date >> $exec_log_file\n",
    "echo \"apt install python3.8\" >> $exec_log_file\n",
    "apt install -y python3.8 >> $detail_log_file\n",
    "\n",
    "# Python3.8 をデフォルトに\n",
    "update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.8 1\n",
    "update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.6 2\n",
    "echo 2 | update-alternatives --config python3\n",
    "python3 --version >> $exec_log_file\n",
    "echo completed  >> $exec_log_file\n",
    "date >> $exec_log_file\n",
    "\n",
    "\"\"\"\n",
    "print(user_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10c8f438",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ubuntu の AMI 集\n",
    "ami_map = {\n",
    "    'af-south-1':'ami-075520a13dc442833',\n",
    "    'ap-east-1':'ami-0a4e8c6468d92c210',\n",
    "    'ap-northeast-1':'ami-0620ef6ee255ea559',\n",
    "    'ap-south-1':'ami-0c0490c60db84298f',\n",
    "    'ap-southeast-1':'ami-09b5aaebc21c273eb',\n",
    "    'ca-central-1':'ami-05c0545bdc0bbff91',\n",
    "    'eu-central-1':'ami-08db015a4afd75546',\n",
    "    'eu-north-1':'ami-05916395833cb9690',\n",
    "    'eu-south-1':'ami-03839a9ab91b19abc',\n",
    "    'eu-west-1':'ami-0f5b07b31937d4275',\n",
    "    'me-south-1':'ami-019ba1b47cfb5211e',\n",
    "    'sa-east-1':'ami-092dc94ff1b65b5c2',\n",
    "    'us-east-1':'ami-0e4d932065378fd3d',\n",
    "    'us-west-1':'ami-05620e35978c63272',\n",
    "    'ap-northeast-2':'ami-0f77aba17625db03b',\n",
    "    'ap-southeast-2':'ami-0b643a2ce5f48199a',\n",
    "    'eu-west-2':'ami-0a14509f661bf2964',\n",
    "    'us-east-2':'ami-063e88ad6c9af427d',\n",
    "    'us-west-2':'ami-0b7d93899b51ff83b',\n",
    "    'ap-northeast-3':'ami-06947c2f8a47debf3',\n",
    "    'eu-west-3':'ami-0b722faecb702e094',\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2ae69ed5",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "sleep(30) # instance profile が反映されるまで待つ\n",
    "ec2_instance = ec2_client.run_instances(\n",
    "    BlockDeviceMappings=[{'Ebs':{'VolumeSize':20},\"DeviceName\" : '/dev/xvda'}],\n",
    "    ImageId=ami_map[region],\n",
    "    MinCount=len(device_name_list),# [development/staging/production*2]\n",
    "    MaxCount=len(device_name_list),\n",
    "    InstanceType='c5.large',\n",
    "    KeyName=key_pair_name,\n",
    "    IamInstanceProfile={'Name': ec2_profile},\n",
    "    UserData = user_data\n",
    ")\n",
    "instance_id_list = [instance['InstanceId'] for instance in ec2_instance['Instances']]\n",
    "except_counter = 0\n",
    "while True:\n",
    "    try:\n",
    "        status_list = ec2_client.describe_instance_status(InstanceIds=instance_id_list)['InstanceStatuses']\n",
    "        print([status['InstanceState']['Name'] for status in status_list])\n",
    "        if [status['InstanceState']['Code'] for status in status_list] == [16]*4:\n",
    "            break\n",
    "        else:\n",
    "            sleep(5)\n",
    "    except:\n",
    "        print('except')\n",
    "        except_counter+=1\n",
    "        if except_counter > 60:\n",
    "            print('maximum except')\n",
    "            break\n",
    "        else:\n",
    "            sleep(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a09eacea",
   "metadata": {},
   "outputs": [],
   "source": [
    "# EC2の名前のタグを設定して、EC2のコンソールから見やすくする\n",
    "for instance_id,tag_value in zip(instance_id_list,device_name_list):\n",
    "    ec2_client.create_tags(\n",
    "        Resources = [instance_id],\n",
    "        Tags = [{'Key':'Name','Value':tag_value}]\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "427e2a68",
   "metadata": {},
   "source": [
    "### Greengrass core が作成・使用するロールとポリシーが存在する場合の事前削除\n",
    "* 本ハンズオンの本筋とは関係ない\n",
    "* 実行前に環境をクリーニング\n",
    "* 既存の場合を想定してロールとポリシーは削除して、そのあと作り直す"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b6ba7e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "greengrass_role_name = 'MLIoTGGV2TESRole'\n",
    "greengrass_policy_names = [\n",
    "    'MLIoTGGV2TESArtifactPolicy',   # ユーザ作成\n",
    "    'MLIoTGGV2TESRoleAccess'        # システム作成\n",
    "]\n",
    "greengrass_role_alias = greengrass_role_name + \"Alias\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "23c1079a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Role Aliasの削除\n",
    "try:\n",
    "    response = iot_client.delete_role_alias(\n",
    "        roleAlias=greengrass_role_alias\n",
    "    )\n",
    "except:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "09c4959a",
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    response = iam_client.list_attached_role_policies(\n",
    "        RoleName=greengrass_role_name\n",
    "    )\n",
    "    print(response[\"AttachedPolicies\"])\n",
    "except:\n",
    "    print('No attachedPolicies')\n",
    "    response = None\n",
    "\n",
    "    \n",
    "if response: \n",
    "    for policy in response[\"AttachedPolicies\"]:\n",
    "        policy_arn = policy[\"PolicyArn\"]\n",
    "        try:\n",
    "            response = iam_client.detach_role_policy(\n",
    "                RoleName=greengrass_role_name,\n",
    "                PolicyArn=policy_arn\n",
    "            )\n",
    "            print(f'detach {greengrass_role_name} {policy_arn}')\n",
    "            print(respoonse)\n",
    "        except:\n",
    "            print('did not detach role policy')\n",
    "\n",
    "        try:\n",
    "            for policy_version in iam_client.list_policy_versions(PolicyArn=policy_arn)['Versions']:\n",
    "                response = iam_client.delete_policy_version(PolicyArn=policy_arn,VersionId=policy_version['VersionId'])\n",
    "                print(f'delete policy version {policy_arn} {policy_version[\"VersionId\"]}')\n",
    "                print(response)\n",
    "        except:\n",
    "            print('did not delete policy version')\n",
    "\n",
    "        try:\n",
    "            response = iam_client.delete_policy(PolicyArn=policy_arn)\n",
    "            print(f'delete policy {policy_arn}')\n",
    "            print(response)\n",
    "        except:\n",
    "            print('did not delete policy')\n",
    "else:\n",
    "    pass\n",
    "\n",
    "# ロールを削除\n",
    "try:\n",
    "    response = iam_client.delete_role(RoleName = greengrass_role_name)\n",
    "    print(f'delete role {greengrass_role_name}')\n",
    "    print(response)\n",
    "except:\n",
    "    print('did not delete role')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b7b80ff",
   "metadata": {},
   "source": [
    "### Greengrass CLI が使用するロール作成\n",
    "* Greengrass CLI が Greengrass のサービスや、ECR や S3 を利用できるようにするためのロールの作成とポリシーのアタッチ\n",
    "* 使用している API は以下の通り(すべて IAM)\n",
    "  * [create_role](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/iam.html#IAM.Client.create_role)\n",
    "  * [create_policy](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/iam.html#IAM.ServiceResource.create_policy)\n",
    "  * [attach_role_policy](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/iam.html#IAM.Client.attach_role_policy)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a4f25c3e",
   "metadata": {},
   "outputs": [],
   "source": [
    "assume_role_policy_document = {\n",
    "  \"Version\": \"2012-10-17\",\n",
    "  \"Statement\": [{\"Sid\": \"\",\"Effect\": \"Allow\",\"Principal\": {\"Service\": \"credentials.iot.amazonaws.com\"},\"Action\": \"sts:AssumeRole\"}]\n",
    "}\n",
    "response = iam_client.create_role(\n",
    "    RoleName = greengrass_role_name,\n",
    "    AssumeRolePolicyDocument = json.dumps(assume_role_policy_document),\n",
    "    Description='using Greengrass ml inference',\n",
    "    MaxSessionDuration=3600*12 # 12時間\n",
    ")\n",
    "role_arn = response['Role']['Arn']\n",
    "role_name = response['Role']['RoleName']\n",
    "\n",
    "print(f'作成した role の\\narn は \"{role_arn}\"')\n",
    "print(f'名前は \"{role_name}\"\\nです')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b938a78c",
   "metadata": {},
   "outputs": [],
   "source": [
    "gg_role_policy_document = {\n",
    "  \"Version\": \"2012-10-17\",\n",
    "  \"Statement\": [\n",
    "      {\"Effect\": \"Allow\",\"Action\": [\"s3:*\"],\"Resource\": f\"arn:aws:s3:::{bucket}/*\"},\n",
    "      {\"Effect\": \"Allow\",\"Action\": [\"ecr:GetAuthorizationToken\",\"ecr:BatchGetImage\",\"ecr:GetDownloadUrlForLayer\",\"iot:Connect\",\"iot:Publish\"],\"Resource\": [\"*\"]}\n",
    "\n",
    "  ]\n",
    "}\n",
    "\n",
    "response = iam_client.create_policy(\n",
    "    PolicyName=greengrass_policy_names[0],\n",
    "    PolicyDocument=json.dumps(gg_role_policy_document),\n",
    "    Description='for GreengrassV2Role policy',\n",
    ")\n",
    "policy_arn = response['Policy']['Arn']\n",
    "policy_name = response['Policy']['PolicyName']\n",
    "print(f'作成した policy の\\narn は \"{policy_arn}\"')\n",
    "print(f'名前は \"{policy_name}\"\\nです')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "03c28790",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Policy attach\n",
    "response = iam_client.attach_role_policy(\n",
    "    RoleName=role_name,\n",
    "    PolicyArn=policy_arn\n",
    ")\n",
    "print(json.dumps(response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00ccf86c",
   "metadata": {},
   "source": [
    "### RoleAlias の作成\n",
    "* Greengrass のデバイスは AWS IoT のX.509証明書を利用したロールエイリアスで、先程作成したロールを代替させることができる  \n",
    "  = ロールエイリアスで、AWS のサービス外でもロールと同じ機能を実現できるようになる\n",
    "* ロールエイリアスは事前に作成しておく\n",
    "* 使用している API は [create_role_alias](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/iot.html#IoT.Client.create_role_alias) (IoT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "308d8fdb",
   "metadata": {},
   "outputs": [],
   "source": [
    "response = iot_client.create_role_alias(\n",
    "    roleAlias=greengrass_role_alias,\n",
    "    roleArn=role_arn\n",
    ")\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ff85a30",
   "metadata": {},
   "source": [
    "## Greengrass CLI をエッジ (EC2) にインストール\n",
    "合計 4 台にインストール\n",
    "\n",
    "* 開発機\n",
    "* ステージング機\n",
    "* 本番機 2 台  \n",
    "![Greengrass CLI をインストール](./image/image04.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60977ac9",
   "metadata": {},
   "source": [
    "### 既存 thing と group を削除\n",
    "* ハンズオンの本筋とは関係ない\n",
    "* Greengrass CLI をインストールする際に thing と group が作成されるが、同名の thing と group は作成できないので事前に削除する"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "99199e56",
   "metadata": {},
   "outputs": [],
   "source": [
    "# thing と type と certification を削除するヘルパー関数\n",
    "def del_thing_and_group_and_cert(thing_name,thing_group_name):\n",
    "    try:\n",
    "        print('try deleting gg core')\n",
    "        response = ggv2_client.delete_core_device(\n",
    "            coreDeviceThingName=thing_name\n",
    "        )\n",
    "    except:\n",
    "        print(f\"Core don't exists: {thing_name}\")\n",
    "\n",
    "    try:\n",
    "        print('try deleting certificate')\n",
    "        for principal in iot_client.list_thing_principals(thingName=thing_name)['principals']:\n",
    "            response = iot_client.detach_thing_principal(thingName=thing_name,principal=principal)\n",
    "            print(response)\n",
    "            response = iot_client.update_certificate(certificateId=principal.split('/')[-1],newStatus='INACTIVE')\n",
    "            print(response)\n",
    "            response = iot_client.delete_certificate(certificateId=principal.split('/')[-1],forceDelete=True)\n",
    "            print(response)\n",
    "    except:\n",
    "        pass\n",
    "\n",
    "    try:\n",
    "        print('try deleting thing')\n",
    "        response = iot_client.delete_thing(thingName=thing_name)\n",
    "        print(response)\n",
    "    except:\n",
    "        pass\n",
    "\n",
    "    try:\n",
    "        print('try deleting thing group')\n",
    "        response = iot_client.delete_thing_group(thingGroupName=thing_group_name)\n",
    "        print(response)\n",
    "    except:\n",
    "        pass\n",
    "\n",
    "    # 作成しようとしている名前のthingがあったら削除\n",
    "    try:\n",
    "        iot_client.delete_thing(thingName=thing_name)\n",
    "    except:\n",
    "        pass\n",
    "    try:\n",
    "        iot_client.delete_thing_group(thingName=thing_group_name)\n",
    "    except:\n",
    "        pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "36d015fb",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Thing groupとthingnameを組み合わせる\n",
    "for thing_group in all_groups:\n",
    "    for thing_name in thing_group[\"thing_names\"]:\n",
    "        print(\"Group: {} Thing: {}\".format(thing_group[\"name\"], thing_name))\n",
    "        del_thing_and_group_and_cert(thing_name,thing_group[\"name\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6103329",
   "metadata": {},
   "source": [
    "### EC2 に Greengrass CLI をインストールするコマンドの準備\n",
    "* java を使ってインストールする\n",
    "* あらかじめ設定してある下記を引数にインストールを行う\n",
    "  * thing name\n",
    "  * thing group\n",
    "  * role\n",
    "  * role alias\n",
    "* dev toolsは本番機には入れない\n",
    "  * ローカルデプロイすることはないため\n",
    "  * staging はデバッグ用に念の為入れておく\n",
    "* Greengrass CLI のインストール時のコマンド等詳細は以下を参照\n",
    "  * [Greengrass CLI のインストール](https://docs.aws.amazon.com/greengrass/v2/developerguide/getting-started.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56ce78ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "gg_install_dir = '/greengrass/v2'\n",
    "\n",
    "command_str_list = []\n",
    "for thing_group in all_groups:\n",
    "    for thing_name in thing_group[\"thing_names\"]:\n",
    "        gg_cli_tool = \"--deploy-dev-tools true\" if \"production\" not in thing_name.lower() else \"\"\n",
    "        command_str=f\"\"\"#!/bin/bash\n",
    "# su ubuntu\n",
    "sudo su -\n",
    "export work_dir={work_dir}\n",
    "mkdir -p {work_dir}\n",
    "cd {work_dir}\n",
    "pwd\n",
    "\n",
    "# userdata の完了向けに 600 秒待機\n",
    "sleep 300\n",
    "\n",
    "# ggv2 で使用する jdk をインストール\n",
    "sudo apt install openjdk-11-jdk -y\n",
    "\n",
    "# ggv2 のインストーラを入手\n",
    "wget https://d2s8p88vqu9w66.cloudfront.net/releases/greengrass-nucleus-latest.zip\n",
    "unzip greengrass-nucleus-latest.zip -d GreengrassCore\n",
    "# バージョン指定する場合はこちら\n",
    "# wget https://d2s8p88vqu9w66.cloudfront.net/releases/greengrass-2.1.0.zip\n",
    "# unzip greengrass-2.1.0.zip -d GreengrassCore\n",
    "\n",
    "# ggv2 をインストール\n",
    "sudo -E java -Droot={gg_install_dir} -Dlog.store=FILE -jar ./GreengrassCore/lib/Greengrass.jar --aws-region {region} --thing-name {thing_name} --thing-group-name {thing_group[\"name\"]} --component-default-user ggc_user:ggc_group --provision true --setup-system-service true  --tes-role-name {greengrass_role_name} --tes-role-alias-name {greengrass_role_alias} {gg_cli_tool}\n",
    "\n",
    "# ggv2 のインストールはすぐに反映されないので、sleepで待つ\n",
    "sleep 180            \n",
    "\n",
    "# docker を ggc_user から実行するためにgroupに加えておく\n",
    "sudo usermod -aG docker ggc_user\n",
    "\n",
    "# ggv2 cli のバージョン確認を持ってインストールを確認する\n",
    "{gg_install_dir}/bin/greengrass-cli -V\n",
    "\"\"\"\n",
    "        command_str_list.append(command_str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d934d43",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# 開発機に流すコマンド\n",
    "print(command_str_list[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "257d5ea4",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# ステージング機に流すコマンド\n",
    "print(command_str_list[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa5e3f7a",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# 本番機 1 に流すコマンド\n",
    "print(command_str_list[2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4de950a7",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# 本番機 2 に流すコマンド\n",
    "print(command_str_list[3])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "379717d2",
   "metadata": {},
   "source": [
    "### インストールコマンドを実行\n",
    "* 作成したコマンドをそれぞれのエッジデバイスで実行する\n",
    "* 今回は EC2 を利用しているので、ssm の send_command API を利用して実行する\n",
    "* 実際はエッジデバイスのコンソールに入って同じコマンドを実行していく"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0656ab6c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# シェルコマンドをEC2に送りこみ、結果を確認するヘルパー\n",
    "def send_command(bucket,instance_id,command_list):\n",
    "    print(f\"send_command {instance_id}\")\n",
    "    document_name = \"AWS-RunShellScript\"\n",
    "    response = ssm_client.send_command(\n",
    "        InstanceIds=[instance_id],\n",
    "        DocumentName=document_name,\n",
    "        OutputS3BucketName=bucket,\n",
    "        Parameters={\n",
    "            'commands': command_list\n",
    "        }\n",
    "    )\n",
    "    print(response)\n",
    "    return response['Command']['CommandId']\n",
    "\n",
    "def check_command_result(command_id,instance_id,sleep_time=30,retry_count=60):\n",
    "    print(f\"check_command_result {instance_id}\")\n",
    "    counter = 0\n",
    "    while counter < retry_count:\n",
    "        try:\n",
    "            response = ssm_client.get_command_invocation(\n",
    "                CommandId=command_id,\n",
    "                InstanceId=instance_id,\n",
    "            )\n",
    "            print(response['Status'])\n",
    "            if response['Status'] in ['Success','Failed']:\n",
    "                print(response['StandardOutputContent'])\n",
    "                break\n",
    "            else:\n",
    "                counter += 1\n",
    "                sleep(sleep_time)\n",
    "        except:\n",
    "            print('invocation does not exist yet')\n",
    "            counter += 1\n",
    "            sleep(sleep_time)\n",
    "    \n",
    "    prefix=f'{command_id}/{instance_id}'\n",
    "    counter = 0\n",
    "    retry_count=3\n",
    "    while counter < retry_count:\n",
    "        response = s3_client.list_objects(\n",
    "            Bucket=bucket,\n",
    "            Prefix=prefix\n",
    "        )\n",
    "        if \"Contents\" in response:\n",
    "            break\n",
    "        counter += 1\n",
    "\n",
    "    # 標準エラー出力を確認\n",
    "    key=f'{prefix}/awsrunShellScript/0.awsrunShellScript/stderr'\n",
    "    print(key)\n",
    "    std = {\n",
    "        'stderr':'',\n",
    "        'stdout':''\n",
    "    }\n",
    "    try:\n",
    "        txt = s3_client.get_object(Bucket=bucket,Key=key)['Body'].read().decode('utf-8')\n",
    "        std['stderr'] = txt\n",
    "    except:\n",
    "        print('Not found stderr')\n",
    "\n",
    "    # 標準出力を確認\n",
    "    key=f'{prefix}/awsrunShellScript/0.awsrunShellScript/stdout'\n",
    "    print(key)\n",
    "    try:\n",
    "        txt = s3_client.get_object(Bucket=bucket,Key=key)['Body'].read().decode('utf-8')\n",
    "        std['stdout'] = txt\n",
    "    except:\n",
    "        print('Not found stdout')\n",
    "    return std"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b8bda1ec",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "sleep(120) # session manager が有効化されるまで待つ\n",
    "command_ids = []\n",
    "for command_str,instance_id in zip(command_str_list,instance_id_list):\n",
    "    print(instance_id)\n",
    "    command_id = send_command(bucket, instance_id, command_str.split('\\n'))\n",
    "    command_ids.append(command_id)\n",
    "for command_id,instance_id in zip(command_ids,instance_id_list):\n",
    "    response = check_command_result(command_id,instance_id)\n",
    "    print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5d971b1",
   "metadata": {},
   "source": [
    "## Publisher / Subscriber の開発\n",
    "### Publisher のローカルデプロイ\n",
    "![Publisher のローカルデプロイ](./image/image05.png)\n",
    "#### 開発機で Publisher のローカルコンポーネント作成とローカルデプロイ\n",
    "* Publisher で画像を撮像し、その画像のファイルパスを IPC で Publish する\n",
    "* 手動で開発した体で、development に開発済みのコードとモデルをエッジ側で githubから wget で取ってきて、deployment コマンドを実行する\n",
    "* コマンド自体は session manager を使ってコマンドを遠隔で送り込む（実際はエッジデバイスのコンソールに入ってコマンドを実行する）\n",
    "* ローカルでデプロイする場合は `greengrass-cli deployment create` コマンドを使う。詳細は[deployment](https://docs.aws.amazon.com/greengrass/v2/developerguide/gg-cli-deployment.html)を参照"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1683156b",
   "metadata": {},
   "outputs": [],
   "source": [
    "base_dir = '/ggv2'\n",
    "artifact_base_dir = f'{base_dir}/components/artifacts'\n",
    "artifact_dir = f'{artifact_base_dir}/com.example.Publisher/1.0.0'\n",
    "recipe_dir = f'{base_dir}/components/recipes'\n",
    "component_name = 'com.example.Publisher=1.0.0'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c7c4f9c6",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "command_str=f\"\"\"#!/bin/bash\n",
    "su ubuntu\n",
    "sudo mkdir -p {base_dir}\n",
    "mkdir -p {artifact_dir}\n",
    "cd {artifact_dir}\n",
    "wget https://raw.githubusercontent.com/kazuhitogo/greengrass-ml-inference/main/src/ggv2/components/artifacts/com.example.Publisher/1.0.0/publisher.py\n",
    "wget https://raw.githubusercontent.com/kazuhitogo/greengrass-ml-inference/main/src/ggv2/components/artifacts/com.example.Publisher/1.0.0/1.h5\n",
    "mkdir {recipe_dir}\n",
    "cd {recipe_dir}\n",
    "wget https://raw.githubusercontent.com/kazuhitogo/greengrass-ml-inference/main/src/ggv2/components/recipes/com.example.Publisher-1.0.0.yaml\n",
    "sudo chown -R ubuntu:ubuntu {base_dir}\n",
    "sudo /greengrass/v2/bin/greengrass-cli deployment create --recipeDir {recipe_dir} --artifactDir {artifact_base_dir} --merge \"{component_name}\"\n",
    "\"\"\"\n",
    "command_list = command_str.split('\\n')\n",
    "print('EC2 に流すコマンド\\n')\n",
    "for command in command_list:\n",
    "    print(command)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f1dc6810",
   "metadata": {},
   "outputs": [],
   "source": [
    "command_id = send_command(bucket,instance_id_list[0],command_list)\n",
    "response = check_command_result(command_id,instance_id_list[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5281e09",
   "metadata": {},
   "source": [
    "### Publisher のデプロイ結果を確認\n",
    "* `greengrass-cli deployment status` コマンドで確認可能\n",
    "* deployment の ID が必要なため、deployment 時の出力から抽出\n",
    "* あくまで Greengrass としてデプロイがうまくいったかであり、コンポーネントが正常に動作しているかは関与しない(動き始めて何秒か経過した後エラーで落ちたりしても deployment 上は SUCCEEDED と表示される)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2510c64c",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# deployment id を標準出力から取得する\n",
    "deployment_id = response['stdout'].split(' ')[-1]\n",
    "print(f'Deployment Id: {deployment_id}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33662330",
   "metadata": {},
   "outputs": [],
   "source": [
    "command_str=f\"\"\"#!/bin/bash\n",
    "su ubuntu\n",
    "sudo /greengrass/v2/bin/greengrass-cli deployment status -i {deployment_id}\n",
    "\"\"\"\n",
    "command_list = command_str.split('\\n')\n",
    "print('EC2 に流すコマンド\\n')\n",
    "for command in command_list:\n",
    "    print(command)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2495d65d",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "sleep(60) # デプロイ完了を待つ\n",
    "command_id = send_command(bucket,instance_id_list[0],command_list)\n",
    "response = check_command_result(command_id,instance_id_list[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "671f2cfc",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(response['stdout'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f8fd417",
   "metadata": {},
   "source": [
    "### デプロイした Publisher コンポーネントの動作を確認\n",
    "* 1 分おきに画像を`/tmp`に png 形式で出力されるはずなので、出力された画像を、エッジ → S3 → SageMaker Notebook に転送して確認する"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b0f9c669",
   "metadata": {},
   "outputs": [],
   "source": [
    "prefix='/publisher/output/development/'\n",
    "\n",
    "command_str=f\"\"\"#!/bin/bash\n",
    "latest_file=`ls -t /tmp/*png | head -n1`\n",
    "aws s3 cp $latest_file s3://{bucket}{prefix}\n",
    "echo $latest_file\"\"\"\n",
    "command_list = command_str.split('\\n')\n",
    "print('EC2 に流すコマンド\\n')\n",
    "for command in command_list:\n",
    "    print(command)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "791a9f2d",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "sleep(60) # 60秒待てば画像はできる\n",
    "command_id = send_command(bucket,instance_id_list[0],command_list)\n",
    "response = check_command_result(command_id,instance_id_list[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "05bd0cba",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 画像の確認、何かしらの画像が表示される\n",
    "\n",
    "output_file_name = response['stdout'].split('\\n')[-2][5:]\n",
    "!aws s3 cp s3://{bucket}{prefix}{output_file_name} .\n",
    "from PIL import Image\n",
    "Image.open(f'./{output_file_name}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "833e13c1",
   "metadata": {},
   "source": [
    "### Subscriber のローカルデプロイ\n",
    "![Subscriber のローカルデプロイ](./image/image06.png)\n",
    "\n",
    "* 先程作成した Publisher は IPC で出力した画像のパスをパブリッシュしている\n",
    "* Subscriber で Publisher のメッセージをサブスクライブして、MLを用いた画像分類器にかける\n",
    "  \n",
    "#### 開発機で Subscriber のローカルコンポーネント作成とローカルデプロイ\n",
    "* ローカルデプロイの方法は Publisher と全く同じ\n",
    "* recipe ファイルとコード一式（Artifact) をエッジ側に持っていき、デプロイする"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac1aa400",
   "metadata": {},
   "outputs": [],
   "source": [
    "artifact_dir = f'{artifact_base_dir}/com.example.Subscriber/1.0.0'\n",
    "recipe_dir = f'{base_dir}/components/recipes'\n",
    "component_name = 'com.example.Subscriber=1.0.0'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d792c921",
   "metadata": {},
   "outputs": [],
   "source": [
    "command_str=f\"\"\"#!/bin/bash\n",
    "su ubuntu\n",
    "sudo mkdir -p {base_dir}\n",
    "mkdir -p {artifact_dir}\n",
    "cd {artifact_dir}\n",
    "sudo rm -f subscriber.py\n",
    "wget https://raw.githubusercontent.com/kazuhitogo/greengrass-ml-inference/main/src/ggv2/components/artifacts/com.example.Subscriber/1.0.0/subscriber.py\n",
    "sudo rm -f 1.h5\n",
    "wget https://raw.githubusercontent.com/kazuhitogo/greengrass-ml-inference/main/src/ggv2/components/artifacts/com.example.Subscriber/1.0.0/1.h5\n",
    "mkdir {recipe_dir}\n",
    "cd {recipe_dir}\n",
    "rm -f com.example.Subscriber-1.0.0.yaml\n",
    "wget https://raw.githubusercontent.com/kazuhitogo/greengrass-ml-inference/main/src/ggv2/components/recipes/com.example.Subscriber-1.0.0.yaml\n",
    "sudo chown -R ubuntu:ubuntu {base_dir}\n",
    "sudo /greengrass/v2/bin/greengrass-cli deployment create --recipeDir {recipe_dir} --artifactDir {artifact_base_dir} --merge \"{component_name}\"\n",
    "\"\"\"\n",
    "command_list = command_str.split('}\\n')\n",
    "print('EC2 に流すコマンド\\n')\n",
    "for command in command_list:\n",
    "    print(command)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3542774a",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "command_id = send_command(bucket,instance_id_list[0],command_list)\n",
    "response = check_command_result(command_id,instance_id_list[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dac3b540",
   "metadata": {},
   "source": [
    "#### Subscriber のデプロイ結果を確認\n",
    "* デプロイの結果確認も Publisher と同じ"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10d7eca4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# deployment id を標準出力から取得する\n",
    "deployment_id = response['stdout'].split(' ')[-1]\n",
    "print(f'Deployment Id: {deployment_id}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "19224b9f",
   "metadata": {},
   "outputs": [],
   "source": [
    "command_str=f\"\"\"#!/bin/bash\n",
    "su ubuntu\n",
    "sudo /greengrass/v2/bin/greengrass-cli deployment status -i {deployment_id}\n",
    "\"\"\"\n",
    "command_list = command_str.split('\\n')\n",
    "print('EC2 に流すコマンド\\n')\n",
    "for command in command_list:\n",
    "    print(command)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "25071655",
   "metadata": {},
   "outputs": [],
   "source": [
    "sleep(60)\n",
    "command_id = send_command(bucket,instance_id_list[0],command_list)\n",
    "response = check_command_result(command_id,instance_id_list[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27483b5d",
   "metadata": {},
   "source": [
    "#### デプロイした Subscriber コンポーネントの動作を確認\n",
    "推論結果は`/tmp/Greengrass_Subscriber.log`に保存するようにアプリを作っているので、中身を確認する"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "48c05333",
   "metadata": {},
   "outputs": [],
   "source": [
    "command_str=f\"\"\"#!/bin/bash\n",
    "cat /tmp/Greengrass_Subscriber.log\"\"\"\n",
    "command_list = command_str.split('\\n')\n",
    "print('EC2 に流すコマンド\\n')\n",
    "for command in command_list:\n",
    "    print(command)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8aa2ba2a",
   "metadata": {},
   "outputs": [],
   "source": [
    "command_id = send_command(bucket,instance_id_list[0],command_list)\n",
    "response = check_command_result(command_id,instance_id_list[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a324a46",
   "metadata": {},
   "source": [
    "### Greengrass クラウドからステージングにデプロイ\n",
    "\n",
    "![Greengrass クラウドからステージングにデプロイ](./image/image07.png)\n",
    "\n",
    "* Staging を利用してデプロイのテスト\n",
    "* 手順としては、クラウド側にコンポーネントを作成してデプロイ先を選択してデプロイ\n",
    "* Publisher と Subscriber をそれぞれ行う"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59e5cd20",
   "metadata": {},
   "source": [
    "#### 既存コンポーネントが存在する場合の削除\n",
    "* ハンズオンの本筋とは関係ない\n",
    "* 名前の重複は許されないため事前に削除しておく"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fec79cb0",
   "metadata": {},
   "outputs": [],
   "source": [
    "result = ggv2_client.list_components()\n",
    "\n",
    "for component in result[\"components\"]:\n",
    "    if component[\"componentName\"] in [\"com.example.Subscriber\", \"com.example.Publisher\"]:\n",
    "        response = ggv2_client.list_component_versions(\n",
    "            arn=component[\"arn\"]\n",
    "        )\n",
    "        for version in response[\"componentVersions\"]:\n",
    "            print(f\"delete: {version['componentName']} {version['componentVersion']}\")\n",
    "            response = ggv2_client.delete_component(\n",
    "                arn=version[\"arn\"]\n",
    "            )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffd07729",
   "metadata": {},
   "source": [
    "#### Publisher のコンポーネントを作成\n",
    "* クラウドでコンポーネントを作成する場合、recipe をファイルで渡せないため、yaml形式の文字列を作成する\n",
    "* 使用する API は Greengrass v2 の [create_component_version](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/greengrassv2.html#GreengrassV2.Client.create_component_version) と [get_component](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/greengrassv2.html#GreengrassV2.Client.get_component)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "abbea920",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Publisher に必要なスクリプトとモデルを S3 にアップロード\n",
    "publisher_artifact_dir = f's3://{bucket}/components/artifacts/com.example.Publisher/1.0.0'\n",
    "!aws s3 cp ./src/ggv2/components/artifacts/com.example.Publisher/1.0.0/ {publisher_artifact_dir} --recursive\n",
    "# Subscriber に必要なスクリプトとモデルを S3 にアップロード\n",
    "subscriber_artifact_dir = f's3://{bucket}/components/artifacts/com.example.Subscriber/1.0.0'\n",
    "!aws s3 cp ./src/ggv2/components/artifacts/com.example.Subscriber/1.0.0/ {subscriber_artifact_dir} --recursive\n",
    "\n",
    "\n",
    "artifact_path = '{artifacts:path}'\n",
    "recipe = f\"\"\"\n",
    "---\n",
    "RecipeFormatVersion: '2020-01-25'\n",
    "ComponentName: com.example.Publisher\n",
    "ComponentVersion: '1.0.0'\n",
    "ComponentDescription: A component that publishes messages.\n",
    "ComponentPublisher: Amazon\n",
    "ComponentConfiguration:\n",
    "  DefaultConfiguration:\n",
    "    accessControl:\n",
    "      aws.greengrass.ipc.pubsub:\n",
    "        'com.example.Publisher:pubsub:1':\n",
    "          policyDescription: Allows access to publish to all topics.\n",
    "          operations:\n",
    "            - 'aws.greengrass#PublishToTopic'\n",
    "          resources:\n",
    "            - '*'\n",
    "Manifests:\n",
    "- Name: Linux\n",
    "  Platform:\n",
    "    os: linux\n",
    "  Lifecycle:\n",
    "    Install:\n",
    "        python3 -m pip install pip & pip3 install awsiotsdk numpy tensorflow-cpu==2.4.1 Pillow -U\n",
    "    Run: |-\n",
    "        python3 {artifact_path}/publisher.py {artifact_path}\n",
    "  Artifacts:\n",
    "  - Uri: \"{publisher_artifact_dir}/publisher.py\"\n",
    "  - Uri: \"{publisher_artifact_dir}/1.h5\"\n",
    "\"\"\"\n",
    "publisher_component_name = yaml.safe_load(recipe)['ComponentName']\n",
    "publisher_component_version = yaml.safe_load(recipe)['ComponentVersion']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c9f9f82",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# コンポーネント作成\n",
    "response = ggv2_client.create_component_version(\n",
    "    inlineRecipe=recipe.encode(),\n",
    "    tags={\n",
    "        'Name': 'Publisher'\n",
    "    }\n",
    ")\n",
    "publisher_component_vesrion_arn = response['arn']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d5f6678c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 作成したコンポーネントを確認\n",
    "response = ggv2_client.get_component(\n",
    "    recipeOutputFormat='YAML',\n",
    "    arn=publisher_component_vesrion_arn\n",
    ")\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1534d85c",
   "metadata": {},
   "source": [
    "#### Subscriber のコンポーネントを作成\n",
    "* Subscriber も Publisher 同様に yaml 形式で recipe を作成する"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6ab11b40",
   "metadata": {},
   "outputs": [],
   "source": [
    "artifact_path = '{artifacts:path}'\n",
    "recipe = f\"\"\"\n",
    "---\n",
    "RecipeFormatVersion: \"2020-01-25\"\n",
    "ComponentName: \"com.example.Subscriber\"\n",
    "ComponentVersion: \"1.0.0\"\n",
    "ComponentType: \"aws.greengrass.generic\"\n",
    "ComponentDescription: \"A component that subscribes to messages.\"\n",
    "ComponentPublisher: \"Amazon\"\n",
    "ComponentConfiguration:\n",
    "  DefaultConfiguration:\n",
    "    accessControl:\n",
    "      aws.greengrass.ipc.pubsub:\n",
    "        'com.example.Subscriber:pubsub:1':\n",
    "          policyDescription: \"Allows access to subscribe to all topics.\"\n",
    "          operations:\n",
    "          - \"aws.greengrass#SubscribeToTopic\"\n",
    "          resources:\n",
    "          - \"*\"\n",
    "Manifests:\n",
    "- Name: Linux\n",
    "  Platform:\n",
    "    os: linux\n",
    "  Lifecycle:\n",
    "    Install:\n",
    "        python3 -m pip install pip & pip3 install awsiotsdk numpy tensorflow-cpu==2.4.1 Pillow -U\n",
    "    Run: |-\n",
    "        python3 {artifact_path}/subscriber.py {artifact_path}\n",
    "  Artifacts:\n",
    "  - Uri: \"{subscriber_artifact_dir}/subscriber.py\"\n",
    "  - Uri: \"{subscriber_artifact_dir}/1.h5\"\n",
    "\"\"\"\n",
    "subscriber_component_name = yaml.safe_load(recipe)['ComponentName']\n",
    "subscriber_component_version = yaml.safe_load(recipe)['ComponentVersion']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1eadf720",
   "metadata": {},
   "outputs": [],
   "source": [
    "response = ggv2_client.create_component_version(\n",
    "    inlineRecipe=recipe.encode(),\n",
    "    tags={\n",
    "        'Name': 'subscriber'\n",
    "    }\n",
    ")\n",
    "subscriber_component_vesrion_arn = response['arn']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3fdaaa8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "response = ggv2_client.get_component(\n",
    "    recipeOutputFormat='YAML',\n",
    "    arn=subscriber_component_vesrion_arn\n",
    ")\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2aa4b118",
   "metadata": {},
   "source": [
    "#### Publisher / Subscriber のクラウドデプロイ\n",
    "* 事前に deploy と deploy 結果の確認用ヘルパー関数を用意しておく\n",
    "* デプロイはリスト形式で複数のコンポーネントを同時に指定できる\n",
    "* デプロイ先の指定は thing 単体でも thing group でも対象にできる。\n",
    "* ヘルパー関数で利用する API は以下の通り\n",
    "  * デプロイの作成 (Greengrass v2) : [create_deployment](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/greengrassv2.html#GreengrassV2.Client.create_deployment)\n",
    "  * ジョブの確認 (IoT) : [describe_job_execution](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/iot.html#IoT.Client.describe_job_execution)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa28c0f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# deployのヘルパー\n",
    "def deploy_component(target_arn, deployment_name, components):\n",
    "    response = ggv2_client.create_deployment(\n",
    "        targetArn=target_arn, # デプロイ先のIoT thing か group\n",
    "        deploymentName=deployment_name, # デプロイの名前\n",
    "        components=components,\n",
    "        iotJobConfiguration={\n",
    "            'timeoutConfig': {'inProgressTimeoutInMinutes': 600}\n",
    "        },\n",
    "        deploymentPolicies={\n",
    "            'failureHandlingPolicy': 'ROLLBACK',\n",
    "            'componentUpdatePolicy': {\n",
    "                'timeoutInSeconds': 600,\n",
    "                'action': 'NOTIFY_COMPONENTS'\n",
    "            },\n",
    "            'configurationValidationPolicy': {\n",
    "                'timeoutInSeconds': 600\n",
    "            }\n",
    "        },\n",
    "        tags={\n",
    "            'Name': deployment_name\n",
    "        }\n",
    "    )\n",
    "    return response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be2a55f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Job確認のヘルパー\n",
    "def check_job_status(job_id, thing_list):\n",
    "    sleep(10)\n",
    "    for target in thing_list:\n",
    "        retry = 20 # wait a while to the job finish\n",
    "        count=0\n",
    "        while True: # check for  2 minuets\n",
    "            response = iot_client.describe_job_execution(\n",
    "                jobId=job_id,\n",
    "                thingName=target\n",
    "            )\n",
    "            if response[\"execution\"][\"status\"] in [\"QUEUED\", \"IN_PROGRESS\"]:\n",
    "                print(f'target: {target} status: {response[\"execution\"][\"status\"]}')\n",
    "                sleep(10)\n",
    "            else:\n",
    "                print(f'target: {target} status: {response[\"execution\"][\"status\"]}')\n",
    "                break\n",
    "            if retry <= count:\n",
    "                      print(\"time out\")\n",
    "                      break"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb2e7b56",
   "metadata": {},
   "source": [
    "##### ステージング機へのデプロイ\n",
    "* デプロイ先の thin arn を取得した後、作成した deploy_component メソッド(の中で呼び出される create_deployment メソッド)でデプロイを作成する"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d404331",
   "metadata": {},
   "outputs": [],
   "source": [
    "# デプロイ先の thing arn を取得\n",
    "response = iot_client.describe_thing_group(\n",
    "    thingGroupName=staging_group[\"name\"]\n",
    ")\n",
    "staging_group_arn=response[\"thingGroupArn\"]\n",
    "print(staging_group_arn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11404672",
   "metadata": {},
   "outputs": [],
   "source": [
    "# デプロイの作成\n",
    "deployment_name = 'staging_deployment'\n",
    "components={\n",
    "    publisher_component_name: { # コンポーネントの名前\n",
    "        'componentVersion': publisher_component_version,\n",
    "    },\n",
    "    subscriber_component_name: { # コンポーネントの名前\n",
    "        'componentVersion': subscriber_component_version,\n",
    "    },\n",
    "}\n",
    "\n",
    "response = deploy_component(staging_group_arn, deployment_name, components)\n",
    "check_job_status(response[\"iotJobId\"], staging_group[\"thing_names\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b911322e",
   "metadata": {},
   "source": [
    "#### ステージング機へのデプロイ結果確認\n",
    "* Subscriber の動作だけ確認する(Publisherが動いていないとSubscriberは何も出力しないため、Subscriber が動いていることが確認取れればPublisherも動いている）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "448869cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Subscriber の確認\n",
    "command_str=f\"\"\"#!/bin/bash\n",
    "cat /tmp/Greengrass_Subscriber.log\"\"\"\n",
    "command_list = command_str.split('\\n')\n",
    "print('EC2 に流すコマンド\\n')\n",
    "for command in command_list:\n",
    "    print(command)\n",
    "print('------')\n",
    "command_id = send_command(bucket,instance_id_list[0],command_list)\n",
    "response = check_command_result(command_id,instance_id_list[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95e0ac93",
   "metadata": {},
   "source": [
    "##### 本番機へのデプロイ\n",
    "![Greengrass クラウドから本番機にデプロイ](./image/image08.png)\n",
    "\n",
    "* 構成はステージングの一緒のため、ステージングの図の矢印を一部省略している\n",
    "* deployment の向き先だけを変える"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8074d0fe",
   "metadata": {},
   "outputs": [],
   "source": [
    "response = iot_client.describe_thing_group(\n",
    "    thingGroupName=production_group[\"name\"]\n",
    ")\n",
    "production_group_arn=response[\"thingGroupArn\"]\n",
    "print(production_group_arn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5522913c",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "deployment_name = 'production_deployment'\n",
    "components={\n",
    "    publisher_component_name: { # コンポーネントの名前\n",
    "        'componentVersion': publisher_component_version,\n",
    "    },\n",
    "    subscriber_component_name: { # コンポーネントの名前\n",
    "        'componentVersion': subscriber_component_version,\n",
    "    },\n",
    "}\n",
    "\n",
    "response = deploy_component(production_group_arn, deployment_name, components)\n",
    "check_job_status(response[\"iotJobId\"], production_group[\"thing_names\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ffd5a4f",
   "metadata": {},
   "source": [
    "#### 本番機へのデプロイ結果確認\n",
    "* Staging と同じなので省略\n",
    "* 同じコマンドを打ってみてください  \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5264744e",
   "metadata": {},
   "source": [
    "## コンテナを利用したコンポーネントの開発\n",
    "* Greengrass で動かすアプリはコンテナでも動かすことができる\n",
    "* コンテナで下記の動作をするアプリを動かす\n",
    "    1. GAN で画像を生成(撮影相当)\n",
    "    2. 撮影した画像をMLのモデルで異常があるか判定\n",
    "    3. MQTT で判定内容をクラウドにパブリッシュ"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "198e8f5e",
   "metadata": {},
   "source": [
    "### コンテナイメージのビルド\n",
    "* ビルドする内容は`./src/ggv2/components/artifacts/com.example.IoTPublisher/1.0.0/`に配置済なので、Dockerfile の中身を最初に確認する\n",
    "* ビルドは docker の build コマンドを利用する  \n",
    "\n",
    "![イメージのビルド](./image/image09.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "32cbf2b0",
   "metadata": {},
   "outputs": [],
   "source": [
    "!cat ./src/ggv2/components/artifacts/com.example.IoTPublisher/1.0.0/Dockerfile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "437bbefd",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!pygmentize ./src/ggv2/components/artifacts/com.example.IoTPublisher/1.0.0/IoTPublisher.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "08e0751f",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Image のビルド\n",
    "image_name = 'com-example-iotpublisher'\n",
    "tag = ':1'\n",
    "\n",
    "%cd ./src/ggv2/components/artifacts/com.example.IoTPublisher/1.0.0/\n",
    "!cp ../../com.example.Publisher/1.0.0/1.h5 ./gan.h5\n",
    "!cp ../../com.example.Subscriber/1.0.0/1.h5 ./classifier.h5\n",
    "!docker rmi $(docker images -a -q)\n",
    "!docker build -t {image_name}{tag} .\n",
    "%cd ../../../../../../"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2666c5c9",
   "metadata": {},
   "source": [
    "### ECR へコンテナイメージをプッシュ\n",
    "* 使用する ECR の API は以下\n",
    "  * [get-login-password](https://docs.aws.amazon.com/cli/latest/reference/ecr/get-login-password.html)\n",
    "  * [(delete-repository)](https://docs.aws.amazon.com/cli/latest/reference/ecr/delete-repository.html) ※ハンズオンの本筋とは関係ない\n",
    "  * [create-repository](https://docs.aws.amazon.com/cli/latest/reference/ecr/create-repository.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3a966a18",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# boto3の機能を使ってリポジトリ名に必要な情報を取得する\n",
    "account_id = boto3.client('sts').get_caller_identity().get('Account')\n",
    "region = boto3.session.Session().region_name\n",
    "ecr_endpoint = f'{account_id}.dkr.ecr.{region}.amazonaws.com/' \n",
    "repository_uri = f'{ecr_endpoint}{image_name}'\n",
    "image_uri = f'{repository_uri}{tag}'\n",
    "\n",
    "!aws ecr get-login-password --region {region} | docker login --username AWS --password-stdin {ecr_endpoint}\n",
    "!docker tag {image_name}{tag} {image_uri}\n",
    "# 同名のリポジトリがあった場合は削除\n",
    "!aws ecr delete-repository --repository-name $image_name --force\n",
    "# リポジトリを作成\n",
    "!aws ecr create-repository --repository-name $image_name\n",
    "# イメージをプッシュ\n",
    "!docker push {image_uri}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "360d9464",
   "metadata": {},
   "source": [
    "### コンテナを使ったコンポーネントを開発機にローカルデプロイ\n",
    "![コンテナをローカルデプロイ](./image/image10.png)\n",
    "#### コンテナローカルデプロイ用 Recipe を作成\n",
    "* Lifecycle として doccker run で起動するように指定\n",
    "* Artifact として Image の URI を指定\n",
    "* ローカルでデプロイするには、yaml ファイルをローカル側にファイルで配置する必要があるので、S3 を介して配置する\n",
    "* 使用している API は S3 の [cp](https://docs.aws.amazon.com/cli/latest/reference/s3/cp.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "684ba069",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "recipe = f\"\"\"---\n",
    "RecipeFormatVersion: '2020-01-25'\n",
    "ComponentName: com.example.IoTPublisher\n",
    "ComponentVersion: '1.0.0'\n",
    "ComponentDescription: Publish MQTT message to AWS IoT Core in Docker image.\n",
    "ComponentPublisher: Amazon\n",
    "ComponentDependencies:\n",
    "  aws.greengrass.DockerApplicationManager:\n",
    "    VersionRequirement: ~2.0.0\n",
    "  aws.greengrass.TokenExchangeService:\n",
    "    VersionRequirement: ~2.0.0\n",
    "ComponentConfiguration:\n",
    "  DefaultConfiguration:\n",
    "    accessControl:\n",
    "      aws.greengrass.ipc.mqttproxy:\n",
    "        'com.example.IoTPublisher:dockerimage:latest':\n",
    "          policyDescription: Allows access to publish to inference/result.\n",
    "          operations:\n",
    "            - 'aws.greengrass#PublishToIoTCore'\n",
    "          resources:\n",
    "            - 'inference/result'\n",
    "\n",
    "Manifests:\n",
    "  - Platform:\n",
    "      os: all\n",
    "    Lifecycle:\n",
    "      Run: 'docker run --rm -v /greengrass/v2:/greengrass/v2 -e AWS_REGION=$AWS_REGION -e SVCUID=$SVCUID -e AWS_GG_NUCLEUS_DOMAIN_SOCKET_FILEPATH_FOR_COMPONENT=$AWS_GG_NUCLEUS_DOMAIN_SOCKET_FILEPATH_FOR_COMPONENT -e AWS_CONTAINER_AUTHORIZATION_TOKEN=$AWS_CONTAINER_AUTHORIZATION_TOKEN -e AWS_CONTAINER_CREDENTIALS_FULL_URI=$AWS_CONTAINER_CREDENTIALS_FULL_URI {image_uri}'\n",
    "    Artifacts:\n",
    "      - URI: \"docker:{image_uri}\"\n",
    "\n",
    "\"\"\"\n",
    "print(recipe)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dfda7ed8",
   "metadata": {},
   "outputs": [],
   "source": [
    "recipe_path = './src/ggv2/components/recipes/com.example.IotPublisher-1.0.0.yaml'\n",
    "with open(recipe_path,'w') as f:\n",
    "    f.write(recipe)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "714d69bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Publisher に必要なスクリプトとモデルを S3 にアップロード\n",
    "iotpublisher_recipe_uri = f's3://{bucket}/components/recipes/com.example.IoTPublisher-1.0.0.yaml'\n",
    "!aws s3 cp {recipe_path} {iotpublisher_recipe_uri}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5e1fecf",
   "metadata": {},
   "source": [
    "#### 開発機でコンテナコンポーネントのローカルデプロイ\n",
    "* デプロイの仕方は Publisher / Subrisher と同じ"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c573f78d",
   "metadata": {},
   "outputs": [],
   "source": [
    "recipe_dir = f'{base_dir}/components/recipes'\n",
    "component_name = 'com.example.IoTPublisher=1.0.0'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f3215157",
   "metadata": {},
   "outputs": [],
   "source": [
    "command_str=f\"\"\"#!/bin/bash\n",
    "su ubuntu\n",
    "cd {recipe_dir}\n",
    "aws s3 cp {iotpublisher_recipe_uri} ./\n",
    "sudo chown -R ubuntu:ubuntu {base_dir}\n",
    "sudo usermod -aG docker ggc_user\n",
    "sudo /greengrass/v2/bin/greengrass-cli component stop -n com.example.Publisher\n",
    "sudo /greengrass/v2/bin/greengrass-cli component stop -n com.example.Subscriber\n",
    "sudo /greengrass/v2/bin/greengrass-cli deployment create --recipeDir {recipe_dir} --merge \"{component_name}\"\n",
    "\"\"\"\n",
    "command_list = command_str.split('}\\n')\n",
    "print('EC2 に流すコマンド\\n')\n",
    "for command in command_list:\n",
    "    print(command)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "90eccd48",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "command_id = send_command(bucket,instance_id_list[0],command_list)\n",
    "response = check_command_result(command_id,instance_id_list[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3bc828c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9da8f5c",
   "metadata": {},
   "source": [
    "#### ローカルデプロイした IoTPublisher が MQTT でパブリッシュした内容を確認する\n",
    "* AWS マネジメントコンソールでトピックの [MQTT のテスト画面](https://ap-northeast-1.console.aws.amazon.com/iot/home?region=ap-northeast-1#/test)があるので、[./src/ggv2/components/artifacts/com.example.IoTPublisher/1.0.0/IoTPublisher.py](./src/ggv2/components/artifacts/com.example.IoTPublisher/1.0.0/IoTPublisher.py) にある通り、`inference/result` というトピックをサブスクライブする\n",
    "* 実際にはクラウドのサービスでサブスクライブして、後段の処理を行うが、コードでサブスクライブする場合はこちらのURLを参照\n",
    "  https://docs.aws.amazon.com/ja_jp/iot/latest/developerguide/sdk-tutorials.html"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01f9cdba",
   "metadata": {},
   "source": [
    "### コンテナを使ったコンポーネントをステージング機にクラウドデプロイ\n",
    "* 先程ローカルデプロイしたレシピを流用してコンポーネントを作成する\n",
    "![コンテナをステージングにクラウドデプロイ](./image/image11.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b91a5d40",
   "metadata": {},
   "source": [
    "#### 既存コンテナコンポーネントが存在する場合の削除\n",
    "* コンポーネントが重複するとエラーで落ちるので事前に削除する"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c31b08b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(recipe)\n",
    "iotpublisher_component_name = yaml.safe_load(recipe)['ComponentName']\n",
    "iotpublisher_component_version = yaml.safe_load(recipe)['ComponentVersion']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3ff2c7f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "result = ggv2_client.list_components()\n",
    "\n",
    "for component in result[\"components\"]:\n",
    "    if component[\"componentName\"] in [iotpublisher_component_name]:\n",
    "        response = ggv2_client.list_component_versions(\n",
    "            arn=component[\"arn\"]\n",
    "        )\n",
    "        for version in response[\"componentVersions\"]:\n",
    "            print(f\"delete: {version['componentName']} {version['componentVersion']}\")\n",
    "            response = ggv2_client.delete_component(\n",
    "                arn=version[\"arn\"]\n",
    "            )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e24613ba",
   "metadata": {},
   "source": [
    "#### コンテナを用いたコンポーネントの作成"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d5f7d3f",
   "metadata": {},
   "outputs": [],
   "source": [
    "response = ggv2_client.create_component_version(\n",
    "    inlineRecipe=recipe.encode(),\n",
    "    tags={\n",
    "        'Name': iotpublisher_component_name\n",
    "    }\n",
    ")\n",
    "iotpublisher_component_vesrion_arn = response['arn']\n",
    "\n",
    "response = ggv2_client.get_component(\n",
    "    recipeOutputFormat='YAML',\n",
    "    arn=iotpublisher_component_vesrion_arn\n",
    ")\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee7970f7",
   "metadata": {},
   "source": [
    "#### ステージング機のデプロイの改定\n",
    "* deployment はターゲット毎に決まっており、ステージング機へのデプロイは先程 Publisher / Subscriber コンポーネントをデプロイしたものがあるので、そのデプロイを IoTPublisherコンポーネントのデプロイで上書く\n",
    "* デプロイに Publisher / Subscriber を明記しないことで、Publisher / Subscriber の動作も止まる"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a7d2a50",
   "metadata": {},
   "outputs": [],
   "source": [
    "# デプロイの作成\n",
    "deployment_name = 'staging_deployment'\n",
    "components={\n",
    "    iotpublisher_component_name: { # コンポーネントの名前\n",
    "        'componentVersion': iotpublisher_component_version,\n",
    "    }\n",
    "}\n",
    "\n",
    "response = deploy_component(staging_group_arn, deployment_name, components)\n",
    "check_job_status(response[\"iotJobId\"], staging_group[\"thing_names\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d59c08db",
   "metadata": {},
   "source": [
    "#### ステージング機にクラウドデプロイした IoTPublisher が MQTT でパブリッシュした内容を確認する\n",
    "* 先程と同様 [MQTT のテスト画面](https://ap-northeast-1.console.aws.amazon.com/iot/home?region=ap-northeast-1#/test)で、inference/result というトピックをサブスクライブする\n",
    "* しかし、このままサブスクライブすると、困ったことに気づくので、本番機へのデプロイはやめて、コンポーネントを修正する  \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76cfb86f",
   "metadata": {},
   "source": [
    "## コンテナを利用したコンポーネントの更新\n",
    "* v1.0.0 では困ったことが発生し、どのエッジデバイスがデータを送ったのかがわからない。\n",
    "* 区別できるようにプログラムを変更してコンポーネントを更新する\n",
    "* 併せて classifier の ML モデルも更新する(2.h5は epoch を追加して学習しているため精度が少し向上している)  \n",
    "\n",
    "### コンテナイメージの更新\n",
    "![コンテナイメージの更新](./image/image12.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a4ae9b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "!cat ./src/ggv2/components/artifacts/com.example.IoTPublisher/1.0.1/Dockerfile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bc5fd83d",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!pygmentize ./src/ggv2/components/artifacts/com.example.IoTPublisher/1.0.1/IoTPublisher.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "caeffa30",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Image のビルド\n",
    "image_name = 'com-example-iotpublisher'\n",
    "tag = ':2' # タグを更新\n",
    "\n",
    "%cd ./src/ggv2/components/artifacts/com.example.IoTPublisher/1.0.1/\n",
    "!cp ../../com.example.Publisher/1.0.0/1.h5 ./gan.h5\n",
    "!cp ./2.h5 ./classifier.h5 # 分類器のモデルも変更\n",
    "!docker rmi $(docker images -a -q)\n",
    "!docker build -t {image_name}{tag} .\n",
    "%cd ../../../../../../"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7d9b96f",
   "metadata": {},
   "source": [
    "### ECR へ更新したコンテナイメージをプッシュ"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13420a40",
   "metadata": {},
   "outputs": [],
   "source": [
    "# boto3の機能を使ってリポジトリ名に必要な情報を取得する\n",
    "image_uri = f'{repository_uri}{tag}'\n",
    "\n",
    "!aws ecr get-login-password --region {region} | docker login --username AWS --password-stdin {ecr_endpoint}\n",
    "!docker tag {image_name}{tag} {image_uri}\n",
    "# イメージをプッシュ\n",
    "!docker push {image_uri}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca2b9c97",
   "metadata": {},
   "source": [
    "### 更新したコンテナを使ったコンポーネントを開発機にローカルデプロイ\n",
    "#### 更新したコンテナのローカルデプロイ用 Recipe を作成\n",
    "![開発機のコンテナを更新](./image/image13.png)\n",
    "* バージョンを上げている\n",
    "* THING_NAME を環境変数でコンテナに引き渡す\n",
    "* S3 に保存しておく"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a4a91cd",
   "metadata": {},
   "outputs": [],
   "source": [
    "recipe = f\"\"\"---\n",
    "RecipeFormatVersion: '2020-01-25'\n",
    "ComponentName: com.example.IoTPublisher\n",
    "ComponentVersion: '1.0.1'\n",
    "ComponentDescription: Publish MQTT message to AWS IoT Core in Docker image.\n",
    "ComponentPublisher: Amazon\n",
    "ComponentDependencies:\n",
    "  aws.greengrass.DockerApplicationManager:\n",
    "    VersionRequirement: ~2.0.0\n",
    "  aws.greengrass.TokenExchangeService:\n",
    "    VersionRequirement: ~2.0.0\n",
    "ComponentConfiguration:\n",
    "  DefaultConfiguration:\n",
    "    accessControl:\n",
    "      aws.greengrass.ipc.mqttproxy:\n",
    "        'com.example.IoTPublisher:dockerimage:latest':\n",
    "          policyDescription: Allows access to publish all topic.\n",
    "          operations:\n",
    "            - 'aws.greengrass#PublishToIoTCore'\n",
    "          resources:\n",
    "            - 'inference/result'\n",
    "\n",
    "Manifests:\n",
    "  - Platform:\n",
    "      os: all\n",
    "    Lifecycle:\n",
    "      Run: 'docker run --rm -v /greengrass/v2:/greengrass/v2 -e AWS_REGION=$AWS_REGION -e SVCUID=$SVCUID -e AWS_GG_NUCLEUS_DOMAIN_SOCKET_FILEPATH_FOR_COMPONENT=$AWS_GG_NUCLEUS_DOMAIN_SOCKET_FILEPATH_FOR_COMPONENT -e AWS_CONTAINER_AUTHORIZATION_TOKEN=$AWS_CONTAINER_AUTHORIZATION_TOKEN -e AWS_CONTAINER_CREDENTIALS_FULL_URI=$AWS_CONTAINER_CREDENTIALS_FULL_URI -e AWS_IOT_THING_NAME=$AWS_IOT_THING_NAME {image_uri}'\n",
    "    Artifacts:\n",
    "      - URI: \"docker:{image_uri}\"\n",
    "\n",
    "\"\"\"\n",
    "print(recipe)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85cadb15",
   "metadata": {},
   "outputs": [],
   "source": [
    "recipe_path = './src/ggv2/components/recipes/com.example.IotPublisher-1.0.1.yaml'\n",
    "with open(recipe_path,'w') as f:\n",
    "    f.write(recipe)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1298093d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# recipe を S3 にアップロード\n",
    "iotpublisher_recipe_uri = f's3://{bucket}/components/recipes/com.example.IoTPublisher-1.0.1.yaml'\n",
    "!aws s3 cp {recipe_path} {iotpublisher_recipe_uri}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82b40ad1",
   "metadata": {},
   "source": [
    "#### 開発機で更新したコンテナのローカルデプロイ"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "571fd8bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "recipe_dir = f'{base_dir}/components/recipes'\n",
    "component_name = 'com.example.IoTPublisher=1.0.1'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fbd5b418",
   "metadata": {},
   "outputs": [],
   "source": [
    "command_str=f\"\"\"#!/bin/bash\n",
    "su ubuntu\n",
    "cd {recipe_dir}\n",
    "aws s3 cp {iotpublisher_recipe_uri} ./\n",
    "sudo chown -R ubuntu:ubuntu {base_dir}\n",
    "sudo usermod -aG docker ggc_user\n",
    "sudo /greengrass/v2/bin/greengrass-cli deployment create --recipeDir {recipe_dir} --merge \"{component_name}\"\n",
    "\"\"\"\n",
    "command_list = command_str.split('}\\n')\n",
    "print('EC2 に流すコマンド\\n')\n",
    "for command in command_list:\n",
    "    print(command)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "723e9806",
   "metadata": {},
   "outputs": [],
   "source": [
    "command_id = send_command(bucket,instance_id_list[0],command_list)\n",
    "response = check_command_result(command_id,instance_id_list[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0367c32a",
   "metadata": {},
   "source": [
    "#### 更新したコンテナのローカルデプロイ結果を確認（開発機）\n",
    "* 先程と同様 [MQTT のテスト画面](https://ap-northeast-1.console.aws.amazon.com/iot/home?region=ap-northeast-1#/test)で、inference/result というトピックをサブスクライブする\n",
    "* THING_NAME というキーにパブリッシュしたデバイスの THING_NAME が格納されていることを確認する\n",
    "* ステージングも動いているので、THING_NAMEがないメッセージも混在している\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4bbd8da",
   "metadata": {},
   "source": [
    "### 更新したコンテナを使ったコンポーネントをステージング機にクラウドデプロイ\n",
    "![ステージング機のコンテナを更新](./image/image14.png)\n",
    "#### 更新したコンテナを用いたコンポーネントの作成"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7855a811",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(recipe)\n",
    "iotpublisher_component_name = yaml.safe_load(recipe)['ComponentName']\n",
    "iotpublisher_component_version = yaml.safe_load(recipe)['ComponentVersion']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "191d977e",
   "metadata": {},
   "outputs": [],
   "source": [
    "response = ggv2_client.create_component_version(\n",
    "    inlineRecipe=recipe.encode(),\n",
    "    tags={\n",
    "        'Name': iotpublisher_component_name\n",
    "    }\n",
    ")\n",
    "iotpublisher_component_vesrion_arn = response['arn']\n",
    "\n",
    "response = ggv2_client.get_component(\n",
    "    recipeOutputFormat='YAML',\n",
    "    arn=iotpublisher_component_vesrion_arn\n",
    ")\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d664e59",
   "metadata": {},
   "source": [
    "#### 更新したコンテナを用いたステージング機のデプロイの改定"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "86ef7582",
   "metadata": {},
   "outputs": [],
   "source": [
    "# デプロイの作成\n",
    "deployment_name = 'staging_deployment'\n",
    "components={\n",
    "    iotpublisher_component_name: { # コンポーネントの名前\n",
    "        'componentVersion': iotpublisher_component_version,\n",
    "    }\n",
    "}\n",
    "response = deploy_component(staging_group_arn, deployment_name, components)\n",
    "check_job_status(response[\"iotJobId\"], staging_group[\"thing_names\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "449b4b03",
   "metadata": {},
   "source": [
    "#### ステージング機で更新したコンテナの MQTT 動作確認\n",
    "* 先程と同様 [MQTT のテスト画面](https://ap-northeast-1.console.aws.amazon.com/iot/home?region=ap-northeast-1#/test)で、inference/result というトピックをサブスクライブする\n",
    "* 今度は メッセージに THING_NAME が含まれているため、どのデバイスがそのメッセージを送信したかがわかるようになっている\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e152bf33",
   "metadata": {},
   "source": [
    "#### 更新したコンテナを用いた本番機のデプロイの改定\n",
    "ステージングで動作を確認し、メッセージでどのデバイスが発したものかがわかったので、本番機にデプロイする\n",
    "![本番機のコンテナを更新](./image/image15.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1709202",
   "metadata": {},
   "outputs": [],
   "source": [
    "# デプロイの作成\n",
    "deployment_name = 'production_deployment' # 変えるのはここだけ\n",
    "\n",
    "response = deploy_component(production_group_arn, deployment_name, components)\n",
    "check_job_status(response[\"iotJobId\"], production_group[\"thing_names\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff45c83a",
   "metadata": {},
   "source": [
    "#### 本番機で更新したコンテナの MQTT 動作確認\n",
    "* 先程と同様 [MQTT のテスト画面](https://ap-northeast-1.console.aws.amazon.com/iot/home?region=ap-northeast-1#/test)で、inference/result というトピックをサブスクライブする\n",
    "* 今度は メッセージに THING_NAME が含まれているため、どのデバイスがそのメッセージを送信したかがわかるようになっている\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "14ebe291",
   "metadata": {},
   "outputs": [],
   "source": [
    "# デプロイの作成\n",
    "deployment_name = 'production_deployment' # 変えるのはここだけ\n",
    "response = deploy_component(staging_group_arn, deployment_name, components)\n",
    "check_job_status(response[\"iotJobId\"], staging_group[\"thing_names\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3bc9cb00",
   "metadata": {},
   "outputs": [],
   "source": [
    "!date"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dcfd1525",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}