Skip to content

chzbrgr71/image-classification

Repository files navigation

Tensorflow Image Classification Demo

Build training container image and run local

  • Training

    # set image tag depending on target cpu/gpu
    export IMAGE_TAG=2.1
    export IMAGE_TAG=2.1-gpu
    export ACRNAME=briaracr
    
    # build/push (ACR or Docker)
    az acr build -t chzbrgr71/image-retrain:$IMAGE_TAG -r $ACRNAME ./training
    
    docker build -t chzbrgr71/image-retrain:$IMAGE_TAG -f ./training/Dockerfile ./training
    docker push chzbrgr71/image-retrain:$IMAGE_TAG
    
    # run local
    docker run -d --name train --volume /Users/brianredmond/gopath/src/github.com/chzbrgr71/image-classification/tf-output:/tf-output chzbrgr71/image-retrain:$IMAGE_TAG "--how_many_training_steps=1000" "--learning_rate=0.01" "--bottleneck_dir=/tf-output/bottlenecks" "--model_dir=/tf-output/inception" "--summaries_dir=/tf-output/training_summaries/baseline" "--output_graph=/tf-output/retrained_graph.pb" "--output_labels=/tf-output/retrained_labels.txt" "--image_dir=images" "--saved_model_dir=/tf-output/saved_models/1"
  • Tensorboard

    export IMAGE_TAG=2.0
    
    # build/push (ACR or Docker)
    az acr build -t chzbrgr71/tensorboard:$IMAGE_TAG -r $ACRNAME -f ./training/Dockerfile.tensorboard ./training
    
    docker build -t chzbrgr71/tensorboard:$IMAGE_TAG -f ./training/Dockerfile.tensorboard ./training
    docker push chzbrgr71/tensorboard:$IMAGE_TAG
    
    # run
    docker run -d --name tensorboard -p 80:6006 --volume /Users/brianredmond/gopath/src/github.com/chzbrgr71/image-classification/tf-output:/tf-output chzbrgr71/tensorboard:$IMAGE_TAG "--logdir" "/tf-output/training_summaries"
  • Test locally - label Image with trained model

    export IMAGE_TAG=2.0
    
    # build
    docker build -t chzbrgr71/tf-testing:$IMAGE_TAG -f ./label-image/Dockerfile ./label-image
    
    # run
    docker run --rm --name tf-testing --volume /Users/brianredmond/gopath/src/github.com/chzbrgr71/image-classification/label-image:/image chzbrgr71/tf-testing:$IMAGE_TAG /image/edsheeran.jpg
    I am confident this is Ed Sheeran (0.962828)
    
    docker run --rm --name tf-testing --volume /Users/brianredmond/gopath/src/github.com/chzbrgr71/image-classification/label-image:/image chzbrgr71/tf-testing:$IMAGE_TAG /image/bradpitt.jpg
    This is not Ed Sheeran (0.048971623)

Setup Kubernetes

  • Use AKS or acs-engine with kubernetes v1.11.8 or newer

  • Enable Helm

    kubectl create -f ./k8s-setup/tiller-rbac-config.yaml
    
    helm init --service-account tiller --upgrade
  • Installing NVIDIA Device Plugin (AKS with GPU only)

    For AKS v1.11 and above, install NVIDIA Device Plugin using:

    kubectl apply -f https://raw.githubusercontent.com/nvidia/k8s-device-plugin/v1.11/nvidia-device-plugin.yml
  • Scale when needed

    # aks
    az aks scale -n $CLUSTERNAME -g $RGNAME -c 1 --no-wait
    
    # aks-engine
    az vmss scale -n $VMSSNAME -g $RGNAME --new-capacity 1 --no-wait
  • Virtual Kubelet/Node. https://github.com/virtual-kubelet/virtual-kubelet/tree/aci-gpu

    Update Helm Chart YAML

    targetAKS: true
    clientId:
    clientKey:
    tenantId:
    subscriptionId:
    aciResourceGroup: 
    aciRegion: 

    Then install the Virtual Kubelet chart in your cluster

    export VK_RELEASE=virtual-kubelet-latest
    export CHART_URL=https://github.com/virtual-kubelet/virtual-kubelet/raw/master/charts/$VK_RELEASE.tgz
    helm install --name vk "$CHART_URL" -f ./k8s-setup/vk-helm-values.yaml
    
    kubectl get nodes
    ...
    NAME                       STATUS   ROLES   AGE   VERSION
    aks-nodepool1-90656626-0   Ready    agent   15m   v1.11.8
    aks-nodepool1-90656626-1   Ready    agent   15m   v1.11.8
    aks-nodepool1-90656626-2   Ready    agent   15m   v1.11.8
    aks-nodepool1-90656626-3   Ready    agent   15m   v1.11.8
    virtual-kubelet            Ready    agent   4s    v1.11.2

Install Kubeflow

  • First, install ksonnet version 0.13.1.

  • Then run the following commands to deploy Kubeflow in your Kubernetes cluster:

    export KUBEFLOW_SRC=kubeflow
    export KUBEFLOW_TAG=v0.4.1
    
    mkdir ${KUBEFLOW_SRC}
    cd ${KUBEFLOW_SRC}
    
    curl https://raw.githubusercontent.com/kubeflow/kubeflow/${KUBEFLOW_TAG}/scripts/download.sh | bash
    cd ..

    KUBEFLOW_SRC a directory where you want to download the source

    KUBEFLOW_TAG a tag corresponding to the version to check out, such as master for the latest code

    # Initialize a kubeflow app
    KFAPP=mykubeflowapp
    ${KUBEFLOW_SRC}/scripts/kfctl.sh init ${KFAPP} --platform none
    
    # Generate kubeflow app
    cd ${KFAPP}
    ../${KUBEFLOW_SRC}/scripts/kfctl.sh generate k8s
    
    # Deploy Kubeflow app
    ../${KUBEFLOW_SRC}/scripts/kfctl.sh apply k8s
    
    # Validate install
    kubectl get pods -n kubeflow

Setup Azure Storage

Three choices for storage:

  1. Azure Disks

    • Create PVC (using either default or managed-premium storage class)
    kubectl create -f ./k8s-setup/azure-disk-pvc.yaml
    • Check status
    kubectl get pvc
    
    NAME                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
    azure-managed-disk   Bound    pvc-f2f9a107-40f1-11e9-bd08-122f8aa7945e   8Gi        RWO            managed-premium   44s
  2. Azure Files Dynamic

    • Create storage account

    Note: this storage account must be created in the AKS MC_ RG

    export RG_NAME=MC_briar-aks-ml-02_briar-aksgpu-01_eastus
    export LOCATION=eastus
    export STORAGE=briartfjob01 #rename this. must be unique
    
    az storage account create --resource-group $RG_NAME --name $STORAGE --location $LOCATION --sku Standard_LRS
    • Setup StorageClass and PVC's
    kubectl create -f ./k8s-setup/azure-files-dynamic-sc.yaml
    kubectl create -f ./k8s-setup/azure-files-dynamic-pvc.yaml
    • Check status
    kubectl get pvc
    
    NAME                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          AGE
    pvc-azure-files-dynamic   Bound    pvc-d08933c3-40f2-11e9-bd08-122f8aa7945e   10Gi       RWX            azure-files-dynamic   5s
  3. Azure Files Static

    export AKS_PERS_STORAGE_ACCOUNT_NAME=briar$RANDOM
    export AKS_PERS_RESOURCE_GROUP=briar-aks-ml-02
    export AKS_PERS_LOCATION=eastus
    export AKS_PERS_SHARE_NAME=aksshare
    
    # Create the storage account
    az storage account create -n $AKS_PERS_STORAGE_ACCOUNT_NAME -g $AKS_PERS_RESOURCE_GROUP -l $AKS_PERS_LOCATION --sku Standard_LRS
    
    # Export the connection string as an environment variable, this is used when creating the Azure file share
    export AZURE_STORAGE_CONNECTION_STRING=`az storage account show-connection-string -n $AKS_PERS_STORAGE_ACCOUNT_NAME -g $AKS_PERS_RESOURCE_GROUP -o tsv`
    
    # Create the file share
    az storage share create -n $AKS_PERS_SHARE_NAME
    
    # Get storage account key
    STORAGE_KEY=$(az storage account keys list --resource-group $AKS_PERS_RESOURCE_GROUP --account-name $AKS_PERS_STORAGE_ACCOUNT_NAME --query "[0].value" -o tsv)
    
    # Echo storage account name and key
    echo Storage account name: $AKS_PERS_STORAGE_ACCOUNT_NAME
    echo Storage account key: $STORAGE_KEY
    
    kubectl create secret generic azure-file-secret --from-literal=azurestorageaccountname=$AKS_PERS_STORAGE_ACCOUNT_NAME --from-literal=azurestorageaccountkey=$STORAGE_KEY

Run Training on Kubeflow

  • Image Classification re-training k8s Job (Inception)

    kubectl apply -f ./training/training-job-aci.yaml
  • Image Classification re-training TFJob (Inception)

    # TFJob
    helm install --name tfjob-image-training-01 --set container.image=briaracr.azurecr.io/chzbrgr71/image-retrain,container.imageTag=2.0-gpu,container.pvcName=pvc-azure-files-dynamic,container.subPath=tfjob-image-training-01,tfjob.name=tfjob-image-training-01 ./training/chart
    
    helm install --name tfjob-image-training-02 --set container.image=briaracr.azurecr.io/chzbrgr71/image-retrain,container.imageTag=2.1-gpu,container.pvcName=pvc-azure-files-dynamic,container.subPath=tfjob-image-training-02,tfjob.name=tfjob-image-training-02 ./training/chart
    
    helm install --name tfjob-image-training-03 --set container.image=briaracr.azurecr.io/chzbrgr71/image-retrain,container.imageTag=2.1-gpu,container.pvcName=pvc-azure-files-dynamic,container.subPath=tfjob-image-training-03,tfjob.name=tfjob-image-training-03 ./training/chart
    
    # Tensorboard
    helm install --name tb-image-training-01 --set tensorboard.name=tb-image-training-01,container.image=briaracr.azurecr.io/chzbrgr71/tensorboard,container.imageTag=2.0-gpu,container.pvcName=pvc-azure-files-dynamic,container.subPath=tfjob-image-training-01 ./training/tensorboard-chart
    # testing
    export IP=13.68.227.154
    
    curl -F "image.jpg=@/Users/brianredmond/gopath/src/github.com/chzbrgr71/image-classification/label-image/edsheeran.jpg" http://$IP:5000/detect_image
    
    curl -F "image.jpg=@/Users/brianredmond/gopath/src/github.com/chzbrgr71/image-classification/label-image/brianredmond.jpg" http://$IP:5000/detect_image
  • Download model (while TB pod is running)

    # to download model from pod
    PODNAME=
    kubectl cp default/$PODNAME:/tmp/tensorflow/tf-output/retrained_graph.pb ~/Downloads/retrained_graph.pb
    kubectl cp default/$PODNAME:/tmp/tensorflow/tf-output/retrained_labels.txt ~/Downloads/retrained_labels.txt
  • Run Tensorboard manually (using tensorboard-standalone.yaml)

    # exec into pod
    tensorboard --logdir /tf-output/training_summaries
  • Test locally

    docker run -it --rm --name tf \
        --publish 6006:6006 \
        --volume /Users/brianredmond/gopath/src/github.com/chzbrgr71:/brianredmond \
        --workdir /brianredmond  \
        tensorflow/tensorflow:1.9.0 bash
    
    python label-image.py edsheeran.jpg
    python label-image.py bradpitt.jpg
    python label-image.py ed-sheeran-puppet.jpg

Hyperparameter Sweep Demo

This step requires 6-7 nodes in k8s. Uses the same container image as standard image re-training.

helm install --name tfjob-hyperparam1 --set tfjob.name=tfjob-hyperparam1,container.image=briaracr.azurecr.io/chzbrgr71/image-retrain:2.0-gpu,container.pvcName=pvc-azure-files-dynamic,container.subPath=tfjob-hyperparam1 ./hyperparameter/chart

helm install --name tb-hyperparam1 --set tensorboard.name=tb-hyperparam1,container.pvcName=pvc-azure-files-dynamic,container.subPath=tfjob-hyperparam1 ./hyperparameter/tensorboard-chart

ACI + Virtual Kubelet

helm install --name tfjob-hyperparam-vk --set image=chzbrgr71/image-retrain:2.1-gpu,useGPU=true ./hyperparameter/chart-vk

Pytorch Operator

# mnist
kubectl apply -f ./pytorch/pytorch_job_mnist_gloo.yaml

# smoke-dist
kubectl apply -f ./pytorch/pytorch_job_sendrecv.yaml

Chainer

kubectl apply -f ./chainer/chainer-job-mn.yaml

Distributed Tensorflow

This step requires 4 nodes in VMSS.

  • Create Docker image

    export IMAGE_TAG=1.8
    
    # build
    docker build -t chzbrgr71/distributed-tf:$IMAGE_TAG -f ./dist-training/Dockerfile ./dist-training
    
    # push
    docker push chzbrgr71/distributed-tf:$IMAGE_TAG
  • Helm Chart

    helm install --set container.image=briar.azurecr.io/chzbrgr71/distributed-tf,container.imageTag=1.8,training.workercount=2,container.pvcName=azure-files,tfjob.name=tfjob-dist-training2 ./dist-training/chart
    
    helm install --set tensorboard.name=tensorboard-dist-training2,container.pvcName=azure-files,container.subPath=tfjob-dist-training2 ./training/tensorboard-chart

TensorFlow Serving

  • Local

    docker run -d --rm --name serving_base tensorflow/serving:1.9.0
    docker cp ./tf-output/saved_models serving_base:/models/inception
    docker commit --change "ENV MODEL_NAME inception" serving_base chzbrgr71/edsheeran_serving:2.0
    docker kill serving_base
    docker run -p 8500:8500 -t chzbrgr71/edsheeran_serving:2.0 &
    
    python serving/inception_client.py --server localhost:8500 --image ./label-image/edsheeran.jpg
    python serving/inception_client.py --server localhost:8500 --image ./label-image/bradpitt.jpg
    python serving/inception_client.py --server localhost:8500 --image ./label-image/brianredmond.jpg

Model Serving (Flask App)

  • This demo is in a separate repo. https://github.com/chzbrgr71/flask-tf

  • Python Flask App

    Running local:

    FLASK_APP=app.py FLASK_DEBUG=1 python -m flask run
    
    curl -F "image.jpg=@/Users/brianredmond/gopath/src/github.com/chzbrgr71/image-classification/label-image/edsheeran.jpg" http://localhost:5000/detect_image
    
    curl -F "image.jpg=@/Users/brianredmond/gopath/src/github.com/chzbrgr71/image-classification/label-image/bradpitt.jpg" http://localhost:5000/detect_image
    
    curl -F "image.jpg=@/Users/brianredmond/gopath/src/github.com/chzbrgr71/image-classification/label-image/brianredmond.jpg" http://localhost:5000/detect_image

    In container:

    IMAGE_TAG=1.5
    
    docker build -t chzbrgr71/edsheeran-flask-app:$IMAGE_TAG -f ./flask-app/Dockerfile ./flask-app
    
    docker push chzbrgr71/edsheeran-flask-app:$IMAGE_TAG
    docker tag chzbrgr71/edsheeran-flask-app:$IMAGE_TAG briarhackfest.azurecr.io/chzbrgr71/edsheeran-flask-app:$IMAGE_TAG
    docker push briarhackfest.azurecr.io/chzbrgr71/edsheeran-flask-app:$IMAGE_TAG
    
    docker run -d --name flask -p 5000:5000 chzbrgr71/edsheeran-flask-app:$IMAGE_TAG
    
    helm upgrade --install flask-tf --set deploy.image=briarhackfest.azurecr.io/chzbrgr71/edsheeran-flask-app,deploy.imageTag=$IMAGE_TAG ./flask-app/chart

    Testing:

    IMAGE=edsheeran.jpg
    IMAGE=bradpitt.jpg
    IMAGE=brianredmond.jpg
    IMAGE=ed-sheeran-puppet.jpg
    
    IP=13.68.208.133 && curl -F "image.jpg=@/Users/brianredmond/gopath/src/github.com/chzbrgr71/image-classification/label-image/$IMAGE" http://$IP:5000/detect_image

Azure Container Registry Tasks Demo

  • This demo is in a separate repo. https://github.com/chzbrgr71/image-training

    ACR_NAME=briaracr    
    GIT_PAT=
    SLACK_WEBHOOK=
    
    az acr task create \
        --registry $ACR_NAME \
        --name tf-image-training \
        --context https://github.com/chzbrgr71/image-training.git \
        --branch master \
        --file acr-task.yaml \
        --git-access-token $GIT_PAT \
        --set-secret SLACK_WEBHOOK=$SLACK_WEBHOOK
  • Create helm container for deployment with service principal (Steve Lasker) https://github.com/AzureCR/cmd/tree/master/helm

  • ACR Task / Github Webhook

    ACR_NAME=
    GIT_PAT=
    SLACK_WEBHOOK=
    SP=
    PASSWORD=
    TENANT=
    CLUSTER_RESOURCE_GROUP=
    CLUSTER_NAME=
    
    az acr task create \
        --registry $ACR_NAME \
        --name flask-tf \
        --context https://github.com/chzbrgr71/flask-tf.git \
        --branch master \
        --file acr-task.yaml \
        --git-access-token $GIT_PAT \
        --set-secret SLACK_WEBHOOK=$SLACK_WEBHOOK \
        --set-secret SP=$SP \
        --set-secret PASSWORD=$PASSWORD \
        --set-secret TENANT=$TENANT \
        --set-secret CLUSTER_RESOURCE_GROUP=$CLUSTER_RESOURCE_GROUP \
        --set-secret CLUSTER_NAME=$CLUSTER_NAME

Argo Workflow

  • Set environment variables for Argo workflow

    # namespace of all the kubeflow components
    export NAMESPACE=kubeflow
    export AZURE_STORAGEACCOUNT_NAME=minio 
    export AZURE_STORAGEACCOUNT_KEY=minio123
    MINIOIP=$(kubectl get svc minio-service -n ${NAMESPACE} -o jsonpath='{.spec.clusterIP}')
    MINIOPORT=$(kubectl get svc minio-service -n ${NAMESPACE} -o jsonpath='{.spec.ports[0].port}')
    
    export S3_ENDPOINT=${MINIOIP}:$MINIOPORT
    export AWS_ENDPOINT_URL=${S3_ENDPOINT}
    export AWS_ACCESS_KEY_ID=$AZURE_STORAGEACCOUNT_NAME
    export AWS_SECRET_ACCESS_KEY=$AZURE_STORAGEACCOUNT_KEY
    export BUCKET_NAME=mybucket
    
    export DOCKER_BASE_URL=docker.io/chzbrgr71 # Update this to fit your scenario
    export S3_DATA_URL=s3://${BUCKET_NAME}/data/retrain/
    export S3_TRAIN_BASE_URL=s3://${BUCKET_NAME}/models
    export AWS_REGION=us-east-1
    export JOB_NAME=myjob-$(uuidgen  | cut -c -5 | tr '[:upper:]' '[:lower:]')
    export TF_MODEL_IMAGE=${DOCKER_BASE_URL}/image-retrain:2.1-gpu
    export TF_WORKER=3
    export MODEL_TRAIN_STEPS=200
    
    # Create a secret for accessing the Minio server
    kubectl create secret generic aws-creds --from-literal=awsAccessKeyID=${AWS_ACCESS_KEY_ID} \
    --from-literal=awsSecretAccessKey=${AWS_SECRET_ACCESS_KEY} -n ${NAMESPACE}
    
    # Create a user for the workflow
    kubectl apply -f workflow/tf-user.yaml -n ${NAMESPACE}
  • Submit a workflow to Argo

    argo submit workflow/model-train-serve-workflow.yaml -n ${NAMESPACE} --serviceaccount tf-user \
    -p aws-endpoint-url=${AWS_ENDPOINT_URL} \
    -p s3-endpoint=${S3_ENDPOINT} \
    -p aws-region=${AWS_REGION} \
    -p tf-model-image=${TF_MODEL_IMAGE} \
    -p s3-data-url=${S3_DATA_URL} \
    -p s3-train-base-url=${S3_TRAIN_BASE_URL} \
    -p job-name=${JOB_NAME} \
    -p tf-worker=${TF_WORKER} \
    -p model-train-steps=${MODEL_TRAIN_STEPS} \
    -p namespace=${NAMESPACE} \
    -p tf-tensorboard-image=tensorflow/tensorflow:1.7.0 \
    -p s3-use-https=0 \
    -p s3-verify-ssl=0
    
    # Check status of the workflow
    argo list -n ${NAMESPACE}
    NAME                STATUS    AGE    DURATION
    tf-workflow-s8k24   Running   5m     5m 
    
    # Check pods that are created by the workflow
    kubectl get pod -n ${NAMESPACE} -o wide -w
    
    # Monitor training from tensorboard
    PODNAME=$(kubectl get pod -n ${NAMESPACE} -l app=tensorboard-${JOB_NAME} -o jsonpath='{.items[0].metadata.name}')
    kubectl port-forward ${PODNAME} -n ${NAMESPACE} 6006:6006
    
    # Get logs from the training pod(s)
    kubectl logs ${JOB_NAME}-master-0 -n ${NAMESPACE} 

About

Binary image classification tensorflow demo

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published