From 6ce7953b451ef25f1607622aff75936b799506b5 Mon Sep 17 00:00:00 2001 From: Eytan Avisror Date: Mon, 11 Jan 2021 21:33:14 -0800 Subject: [PATCH 01/74] Delete README.md Signed-off-by: Eytan Avisror --- README.md | 84 ------------------------------------------------------- 1 file changed, 84 deletions(-) delete mode 100644 README.md diff --git a/README.md b/README.md deleted file mode 100644 index d5fe45e2..00000000 --- a/README.md +++ /dev/null @@ -1,84 +0,0 @@ -# RollingUpgrade - -![Build Status](https://github.com/keikoproj/upgrade-manager/workflows/Build-Test/badge.svg) ![Build Status](https://github.com/keikoproj/upgrade-manager/workflows/BDD/badge.svg) [![codecov](https://codecov.io/gh/keikoproj/upgrade-manager/branch/master/graph/badge.svg)](https://codecov.io/gh/keikoproj/upgrade-manager) - -> Reliable, extensible rolling-upgrades of Autoscaling groups in Kubernetes - -RollingUpgrade provides a Kubernetes native mechanism for doing rolling-updates of instances in an AutoScaling group using a CRD and a controller. - -## What does it do? - -- RollingUpgrade is highly inspired by the way kops does rolling-updates. - -- It provides similar options for the rolling-updates as kops and more. - -- The RollingUpgrade Kubernetes custom resource has the following options in the spec: - - `asgName`: Name of the autoscaling group to perform the rolling-update. - - `preDrain.script`: The script to run before draining a node. - - `postDrain.script`: The script to run after draining a node. This allows for performing actions such as quiescing network traffic, adding labels, etc. - - `postDrain.waitSeconds`: The seconds to wait after a node is drained. - - `postDrain.postWaitScript`: The script to run after the node is drained and the waitSeconds have passed. This can be used for ensuring that the drained pods actually were able to start elsewhere. - - `nodeIntervalSeconds`: The amount of time in seconds to wait after each node in the ASG is terminated. - - `postTerminate.script`: Optional bash script to execute after the node has terminated. - - `strategy.mode`: This field is optional and allows for two possible modes - - `lazy` - this is the default mode, upgrade will terminate an instance first. - - `eager` - upgrade will launch an instance prior to terminating. - - `strategy.type`: This field is optional and currently two strategies are supported - - `randomUpdate` - Default is type is not specified. Picks nodes randomly for updating. Refer to [random_update_strategy.yaml](examples/random_update_strategy.yaml) for sample custom resource definition. - - `uniformAcrossAzUpdate` - Picks same number of nodes or same percentage of nodes from each AZ for update. Refer to [uniform_across_az_update_strategy.yaml](examples/uniform_across_az_update_strategy.yaml) for sample custom resource definition. - - `strategy.maxUnavailable`: Optional field. The number of nodes that can be unavailable during rolling upgrade, can be specified as number of nodes or the percent of total number of nodes. Default is "1". - - `strategy.drainTimeout`: Optional field. Node will be terminated after drain timeout even if `kubectl drain` has not been completed and value has to be specified in seconds. Default is -1. - -- After performing the rolling-update of the nodes in the ASG, RollingUpgrade puts the following data in the "Status" field. - - `currentStatus`: Whether the rolling-update completed or errored out. - - `startTime`: The RFC3339 timestamp when the rolling-update began. E.g. 2019-01-15T23:51:10Z - - `endTime`: The RFC3339 timestamp when the rolling-update completed. E.g. 2019-01-15T00:35:10Z - - `nodesProcessed`: The number of ec2 instances that were processed. - - `conditions`: Conditions describing the lifecycle of the rolling-update. - -## Design - -For each RollingUpgrade custom resource that is submitted, the following flowchart shows the sequence of actions taken to [perform the rolling-update](docs/RollingUpgradeDesign.png) - -## Dependencies - -- Kubernetes cluster on AWS with nodes in AutoscalingGroups. rolling-upgrades have been tested with Kubernetes clusters v1.12+. -- An IAM role with at least the policy specified below. The upgrade-manager should be run with that IAM role. - -## Installing - -### Complete step by step guide to create a cluster and run rolling-upgrades - -For a complete, step by step guide for creating a cluster with kops, editing it and then running rolling-upgrades, please see [this](docs/step-by-step-example.md) - -### Existing cluster in AWS - -If you already have an existing cluster created using kops, follow the instructions below. - -- Ensure that you have a Kubernetes cluster on AWS. -- Install the CRD using: `kubectl apply -f https://raw.githubusercontent.com/keikoproj/upgrade-manager/master/config/crd/bases/upgrademgr.keikoproj.io_rollingupgrades.yaml` -- Install the controller using: -`kubectl create -f https://raw.githubusercontent.com/keikoproj/upgrade-manager/master/deploy/rolling-upgrade-controller-deploy.yaml` - -- Note that the rolling-upgrade controller requires an IAM role with the following policy - -``` json -{ - "Effect": "Allow", - "Action": [ - "ec2:CreateTags", - "ec2:DescribeInstances", - "autoscaling:EnterStandby", - "autoscaling:DescribeAutoScalingGroups", - "autoscaling:TerminateInstanceInAutoScalingGroup" - ], - "Resource": [ - "*" - ] -} -``` - -- If the rolling-upgrade controller is directly using the IAM role of the node it runs on, the above policy will have to be added to the IAM role of the node. -- If the rolling-upgrade controller is using it's own role created using KIAM, that role should have the above policy in it. - -## For more details and FAQs, refer to [this](docs/faq.md) From 3ad13b846ffb793a653294581cc899c1d853d806 Mon Sep 17 00:00:00 2001 From: Eytan Avisror Date: Mon, 11 Jan 2021 21:34:49 -0800 Subject: [PATCH 02/74] delete all Signed-off-by: Eytan Avisror --- .github/workflows/bdd.yaml | 44 - .github/workflows/ci.yaml | 83 - Dockerfile | 36 - LICENSE | 201 -- Makefile | 75 - PROJECT | 7 - api/v1alpha1/groupversion_info.go | 35 - api/v1alpha1/rollingupgrade_types.go | 163 - api/v1alpha1/rollingupgrade_types_test.go | 83 - api/v1alpha1/suite_test.go | 74 - api/v1alpha1/zz_generated.deepcopy.go | 227 -- config/certmanager/certificate.yaml | 24 - config/certmanager/kustomization.yaml | 26 - config/certmanager/kustomizeconfig.yaml | 16 - ...grademgr.keikoproj.io_rollingupgrades.yaml | 170 - config/crd/kustomization.yaml | 19 - config/crd/kustomizeconfig.yaml | 17 - .../cainjection_in_rollingupgrades.yaml | 8 - .../patches/webhook_in_rollingupgrades.yaml | 17 - config/default/kustomization.yaml | 43 - config/default/manager_auth_proxy_patch.yaml | 24 - config/default/manager_image_patch.yaml | 12 - .../manager_prometheus_metrics_patch.yaml | 19 - config/default/manager_webhook_patch.yaml | 23 - config/default/webhookcainjection_patch.yaml | 15 - config/manager/kustomization.yaml | 2 - config/manager/manager.yaml | 39 - config/rbac/auth_proxy_role.yaml | 13 - config/rbac/auth_proxy_role_binding.yaml | 12 - config/rbac/auth_proxy_service.yaml | 18 - config/rbac/kustomization.yaml | 11 - config/rbac/leader_election_role.yaml | 26 - config/rbac/leader_election_role_binding.yaml | 12 - config/rbac/role.yaml | 69 - config/rbac/role_binding.yaml | 12 - .../upgrademgr_v1alpha1_rollingupgrade.yaml | 7 - config/webhook/kustomization.yaml | 6 - config/webhook/kustomizeconfig.yaml | 25 - config/webhook/manifests.yaml | 0 config/webhook/service.yaml | 12 - controllers/events.go | 75 - controllers/events_test.go | 37 - controllers/helpers.go | 174 - controllers/helpers_test.go | 338 -- controllers/launch_definition.go | 27 - controllers/node_selector.go | 19 - controllers/node_selector_test.go | 88 - controllers/random_node_selector.go | 27 - controllers/rollingupgrade_controller.go | 1185 ------- controllers/rollingupgrade_controller_test.go | 2979 ----------------- controllers/rollup_cluster_state.go | 176 - controllers/rollup_cluster_state_test.go | 157 - controllers/script_runner.go | 184 - controllers/script_runner_test.go | 52 - controllers/suite_test.go | 79 - .../uniform_across_az_node_selector.go | 61 - .../uniform_across_az_node_selector_test.go | 109 - deploy/rolling-upgrade-controller-deploy.yaml | 67 - docs/RollingUpgradeDesign.png | Bin 229125 -> 0 bytes docs/faq.md | 53 - docs/step-by-step-example.md | 193 -- examples/basic.yaml | 9 - examples/pre_post_drain.yaml | 22 - examples/random_update_strategy.yaml | 26 - .../uniform_across_az_update_strategy.yaml | 26 - go.mod | 24 - go.sum | 695 ---- hack/boilerplate.go.txt | 14 - main.go | 204 -- pkg/log/log.go | 205 -- pkg/log/retry_logger.go | 44 - test-bdd/bases/kustomization.yaml | 8 - test-bdd/features/01_create.feature | 18 - test-bdd/main_test.go | 96 - test-bdd/templates/rolling-upgrade.yaml | 23 - 75 files changed, 9219 deletions(-) delete mode 100644 .github/workflows/bdd.yaml delete mode 100644 .github/workflows/ci.yaml delete mode 100644 Dockerfile delete mode 100644 LICENSE delete mode 100644 Makefile delete mode 100644 PROJECT delete mode 100644 api/v1alpha1/groupversion_info.go delete mode 100644 api/v1alpha1/rollingupgrade_types.go delete mode 100644 api/v1alpha1/rollingupgrade_types_test.go delete mode 100644 api/v1alpha1/suite_test.go delete mode 100644 api/v1alpha1/zz_generated.deepcopy.go delete mode 100644 config/certmanager/certificate.yaml delete mode 100644 config/certmanager/kustomization.yaml delete mode 100644 config/certmanager/kustomizeconfig.yaml delete mode 100644 config/crd/bases/upgrademgr.keikoproj.io_rollingupgrades.yaml delete mode 100644 config/crd/kustomization.yaml delete mode 100644 config/crd/kustomizeconfig.yaml delete mode 100644 config/crd/patches/cainjection_in_rollingupgrades.yaml delete mode 100644 config/crd/patches/webhook_in_rollingupgrades.yaml delete mode 100644 config/default/kustomization.yaml delete mode 100644 config/default/manager_auth_proxy_patch.yaml delete mode 100644 config/default/manager_image_patch.yaml delete mode 100644 config/default/manager_prometheus_metrics_patch.yaml delete mode 100644 config/default/manager_webhook_patch.yaml delete mode 100644 config/default/webhookcainjection_patch.yaml delete mode 100644 config/manager/kustomization.yaml delete mode 100644 config/manager/manager.yaml delete mode 100644 config/rbac/auth_proxy_role.yaml delete mode 100644 config/rbac/auth_proxy_role_binding.yaml delete mode 100644 config/rbac/auth_proxy_service.yaml delete mode 100644 config/rbac/kustomization.yaml delete mode 100644 config/rbac/leader_election_role.yaml delete mode 100644 config/rbac/leader_election_role_binding.yaml delete mode 100644 config/rbac/role.yaml delete mode 100644 config/rbac/role_binding.yaml delete mode 100644 config/samples/upgrademgr_v1alpha1_rollingupgrade.yaml delete mode 100644 config/webhook/kustomization.yaml delete mode 100644 config/webhook/kustomizeconfig.yaml delete mode 100644 config/webhook/manifests.yaml delete mode 100644 config/webhook/service.yaml delete mode 100644 controllers/events.go delete mode 100644 controllers/events_test.go delete mode 100644 controllers/helpers.go delete mode 100644 controllers/helpers_test.go delete mode 100644 controllers/launch_definition.go delete mode 100644 controllers/node_selector.go delete mode 100644 controllers/node_selector_test.go delete mode 100644 controllers/random_node_selector.go delete mode 100644 controllers/rollingupgrade_controller.go delete mode 100644 controllers/rollingupgrade_controller_test.go delete mode 100644 controllers/rollup_cluster_state.go delete mode 100644 controllers/rollup_cluster_state_test.go delete mode 100644 controllers/script_runner.go delete mode 100644 controllers/script_runner_test.go delete mode 100644 controllers/suite_test.go delete mode 100644 controllers/uniform_across_az_node_selector.go delete mode 100644 controllers/uniform_across_az_node_selector_test.go delete mode 100644 deploy/rolling-upgrade-controller-deploy.yaml delete mode 100644 docs/RollingUpgradeDesign.png delete mode 100644 docs/faq.md delete mode 100644 docs/step-by-step-example.md delete mode 100644 examples/basic.yaml delete mode 100644 examples/pre_post_drain.yaml delete mode 100644 examples/random_update_strategy.yaml delete mode 100644 examples/uniform_across_az_update_strategy.yaml delete mode 100644 go.mod delete mode 100644 go.sum delete mode 100644 hack/boilerplate.go.txt delete mode 100644 main.go delete mode 100644 pkg/log/log.go delete mode 100644 pkg/log/retry_logger.go delete mode 100644 test-bdd/bases/kustomization.yaml delete mode 100644 test-bdd/features/01_create.feature delete mode 100644 test-bdd/main_test.go delete mode 100644 test-bdd/templates/rolling-upgrade.yaml diff --git a/.github/workflows/bdd.yaml b/.github/workflows/bdd.yaml deleted file mode 100644 index 85a1fbcd..00000000 --- a/.github/workflows/bdd.yaml +++ /dev/null @@ -1,44 +0,0 @@ -name: BDD - -on: - schedule: - - cron: '0 7 * * *' # UTC is being used, 07:00 am would be 12:00am in PT - -jobs: - build: - name: Setup & Run - if: github.repository == 'keikoproj/upgrade-manager' - runs-on: ubuntu-latest - steps: - - - name: Checkout code - uses: actions/checkout@v2 - - - name: Configure AWS Credentials - uses: aws-actions/configure-aws-credentials@v1 - with: - aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} - aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} - aws-region: us-west-2 - - - name: Prerequisites - run: | - # Getting kustomize - sudo snap install kustomize - # Generating resources files - make manifests - kustomize build config/default -o test-bdd/bases/base_crd-rbac-deployment.yaml - kustomize build test-bdd/bases -o test-bdd/crd-rbac-deployment.yaml - # Setting kubeconfig - aws eks update-kubeconfig --name upgrademgr-eks-nightly --region us-west-2 - # Deploying - kubectl apply -f test-bdd/crd-rbac-deployment.yaml - - - name: Run BDD - run: | - go get github.com/cucumber/godog/cmd/godog@v0.10.0 - cd test-bdd - $HOME/go/bin/godog - - - name: Cleanup - run: kubectl delete deployment upgrade-manager-controller-manager -n upgrade-manager-system \ No newline at end of file diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml deleted file mode 100644 index 620e29ff..00000000 --- a/.github/workflows/ci.yaml +++ /dev/null @@ -1,83 +0,0 @@ -name: Build-Test - -on: - push: - branches: - - master - pull_request: - branches: - - master - release: - types: - - published - -jobs: - build: - name: CI # Lint, Test, Codecov, Docker build & Push - runs-on: ubuntu-latest - steps: - - - name: Checkout code - uses: actions/checkout@v2 - - - name: Golangci-lint - uses: golangci/golangci-lint-action@v2 - with: - # Required: the version of golangci-lint is required and must be specified without patch version: we always use the latest patch version. - version: v1.32 - args: --timeout 2m - - - name: Get kubebuilder - env: - version: 1.0.8 # latest stable version - arch: amd64 - run: | - # download the release - curl -L -O "https://github.com/kubernetes-sigs/kubebuilder/releases/download/v${version}/kubebuilder_${version}_linux_${arch}.tar.gz" - # extract the archive - tar -zxvf kubebuilder_${version}_linux_${arch}.tar.gz - mv kubebuilder_${version}_linux_${arch} kubebuilder && sudo mv kubebuilder /usr/local/ - # update your PATH to include /usr/local/kubebuilder/bin - export PATH=$PATH:/usr/local/kubebuilder/bin - - - name: Run Tests - run: make test - - - name: Codecov - uses: codecov/codecov-action@v1 - with: - file: ./coverage.txt # optional - flags: unittests # optional - name: codecov-umbrella # optional - fail_ci_if_error: true # optional (default = false) - - - name: Docker build - if: github.event_name == 'pull_request' || (github.repository != 'keikoproj/upgrade-manager' && github.event_name == 'push') - run: make docker-build - - - name: Build and push Docker image with tag master # only on pushes to keikoproj/upgrade-manager - if: github.event_name == 'push' && github.repository == 'keikoproj/upgrade-manager' - uses: docker/build-push-action@v1 - with: - username: ${{ secrets.DOCKER_USERNAME }} - password: ${{ secrets.DOCKER_PASSWORD }} - repository: keikoproj/rolling-upgrade-controller - tags: master - - - name: Build and push Docker image with tag latest # only on releases of keikoproj/upgrade-manager - if: github.event_name == 'release' && github.repository == 'keikoproj/upgrade-manager' - uses: docker/build-push-action@v1 - with: - username: ${{ secrets.DOCKER_USERNAME }} - password: ${{ secrets.DOCKER_PASSWORD }} - repository: keikoproj/rolling-upgrade-controller - tags: latest - - - name: Build and push Docker image with tag git-tag # only on releases of keikoproj/upgrade-manager - if: github.event_name == 'release' && github.repository == 'keikoproj/upgrade-manager' - uses: docker/build-push-action@v1 - with: - username: ${{ secrets.DOCKER_USERNAME }} - password: ${{ secrets.DOCKER_PASSWORD }} - repository: keikoproj/rolling-upgrade-controller - tag_with_ref: true diff --git a/Dockerfile b/Dockerfile deleted file mode 100644 index a0e7b2d6..00000000 --- a/Dockerfile +++ /dev/null @@ -1,36 +0,0 @@ -# Build the manager binary -FROM golang:1.15.6 as builder - -WORKDIR /workspace -# Copy the Go Modules manifests -COPY go.mod go.mod -COPY go.sum go.sum -# cache deps before building and copying source so that we don't need to re-download as much -# and so that source changes don't invalidate our downloaded layer -COPY pkg pkg -RUN go mod download - -# Copy the go source -COPY main.go main.go -COPY api/ api/ -COPY controllers/ controllers/ - -# Build -RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GO111MODULE=on go build -a -o manager main.go - -# Add kubectl -RUN curl -L https://storage.googleapis.com/kubernetes-release/release/v1.14.10/bin/linux/amd64/kubectl -o /usr/local/bin/kubectl -RUN chmod +x /usr/local/bin/kubectl - -# Add busybox -FROM busybox:1.32.0 as shelladder - -# Use distroless as minimal base image to package the manager binary -# Refer to https://github.com/GoogleContainerTools/distroless for more details -FROM gcr.io/distroless/static:latest -WORKDIR / - -COPY --from=shelladder /bin/sh /bin/sh -COPY --from=builder /workspace/manager . -COPY --from=builder /usr/local/bin/kubectl /usr/local/bin/kubectl -ENTRYPOINT ["/manager"] diff --git a/LICENSE b/LICENSE deleted file mode 100644 index bbfa79ac..00000000 --- a/LICENSE +++ /dev/null @@ -1,201 +0,0 @@ - Apache License - Version 2.0, January 2004 - http://www.apache.org/licenses/ - - TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION - - 1. Definitions. - - "License" shall mean the terms and conditions for use, reproduction, - and distribution as defined by Sections 1 through 9 of this document. - - "Licensor" shall mean the copyright owner or entity authorized by - the copyright owner that is granting the License. - - "Legal Entity" shall mean the union of the acting entity and all - other entities that control, are controlled by, or are under common - control with that entity. For the purposes of this definition, - "control" means (i) the power, direct or indirect, to cause the - direction or management of such entity, whether by contract or - otherwise, or (ii) ownership of fifty percent (50%) or more of the - outstanding shares, or (iii) beneficial ownership of such entity. - - "You" (or "Your") shall mean an individual or Legal Entity - exercising permissions granted by this License. - - "Source" form shall mean the preferred form for making modifications, - including but not limited to software source code, documentation - source, and configuration files. - - "Object" form shall mean any form resulting from mechanical - transformation or translation of a Source form, including but - not limited to compiled object code, generated documentation, - and conversions to other media types. - - "Work" shall mean the work of authorship, whether in Source or - Object form, made available under the License, as indicated by a - copyright notice that is included in or attached to the work - (an example is provided in the Appendix below). - - "Derivative Works" shall mean any work, whether in Source or Object - form, that is based on (or derived from) the Work and for which the - editorial revisions, annotations, elaborations, or other modifications - represent, as a whole, an original work of authorship. For the purposes - of this License, Derivative Works shall not include works that remain - separable from, or merely link (or bind by name) to the interfaces of, - the Work and Derivative Works thereof. - - "Contribution" shall mean any work of authorship, including - the original version of the Work and any modifications or additions - to that Work or Derivative Works thereof, that is intentionally - submitted to Licensor for inclusion in the Work by the copyright owner - or by an individual or Legal Entity authorized to submit on behalf of - the copyright owner. For the purposes of this definition, "submitted" - means any form of electronic, verbal, or written communication sent - to the Licensor or its representatives, including but not limited to - communication on electronic mailing lists, source code control systems, - and issue tracking systems that are managed by, or on behalf of, the - Licensor for the purpose of discussing and improving the Work, but - excluding communication that is conspicuously marked or otherwise - designated in writing by the copyright owner as "Not a Contribution." - - "Contributor" shall mean Licensor and any individual or Legal Entity - on behalf of whom a Contribution has been received by Licensor and - subsequently incorporated within the Work. - - 2. Grant of Copyright License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - copyright license to reproduce, prepare Derivative Works of, - publicly display, publicly perform, sublicense, and distribute the - Work and such Derivative Works in Source or Object form. - - 3. Grant of Patent License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - (except as stated in this section) patent license to make, have made, - use, offer to sell, sell, import, and otherwise transfer the Work, - where such license applies only to those patent claims licensable - by such Contributor that are necessarily infringed by their - Contribution(s) alone or by combination of their Contribution(s) - with the Work to which such Contribution(s) was submitted. If You - institute patent litigation against any entity (including a - cross-claim or counterclaim in a lawsuit) alleging that the Work - or a Contribution incorporated within the Work constitutes direct - or contributory patent infringement, then any patent licenses - granted to You under this License for that Work shall terminate - as of the date such litigation is filed. - - 4. Redistribution. You may reproduce and distribute copies of the - Work or Derivative Works thereof in any medium, with or without - modifications, and in Source or Object form, provided that You - meet the following conditions: - - (a) You must give any other recipients of the Work or - Derivative Works a copy of this License; and - - (b) You must cause any modified files to carry prominent notices - stating that You changed the files; and - - (c) You must retain, in the Source form of any Derivative Works - that You distribute, all copyright, patent, trademark, and - attribution notices from the Source form of the Work, - excluding those notices that do not pertain to any part of - the Derivative Works; and - - (d) If the Work includes a "NOTICE" text file as part of its - distribution, then any Derivative Works that You distribute must - include a readable copy of the attribution notices contained - within such NOTICE file, excluding those notices that do not - pertain to any part of the Derivative Works, in at least one - of the following places: within a NOTICE text file distributed - as part of the Derivative Works; within the Source form or - documentation, if provided along with the Derivative Works; or, - within a display generated by the Derivative Works, if and - wherever such third-party notices normally appear. The contents - of the NOTICE file are for informational purposes only and - do not modify the License. You may add Your own attribution - notices within Derivative Works that You distribute, alongside - or as an addendum to the NOTICE text from the Work, provided - that such additional attribution notices cannot be construed - as modifying the License. - - You may add Your own copyright statement to Your modifications and - may provide additional or different license terms and conditions - for use, reproduction, or distribution of Your modifications, or - for any such Derivative Works as a whole, provided Your use, - reproduction, and distribution of the Work otherwise complies with - the conditions stated in this License. - - 5. Submission of Contributions. Unless You explicitly state otherwise, - any Contribution intentionally submitted for inclusion in the Work - by You to the Licensor shall be under the terms and conditions of - this License, without any additional terms or conditions. - Notwithstanding the above, nothing herein shall supersede or modify - the terms of any separate license agreement you may have executed - with Licensor regarding such Contributions. - - 6. Trademarks. This License does not grant permission to use the trade - names, trademarks, service marks, or product names of the Licensor, - except as required for reasonable and customary use in describing the - origin of the Work and reproducing the content of the NOTICE file. - - 7. Disclaimer of Warranty. Unless required by applicable law or - agreed to in writing, Licensor provides the Work (and each - Contributor provides its Contributions) on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or - implied, including, without limitation, any warranties or conditions - of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A - PARTICULAR PURPOSE. You are solely responsible for determining the - appropriateness of using or redistributing the Work and assume any - risks associated with Your exercise of permissions under this License. - - 8. Limitation of Liability. In no event and under no legal theory, - whether in tort (including negligence), contract, or otherwise, - unless required by applicable law (such as deliberate and grossly - negligent acts) or agreed to in writing, shall any Contributor be - liable to You for damages, including any direct, indirect, special, - incidental, or consequential damages of any character arising as a - result of this License or out of the use or inability to use the - Work (including but not limited to damages for loss of goodwill, - work stoppage, computer failure or malfunction, or any and all - other commercial damages or losses), even if such Contributor - has been advised of the possibility of such damages. - - 9. Accepting Warranty or Additional Liability. While redistributing - the Work or Derivative Works thereof, You may choose to offer, - and charge a fee for, acceptance of support, warranty, indemnity, - or other liability obligations and/or rights consistent with this - License. However, in accepting such obligations, You may act only - on Your own behalf and on Your sole responsibility, not on behalf - of any other Contributor, and only if You agree to indemnify, - defend, and hold each Contributor harmless for any liability - incurred by, or claims asserted against, such Contributor by reason - of your accepting any such warranty or additional liability. - - END OF TERMS AND CONDITIONS - - APPENDIX: How to apply the Apache License to your work. - - To apply the Apache License to your work, attach the following - boilerplate notice, with the fields enclosed by brackets "[]" - replaced with your own identifying information. (Don't include - the brackets!) The text should be enclosed in the appropriate - comment syntax for the file format. We also recommend that a - file or class name and description of purpose be included on the - same "printed page" as the copyright notice for easier - identification within third-party archives. - - Copyright 2019 The KeikoProj Authors - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. diff --git a/Makefile b/Makefile deleted file mode 100644 index ac6a5d88..00000000 --- a/Makefile +++ /dev/null @@ -1,75 +0,0 @@ -VERSION=0.18-dev -# Image URL to use all building/pushing image targets -IMG ?= keikoproj/rolling-upgrade-controller:${VERSION} -# Produce CRDs that work back to Kubernetes 1.11 (no version conversion) -CRD_OPTIONS ?= "crd:trivialVersions=true" - -export GO111MODULE = on - -all: manager - -# Run tests -test: generate fmt vet manifests - go test -v ./api/... ./controllers/... -coverprofile coverage.txt - go tool cover -html=./coverage.txt -o cover.html - -# Run golangci lint tests -lint: - golangci-lint run ./... -.PHONY: lint - -# Build manager binary -manager: generate fmt vet - go build -o bin/manager main.go - -# Run against the configured Kubernetes cluster in ~/.kube/config -run: generate fmt vet - go run ./main.go - -# Install CRDs into a cluster -install: manifests - kubectl apply -f config/crd/bases - -# Deploy controller in the configured Kubernetes cluster in ~/.kube/config -deploy: manifests - kubectl apply -f config/crd/bases - kustomize build config/default | kubectl apply -f - - -# Generate manifests e.g. CRD, RBAC etc. -manifests: controller-gen - $(CONTROLLER_GEN) $(CRD_OPTIONS) rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases - -# Run go fmt against code -fmt: - go fmt ./... - -# Run go vet against code -vet: - go vet ./... - -# Generate code -generate: controller-gen - $(CONTROLLER_GEN) object:headerFile=./hack/boilerplate.go.txt paths=./api/... - -# Build the docker image -docker-build: - docker build . -t ${IMG} - docker tag ${IMG} keikoproj/rolling-upgrade-controller:latest - @echo "updating kustomize image patch file for manager resource" - sed -i'' -e 's@image: .*@image: '"${IMG}"'@' ./config/default/manager_image_patch.yaml - -# Push the docker image -docker-push: - docker push ${IMG} - -# find or download controller-gen -# download controller-gen if necessary -controller-gen: -ifeq (, $(shell which controller-gen)) - export GO111MODULE=off # https://stackoverflow.com/questions/54415733/getting-gopath-error-go-cannot-use-pathversion-syntax-in-gopath-mode-in-ubun - go clean -modcache - go get sigs.k8s.io/controller-tools/cmd/controller-gen@v0.2.4 -CONTROLLER_GEN=$(shell go env GOPATH)/bin/controller-gen -else -CONTROLLER_GEN=$(shell which controller-gen) -endif diff --git a/PROJECT b/PROJECT deleted file mode 100644 index 9a80e1c2..00000000 --- a/PROJECT +++ /dev/null @@ -1,7 +0,0 @@ -version: "2" -domain: keikoproj.io -repo: github.com/keikoproj/upgrade-manager -resources: -- group: upgrademgr - version: v1alpha1 - kind: RollingUpgrade diff --git a/api/v1alpha1/groupversion_info.go b/api/v1alpha1/groupversion_info.go deleted file mode 100644 index 1c1f3f62..00000000 --- a/api/v1alpha1/groupversion_info.go +++ /dev/null @@ -1,35 +0,0 @@ -/* - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -// Package v1alpha1 contains API Schema definitions for the upgrademgr v1alpha1 API group -// +kubebuilder:object:generate=true -// +groupName=upgrademgr.keikoproj.io -package v1alpha1 - -import ( - "k8s.io/apimachinery/pkg/runtime/schema" - "sigs.k8s.io/controller-runtime/pkg/scheme" -) - -var ( - // GroupVersion is group version used to register these objects - GroupVersion = schema.GroupVersion{Group: "upgrademgr.keikoproj.io", Version: "v1alpha1"} - - // SchemeBuilder is used to add go types to the GroupVersionKind scheme - SchemeBuilder = &scheme.Builder{GroupVersion: GroupVersion} - - // AddToScheme adds the types in this group-version to the given scheme. - AddToScheme = SchemeBuilder.AddToScheme -) diff --git a/api/v1alpha1/rollingupgrade_types.go b/api/v1alpha1/rollingupgrade_types.go deleted file mode 100644 index 42a8a6a0..00000000 --- a/api/v1alpha1/rollingupgrade_types.go +++ /dev/null @@ -1,163 +0,0 @@ -/* - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package v1alpha1 - -import ( - "fmt" - - corev1 "k8s.io/api/core/v1" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - "k8s.io/apimachinery/pkg/util/intstr" -) - -// PreDrainSpec contains the fields for actions taken before draining the node. -type PreDrainSpec struct { - Script string `json:"script,omitempty"` -} - -// PostDrainSpec contains the fields for actions taken after draining the node. -type PostDrainSpec struct { - Script string `json:"script,omitempty"` - WaitSeconds int64 `json:"waitSeconds,omitempty"` - PostWaitScript string `json:"postWaitScript,omitempty"` -} - -// PostTerminateSpec contains the fields for actions taken after terminating the node. -type PostTerminateSpec struct { - Script string `json:"script,omitempty"` -} - -// RollingUpgradeSpec defines the desired state of RollingUpgrade -type RollingUpgradeSpec struct { - PostDrainDelaySeconds int `json:"postDrainDelaySeconds,omitempty"` - NodeIntervalSeconds int `json:"nodeIntervalSeconds,omitempty"` - // AsgName is AWS Autoscaling Group name to roll. - AsgName string `json:"asgName,omitempty"` - PreDrain PreDrainSpec `json:"preDrain,omitempty"` - PostDrain PostDrainSpec `json:"postDrain,omitempty"` - PostTerminate PostTerminateSpec `json:"postTerminate,omitempty"` - Strategy UpdateStrategy `json:"strategy,omitempty"` - // IgnoreDrainFailures allows ignoring node drain failures and proceed with rolling upgrade. - IgnoreDrainFailures bool `json:"ignoreDrainFailures,omitempty"` - // ForceRefresh enables draining and terminating the node even if the launch config/template hasn't changed. - ForceRefresh bool `json:"forceRefresh,omitempty"` - // ReadinessGates allow to specify label selectors that node must match to be considered ready. - ReadinessGates []NodeReadinessGate `json:"readinessGates,omitempty"` -} - -type NodeReadinessGate struct { - MatchLabels map[string]string `json:"matchLabels,omitempty" protobuf:"bytes,1,rep,name=matchLabels"` -} - -// RollingUpgradeStatus defines the observed state of RollingUpgrade -type RollingUpgradeStatus struct { - CurrentStatus string `json:"currentStatus,omitempty"` - StartTime string `json:"startTime,omitempty"` - EndTime string `json:"endTime,omitempty"` - TotalProcessingTime string `json:"totalProcessingTime,omitempty"` - NodesProcessed int `json:"nodesProcessed,omitempty"` - TotalNodes int `json:"totalNodes,omitempty"` - - Conditions []RollingUpgradeCondition `json:"conditions,omitempty"` -} - -const ( - // StatusRunning marks the CR to be running. - StatusRunning = "running" - // StatusComplete marks the CR as completed. - StatusComplete = "completed" - // StatusError marks the CR as errored out. - StatusError = "error" -) - -// RollingUpgradeCondition describes the state of the RollingUpgrade -type RollingUpgradeCondition struct { - Type UpgradeConditionType `json:"type,omitempty"` - Status corev1.ConditionStatus `json:"status,omitempty"` -} - -// +kubebuilder:object:root=true -// +kubebuilder:subresource:status -// +kubebuilder:resource:path=rollingupgrades,scope=Namespaced,shortName=ru -// +kubebuilder:printcolumn:name="Status",type="string",JSONPath=".status.currentStatus",description="current status of the rollingupgarde" -// +kubebuilder:printcolumn:name="TotalNodes",type="string",JSONPath=".status.totalNodes",description="total nodes involved in the rollingupgarde" -// +kubebuilder:printcolumn:name="NodesProcessed",type="string",JSONPath=".status.nodesProcessed",description="current number of nodes processed in the rollingupgarde" - -// RollingUpgrade is the Schema for the rollingupgrades API -type RollingUpgrade struct { - metav1.TypeMeta `json:",inline"` - metav1.ObjectMeta `json:"metadata,omitempty"` - - Spec RollingUpgradeSpec `json:"spec,omitempty"` - Status RollingUpgradeStatus `json:"status,omitempty"` -} - -// +kubebuilder:object:root=true - -// RollingUpgradeList contains a list of RollingUpgrade -type RollingUpgradeList struct { - metav1.TypeMeta `json:",inline"` - metav1.ListMeta `json:"metadata,omitempty"` - Items []RollingUpgrade `json:"items"` -} - -func init() { - SchemeBuilder.Register(&RollingUpgrade{}, &RollingUpgradeList{}) -} - -// UpdateStrategyType indicates how the update has to be rolled out -// whether to roll the update AZ wise or all Azs at once -type UpdateStrategyType string - -type UpdateStrategyMode string - -type UpgradeConditionType string - -const ( - // RandomUpdate strategy treats all the availability zones as a single unit and picks random nodes for update. - RandomUpdateStrategy UpdateStrategyType = "randomUpdate" - - // UniformAcrossAzUpdateStrategy Picks same number of nodes or same percentage of nodes from each AZ for update. - UniformAcrossAzUpdateStrategy UpdateStrategyType = "uniformAcrossAzUpdate" - - UpdateStrategyModeLazy UpdateStrategyMode = "lazy" - UpdateStrategyModeEager UpdateStrategyMode = "eager" - - // Other update strategies such as rolling update by AZ or rolling update with a pre-defined instance list - // can be implemented in future by adding more update strategy types - - UpgradeComplete UpgradeConditionType = "Complete" -) - -func (c UpdateStrategyMode) String() string { - return string(c) -} - -// NamespacedName returns namespaced name of the object. -func (r RollingUpgrade) NamespacedName() string { - return fmt.Sprintf("%s/%s", r.Namespace, r.Name) -} - -// UpdateStrategy holds the information needed to perform update based on different update strategies -type UpdateStrategy struct { - Type UpdateStrategyType `json:"type,omitempty"` - Mode UpdateStrategyMode `json:"mode,omitempty"` - // MaxUnavailable can be specified as number of nodes or the percent of total number of nodes - MaxUnavailable intstr.IntOrString `json:"maxUnavailable,omitempty"` - // Node will be terminated after drain timeout even if `kubectl drain` has not been completed - // and value has to be specified in seconds - DrainTimeout int `json:"drainTimeout"` -} diff --git a/api/v1alpha1/rollingupgrade_types_test.go b/api/v1alpha1/rollingupgrade_types_test.go deleted file mode 100644 index c6c737c1..00000000 --- a/api/v1alpha1/rollingupgrade_types_test.go +++ /dev/null @@ -1,83 +0,0 @@ -/* - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package v1alpha1 - -import ( - . "github.com/onsi/ginkgo" - . "github.com/onsi/gomega" - - "golang.org/x/net/context" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - "k8s.io/apimachinery/pkg/types" -) - -// These tests are written in BDD-style using Ginkgo framework. Refer to -// http://onsi.github.io/ginkgo to learn more. - -var _ = Describe("RollingUpgrade", func() { - var ( - key types.NamespacedName - created, fetched *RollingUpgrade - ) - - BeforeEach(func() { - // Add any setup steps that needs to be executed before each test - }) - - AfterEach(func() { - // Add any teardown steps that needs to be executed after each test - }) - - Context("NamespacedName", func() { - It("generates qualified name", func() { - ru := &RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Namespace: "namespace-foo", Name: "object-bar"}} - Expect(ru.NamespacedName()).To(Equal("namespace-foo/object-bar")) - }) - }) - - // Add Tests for OpenAPI validation (or additonal CRD features) specified in - // your API definition. - // Avoid adding tests for vanilla CRUD operations because they would - // test Kubernetes API server, which isn't the goal here. - Context("Create API", func() { - - It("should create an object successfully", func() { - - key = types.NamespacedName{ - Name: "foo", - Namespace: "default", - } - created = &RollingUpgrade{ - ObjectMeta: metav1.ObjectMeta{ - Name: "foo", - Namespace: "default", - }} - - By("creating an API obj") - Expect(k8sClient.Create(context.TODO(), created)).To(Succeed()) - - fetched = &RollingUpgrade{} - Expect(k8sClient.Get(context.TODO(), key, fetched)).To(Succeed()) - Expect(fetched).To(Equal(created)) - - By("deleting the created object") - Expect(k8sClient.Delete(context.TODO(), created)).To(Succeed()) - Expect(k8sClient.Get(context.TODO(), key, created)).ToNot(Succeed()) - }) - - }) - -}) diff --git a/api/v1alpha1/suite_test.go b/api/v1alpha1/suite_test.go deleted file mode 100644 index 16948287..00000000 --- a/api/v1alpha1/suite_test.go +++ /dev/null @@ -1,74 +0,0 @@ -/* - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package v1alpha1 - -import ( - "path/filepath" - "testing" - - . "github.com/onsi/ginkgo" - . "github.com/onsi/gomega" - - "k8s.io/client-go/kubernetes/scheme" - "k8s.io/client-go/rest" - "sigs.k8s.io/controller-runtime/pkg/client" - "sigs.k8s.io/controller-runtime/pkg/envtest" - logf "sigs.k8s.io/controller-runtime/pkg/log" - "sigs.k8s.io/controller-runtime/pkg/log/zap" -) - -// These tests use Ginkgo (BDD-style Go testing framework). Refer to -// http://onsi.github.io/ginkgo/ to learn more about Ginkgo. - -var cfg *rest.Config -var k8sClient client.Client -var testEnv *envtest.Environment - -func TestAPIs(t *testing.T) { - RegisterFailHandler(Fail) - - RunSpecsWithDefaultAndCustomReporters(t, - "v1alpha1 Suite", - []Reporter{envtest.NewlineReporter{}}) -} - -var _ = BeforeSuite(func(done Done) { - logf.SetLogger(zap.New(zap.UseDevMode(true), zap.WriteTo(GinkgoWriter))) - - By("bootstrapping test environment") - testEnv = &envtest.Environment{ - CRDDirectoryPaths: []string{filepath.Join("..", "..", "config", "crd", "bases")}, - } - - err := SchemeBuilder.AddToScheme(scheme.Scheme) - Expect(err).NotTo(HaveOccurred()) - - cfg, err = testEnv.Start() - Expect(err).ToNot(HaveOccurred()) - Expect(cfg).ToNot(BeNil()) - - k8sClient, err = client.New(cfg, client.Options{Scheme: scheme.Scheme}) - Expect(err).ToNot(HaveOccurred()) - Expect(k8sClient).ToNot(BeNil()) - - close(done) -}, 60) - -var _ = AfterSuite(func() { - By("tearing down the test environment") - err := testEnv.Stop() - Expect(err).ToNot(HaveOccurred()) -}) diff --git a/api/v1alpha1/zz_generated.deepcopy.go b/api/v1alpha1/zz_generated.deepcopy.go deleted file mode 100644 index 3f4d426b..00000000 --- a/api/v1alpha1/zz_generated.deepcopy.go +++ /dev/null @@ -1,227 +0,0 @@ -// +build !ignore_autogenerated - -/* - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -// Code generated by controller-gen. DO NOT EDIT. - -package v1alpha1 - -import ( - runtime "k8s.io/apimachinery/pkg/runtime" -) - -// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. -func (in *NodeReadinessGate) DeepCopyInto(out *NodeReadinessGate) { - *out = *in - if in.MatchLabels != nil { - in, out := &in.MatchLabels, &out.MatchLabels - *out = make(map[string]string, len(*in)) - for key, val := range *in { - (*out)[key] = val - } - } -} - -// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new NodeReadinessGate. -func (in *NodeReadinessGate) DeepCopy() *NodeReadinessGate { - if in == nil { - return nil - } - out := new(NodeReadinessGate) - in.DeepCopyInto(out) - return out -} - -// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. -func (in *PostDrainSpec) DeepCopyInto(out *PostDrainSpec) { - *out = *in -} - -// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new PostDrainSpec. -func (in *PostDrainSpec) DeepCopy() *PostDrainSpec { - if in == nil { - return nil - } - out := new(PostDrainSpec) - in.DeepCopyInto(out) - return out -} - -// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. -func (in *PostTerminateSpec) DeepCopyInto(out *PostTerminateSpec) { - *out = *in -} - -// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new PostTerminateSpec. -func (in *PostTerminateSpec) DeepCopy() *PostTerminateSpec { - if in == nil { - return nil - } - out := new(PostTerminateSpec) - in.DeepCopyInto(out) - return out -} - -// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. -func (in *PreDrainSpec) DeepCopyInto(out *PreDrainSpec) { - *out = *in -} - -// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new PreDrainSpec. -func (in *PreDrainSpec) DeepCopy() *PreDrainSpec { - if in == nil { - return nil - } - out := new(PreDrainSpec) - in.DeepCopyInto(out) - return out -} - -// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. -func (in *RollingUpgrade) DeepCopyInto(out *RollingUpgrade) { - *out = *in - out.TypeMeta = in.TypeMeta - in.ObjectMeta.DeepCopyInto(&out.ObjectMeta) - in.Spec.DeepCopyInto(&out.Spec) - in.Status.DeepCopyInto(&out.Status) -} - -// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RollingUpgrade. -func (in *RollingUpgrade) DeepCopy() *RollingUpgrade { - if in == nil { - return nil - } - out := new(RollingUpgrade) - in.DeepCopyInto(out) - return out -} - -// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. -func (in *RollingUpgrade) DeepCopyObject() runtime.Object { - if c := in.DeepCopy(); c != nil { - return c - } - return nil -} - -// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. -func (in *RollingUpgradeCondition) DeepCopyInto(out *RollingUpgradeCondition) { - *out = *in -} - -// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RollingUpgradeCondition. -func (in *RollingUpgradeCondition) DeepCopy() *RollingUpgradeCondition { - if in == nil { - return nil - } - out := new(RollingUpgradeCondition) - in.DeepCopyInto(out) - return out -} - -// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. -func (in *RollingUpgradeList) DeepCopyInto(out *RollingUpgradeList) { - *out = *in - out.TypeMeta = in.TypeMeta - in.ListMeta.DeepCopyInto(&out.ListMeta) - if in.Items != nil { - in, out := &in.Items, &out.Items - *out = make([]RollingUpgrade, len(*in)) - for i := range *in { - (*in)[i].DeepCopyInto(&(*out)[i]) - } - } -} - -// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RollingUpgradeList. -func (in *RollingUpgradeList) DeepCopy() *RollingUpgradeList { - if in == nil { - return nil - } - out := new(RollingUpgradeList) - in.DeepCopyInto(out) - return out -} - -// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. -func (in *RollingUpgradeList) DeepCopyObject() runtime.Object { - if c := in.DeepCopy(); c != nil { - return c - } - return nil -} - -// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. -func (in *RollingUpgradeSpec) DeepCopyInto(out *RollingUpgradeSpec) { - *out = *in - out.PreDrain = in.PreDrain - out.PostDrain = in.PostDrain - out.PostTerminate = in.PostTerminate - out.Strategy = in.Strategy - if in.ReadinessGates != nil { - in, out := &in.ReadinessGates, &out.ReadinessGates - *out = make([]NodeReadinessGate, len(*in)) - for i := range *in { - (*in)[i].DeepCopyInto(&(*out)[i]) - } - } -} - -// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RollingUpgradeSpec. -func (in *RollingUpgradeSpec) DeepCopy() *RollingUpgradeSpec { - if in == nil { - return nil - } - out := new(RollingUpgradeSpec) - in.DeepCopyInto(out) - return out -} - -// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. -func (in *RollingUpgradeStatus) DeepCopyInto(out *RollingUpgradeStatus) { - *out = *in - if in.Conditions != nil { - in, out := &in.Conditions, &out.Conditions - *out = make([]RollingUpgradeCondition, len(*in)) - copy(*out, *in) - } -} - -// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RollingUpgradeStatus. -func (in *RollingUpgradeStatus) DeepCopy() *RollingUpgradeStatus { - if in == nil { - return nil - } - out := new(RollingUpgradeStatus) - in.DeepCopyInto(out) - return out -} - -// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. -func (in *UpdateStrategy) DeepCopyInto(out *UpdateStrategy) { - *out = *in - out.MaxUnavailable = in.MaxUnavailable -} - -// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new UpdateStrategy. -func (in *UpdateStrategy) DeepCopy() *UpdateStrategy { - if in == nil { - return nil - } - out := new(UpdateStrategy) - in.DeepCopyInto(out) - return out -} diff --git a/config/certmanager/certificate.yaml b/config/certmanager/certificate.yaml deleted file mode 100644 index 9d6bad1e..00000000 --- a/config/certmanager/certificate.yaml +++ /dev/null @@ -1,24 +0,0 @@ -# The following manifests contain a self-signed issuer CR and a certificate CR. -# More document can be found at https://docs.cert-manager.io -apiVersion: certmanager.k8s.io/v1alpha1 -kind: Issuer -metadata: - name: selfsigned-issuer - namespace: system -spec: - selfSigned: {} ---- -apiVersion: certmanager.k8s.io/v1alpha1 -kind: Certificate -metadata: - name: serving-cert # this name should match the one appeared in kustomizeconfig.yaml - namespace: system -spec: - # $(SERVICENAME) and $(NAMESPACE) will be substituted by kustomize - commonName: $(SERVICENAME).$(NAMESPACE).svc - dnsNames: - - $(SERVICENAME).$(NAMESPACE).svc.cluster.local - issuerRef: - kind: Issuer - name: selfsigned-issuer - secretName: webhook-server-cert # this secret will not be prefixed, since it's not managed by kustomize diff --git a/config/certmanager/kustomization.yaml b/config/certmanager/kustomization.yaml deleted file mode 100644 index 8181bc3a..00000000 --- a/config/certmanager/kustomization.yaml +++ /dev/null @@ -1,26 +0,0 @@ -resources: -- certificate.yaml - -# the following config is for teaching kustomize how to do var substitution -vars: -- name: NAMESPACE # namespace of the service and the certificate CR - objref: - kind: Service - version: v1 - name: webhook-service - fieldref: - fieldpath: metadata.namespace -- name: CERTIFICATENAME - objref: - kind: Certificate - group: certmanager.k8s.io - version: v1alpha1 - name: serving-cert # this name should match the one in certificate.yaml -- name: SERVICENAME - objref: - kind: Service - version: v1 - name: webhook-service - -configurations: -- kustomizeconfig.yaml diff --git a/config/certmanager/kustomizeconfig.yaml b/config/certmanager/kustomizeconfig.yaml deleted file mode 100644 index 49e0b1e7..00000000 --- a/config/certmanager/kustomizeconfig.yaml +++ /dev/null @@ -1,16 +0,0 @@ -# This configuration is for teaching kustomize how to update name ref and var substitution -nameReference: -- kind: Issuer - group: certmanager.k8s.io - fieldSpecs: - - kind: Certificate - group: certmanager.k8s.io - path: spec/issuerRef/name - -varReference: -- kind: Certificate - group: certmanager.k8s.io - path: spec/commonName -- kind: Certificate - group: certmanager.k8s.io - path: spec/dnsNames diff --git a/config/crd/bases/upgrademgr.keikoproj.io_rollingupgrades.yaml b/config/crd/bases/upgrademgr.keikoproj.io_rollingupgrades.yaml deleted file mode 100644 index 04fa959a..00000000 --- a/config/crd/bases/upgrademgr.keikoproj.io_rollingupgrades.yaml +++ /dev/null @@ -1,170 +0,0 @@ - ---- -apiVersion: apiextensions.k8s.io/v1beta1 -kind: CustomResourceDefinition -metadata: - annotations: - controller-gen.kubebuilder.io/version: v0.2.4 - creationTimestamp: null - name: rollingupgrades.upgrademgr.keikoproj.io -spec: - additionalPrinterColumns: - - JSONPath: .status.currentStatus - description: current status of the rollingupgarde - name: Status - type: string - - JSONPath: .status.totalNodes - description: total nodes involved in the rollingupgarde - name: TotalNodes - type: string - - JSONPath: .status.nodesProcessed - description: current number of nodes processed in the rollingupgarde - name: NodesProcessed - type: string - group: upgrademgr.keikoproj.io - names: - kind: RollingUpgrade - listKind: RollingUpgradeList - plural: rollingupgrades - shortNames: - - ru - singular: rollingupgrade - scope: Namespaced - subresources: - status: {} - validation: - openAPIV3Schema: - description: RollingUpgrade is the Schema for the rollingupgrades API - properties: - apiVersion: - description: 'APIVersion defines the versioned schema of this representation - of an object. Servers should convert recognized schemas to the latest - internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' - type: string - kind: - description: 'Kind is a string value representing the REST resource this - object represents. Servers may infer this from the endpoint the client - submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' - type: string - metadata: - type: object - spec: - description: RollingUpgradeSpec defines the desired state of RollingUpgrade - properties: - asgName: - description: AsgName is AWS Autoscaling Group name to roll. - type: string - forceRefresh: - description: ForceRefresh enables draining and terminating the node - even if the launch config/template hasn't changed. - type: boolean - ignoreDrainFailures: - description: IgnoreDrainFailures allows ignoring node drain failures - and proceed with rolling upgrade. - type: boolean - nodeIntervalSeconds: - type: integer - postDrain: - description: PostDrainSpec contains the fields for actions taken after - draining the node. - properties: - postWaitScript: - type: string - script: - type: string - waitSeconds: - format: int64 - type: integer - type: object - postDrainDelaySeconds: - type: integer - postTerminate: - description: PostTerminateSpec contains the fields for actions taken - after terminating the node. - properties: - script: - type: string - type: object - preDrain: - description: PreDrainSpec contains the fields for actions taken before - draining the node. - properties: - script: - type: string - type: object - readinessGates: - description: ReadinessGates allow to specify label selectors that node - must match to be considered ready. - items: - properties: - matchLabels: - additionalProperties: - type: string - type: object - type: object - type: array - strategy: - description: UpdateStrategy holds the information needed to perform - update based on different update strategies - properties: - drainTimeout: - description: Node will be terminated after drain timeout even if - `kubectl drain` has not been completed and value has to be specified - in seconds - type: integer - maxUnavailable: - anyOf: - - type: integer - - type: string - description: MaxUnavailable can be specified as number of nodes - or the percent of total number of nodes - x-kubernetes-int-or-string: true - mode: - type: string - type: - description: UpdateStrategyType indicates how the update has to - be rolled out whether to roll the update AZ wise or all Azs at - once - type: string - required: - - drainTimeout - type: object - type: object - status: - description: RollingUpgradeStatus defines the observed state of RollingUpgrade - properties: - conditions: - items: - description: RollingUpgradeCondition describes the state of the RollingUpgrade - properties: - status: - type: string - type: - type: string - type: object - type: array - currentStatus: - type: string - endTime: - type: string - nodesProcessed: - type: integer - startTime: - type: string - totalNodes: - type: integer - totalProcessingTime: - type: string - type: object - type: object - version: v1alpha1 - versions: - - name: v1alpha1 - served: true - storage: true -status: - acceptedNames: - kind: "" - plural: "" - conditions: [] - storedVersions: [] diff --git a/config/crd/kustomization.yaml b/config/crd/kustomization.yaml deleted file mode 100644 index b02f5d3f..00000000 --- a/config/crd/kustomization.yaml +++ /dev/null @@ -1,19 +0,0 @@ -# This kustomization.yaml is not intended to be run by itself, -# since it depends on service name and namespace that are out of this kustomize package. -# It should be run by config/default -resources: -- bases/upgrademgr.keikoproj.io_rollingupgrades.yaml -# +kubebuilder:scaffold:crdkustomizeresource - -patches: -# [WEBHOOK] patches here are for enabling the conversion webhook for each CRD -#- patches/webhook_in_rollingupgrades.yaml -# +kubebuilder:scaffold:crdkustomizewebhookpatch - -# [CAINJECTION] patches here are for enabling the CA injection for each CRD -#- patches/cainjection_in_rollingupgrades.yaml -# +kubebuilder:scaffold:crdkustomizecainjectionpatch - -# the following config is for teaching kustomize how to do kustomization for CRDs. -configurations: -- kustomizeconfig.yaml diff --git a/config/crd/kustomizeconfig.yaml b/config/crd/kustomizeconfig.yaml deleted file mode 100644 index 6f83d9a9..00000000 --- a/config/crd/kustomizeconfig.yaml +++ /dev/null @@ -1,17 +0,0 @@ -# This file is for teaching kustomize how to substitute name and namespace reference in CRD -nameReference: -- kind: Service - version: v1 - fieldSpecs: - - kind: CustomResourceDefinition - group: apiextensions.k8s.io - path: spec/conversion/webhookClientConfig/service/name - -namespace: -- kind: CustomResourceDefinition - group: apiextensions.k8s.io - path: spec/conversion/webhookClientConfig/service/namespace - create: false - -varReference: -- path: metadata/annotations diff --git a/config/crd/patches/cainjection_in_rollingupgrades.yaml b/config/crd/patches/cainjection_in_rollingupgrades.yaml deleted file mode 100644 index 34e9b4c6..00000000 --- a/config/crd/patches/cainjection_in_rollingupgrades.yaml +++ /dev/null @@ -1,8 +0,0 @@ -# The following patch adds a directive for certmanager to inject CA into the CRD -# CRD conversion requires k8s 1.13 or later. -apiVersion: apiextensions.k8s.io/v1beta1 -kind: CustomResourceDefinition -metadata: - annotations: - certmanager.k8s.io/inject-ca-from: $(NAMESPACE)/$(CERTIFICATENAME) - name: rollingupgrades.upgrademgr.keikoproj.io diff --git a/config/crd/patches/webhook_in_rollingupgrades.yaml b/config/crd/patches/webhook_in_rollingupgrades.yaml deleted file mode 100644 index 7b230fec..00000000 --- a/config/crd/patches/webhook_in_rollingupgrades.yaml +++ /dev/null @@ -1,17 +0,0 @@ -# The following patch enables conversion webhook for CRD -# CRD conversion requires k8s 1.13 or later. -apiVersion: apiextensions.k8s.io/v1beta1 -kind: CustomResourceDefinition -metadata: - name: rollingupgrades.upgrademgr.keikoproj.io -spec: - conversion: - strategy: Webhook - webhookClientConfig: - # this is "\n" used as a placeholder, otherwise it will be rejected by the apiserver for being blank, - # but we're going to set it later using the cert-manager (or potentially a patch if not using cert-manager) - caBundle: Cg== - service: - namespace: system - name: webhook-service - path: /convert diff --git a/config/default/kustomization.yaml b/config/default/kustomization.yaml deleted file mode 100644 index 6fd0f327..00000000 --- a/config/default/kustomization.yaml +++ /dev/null @@ -1,43 +0,0 @@ -# Adds namespace to all resources. -namespace: upgrade-manager-system - -# Value of this field is prepended to the -# names of all resources, e.g. a deployment named -# "wordpress" becomes "alices-wordpress". -# Note that it should also match with the prefix (text before '-') of the namespace -# field above. -namePrefix: upgrade-manager- - -# Labels to add to all resources and selectors. -#commonLabels: -# someName: someValue - -bases: -- ../crd -- ../rbac -- ../manager -# [WEBHOOK] To enable webhook, uncomment all the sections with [WEBHOOK] prefix including the one in crd/kustomization.yaml -#- ../webhook -# [CERTMANAGER] To enable cert-manager, uncomment next line. 'WEBHOOK' components are required. -#- ../certmanager - -patches: -- manager_image_patch.yaml - # Protect the /metrics endpoint by putting it behind auth. - # Only one of manager_auth_proxy_patch.yaml and - # manager_prometheus_metrics_patch.yaml should be enabled. -- manager_auth_proxy_patch.yaml - # If you want your controller-manager to expose the /metrics - # endpoint w/o any authn/z, uncomment the following line and - # comment manager_auth_proxy_patch.yaml. - # Only one of manager_auth_proxy_patch.yaml and - # manager_prometheus_metrics_patch.yaml should be enabled. -#- manager_prometheus_metrics_patch.yaml - -# [WEBHOOK] To enable webhook, uncomment all the sections with [WEBHOOK] prefix including the one in crd/kustomization.yaml -#- manager_webhook_patch.yaml - -# [CAINJECTION] Uncomment next line to enable the CA injection in the admission webhooks. -# Uncomment 'CAINJECTION' in crd/kustomization.yaml to enable the CA injection in the admission webhooks. -# 'CERTMANAGER' needs to be enabled to use ca injection -#- webhookcainjection_patch.yaml diff --git a/config/default/manager_auth_proxy_patch.yaml b/config/default/manager_auth_proxy_patch.yaml deleted file mode 100644 index d3994fb9..00000000 --- a/config/default/manager_auth_proxy_patch.yaml +++ /dev/null @@ -1,24 +0,0 @@ -# This patch inject a sidecar container which is a HTTP proxy for the controller manager, -# it performs RBAC authorization against the Kubernetes API using SubjectAccessReviews. -apiVersion: apps/v1 -kind: Deployment -metadata: - name: controller-manager - namespace: system -spec: - template: - spec: - containers: - - name: kube-rbac-proxy - image: gcr.io/kubebuilder/kube-rbac-proxy:v0.4.0 - args: - - "--secure-listen-address=0.0.0.0:8443" - - "--upstream=http://127.0.0.1:8080/" - - "--logtostderr=true" - - "--v=10" - ports: - - containerPort: 8443 - name: https - - name: manager - args: - - "--metrics-addr=127.0.0.1:8080" diff --git a/config/default/manager_image_patch.yaml b/config/default/manager_image_patch.yaml deleted file mode 100644 index 09fb22d4..00000000 --- a/config/default/manager_image_patch.yaml +++ /dev/null @@ -1,12 +0,0 @@ -apiVersion: apps/v1 -kind: Deployment -metadata: - name: controller-manager - namespace: system -spec: - template: - spec: - containers: - # Change the value of image field below to your controller image URL - - image: keikoproj/rolling-upgrade-controller:0.18-dev - name: manager diff --git a/config/default/manager_prometheus_metrics_patch.yaml b/config/default/manager_prometheus_metrics_patch.yaml deleted file mode 100644 index 0b96c681..00000000 --- a/config/default/manager_prometheus_metrics_patch.yaml +++ /dev/null @@ -1,19 +0,0 @@ -# This patch enables Prometheus scraping for the manager pod. -apiVersion: apps/v1 -kind: Deployment -metadata: - name: controller-manager - namespace: system -spec: - template: - metadata: - annotations: - prometheus.io/scrape: 'true' - spec: - containers: - # Expose the prometheus metrics on default port - - name: manager - ports: - - containerPort: 8080 - name: metrics - protocol: TCP diff --git a/config/default/manager_webhook_patch.yaml b/config/default/manager_webhook_patch.yaml deleted file mode 100644 index f2f7157b..00000000 --- a/config/default/manager_webhook_patch.yaml +++ /dev/null @@ -1,23 +0,0 @@ -apiVersion: apps/v1 -kind: Deployment -metadata: - name: controller-manager - namespace: system -spec: - template: - spec: - containers: - - name: manager - ports: - - containerPort: 443 - name: webhook-server - protocol: TCP - volumeMounts: - - mountPath: /tmp/k8s-webhook-server/serving-certs - name: cert - readOnly: true - volumes: - - name: cert - secret: - defaultMode: 420 - secretName: webhook-server-cert diff --git a/config/default/webhookcainjection_patch.yaml b/config/default/webhookcainjection_patch.yaml deleted file mode 100644 index f6d71cb7..00000000 --- a/config/default/webhookcainjection_patch.yaml +++ /dev/null @@ -1,15 +0,0 @@ -# This patch add annotation to admission webhook config and -# the variables $(NAMESPACE) and $(CERTIFICATENAME) will be substituted by kustomize. -apiVersion: admissionregistration.k8s.io/v1beta1 -kind: MutatingWebhookConfiguration -metadata: - name: mutating-webhook-configuration - annotations: - certmanager.k8s.io/inject-ca-from: $(NAMESPACE)/$(CERTIFICATENAME) ---- -apiVersion: admissionregistration.k8s.io/v1beta1 -kind: ValidatingWebhookConfiguration -metadata: - name: validating-webhook-configuration - annotations: - certmanager.k8s.io/inject-ca-from: $(NAMESPACE)/$(CERTIFICATENAME) diff --git a/config/manager/kustomization.yaml b/config/manager/kustomization.yaml deleted file mode 100644 index 5c5f0b84..00000000 --- a/config/manager/kustomization.yaml +++ /dev/null @@ -1,2 +0,0 @@ -resources: -- manager.yaml diff --git a/config/manager/manager.yaml b/config/manager/manager.yaml deleted file mode 100644 index b6c85a52..00000000 --- a/config/manager/manager.yaml +++ /dev/null @@ -1,39 +0,0 @@ -apiVersion: v1 -kind: Namespace -metadata: - labels: - control-plane: controller-manager - name: system ---- -apiVersion: apps/v1 -kind: Deployment -metadata: - name: controller-manager - namespace: system - labels: - control-plane: controller-manager -spec: - selector: - matchLabels: - control-plane: controller-manager - replicas: 1 - template: - metadata: - labels: - control-plane: controller-manager - spec: - containers: - - command: - - /manager - args: - - --enable-leader-election - image: controller:latest - name: manager - resources: - limits: - cpu: 100m - memory: 30Mi - requests: - cpu: 100m - memory: 20Mi - terminationGracePeriodSeconds: 10 diff --git a/config/rbac/auth_proxy_role.yaml b/config/rbac/auth_proxy_role.yaml deleted file mode 100644 index 618f5e41..00000000 --- a/config/rbac/auth_proxy_role.yaml +++ /dev/null @@ -1,13 +0,0 @@ -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRole -metadata: - name: proxy-role -rules: -- apiGroups: ["authentication.k8s.io"] - resources: - - tokenreviews - verbs: ["create"] -- apiGroups: ["authorization.k8s.io"] - resources: - - subjectaccessreviews - verbs: ["create"] diff --git a/config/rbac/auth_proxy_role_binding.yaml b/config/rbac/auth_proxy_role_binding.yaml deleted file mode 100644 index 48ed1e4b..00000000 --- a/config/rbac/auth_proxy_role_binding.yaml +++ /dev/null @@ -1,12 +0,0 @@ -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRoleBinding -metadata: - name: proxy-rolebinding -roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: proxy-role -subjects: -- kind: ServiceAccount - name: default - namespace: system diff --git a/config/rbac/auth_proxy_service.yaml b/config/rbac/auth_proxy_service.yaml deleted file mode 100644 index d61e5469..00000000 --- a/config/rbac/auth_proxy_service.yaml +++ /dev/null @@ -1,18 +0,0 @@ -apiVersion: v1 -kind: Service -metadata: - annotations: - prometheus.io/port: "8443" - prometheus.io/scheme: https - prometheus.io/scrape: "true" - labels: - control-plane: controller-manager - name: controller-manager-metrics-service - namespace: system -spec: - ports: - - name: https - port: 8443 - targetPort: https - selector: - control-plane: controller-manager diff --git a/config/rbac/kustomization.yaml b/config/rbac/kustomization.yaml deleted file mode 100644 index 817f1fe6..00000000 --- a/config/rbac/kustomization.yaml +++ /dev/null @@ -1,11 +0,0 @@ -resources: -- role.yaml -- role_binding.yaml -- leader_election_role.yaml -- leader_election_role_binding.yaml -# Comment the following 3 lines if you want to disable -# the auth proxy (https://github.com/brancz/kube-rbac-proxy) -# which protects your /metrics endpoint. -- auth_proxy_service.yaml -- auth_proxy_role.yaml -- auth_proxy_role_binding.yaml diff --git a/config/rbac/leader_election_role.yaml b/config/rbac/leader_election_role.yaml deleted file mode 100644 index 85093a8c..00000000 --- a/config/rbac/leader_election_role.yaml +++ /dev/null @@ -1,26 +0,0 @@ -# permissions to do leader election. -apiVersion: rbac.authorization.k8s.io/v1 -kind: Role -metadata: - name: leader-election-role -rules: -- apiGroups: - - "" - resources: - - configmaps - verbs: - - get - - list - - watch - - create - - update - - patch - - delete -- apiGroups: - - "" - resources: - - configmaps/status - verbs: - - get - - update - - patch diff --git a/config/rbac/leader_election_role_binding.yaml b/config/rbac/leader_election_role_binding.yaml deleted file mode 100644 index eed16906..00000000 --- a/config/rbac/leader_election_role_binding.yaml +++ /dev/null @@ -1,12 +0,0 @@ -apiVersion: rbac.authorization.k8s.io/v1 -kind: RoleBinding -metadata: - name: leader-election-rolebinding -roleRef: - apiGroup: rbac.authorization.k8s.io - kind: Role - name: leader-election-role -subjects: -- kind: ServiceAccount - name: default - namespace: system diff --git a/config/rbac/role.yaml b/config/rbac/role.yaml deleted file mode 100644 index 5c4ea57f..00000000 --- a/config/rbac/role.yaml +++ /dev/null @@ -1,69 +0,0 @@ - ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRole -metadata: - creationTimestamp: null - name: manager-role -rules: -- apiGroups: - - apps - - extensions - resources: - - daemonsets - - replicasets - - statefulsets - verbs: - - get -- apiGroups: - - batch - resources: - - jobs - verbs: - - get -- apiGroups: - - "" - resources: - - events - verbs: - - create -- apiGroups: - - "" - resources: - - nodes - verbs: - - get - - list - - patch -- apiGroups: - - "" - resources: - - pods - verbs: - - list -- apiGroups: - - "" - resources: - - pods/eviction - verbs: - - create -- apiGroups: - - upgrademgr.keikoproj.io - resources: - - rollingupgrades - verbs: - - create - - delete - - get - - list - - patch - - update - - watch -- apiGroups: - - upgrademgr.keikoproj.io - resources: - - rollingupgrades/status - verbs: - - get - - patch - - update diff --git a/config/rbac/role_binding.yaml b/config/rbac/role_binding.yaml deleted file mode 100644 index 8f265870..00000000 --- a/config/rbac/role_binding.yaml +++ /dev/null @@ -1,12 +0,0 @@ -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRoleBinding -metadata: - name: manager-rolebinding -roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: manager-role -subjects: -- kind: ServiceAccount - name: default - namespace: system diff --git a/config/samples/upgrademgr_v1alpha1_rollingupgrade.yaml b/config/samples/upgrademgr_v1alpha1_rollingupgrade.yaml deleted file mode 100644 index e60a88b5..00000000 --- a/config/samples/upgrademgr_v1alpha1_rollingupgrade.yaml +++ /dev/null @@ -1,7 +0,0 @@ -apiVersion: upgrademgr.keikoproj.io/v1alpha1 -kind: RollingUpgrade -metadata: - generateName: rollingupgrade-sample- -spec: - # Add fields here - foo: bar diff --git a/config/webhook/kustomization.yaml b/config/webhook/kustomization.yaml deleted file mode 100644 index 9cf26134..00000000 --- a/config/webhook/kustomization.yaml +++ /dev/null @@ -1,6 +0,0 @@ -resources: -- manifests.yaml -- service.yaml - -configurations: -- kustomizeconfig.yaml diff --git a/config/webhook/kustomizeconfig.yaml b/config/webhook/kustomizeconfig.yaml deleted file mode 100644 index 25e21e3c..00000000 --- a/config/webhook/kustomizeconfig.yaml +++ /dev/null @@ -1,25 +0,0 @@ -# the following config is for teaching kustomize where to look at when substituting vars. -# It requires kustomize v2.1.0 or newer to work properly. -nameReference: -- kind: Service - version: v1 - fieldSpecs: - - kind: MutatingWebhookConfiguration - group: admissionregistration.k8s.io - path: webhooks/clientConfig/service/name - - kind: ValidatingWebhookConfiguration - group: admissionregistration.k8s.io - path: webhooks/clientConfig/service/name - -namespace: -- kind: MutatingWebhookConfiguration - group: admissionregistration.k8s.io - path: webhooks/clientConfig/service/namespace - create: true -- kind: ValidatingWebhookConfiguration - group: admissionregistration.k8s.io - path: webhooks/clientConfig/service/namespace - create: true - -varReference: -- path: metadata/annotations diff --git a/config/webhook/manifests.yaml b/config/webhook/manifests.yaml deleted file mode 100644 index e69de29b..00000000 diff --git a/config/webhook/service.yaml b/config/webhook/service.yaml deleted file mode 100644 index b4861025..00000000 --- a/config/webhook/service.yaml +++ /dev/null @@ -1,12 +0,0 @@ - -apiVersion: v1 -kind: Service -metadata: - name: webhook-service - namespace: system -spec: - ports: - - port: 443 - targetPort: 443 - selector: - control-plane: controller-manager diff --git a/controllers/events.go b/controllers/events.go deleted file mode 100644 index 9265b8cb..00000000 --- a/controllers/events.go +++ /dev/null @@ -1,75 +0,0 @@ -package controllers - -import ( - "encoding/json" - "fmt" - "math/rand" - "time" - - upgrademgrv1alpha1 "github.com/keikoproj/upgrade-manager/api/v1alpha1" - "github.com/keikoproj/upgrade-manager/pkg/log" - - v1 "k8s.io/api/core/v1" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" -) - -// EventReason defines the reason of an event -type EventReason string - -// EventLevel defines the level of an event -type EventLevel string - -const ( - // EventLevelNormal is the level of a normal event - EventLevelNormal = "Normal" - // EventLevelWarning is the level of a warning event - EventLevelWarning = "Warning" - // EventReasonRUStarted Rolling Upgrade Started - EventReasonRUStarted EventReason = "RollingUpgradeStarted" - // EventReasonRUInstanceStarted Rolling Upgrade for Instance has started - EventReasonRUInstanceStarted EventReason = "RollingUpgradeInstanceStarted" - // EventReasonRUInstanceFinished Rolling Upgrade for Instance has finished - EventReasonRUInstanceFinished EventReason = "RollingUpgradeInstanceFinished" - // EventReasonRUFinished Rolling Upgrade Finished - EventReasonRUFinished EventReason = "RollingUpgradeFinished" -) - -func (r *RollingUpgradeReconciler) createK8sV1Event(objMeta *upgrademgrv1alpha1.RollingUpgrade, reason EventReason, level string, msgFields map[string]string) *v1.Event { - // Marshal as JSON - // I think it is very tough to trigger this error since json.Marshal function can return two types of errors - // UnsupportedTypeError or UnsupportedValueError. Since our type is very rigid, these errors won't be triggered. - b, _ := json.Marshal(msgFields) - msgPayload := string(b) - t := metav1.Time{Time: time.Now()} - event := &v1.Event{ - ObjectMeta: metav1.ObjectMeta{ - Name: fmt.Sprintf("%v-%v.%v", objMeta.Name, time.Now().Unix(), rand.Int()), - }, - Source: v1.EventSource{ - // TODO(vigith): get it from GVK? - Component: "upgrade-manager", - }, - InvolvedObject: v1.ObjectReference{ - Kind: "RollingUpgrade", - Name: objMeta.Name, - Namespace: objMeta.Namespace, - ResourceVersion: objMeta.ResourceVersion, - APIVersion: upgrademgrv1alpha1.GroupVersion.Version, - UID: objMeta.UID, - }, - Reason: string(reason), - Message: msgPayload, - Type: level, - Count: 1, - FirstTimestamp: t, - LastTimestamp: t, - } - - log.Debugf("Publishing event: %v", event) - _event, err := r.generatedClient.CoreV1().Events(objMeta.Namespace).Create(event) - if err != nil { - log.Errorf("Create Events Failed %v, %v", event, err) - } - - return _event -} diff --git a/controllers/events_test.go b/controllers/events_test.go deleted file mode 100644 index 24126029..00000000 --- a/controllers/events_test.go +++ /dev/null @@ -1,37 +0,0 @@ -package controllers - -import ( - "github.com/keikoproj/aws-sdk-go-cache/cache" - upgrademgrv1alpha1 "github.com/keikoproj/upgrade-manager/api/v1alpha1" - "github.com/onsi/gomega" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - "k8s.io/client-go/kubernetes" - log2 "sigs.k8s.io/controller-runtime/pkg/log" - "testing" - "time" -) - -func Test_createK8sV1Event(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}} - - mgr, err := buildManager() - g.Expect(err).NotTo(gomega.HaveOccurred()) - - rcRollingUpgrade := &RollingUpgradeReconciler{ - Client: mgr.GetClient(), - Log: log2.NullLogger{}, - ASGClient: MockAutoscalingGroup{}, - EC2Client: MockEC2{}, - generatedClient: kubernetes.NewForConfigOrDie(mgr.GetConfig()), - ClusterState: NewClusterState(), - CacheConfig: cache.NewConfig(0*time.Second, 0, 0), - ScriptRunner: NewScriptRunner(log2.NullLogger{}), - } - - event := rcRollingUpgrade.createK8sV1Event(ruObj, EventReasonRUStarted, EventLevelNormal, map[string]string{}) - g.Expect(EventReason(event.Reason)).To(gomega.Equal(EventReasonRUStarted)) - - g.Expect(err).To(gomega.BeNil()) -} diff --git a/controllers/helpers.go b/controllers/helpers.go deleted file mode 100644 index 4a7f14bd..00000000 --- a/controllers/helpers.go +++ /dev/null @@ -1,174 +0,0 @@ -package controllers - -import ( - "fmt" - "k8s.io/apimachinery/pkg/util/intstr" - - "github.com/aws/aws-sdk-go/aws" - "github.com/aws/aws-sdk-go/service/autoscaling" - "github.com/aws/aws-sdk-go/service/ec2" - "github.com/aws/aws-sdk-go/service/ec2/ec2iface" - corev1 "k8s.io/api/core/v1" - - upgrademgrv1alpha1 "github.com/keikoproj/upgrade-manager/api/v1alpha1" - "log" -) - -// getMaxUnavailable calculates and returns the maximum unavailable nodes -// takes an update strategy and total number of nodes as input -func getMaxUnavailable(strategy upgrademgrv1alpha1.UpdateStrategy, totalNodes int) int { - maxUnavailable, _ := intstr.GetValueFromIntOrPercent(&strategy.MaxUnavailable, totalNodes, false) - // setting maxUnavailable to total number of nodes when maxUnavailable is greater than total node count - if totalNodes < maxUnavailable { - log.Printf("Reducing maxUnavailable count from %d to %d as total nodes count is %d", - maxUnavailable, totalNodes, totalNodes) - maxUnavailable = totalNodes - } - // maxUnavailable has to be at least 1 when there are nodes in the ASG - if totalNodes > 0 && maxUnavailable < 1 { - maxUnavailable = 1 - } - return maxUnavailable -} - -func isNodeReady(node corev1.Node) bool { - for _, condition := range node.Status.Conditions { - if condition.Type == corev1.NodeReady && condition.Status == corev1.ConditionTrue { - return true - } - } - return false -} - -func IsNodePassesReadinessGates(node corev1.Node, requiredReadinessGates []upgrademgrv1alpha1.NodeReadinessGate) bool { - - if len(requiredReadinessGates) == 0 { - return true - } - for _, gate := range requiredReadinessGates { - for key, value := range gate.MatchLabels { - if node.Labels[key] != value { - return false - } - } - } - return true -} - -func getInServiceCount(instances []*autoscaling.Instance) int64 { - var count int64 - for _, instance := range instances { - if aws.StringValue(instance.LifecycleState) == autoscaling.LifecycleStateInService { - count++ - } - } - return count -} - -func getInServiceIds(instances []*autoscaling.Instance) []string { - list := []string{} - for _, instance := range instances { - if aws.StringValue(instance.LifecycleState) == autoscaling.LifecycleStateInService { - list = append(list, aws.StringValue(instance.InstanceId)) - } - } - return list -} - -func getInstanceStateInASG(group *autoscaling.Group, instanceID string) (string, error) { - for _, instance := range group.Instances { - if aws.StringValue(instance.InstanceId) == instanceID { - return aws.StringValue(instance.LifecycleState), nil - } - } - return "", fmt.Errorf("could not get instance group state, instance %s not found", instanceID) -} - -func isInServiceLifecycleState(state string) bool { - return state == autoscaling.LifecycleStateInService -} - -func tagEC2instance(instanceID, tagKey, tagValue string, client ec2iface.EC2API) error { - input := &ec2.CreateTagsInput{ - Resources: aws.StringSlice([]string{instanceID}), - Tags: []*ec2.Tag{ - { - Key: aws.String(tagKey), - Value: aws.String(tagValue), - }, - }, - } - _, err := client.CreateTags(input) - return err -} - -func getTaggedInstances(tagKey, tagValue string, client ec2iface.EC2API) ([]string, error) { - instances := []string{} - key := fmt.Sprintf("tag:%v", tagKey) - input := &ec2.DescribeInstancesInput{ - Filters: []*ec2.Filter{ - { - Name: aws.String(key), - Values: aws.StringSlice([]string{tagValue}), - }, - }, - } - - err := client.DescribeInstancesPages(input, func(page *ec2.DescribeInstancesOutput, lastPage bool) bool { - for _, res := range page.Reservations { - for _, instance := range res.Instances { - instances = append(instances, aws.StringValue(instance.InstanceId)) - } - } - return page.NextToken != nil - }) - return instances, err -} - -func contains(s []string, e string) bool { - for _, a := range s { - if a == e { - return true - } - } - return false -} - -// getNextAvailableInstances checks the cluster state store for the instance state -// and returns the next set of instances available for update -func getNextAvailableInstances( - asgName string, - numberOfInstances int, - instances []*autoscaling.Instance, - state ClusterState) []*autoscaling.Instance { - return getNextSetOfAvailableInstancesInAz(asgName, "", numberOfInstances, instances, state) -} - -// getNextSetOfAvailableInstancesInAz checks the cluster state store for the instance state -// and returns the next set of instances available for update in the given AX -func getNextSetOfAvailableInstancesInAz( - asgName string, - azName string, - numberOfInstances int, - instances []*autoscaling.Instance, - state ClusterState, -) []*autoscaling.Instance { - - var instancesForUpdate []*autoscaling.Instance - for instancesFound := 0; instancesFound < numberOfInstances; { - instanceId := state.getNextAvailableInstanceIdInAz(asgName, azName) - if len(instanceId) == 0 { - // All instances are updated, no more instance to update in this AZ - break - } - - // check if the instance picked is part of ASG - for _, instance := range instances { - if *instance.InstanceId == instanceId { - instancesForUpdate = append(instancesForUpdate, instance) - instancesFound++ - } - } - } - return instancesForUpdate -} diff --git a/controllers/helpers_test.go b/controllers/helpers_test.go deleted file mode 100644 index 00b0c244..00000000 --- a/controllers/helpers_test.go +++ /dev/null @@ -1,338 +0,0 @@ -package controllers - -import ( - v1 "k8s.io/apimachinery/pkg/apis/meta/v1" - "k8s.io/apimachinery/pkg/util/intstr" - - "github.com/aws/aws-sdk-go/aws" - - "github.com/aws/aws-sdk-go/service/autoscaling" - upgrademgrv1alpha1 "github.com/keikoproj/upgrade-manager/api/v1alpha1" - "github.com/onsi/gomega" - corev1 "k8s.io/api/core/v1" - "testing" -) - -func TestGetMaxUnavailableWithPercentageValue(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - strategy := upgrademgrv1alpha1.UpdateStrategy{ - MaxUnavailable: intstr.Parse("75%"), - } - g.Expect(getMaxUnavailable(strategy, 200)).To(gomega.Equal(150)) -} - -func TestIsNodeReady(t *testing.T) { - g := gomega.NewGomegaWithT(t) - tt := map[corev1.NodeCondition]bool{ - corev1.NodeCondition{Type: corev1.NodeReady, Status: corev1.ConditionTrue}: true, - corev1.NodeCondition{Type: corev1.NodeReady, Status: corev1.ConditionFalse}: false, - } - - for condition, val := range tt { - node := corev1.Node{ - Status: corev1.NodeStatus{ - Conditions: []corev1.NodeCondition{ - condition, - }, - }, - } - g.Expect(isNodeReady(node)).To(gomega.Equal(val)) - } -} - -func TestIsNodePassesReadinessGates(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - type test struct { - gate []map[string]string - labels map[string]string - want bool - } - tests := []test{ - { - gate: []map[string]string{ - { - "healthy": "true", - }, - }, - labels: map[string]string{ - "healthy": "true", - }, - want: true, - }, - - { - gate: []map[string]string{}, - labels: map[string]string{ - "healthy": "true", - }, - want: true, - }, - - { - gate: []map[string]string{ - {"healthy": "true"}, - }, - labels: map[string]string{ - "healthy": "false", - }, - want: false, - }, - - { - gate: []map[string]string{ - {"healthy": "true"}, - }, - labels: map[string]string{}, - want: false, - }, - - { - gate: []map[string]string{ - {"healthy": "true"}, - {"second-check": "true"}, - }, - labels: map[string]string{ - "healthy": "true", - }, - want: false, - }, - { - gate: []map[string]string{ - {"healthy": "true"}, - {"second-check": "true"}, - }, - labels: map[string]string{ - "healthy": "true", - "second-check": "true", - }, - want: true, - }, - { - gate: []map[string]string{ - {"healthy": "true"}, - {"second-check": "true"}, - }, - labels: map[string]string{ - "healthy": "true", - "second-check": "false", - }, - want: false, - }} - - for _, tt := range tests { - readinessGates := make([]upgrademgrv1alpha1.NodeReadinessGate, len(tt.gate)) - for i, g := range tt.gate { - readinessGates[i] = upgrademgrv1alpha1.NodeReadinessGate{ - MatchLabels: g, - } - } - node := corev1.Node{ - ObjectMeta: v1.ObjectMeta{ - Labels: tt.labels, - }, - } - g.Expect(IsNodePassesReadinessGates(node, readinessGates)).To(gomega.Equal(tt.want)) - } - -} - -func TestGetInServiceCount(t *testing.T) { - g := gomega.NewGomegaWithT(t) - tt := map[*autoscaling.Instance]int64{ - &autoscaling.Instance{LifecycleState: aws.String(autoscaling.LifecycleStateInService)}: 1, - &autoscaling.Instance{LifecycleState: aws.String(autoscaling.LifecycleStateDetached)}: 0, - &autoscaling.Instance{LifecycleState: aws.String(autoscaling.LifecycleStateDetaching)}: 0, - &autoscaling.Instance{LifecycleState: aws.String(autoscaling.LifecycleStateEnteringStandby)}: 0, - &autoscaling.Instance{LifecycleState: aws.String(autoscaling.LifecycleStatePending)}: 0, - &autoscaling.Instance{LifecycleState: aws.String(autoscaling.LifecycleStatePendingProceed)}: 0, - &autoscaling.Instance{LifecycleState: aws.String(autoscaling.LifecycleStatePendingWait)}: 0, - &autoscaling.Instance{LifecycleState: aws.String(autoscaling.LifecycleStateQuarantined)}: 0, - &autoscaling.Instance{LifecycleState: aws.String(autoscaling.LifecycleStateStandby)}: 0, - &autoscaling.Instance{LifecycleState: aws.String(autoscaling.LifecycleStateTerminated)}: 0, - &autoscaling.Instance{LifecycleState: aws.String(autoscaling.LifecycleStateTerminating)}: 0, - &autoscaling.Instance{LifecycleState: aws.String(autoscaling.LifecycleStateTerminatingProceed)}: 0, - &autoscaling.Instance{LifecycleState: aws.String(autoscaling.LifecycleStateTerminatingWait)}: 0, - } - - // test every condition - for instance, expectedCount := range tt { - instances := []*autoscaling.Instance{ - instance, - } - g.Expect(getInServiceCount(instances)).To(gomega.Equal(expectedCount)) - } - - // test all instances - instances := []*autoscaling.Instance{} - for instance := range tt { - instances = append(instances, instance) - } - g.Expect(getInServiceCount(instances)).To(gomega.Equal(int64(1))) -} - -func TestGetInServiceIds(t *testing.T) { - g := gomega.NewGomegaWithT(t) - tt := map[*autoscaling.Instance][]string{ - &autoscaling.Instance{InstanceId: aws.String("i-1"), LifecycleState: aws.String(autoscaling.LifecycleStateInService)}: {"i-1"}, - &autoscaling.Instance{InstanceId: aws.String("i-2"), LifecycleState: aws.String(autoscaling.LifecycleStateDetached)}: {}, - &autoscaling.Instance{InstanceId: aws.String("i-3"), LifecycleState: aws.String(autoscaling.LifecycleStateDetaching)}: {}, - &autoscaling.Instance{InstanceId: aws.String("i-4"), LifecycleState: aws.String(autoscaling.LifecycleStateEnteringStandby)}: {}, - &autoscaling.Instance{InstanceId: aws.String("i-5"), LifecycleState: aws.String(autoscaling.LifecycleStatePending)}: {}, - &autoscaling.Instance{InstanceId: aws.String("i-6"), LifecycleState: aws.String(autoscaling.LifecycleStatePendingProceed)}: {}, - &autoscaling.Instance{InstanceId: aws.String("i-7"), LifecycleState: aws.String(autoscaling.LifecycleStatePendingWait)}: {}, - &autoscaling.Instance{InstanceId: aws.String("i-8"), LifecycleState: aws.String(autoscaling.LifecycleStateQuarantined)}: {}, - &autoscaling.Instance{InstanceId: aws.String("i-9"), LifecycleState: aws.String(autoscaling.LifecycleStateStandby)}: {}, - &autoscaling.Instance{InstanceId: aws.String("i-10"), LifecycleState: aws.String(autoscaling.LifecycleStateTerminated)}: {}, - &autoscaling.Instance{InstanceId: aws.String("i-11"), LifecycleState: aws.String(autoscaling.LifecycleStateTerminating)}: {}, - &autoscaling.Instance{InstanceId: aws.String("i-12"), LifecycleState: aws.String(autoscaling.LifecycleStateTerminatingProceed)}: {}, - &autoscaling.Instance{InstanceId: aws.String("i-13"), LifecycleState: aws.String(autoscaling.LifecycleStateTerminatingWait)}: {}, - } - - // test every condition - for instance, expectedList := range tt { - instances := []*autoscaling.Instance{ - instance, - } - g.Expect(getInServiceIds(instances)).To(gomega.Equal(expectedList)) - } - - // test all instances - instances := []*autoscaling.Instance{} - for instance := range tt { - instances = append(instances, instance) - } - g.Expect(getInServiceIds(instances)).To(gomega.Equal([]string{"i-1"})) -} - -func TestGetMaxUnavailableWithPercentageValue33(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - strategy := upgrademgrv1alpha1.UpdateStrategy{ - MaxUnavailable: intstr.Parse("67%"), - } - g.Expect(getMaxUnavailable(strategy, 3)).To(gomega.Equal(2)) -} - -func TestGetMaxUnavailableWithPercentageAndSingleInstance(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - totalNodes := 1 - strategy := upgrademgrv1alpha1.UpdateStrategy{ - MaxUnavailable: intstr.Parse("67%"), - } - g.Expect(getMaxUnavailable(strategy, totalNodes)).To(gomega.Equal(1)) -} - -func TestGetMaxUnavailableWithPercentageNonIntResult(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - strategy := upgrademgrv1alpha1.UpdateStrategy{ - MaxUnavailable: intstr.Parse("37%"), - } - g.Expect(getMaxUnavailable(strategy, 50)).To(gomega.Equal(18)) -} - -func TestGetMaxUnavailableWithIntValue(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - strategy := upgrademgrv1alpha1.UpdateStrategy{ - MaxUnavailable: intstr.Parse("75"), - } - g.Expect(getMaxUnavailable(strategy, 200)).To(gomega.Equal(75)) -} - -func TestGetNextAvailableInstance(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - mockAsgName := "some-asg" - mockInstanceName1 := "foo1" - mockInstanceName2 := "bar1" - az := "az-1" - instance1 := autoscaling.Instance{InstanceId: &mockInstanceName1, AvailabilityZone: &az} - instance2 := autoscaling.Instance{InstanceId: &mockInstanceName2, AvailabilityZone: &az} - - instancesList := []*autoscaling.Instance{&instance1, &instance2} - rcRollingUpgrade := &RollingUpgradeReconciler{ClusterState: clusterState} - rcRollingUpgrade.ClusterState.initializeAsg(mockAsgName, instancesList) - available := getNextAvailableInstances(mockAsgName, 1, instancesList, rcRollingUpgrade.ClusterState) - - g.Expect(1).Should(gomega.Equal(len(available))) - g.Expect(rcRollingUpgrade.ClusterState.deleteAllInstancesInAsg(mockAsgName)).To(gomega.BeTrue()) - -} - -func TestGetNextAvailableInstanceNoInstanceFound(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - mockAsgName := "some-asg" - mockInstanceName1 := "foo1" - mockInstanceName2 := "bar1" - az := "az-1" - instance1 := autoscaling.Instance{InstanceId: &mockInstanceName1, AvailabilityZone: &az} - instance2 := autoscaling.Instance{InstanceId: &mockInstanceName2, AvailabilityZone: &az} - - instancesList := []*autoscaling.Instance{&instance1, &instance2} - rcRollingUpgrade := &RollingUpgradeReconciler{ClusterState: clusterState} - rcRollingUpgrade.ClusterState.initializeAsg(mockAsgName, instancesList) - available := getNextAvailableInstances("asg2", 1, instancesList, rcRollingUpgrade.ClusterState) - - g.Expect(0).Should(gomega.Equal(len(available))) - g.Expect(rcRollingUpgrade.ClusterState.deleteAllInstancesInAsg(mockAsgName)).To(gomega.BeTrue()) - -} - -func TestGetNextAvailableInstanceInAz(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - mockAsgName := "some-asg" - mockInstanceName1 := "foo1" - mockInstanceName2 := "bar1" - az := "az-1" - az2 := "az-2" - instance1 := autoscaling.Instance{InstanceId: &mockInstanceName1, AvailabilityZone: &az} - instance2 := autoscaling.Instance{InstanceId: &mockInstanceName2, AvailabilityZone: &az2} - - instancesList := []*autoscaling.Instance{&instance1, &instance2} - rcRollingUpgrade := &RollingUpgradeReconciler{ClusterState: clusterState} - rcRollingUpgrade.ClusterState.initializeAsg(mockAsgName, instancesList) - - instances := getNextSetOfAvailableInstancesInAz(mockAsgName, az, 1, instancesList, rcRollingUpgrade.ClusterState) - g.Expect(1).Should(gomega.Equal(len(instances))) - g.Expect(mockInstanceName1).Should(gomega.Equal(*instances[0].InstanceId)) - - instances = getNextSetOfAvailableInstancesInAz(mockAsgName, az2, 1, instancesList, rcRollingUpgrade.ClusterState) - g.Expect(1).Should(gomega.Equal(len(instances))) - g.Expect(mockInstanceName2).Should(gomega.Equal(*instances[0].InstanceId)) - - instances = getNextSetOfAvailableInstancesInAz(mockAsgName, "az3", 1, instancesList, rcRollingUpgrade.ClusterState) - g.Expect(0).Should(gomega.Equal(len(instances))) - - g.Expect(rcRollingUpgrade.ClusterState.deleteAllInstancesInAsg(mockAsgName)).To(gomega.BeTrue()) - -} - -func TestGetNextAvailableInstanceInAzGetMultipleInstances(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - mockAsgName := "some-asg" - mockInstanceName1 := "foo1" - mockInstanceName2 := "bar1" - az := "az-1" - instance1 := autoscaling.Instance{InstanceId: &mockInstanceName1, AvailabilityZone: &az} - instance2 := autoscaling.Instance{InstanceId: &mockInstanceName2, AvailabilityZone: &az} - - instancesList := []*autoscaling.Instance{&instance1, &instance2} - rcRollingUpgrade := &RollingUpgradeReconciler{ClusterState: clusterState} - rcRollingUpgrade.ClusterState.initializeAsg(mockAsgName, instancesList) - - instances := getNextSetOfAvailableInstancesInAz(mockAsgName, az, 3, instancesList, rcRollingUpgrade.ClusterState) - - // Even though the request is for 3 instances, only 2 should be returned as there are only 2 nodes in the ASG - g.Expect(2).Should(gomega.Equal(len(instances))) - instanceIds := []string{*instances[0].InstanceId, *instances[1].InstanceId} - g.Expect(instanceIds).Should(gomega.ConsistOf(mockInstanceName1, mockInstanceName2)) - - g.Expect(rcRollingUpgrade.ClusterState.deleteAllInstancesInAsg(mockAsgName)).To(gomega.BeTrue()) -} diff --git a/controllers/launch_definition.go b/controllers/launch_definition.go deleted file mode 100644 index 4fb902be..00000000 --- a/controllers/launch_definition.go +++ /dev/null @@ -1,27 +0,0 @@ -package controllers - -import ( - "github.com/aws/aws-sdk-go/service/autoscaling" -) - -// launchDefinition describes how instances are launched in ASG. -// Supports LaunchConfiguration and LaunchTemplate. -type launchDefinition struct { - // launchConfigurationName is name of LaunchConfiguration used by ASG. - // +optional - launchConfigurationName *string - // launchTemplate is Launch template definition used for ASG. - // +optional - launchTemplate *autoscaling.LaunchTemplateSpecification -} - -func NewLaunchDefinition(asg *autoscaling.Group) *launchDefinition { - template := asg.LaunchTemplate - if template == nil && asg.MixedInstancesPolicy != nil { - template = asg.MixedInstancesPolicy.LaunchTemplate.LaunchTemplateSpecification - } - return &launchDefinition{ - launchConfigurationName: asg.LaunchConfigurationName, - launchTemplate: template, - } -} diff --git a/controllers/node_selector.go b/controllers/node_selector.go deleted file mode 100644 index 8d2cd241..00000000 --- a/controllers/node_selector.go +++ /dev/null @@ -1,19 +0,0 @@ -package controllers - -import ( - "github.com/aws/aws-sdk-go/service/autoscaling" - upgrademgrv1alpha1 "github.com/keikoproj/upgrade-manager/api/v1alpha1" -) - -type NodeSelector interface { - SelectNodesForRestack(state ClusterState) []*autoscaling.Instance -} - -func getNodeSelector(asg *autoscaling.Group, ruObj *upgrademgrv1alpha1.RollingUpgrade) NodeSelector { - switch ruObj.Spec.Strategy.Type { - case upgrademgrv1alpha1.UniformAcrossAzUpdateStrategy: - return NewUniformAcrossAzNodeSelector(asg, ruObj) - default: - return NewRandomNodeSelector(asg, ruObj) - } -} diff --git a/controllers/node_selector_test.go b/controllers/node_selector_test.go deleted file mode 100644 index 4eeae7a1..00000000 --- a/controllers/node_selector_test.go +++ /dev/null @@ -1,88 +0,0 @@ -package controllers - -import ( - "github.com/aws/aws-sdk-go/service/autoscaling" - upgrademgrv1alpha1 "github.com/keikoproj/upgrade-manager/api/v1alpha1" - "github.com/onsi/gomega" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - "testing" -) - -func TestGetRandomNodeSelector(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - someAsg := "some-asg" - mockID := "some-id" - someLaunchConfig := "some-launch-config" - diffLaunchConfig := "different-launch-config" - az := "az-1" - mockInstance := autoscaling.Instance{InstanceId: &mockID, LaunchConfigurationName: &diffLaunchConfig, AvailabilityZone: &az} - mockAsg := autoscaling.Group{AutoScalingGroupName: &someAsg, - LaunchConfigurationName: &someLaunchConfig, - Instances: []*autoscaling.Instance{&mockInstance}} - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ - ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{AsgName: someAsg}, - } - - nodeSelector := getNodeSelector(&mockAsg, ruObj) - - g.Expect(nodeSelector).Should(gomega.BeAssignableToTypeOf(&RandomNodeSelector{})) -} - -func TestGetUniformAcrossAzNodeSelector(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - someAsg := "some-asg" - mockID := "some-id" - someLaunchConfig := "some-launch-config" - diffLaunchConfig := "different-launch-config" - az := "az-1" - mockInstance := autoscaling.Instance{InstanceId: &mockID, LaunchConfigurationName: &diffLaunchConfig, AvailabilityZone: &az} - mockAsg := autoscaling.Group{AutoScalingGroupName: &someAsg, - LaunchConfigurationName: &someLaunchConfig, - Instances: []*autoscaling.Instance{&mockInstance}} - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ - ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{ - AsgName: someAsg, - Strategy: upgrademgrv1alpha1.UpdateStrategy{ - Type: upgrademgrv1alpha1.UniformAcrossAzUpdateStrategy, - }, - }, - } - - nodeSelector := getNodeSelector(&mockAsg, ruObj) - - g.Expect(nodeSelector).Should(gomega.BeAssignableToTypeOf(&UniformAcrossAzNodeSelector{})) -} - -func TestGetNodeSelectorWithInvalidStrategy(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - someAsg := "some-asg" - mockID := "some-id" - someLaunchConfig := "some-launch-config" - diffLaunchConfig := "different-launch-config" - az := "az-1" - mockInstance := autoscaling.Instance{InstanceId: &mockID, LaunchConfigurationName: &diffLaunchConfig, AvailabilityZone: &az} - mockAsg := autoscaling.Group{AutoScalingGroupName: &someAsg, - LaunchConfigurationName: &someLaunchConfig, - Instances: []*autoscaling.Instance{&mockInstance}} - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ - ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{ - AsgName: someAsg, - Strategy: upgrademgrv1alpha1.UpdateStrategy{ - Type: "invalid", - }, - }, - } - - nodeSelector := getNodeSelector(&mockAsg, ruObj) - - g.Expect(nodeSelector).Should(gomega.BeAssignableToTypeOf(&RandomNodeSelector{})) -} diff --git a/controllers/random_node_selector.go b/controllers/random_node_selector.go deleted file mode 100644 index 0916de45..00000000 --- a/controllers/random_node_selector.go +++ /dev/null @@ -1,27 +0,0 @@ -package controllers - -import ( - "github.com/aws/aws-sdk-go/service/autoscaling" - upgrademgrv1alpha1 "github.com/keikoproj/upgrade-manager/api/v1alpha1" - "log" -) - -type RandomNodeSelector struct { - maxUnavailable int - ruObj *upgrademgrv1alpha1.RollingUpgrade - asg *autoscaling.Group -} - -func NewRandomNodeSelector(asg *autoscaling.Group, ruObj *upgrademgrv1alpha1.RollingUpgrade) *RandomNodeSelector { - maxUnavailable := getMaxUnavailable(ruObj.Spec.Strategy, len(asg.Instances)) - log.Printf("Max unavailable calculated for %s is %d", ruObj.Name, maxUnavailable) - return &RandomNodeSelector{ - maxUnavailable: maxUnavailable, - ruObj: ruObj, - asg: asg, - } -} - -func (selector *RandomNodeSelector) SelectNodesForRestack(state ClusterState) []*autoscaling.Instance { - return getNextAvailableInstances(selector.ruObj.Spec.AsgName, selector.maxUnavailable, selector.asg.Instances, state) -} diff --git a/controllers/rollingupgrade_controller.go b/controllers/rollingupgrade_controller.go deleted file mode 100644 index 72c421d7..00000000 --- a/controllers/rollingupgrade_controller.go +++ /dev/null @@ -1,1185 +0,0 @@ -/* - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package controllers - -import ( - "context" - "fmt" - "strconv" - "strings" - "sync" - "time" - - "github.com/aws/aws-sdk-go/aws" - "github.com/aws/aws-sdk-go/aws/awserr" - "github.com/aws/aws-sdk-go/service/autoscaling" - "github.com/aws/aws-sdk-go/service/autoscaling/autoscalingiface" - "github.com/aws/aws-sdk-go/service/ec2" - "github.com/aws/aws-sdk-go/service/ec2/ec2iface" - "github.com/go-logr/logr" - "github.com/keikoproj/aws-sdk-go-cache/cache" - iebackoff "github.com/keikoproj/inverse-exp-backoff" - corev1 "k8s.io/api/core/v1" - k8serrors "k8s.io/apimachinery/pkg/api/errors" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - "k8s.io/apimachinery/pkg/util/intstr" - "k8s.io/client-go/kubernetes" - v1 "k8s.io/client-go/kubernetes/typed/core/v1" - ctrl "sigs.k8s.io/controller-runtime" - "sigs.k8s.io/controller-runtime/pkg/client" - "sigs.k8s.io/controller-runtime/pkg/controller" - "sigs.k8s.io/controller-runtime/pkg/reconcile" - - upgrademgrv1alpha1 "github.com/keikoproj/upgrade-manager/api/v1alpha1" -) - -const ( - // JanitorAnnotation is for completed objects. - JanitorAnnotation = "janitor/ttl" - // ClearCompletedFrequency is the time after which a completed rollingUpgrade object is deleted. - ClearCompletedFrequency = "1d" - // ClearErrorFrequency is the time after which an errored rollingUpgrade object is deleted. - ClearErrorFrequency = "7d" - // EC2StateTagKey is the EC2 tag key for indicating the state - EC2StateTagKey = "upgrademgr.keikoproj.io/state" - - // Environment variable keys - asgNameKey = "ASG_NAME" - instanceIDKey = "INSTANCE_ID" - instanceNameKey = "INSTANCE_NAME" - - // InService is a state of an instance - InService = "InService" -) - -var ( - // TerminationTimeoutSeconds is the timeout threshold for waiting for a node object unjoin - TerminationTimeoutSeconds = 3600 - // TerminationSleepIntervalSeconds is the polling interval for checking if a node object is unjoined - TerminationSleepIntervalSeconds = 30 - // WaiterMaxDelay is the maximum delay for waiters inverse exponential backoff - WaiterMaxDelay = time.Second * 90 - // WaiterMinDelay is the minimum delay for waiters inverse exponential backoff - WaiterMinDelay = time.Second * 15 - // WaiterFactor is the delay reduction factor per retry - WaiterFactor = 0.5 - // WaiterMaxAttempts is the maximum number of retries for waiters - WaiterMaxAttempts = uint32(32) - // CacheTTL is ttl for ASG cache. - CacheTTL = 30 * time.Second -) - -// RollingUpgradeReconciler reconciles a RollingUpgrade object -type RollingUpgradeReconciler struct { - client.Client - Log logr.Logger - EC2Client ec2iface.EC2API - ASGClient autoscalingiface.AutoScalingAPI - generatedClient *kubernetes.Clientset - NodeList *corev1.NodeList - LaunchTemplates []*ec2.LaunchTemplate - inProcessASGs sync.Map - admissionMap sync.Map - ruObjNameToASG AsgCache - ClusterState ClusterState - maxParallel int - CacheConfig *cache.Config - ScriptRunner ScriptRunner -} - -type AsgCache struct { - cache sync.Map -} - -func (c *AsgCache) IsExpired(name string) bool { - if val, ok := c.cache.Load(name); !ok { - return true - } else { - cached := val.(CachedValue) - return time.Now().After(cached.expiration) - } -} - -func (c *AsgCache) Load(name string) (*autoscaling.Group, bool) { - if val, ok := c.cache.Load(name); !ok { - return nil, false - } else { - cached := val.(CachedValue) - return cached.val.(*autoscaling.Group), true - } -} - -func (c *AsgCache) Store(name string, asg *autoscaling.Group) { - c.cache.Store(name, CachedValue{asg, time.Now().Add(CacheTTL)}) -} - -func (c *AsgCache) Delete(name string) { - c.cache.Delete(name) -} - -type CachedValue struct { - val interface{} - expiration time.Time -} - -func (r *RollingUpgradeReconciler) SetMaxParallel(max int) { - if max >= 1 { - r.Log.Info(fmt.Sprintf("max parallel reconciles = %v", max)) - r.maxParallel = max - } -} - -func (r *RollingUpgradeReconciler) preDrainHelper(instanceID, nodeName string, ruObj *upgrademgrv1alpha1.RollingUpgrade) error { - return r.ScriptRunner.PreDrain(instanceID, nodeName, ruObj) -} - -// Operates on any scripts that were provided after the draining of the node. -// kubeCtlCall is provided as an argument to decouple the method from the actual kubectl call -func (r *RollingUpgradeReconciler) postDrainHelper(instanceID, nodeName string, ruObj *upgrademgrv1alpha1.RollingUpgrade) error { - err := r.ScriptRunner.PostDrain(instanceID, nodeName, ruObj) - if err != nil { - return err - } - - r.info(ruObj, "Waiting for postDrainDelay", "postDrainDelay", ruObj.Spec.PostDrainDelaySeconds) - time.Sleep(time.Duration(ruObj.Spec.PostDrainDelaySeconds) * time.Second) - - return r.ScriptRunner.PostWait(instanceID, nodeName, ruObj) - -} - -// DrainNode runs "kubectl drain" on the given node -// kubeCtlCall is provided as an argument to decouple the method from the actual kubectl call -func (r *RollingUpgradeReconciler) DrainNode(ruObj *upgrademgrv1alpha1.RollingUpgrade, - nodeName string, - instanceID string, - drainTimeout int) error { - // Running kubectl drain node. - err := r.preDrainHelper(instanceID, nodeName, ruObj) - if err != nil { - return fmt.Errorf("%s: pre-drain script failed: %w", ruObj.NamespacedName(), err) - } - - errChan := make(chan error) - ctx := context.TODO() - var cancel context.CancelFunc - - // Add a context with timeout only if a valid drain timeout value is specified - // default value used for drain timeout is -1 - if drainTimeout >= 0 { - r.info(ruObj, "Creating a context with timeout", "drainTimeout", drainTimeout) - // Define a cancellation after drainTimeout - ctx, cancel = context.WithTimeout(ctx, time.Duration(drainTimeout)*time.Second) - defer cancel() - } else { - r.info(ruObj, "Skipped creating context with timeout.", "drainTimeout", drainTimeout) - } - - r.info(ruObj, "Invoking kubectl drain for the node", "nodeName", nodeName) - go r.CallKubectlDrain(nodeName, ruObj, errChan) - - // Listening to signals from the CallKubectlDrain go routine - select { - case <-ctx.Done(): - r.error(ruObj, ctx.Err(), "Kubectl drain timed out for node", "nodeName", nodeName) - case err := <-errChan: - if err != nil { - r.error(ruObj, err, "Kubectl drain errored for node", "nodeName", nodeName) - return err - } - r.info(ruObj, "Kubectl drain completed for node", "nodeName", nodeName) - } - - return r.postDrainHelper(instanceID, nodeName, ruObj) -} - -// CallKubectlDrain runs the "kubectl drain" for a given node -// Node will be terminated even if pod eviction is not completed when the drain timeout is exceeded -func (r *RollingUpgradeReconciler) CallKubectlDrain(nodeName string, ruObj *upgrademgrv1alpha1.RollingUpgrade, errChan chan error) { - out, err := r.ScriptRunner.drainNode(nodeName, ruObj) - if err != nil { - if strings.Contains(out, "Error from server (NotFound): nodes") { - r.error(ruObj, err, "Not executing postDrainHelper. Node not found.", "output", out) - errChan <- nil - return - } - errChan <- fmt.Errorf("%s failed to drain: %w", ruObj.NamespacedName(), err) - return - } - errChan <- nil -} - -func (r *RollingUpgradeReconciler) WaitForDesiredInstances(ruObj *upgrademgrv1alpha1.RollingUpgrade) error { - var err error - var ieb *iebackoff.IEBackoff - for ieb, err = iebackoff.NewIEBackoff(WaiterMaxDelay, WaiterMinDelay, 0.5, WaiterMaxAttempts); err == nil; err = ieb.Next() { - err = r.populateAsg(ruObj) - if err != nil { - return err - } - - asg, err := r.GetAutoScalingGroup(ruObj.NamespacedName()) - if err != nil { - return fmt.Errorf("Unable to load ASG with name: %s", ruObj.Name) - } - - inServiceCount := getInServiceCount(asg.Instances) - if inServiceCount == aws.Int64Value(asg.DesiredCapacity) { - r.info(ruObj, "desired capacity is met", "inServiceCount", inServiceCount) - return nil - } - - r.info(ruObj, "new instance has not yet joined the scaling group") - } - return fmt.Errorf("%s: WaitForDesiredInstances timed out while waiting for instance to be added: %w", ruObj.NamespacedName(), err) -} - -func (r *RollingUpgradeReconciler) WaitForDesiredNodes(ruObj *upgrademgrv1alpha1.RollingUpgrade) error { - var err error - var ieb *iebackoff.IEBackoff - for ieb, err = iebackoff.NewIEBackoff(WaiterMaxDelay, WaiterMinDelay, 0.5, WaiterMaxAttempts); err == nil; err = ieb.Next() { - err = r.populateAsg(ruObj) - if err != nil { - return err - } - - err = r.populateNodeList(ruObj, r.generatedClient.CoreV1().Nodes()) - if err != nil { - r.error(ruObj, err, "unable to populate node list") - } - - asg, err := r.GetAutoScalingGroup(ruObj.NamespacedName()) - if err != nil { - return fmt.Errorf("Unable to load ASG with name: %s", ruObj.Name) - } - - // get list of inService instance IDs - inServiceInstances := getInServiceIds(asg.Instances) - desiredCapacity := aws.Int64Value(asg.DesiredCapacity) - - // check all of them are nodes and are ready - var foundCount int64 = 0 - for _, node := range r.NodeList.Items { - tokens := strings.Split(node.Spec.ProviderID, "/") - instanceID := tokens[len(tokens)-1] - if contains(inServiceInstances, instanceID) && isNodeReady(node) && IsNodePassesReadinessGates(node, ruObj.Spec.ReadinessGates) { - foundCount++ - } - } - - if foundCount == desiredCapacity { - r.info(ruObj, "desired capacity is met", "inServiceCount", foundCount) - return nil - } - - r.info(ruObj, "new node has not yet joined the cluster") - } - return fmt.Errorf("%s: WaitForDesiredNodes timed out while waiting for nodes to join: %w", ruObj.NamespacedName(), err) -} - -func (r *RollingUpgradeReconciler) WaitForTermination(ruObj *upgrademgrv1alpha1.RollingUpgrade, nodeName string, nodeInterface v1.NodeInterface) (bool, error) { - if nodeName == "" { - return true, nil - } - - started := time.Now() - for { - if time.Since(started) >= (time.Second * time.Duration(TerminationTimeoutSeconds)) { - r.info(ruObj, "WaitForTermination timed out while waiting for node to unjoin") - return false, nil - } - - _, err := nodeInterface.Get(nodeName, metav1.GetOptions{}) - if k8serrors.IsNotFound(err) { - r.info(ruObj, "node is unjoined from cluster, upgrade will proceed", "nodeName", nodeName) - break - } - - r.info(ruObj, "node is still joined to cluster, will wait and retry", - "nodeName", nodeName, "terminationSleepIntervalSeconds", TerminationSleepIntervalSeconds) - - time.Sleep(time.Duration(TerminationSleepIntervalSeconds) * time.Second) - } - return true, nil -} - -func (r *RollingUpgradeReconciler) GetAutoScalingGroup(namespacedName string) (*autoscaling.Group, error) { - val, ok := r.ruObjNameToASG.Load(namespacedName) - if !ok { - return &autoscaling.Group{}, fmt.Errorf("Unable to load ASG with name: %s", namespacedName) - } - return val, nil -} - -// SetStandby sets the autoscaling instance to standby mode. -func (r *RollingUpgradeReconciler) SetStandby(ruObj *upgrademgrv1alpha1.RollingUpgrade, instanceID string) error { - r.info(ruObj, "Setting to stand-by", ruObj.Name, instanceID) - - asg, err := r.GetAutoScalingGroup(ruObj.NamespacedName()) - if err != nil { - return err - } - - instanceState, err := getInstanceStateInASG(asg, instanceID) - if err != nil { - r.info(ruObj, fmt.Sprintf("WARNING: %v", err)) - return nil - } - - if instanceState == autoscaling.LifecycleStateStandby { - return nil - } - - if !isInServiceLifecycleState(instanceState) { - r.info(ruObj, "Cannot set instance to stand-by, instance is in state", "instanceState", instanceState, "instanceID", instanceID) - return nil - } - - input := &autoscaling.EnterStandbyInput{ - AutoScalingGroupName: aws.String(ruObj.Spec.AsgName), - InstanceIds: aws.StringSlice([]string{instanceID}), - ShouldDecrementDesiredCapacity: aws.Bool(false), - } - - _, err = r.ASGClient.EnterStandby(input) - if err != nil { - r.error(ruObj, err, "Failed to enter standby", "instanceID", instanceID) - } - return nil -} - -// TerminateNode actually terminates the given node. -func (r *RollingUpgradeReconciler) TerminateNode(ruObj *upgrademgrv1alpha1.RollingUpgrade, instanceID string, nodeName string) error { - - input := &autoscaling.TerminateInstanceInAutoScalingGroupInput{ - InstanceId: aws.String(instanceID), - ShouldDecrementDesiredCapacity: aws.Bool(false), - } - var err error - var ieb *iebackoff.IEBackoff - for ieb, err = iebackoff.NewIEBackoff(WaiterMaxDelay, WaiterMinDelay, 0.5, WaiterMaxAttempts); err == nil; err = ieb.Next() { - _, err := r.ASGClient.TerminateInstanceInAutoScalingGroup(input) - if err == nil { - break - } - if aerr, ok := err.(awserr.Error); ok { - if strings.Contains(aerr.Message(), "not found") { - r.info(ruObj, "Instance not found. Moving on", "instanceID", instanceID) - return nil - } - switch aerr.Code() { - case autoscaling.ErrCodeScalingActivityInProgressFault: - r.error(ruObj, aerr, autoscaling.ErrCodeScalingActivityInProgressFault, "instanceID", instanceID) - case autoscaling.ErrCodeResourceContentionFault: - r.error(ruObj, aerr, autoscaling.ErrCodeResourceContentionFault, "instanceID", instanceID) - default: - r.error(ruObj, aerr, aerr.Code(), "instanceID", instanceID) - return err - } - } - } - if err != nil { - return err - } - r.info(ruObj, "Instance terminated.", "instanceID", instanceID) - r.info(ruObj, "starting post termination sleep", "instanceID", instanceID, "nodeIntervalSeconds", ruObj.Spec.NodeIntervalSeconds) - time.Sleep(time.Duration(ruObj.Spec.NodeIntervalSeconds) * time.Second) - return r.ScriptRunner.PostTerminate(instanceID, nodeName, ruObj) -} - -func (r *RollingUpgradeReconciler) getNodeName(i *autoscaling.Instance, nodeList *corev1.NodeList, ruObj *upgrademgrv1alpha1.RollingUpgrade) string { - node := r.getNodeFromAsg(i, nodeList, ruObj) - if node == nil { - r.info(ruObj, "Node name for instance not found", "instanceID", *i.InstanceId) - return "" - } - return node.Name -} - -func (r *RollingUpgradeReconciler) getNodeFromAsg(i *autoscaling.Instance, nodeList *corev1.NodeList, ruObj *upgrademgrv1alpha1.RollingUpgrade) *corev1.Node { - for _, n := range nodeList.Items { - tokens := strings.Split(n.Spec.ProviderID, "/") - justID := tokens[len(tokens)-1] - if *i.InstanceId == justID { - r.info(ruObj, "Found instance", "instanceID", justID, "instanceName", n.Name) - return &n - } - } - - r.info(ruObj, "Node for instance not found", "instanceID", *i.InstanceId) - return nil -} - -func (r *RollingUpgradeReconciler) populateAsg(ruObj *upgrademgrv1alpha1.RollingUpgrade) error { - // if value is still in cache, do nothing. - if !r.ruObjNameToASG.IsExpired(ruObj.NamespacedName()) { - return nil - } - - input := &autoscaling.DescribeAutoScalingGroupsInput{ - AutoScalingGroupNames: []*string{ - aws.String(ruObj.Spec.AsgName), - }, - } - result, err := r.ASGClient.DescribeAutoScalingGroups(input) - if err != nil { - r.error(ruObj, err, "Failed to describe autoscaling group") - return fmt.Errorf("%s: failed to describe autoscaling group: %w", ruObj.NamespacedName(), err) - } - - if len(result.AutoScalingGroups) == 0 { - r.info(ruObj, "%s: No ASG found with name %s!\n", ruObj.Name, ruObj.Spec.AsgName) - return fmt.Errorf("%s: no ASG found", ruObj.NamespacedName()) - } else if len(result.AutoScalingGroups) > 1 { - r.info(ruObj, "%s: Too many asgs found with name %d!\n", ruObj.Name, len(result.AutoScalingGroups)) - return fmt.Errorf("%s: Too many ASGs: %d", ruObj.NamespacedName(), len(result.AutoScalingGroups)) - } - - asg := result.AutoScalingGroups[0] - r.ruObjNameToASG.Store(ruObj.NamespacedName(), asg) - - return nil -} - -func (r *RollingUpgradeReconciler) populateLaunchTemplates(ruObj *upgrademgrv1alpha1.RollingUpgrade) error { - launchTemplates := []*ec2.LaunchTemplate{} - err := r.EC2Client.DescribeLaunchTemplatesPages(&ec2.DescribeLaunchTemplatesInput{}, func(page *ec2.DescribeLaunchTemplatesOutput, lastPage bool) bool { - launchTemplates = append(launchTemplates, page.LaunchTemplates...) - return page.NextToken != nil - }) - if err != nil { - r.error(ruObj, err, "Failed to populate launch template list") - return fmt.Errorf("failed to populate launch template list for %s: %w", ruObj.NamespacedName(), err) - } - r.LaunchTemplates = launchTemplates - return nil -} - -func (r *RollingUpgradeReconciler) populateNodeList(ruObj *upgrademgrv1alpha1.RollingUpgrade, nodeInterface v1.NodeInterface) error { - nodeList, err := nodeInterface.List(metav1.ListOptions{}) - if err != nil { - msg := "Failed to get all nodes in the cluster: " + err.Error() - r.info(ruObj, msg) - return fmt.Errorf("%s: Failed to get all nodes in the cluster: %w", ruObj.NamespacedName(), err) - } - r.NodeList = nodeList - return nil -} - -func (r *RollingUpgradeReconciler) getInProgressInstances(instances []*autoscaling.Instance) ([]*autoscaling.Instance, error) { - var inProgressInstances []*autoscaling.Instance - taggedInstances, err := getTaggedInstances(EC2StateTagKey, "in-progress", r.EC2Client) - if err != nil { - return inProgressInstances, err - } - for _, instance := range instances { - if contains(taggedInstances, aws.StringValue(instance.InstanceId)) { - inProgressInstances = append(inProgressInstances, instance) - } - } - return inProgressInstances, nil -} - -// runRestack performs rollout of new nodes. -// returns number of processed instances and optional error. -func (r *RollingUpgradeReconciler) runRestack(ctx *context.Context, ruObj *upgrademgrv1alpha1.RollingUpgrade) (int, error) { - - asg, err := r.GetAutoScalingGroup(ruObj.NamespacedName()) - if err != nil { - return 0, fmt.Errorf("Unable to load ASG with name: %s", ruObj.Name) - } - - r.info(ruObj, "Nodes in ASG that *might* need to be updated", "asgName", *asg.AutoScalingGroupName, "asgSize", len(asg.Instances)) - - totalNodes := len(asg.Instances) - // No further processing is required if ASG doesn't have an instance running - if totalNodes == 0 { - r.info(ruObj, fmt.Sprintf("Total nodes needing update for %s is 0. Restack complete.", *asg.AutoScalingGroupName)) - return 0, nil - } - - nodeSelector := getNodeSelector(asg, ruObj) - - r.inProcessASGs.Store(*asg.AutoScalingGroupName, "running") - r.ClusterState.initializeAsg(*asg.AutoScalingGroupName, asg.Instances) - defer r.ClusterState.deleteAllInstancesInAsg(*asg.AutoScalingGroupName) - - launchDefinition := NewLaunchDefinition(asg) - - processedInstances := 0 - - inProgress, err := r.getInProgressInstances(asg.Instances) - if err != nil { - r.error(ruObj, err, "Failed to acquire in-progress instances") - } - - for processedInstances < totalNodes { - var instances []*autoscaling.Instance - if len(inProgress) == 0 { - // Fetch instances to update from node selector - instances = nodeSelector.SelectNodesForRestack(r.ClusterState) - r.info(ruObj, fmt.Sprintf("selected instances for rotation: %+v", instances)) - } else { - // Prefer in progress instances over new ones - instances = inProgress - inProgress = []*autoscaling.Instance{} - r.info(ruObj, fmt.Sprintf("found in progress instances: %+v", instances)) - } - - if instances == nil { - errorMessage := fmt.Sprintf( - "No instances available for update across all AZ's for %s. Processed %d of total %d instances", - ruObj.Name, processedInstances, totalNodes) - // No instances fetched from any AZ, stop processing - r.info(ruObj, errorMessage) - - // this should never be case, return error - return processedInstances, fmt.Errorf(errorMessage) - } - - // update the instances - err := r.UpdateInstances(ctx, ruObj, instances, launchDefinition) - processedInstances += len(instances) - if err != nil { - return processedInstances, err - } - } - return processedInstances, nil -} - -func (r *RollingUpgradeReconciler) finishExecution(err error, nodesProcessed int, ctx *context.Context, ruObj *upgrademgrv1alpha1.RollingUpgrade) { - var level string - var finalStatus string - - if err == nil { - finalStatus = upgrademgrv1alpha1.StatusComplete - level = EventLevelNormal - r.info(ruObj, "Marked object as", "finalStatus", finalStatus) - } else { - finalStatus = upgrademgrv1alpha1.StatusError - level = EventLevelWarning - r.error(ruObj, err, "Marked object as", "finalStatus", finalStatus) - } - - endTime := time.Now() - ruObj.Status.EndTime = endTime.Format(time.RFC3339) - ruObj.Status.CurrentStatus = finalStatus - ruObj.Status.NodesProcessed = nodesProcessed - - ruObj.Status.Conditions = append(ruObj.Status.Conditions, - upgrademgrv1alpha1.RollingUpgradeCondition{ - Type: upgrademgrv1alpha1.UpgradeComplete, - Status: corev1.ConditionTrue, - }) - - startTime, err := time.Parse(time.RFC3339, ruObj.Status.StartTime) - if err != nil { - r.info(ruObj, "Failed to calculate totalProcessingTime") - } else { - ruObj.Status.TotalProcessingTime = endTime.Sub(startTime).String() - } - // end event - - r.createK8sV1Event(ruObj, EventReasonRUFinished, level, map[string]string{ - "status": finalStatus, - "asgName": ruObj.Spec.AsgName, - "strategy": string(ruObj.Spec.Strategy.Type), - "info": fmt.Sprintf("Rolling Upgrade as finished (status=%s)", finalStatus), - }) - - MarkObjForCleanup(ruObj) - if err := r.Status().Update(*ctx, ruObj); err != nil { - // Check if the err is "StorageError: invalid object". If so, the object was deleted... - if strings.Contains(err.Error(), "StorageError: invalid object") { - r.info(ruObj, "Object most likely deleted") - } else { - r.error(ruObj, err, "failed to update status") - } - } - - r.ClusterState.deleteAllInstancesInAsg(ruObj.Spec.AsgName) - r.info(ruObj, "Deleted the entries of ASG in the cluster store", "asgName", ruObj.Spec.AsgName) - r.inProcessASGs.Delete(ruObj.Spec.AsgName) - r.admissionMap.Delete(ruObj.NamespacedName()) - r.info(ruObj, "Deleted from admission map ", "admissionMap", &r.admissionMap) -} - -// Process actually performs the ec2-instance restacking. -func (r *RollingUpgradeReconciler) Process(ctx *context.Context, - ruObj *upgrademgrv1alpha1.RollingUpgrade) { - - if ruObj.Status.CurrentStatus == upgrademgrv1alpha1.StatusComplete || - ruObj.Status.CurrentStatus == upgrademgrv1alpha1.StatusError { - r.info(ruObj, "No more processing", "currentStatus", ruObj.Status.CurrentStatus) - - if exists := ruObj.ObjectMeta.Annotations[JanitorAnnotation]; exists == "" { - r.info(ruObj, "Marking object for deletion") - MarkObjForCleanup(ruObj) - } - - r.admissionMap.Delete(ruObj.NamespacedName()) - r.info(ruObj, "Deleted object from admission map") - return - } - // start event - r.createK8sV1Event(ruObj, EventReasonRUStarted, EventLevelNormal, map[string]string{ - "status": "started", - "asgName": ruObj.Spec.AsgName, - "strategy": string(ruObj.Spec.Strategy.Type), - "msg": "Rolling Upgrade has started", - }) - r.CacheConfig.FlushCache("autoscaling") - err := r.populateAsg(ruObj) - if err != nil { - r.finishExecution(err, 0, ctx, ruObj) - return - } - - //TODO(shri): Ensure that no node is Unschedulable at this time. - err = r.populateNodeList(ruObj, r.generatedClient.CoreV1().Nodes()) - if err != nil { - r.finishExecution(err, 0, ctx, ruObj) - return - } - - if err := r.populateLaunchTemplates(ruObj); err != nil { - r.finishExecution(err, 0, ctx, ruObj) - return - } - - asg, err := r.GetAutoScalingGroup(ruObj.NamespacedName()) - if err != nil { - r.error(ruObj, err, "Unable to load ASG for rolling upgrade") - r.finishExecution(err, 0, ctx, ruObj) - return - } - - // Update the CR with some basic info before staring the restack. - ruObj.Status.StartTime = time.Now().Format(time.RFC3339) - ruObj.Status.CurrentStatus = upgrademgrv1alpha1.StatusRunning - ruObj.Status.NodesProcessed = 0 - ruObj.Status.TotalNodes = len(asg.Instances) - - if err := r.Status().Update(*ctx, ruObj); err != nil { - r.error(ruObj, err, "failed to update status") - } - - // Run the restack that actually performs the rolling update. - nodesProcessed, err := r.runRestack(ctx, ruObj) - if err != nil { - r.error(ruObj, err, "Failed to runRestack") - r.finishExecution(err, nodesProcessed, ctx, ruObj) - return - } - - //Validation step: check if all the nodes have the latest launchconfig. - r.info(ruObj, "Validating the launch definition of nodes and ASG") - if err := r.validateNodesLaunchDefinition(ruObj); err != nil { - r.error(ruObj, err, "Launch definition validation failed") - r.finishExecution(err, nodesProcessed, ctx, ruObj) - return - } - - // no error -> report success - r.finishExecution(nil, nodesProcessed, ctx, ruObj) -} - -//Check if ec2Instances and the ASG have same launch config. -func (r *RollingUpgradeReconciler) validateNodesLaunchDefinition(ruObj *upgrademgrv1alpha1.RollingUpgrade) error { - //Get ASG launch config - var err error - err = r.populateAsg(ruObj) - if err != nil { - return fmt.Errorf("%s: Unable to populate the ASG object: %w", ruObj.NamespacedName(), err) - } - asg, err := r.GetAutoScalingGroup(ruObj.NamespacedName()) - if err != nil { - return fmt.Errorf("%s: Unable to load ASG with name: %w", ruObj.NamespacedName(), err) - } - launchDefinition := NewLaunchDefinition(asg) - launchConfigASG, launchTemplateASG := launchDefinition.launchConfigurationName, launchDefinition.launchTemplate - - //Get ec2 instances and their launch configs. - ec2instances := asg.Instances - for _, ec2Instance := range ec2instances { - ec2InstanceID, ec2InstanceLaunchConfig, ec2InstanceLaunchTemplate := ec2Instance.InstanceId, ec2Instance.LaunchConfigurationName, ec2Instance.LaunchTemplate - if aws.StringValue(ec2Instance.LifecycleState) == InService { - continue - } - if aws.StringValue(launchConfigASG) != aws.StringValue(ec2InstanceLaunchConfig) { - return fmt.Errorf("launch config mismatch, %s instance config - %s, does not match the asg config", aws.StringValue(ec2InstanceID), aws.StringValue(ec2InstanceLaunchConfig)) - } else if launchTemplateASG != nil && ec2InstanceLaunchTemplate != nil { - if aws.StringValue(launchTemplateASG.LaunchTemplateId) != aws.StringValue(ec2InstanceLaunchTemplate.LaunchTemplateId) { - return fmt.Errorf("launch template mismatch, %s instance template - %s, does not match the asg template", aws.StringValue(ec2InstanceID), aws.StringValue(ec2InstanceLaunchTemplate.LaunchTemplateId)) - } - } - } - return nil -} - -// MarkObjForCleanup sets the annotation on the given object for deletion. -func MarkObjForCleanup(ruObj *upgrademgrv1alpha1.RollingUpgrade) { - if ruObj.ObjectMeta.Annotations == nil { - ruObj.ObjectMeta.Annotations = map[string]string{} - } - - switch ruObj.Status.CurrentStatus { - case upgrademgrv1alpha1.StatusComplete: - ruObj.ObjectMeta.Annotations[JanitorAnnotation] = ClearCompletedFrequency - case upgrademgrv1alpha1.StatusError: - ruObj.ObjectMeta.Annotations[JanitorAnnotation] = ClearErrorFrequency - } -} - -// +kubebuilder:rbac:groups=upgrademgr.keikoproj.io,resources=rollingupgrades,verbs=get;list;watch;create;update;patch;delete -// +kubebuilder:rbac:groups=upgrademgr.keikoproj.io,resources=rollingupgrades/status,verbs=get;update;patch -// +kubebuilder:rbac:groups=core,resources=nodes,verbs=get;list;patch -// +kubebuilder:rbac:groups=core,resources=pods,verbs=list -// +kubebuilder:rbac:groups=core,resources=events,verbs=create -// +kubebuilder:rbac:groups=core,resources=pods/eviction,verbs=create -// +kubebuilder:rbac:groups=extensions;apps,resources=daemonsets;replicasets;statefulsets,verbs=get -// +kubebuilder:rbac:groups=batch,resources=jobs,verbs=get - -// Reconcile reads that state of the cluster for a RollingUpgrade object and makes changes based on the state read -// and the details in the RollingUpgrade.Spec -func (r *RollingUpgradeReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) { - ctx := context.Background() - - // Fetch the RollingUpgrade instance - ruObj := &upgrademgrv1alpha1.RollingUpgrade{} - err := r.Get(ctx, req.NamespacedName, ruObj) - if err != nil { - if k8serrors.IsNotFound(err) { - // Object not found, return. Created objects are automatically garbage collected. - // For additional cleanup logic use finalizers. - r.admissionMap.Delete(req.NamespacedName) - r.info(ruObj, "Deleted object from map", "name", req.NamespacedName) - return ctrl.Result{}, nil - } - // Error reading the object - requeue the request. - return ctrl.Result{}, err - } - - // If the resource is being deleted, remove it from the admissionMap - if !ruObj.DeletionTimestamp.IsZero() { - r.info(ruObj, "Object is being deleted. No more processing") - r.admissionMap.Delete(ruObj.NamespacedName()) - r.ruObjNameToASG.Delete(ruObj.NamespacedName()) - r.info(ruObj, "Deleted object from admission map") - return reconcile.Result{}, nil - } - - // set the state of instances in the ASG to new in the cluster store - _, exists := r.inProcessASGs.Load(ruObj.Spec.AsgName) - if exists { - r.info(ruObj, "ASG "+ruObj.Spec.AsgName+" is being processed. Requeuing") - return reconcile.Result{Requeue: true, RequeueAfter: time.Duration(60) * time.Second}, nil - } - - // Setting default values for the Strategy in rollup object - r.setDefaultsForRollingUpdateStrategy(ruObj) - r.info(ruObj, "Default strategy settings applied.", "updateStrategy", ruObj.Spec.Strategy) - - err = r.validateRollingUpgradeObj(ruObj) - if err != nil { - r.error(ruObj, err, "Validation failed") - return reconcile.Result{}, err - } - - result, ok := r.admissionMap.Load(ruObj.NamespacedName()) - if ok { - if result == "processing" { - r.info(ruObj, "Found obj in map:", "name", ruObj.NamespacedName()) - r.info(ruObj, "Object already being processed", "name", ruObj.NamespacedName()) - } else { - r.info(ruObj, "Sync map with invalid entry for ", "name", ruObj.NamespacedName()) - } - } else { - r.info(ruObj, "Adding obj to map: ", "name", ruObj.NamespacedName()) - r.admissionMap.Store(ruObj.NamespacedName(), "processing") - go r.Process(&ctx, ruObj) - } - - return ctrl.Result{}, nil -} - -// SetupWithManager creates a new manager. -func (r *RollingUpgradeReconciler) SetupWithManager(mgr ctrl.Manager) error { - r.generatedClient = kubernetes.NewForConfigOrDie(mgr.GetConfig()) - return ctrl.NewControllerManagedBy(mgr). - For(&upgrademgrv1alpha1.RollingUpgrade{}). - WithOptions(controller.Options{MaxConcurrentReconciles: r.maxParallel}). - Complete(r) -} - -func (r *RollingUpgradeReconciler) setStateTag(ruObj *upgrademgrv1alpha1.RollingUpgrade, instanceID string, state string) error { - r.info(ruObj, "setting instance state", "instanceID", instanceID, "instanceState", state) - return tagEC2instance(instanceID, EC2StateTagKey, state, r.EC2Client) -} - -// validateRollingUpgradeObj validates rollup object for the type, maxUnavailable and drainTimeout -func (r *RollingUpgradeReconciler) validateRollingUpgradeObj(ruObj *upgrademgrv1alpha1.RollingUpgrade) error { - strategy := ruObj.Spec.Strategy - - var nilStrategy = upgrademgrv1alpha1.UpdateStrategy{} - if strategy == nilStrategy { - return nil - } - // validating the maxUnavailable value - if strategy.MaxUnavailable.Type == intstr.Int { - if strategy.MaxUnavailable.IntVal <= 0 { - err := fmt.Errorf("%s: Invalid value for maxUnavailable - %d", - ruObj.Name, strategy.MaxUnavailable.IntVal) - r.error(ruObj, err, "Invalid value for maxUnavailable", "value", strategy.MaxUnavailable.IntVal) - return err - } - } else if strategy.MaxUnavailable.Type == intstr.String { - strVal := strategy.MaxUnavailable.StrVal - intValue, _ := strconv.Atoi(strings.Trim(strVal, "%")) - if intValue <= 0 || intValue > 100 { - err := fmt.Errorf("%s: Invalid value for maxUnavailable - %s", - ruObj.Name, strategy.MaxUnavailable.StrVal) - r.error(ruObj, err, "Invalid value for maxUnavailable", "value", strategy.MaxUnavailable.StrVal) - return err - } - } - - // validating the strategy type - if strategy.Type != upgrademgrv1alpha1.RandomUpdateStrategy && - strategy.Type != upgrademgrv1alpha1.UniformAcrossAzUpdateStrategy { - err := fmt.Errorf("%s: Invalid value for strategy type - %s", ruObj.NamespacedName(), strategy.Type) - r.error(ruObj, err, "Invalid value for strategy type", "value", strategy.Type) - return err - } - return nil -} - -// setDefaultsForRollingUpdateStrategy sets the default values for type, maxUnavailable and drainTimeout -func (r *RollingUpgradeReconciler) setDefaultsForRollingUpdateStrategy(ruObj *upgrademgrv1alpha1.RollingUpgrade) { - if ruObj.Spec.Strategy.Type == "" { - ruObj.Spec.Strategy.Type = upgrademgrv1alpha1.RandomUpdateStrategy - } - if ruObj.Spec.Strategy.Mode == "" { - // default to lazy mode - ruObj.Spec.Strategy.Mode = upgrademgrv1alpha1.UpdateStrategyModeLazy - } - // Set default max unavailable to 1. - if ruObj.Spec.Strategy.MaxUnavailable.Type == intstr.Int && ruObj.Spec.Strategy.MaxUnavailable.IntVal == 0 { - ruObj.Spec.Strategy.MaxUnavailable.IntVal = 1 - } - if ruObj.Spec.Strategy.DrainTimeout == 0 { - ruObj.Spec.Strategy.DrainTimeout = -1 - } -} - -type UpdateInstancesError struct { - InstanceUpdateErrors []error -} - -func (error UpdateInstancesError) Error() string { - return fmt.Sprintf("Error updating instances, ErrorCount: %d, Errors: %v", - len(error.InstanceUpdateErrors), error.InstanceUpdateErrors) -} - -func NewUpdateInstancesError(instanceUpdateErrors []error) *UpdateInstancesError { - return &UpdateInstancesError{InstanceUpdateErrors: instanceUpdateErrors} -} - -func (r *RollingUpgradeReconciler) UpdateInstances(ctx *context.Context, - ruObj *upgrademgrv1alpha1.RollingUpgrade, - instances []*autoscaling.Instance, - launchDefinition *launchDefinition) error { - - totalNodes := len(instances) - if totalNodes == 0 { - return nil - } - - ch := make(chan error) - - for _, instance := range instances { - // log it before we start updating the instance - r.createK8sV1Event(ruObj, EventReasonRUInstanceStarted, EventLevelNormal, map[string]string{ - "status": "in-progress", - "asgName": ruObj.Spec.AsgName, - "strategy": string(ruObj.Spec.Strategy.Type), - "msg": fmt.Sprintf("Started Updating Instance %s, in AZ: %s", *instance.InstanceId, *instance.AvailabilityZone), - }) - go r.UpdateInstance(ctx, ruObj, instance, launchDefinition, ch) - } - - // wait for upgrades to complete - nodesProcessed := 0 - var instanceUpdateErrors []error - - for err := range ch { - nodesProcessed++ - switch err { - case nil: - // do nothing - default: - instanceUpdateErrors = append(instanceUpdateErrors, err) - } - // log the event - r.createK8sV1Event(ruObj, EventReasonRUInstanceFinished, EventLevelNormal, map[string]string{ - "status": "in-progress", - "asgName": ruObj.Spec.AsgName, - "strategy": string(ruObj.Spec.Strategy.Type), - "msg": fmt.Sprintf("Finished Updating Instance %d/%d (Errors=%d)", nodesProcessed, totalNodes, len(instanceUpdateErrors)), - }) - // break if we are done with all the nodes - if nodesProcessed == totalNodes { - break - } - } - - if len(instanceUpdateErrors) > 0 { - return NewUpdateInstancesError(instanceUpdateErrors) - } - return nil -} - -func (r *RollingUpgradeReconciler) UpdateInstanceEager( - ruObj *upgrademgrv1alpha1.RollingUpgrade, - nodeName, - targetInstanceID string) error { - - // Set instance to standby - err := r.SetStandby(ruObj, targetInstanceID) - if err != nil { - return err - } - - // Wait for new instance to be created - err = r.WaitForDesiredInstances(ruObj) - if err != nil { - return err - } - - // Wait for in-service nodes to be ready and match desired - err = r.WaitForDesiredNodes(ruObj) - if err != nil { - return err - } - - // Drain and wait for draining node. - return r.DrainTerminate(ruObj, nodeName, targetInstanceID) -} - -func (r *RollingUpgradeReconciler) DrainTerminate( - ruObj *upgrademgrv1alpha1.RollingUpgrade, - nodeName, - targetInstanceID string) error { - - // Drain and wait for draining node. - if nodeName != "" { - err := r.DrainNode(ruObj, nodeName, targetInstanceID, ruObj.Spec.Strategy.DrainTimeout) - if err != nil && !ruObj.Spec.IgnoreDrainFailures { - return err - } - } - - // Terminate instance. - err := r.TerminateNode(ruObj, targetInstanceID, nodeName) - if err != nil { - return err - } - - return nil -} - -// UpdateInstance runs the rolling upgrade on one instance from an autoscaling group -func (r *RollingUpgradeReconciler) UpdateInstance(ctx *context.Context, - ruObj *upgrademgrv1alpha1.RollingUpgrade, - i *autoscaling.Instance, - launchDefinition *launchDefinition, - ch chan error) { - targetInstanceID := aws.StringValue(i.InstanceId) - // If an instance was marked as "in-progress" in ClusterState, it has to be marked - // completed so that it can get considered again in a subsequent rollup CR. - defer r.ClusterState.markUpdateCompleted(targetInstanceID) - - // Check if the rollingupgrade object still exists - _, ok := r.admissionMap.Load(ruObj.NamespacedName()) - if !ok { - r.info(ruObj, "Object either force completed or deleted. Ignoring node update") - ruObj.Status.NodesProcessed = ruObj.Status.NodesProcessed + 1 - ch <- nil - return - } - - // If the running node has the same launchconfig as the asg, - // there is no need to refresh it. - if !r.requiresRefresh(ruObj, i, launchDefinition) { - ruObj.Status.NodesProcessed = ruObj.Status.NodesProcessed + 1 - if err := r.Status().Update(*ctx, ruObj); err != nil { - r.error(ruObj, err, "failed to update status") - } - ch <- nil - return - } - - nodeName := r.getNodeName(i, r.NodeList, ruObj) - - // set the EC2 tag indicating the state to in-progress - err := r.setStateTag(ruObj, targetInstanceID, "in-progress") - if err != nil { - if awsErr, ok := err.(awserr.Error); ok { - if awsErr.Code() == "InvalidInstanceID.NotFound" { - ch <- nil - return - } - } - ch <- err - return - } - - mode := ruObj.Spec.Strategy.Mode.String() - if strings.ToLower(mode) == upgrademgrv1alpha1.UpdateStrategyModeEager.String() { - r.info(ruObj, "starting replacement with eager mode", "mode", mode) - err = r.UpdateInstanceEager(ruObj, nodeName, targetInstanceID) - } else if strings.ToLower(mode) == upgrademgrv1alpha1.UpdateStrategyModeLazy.String() { - r.info(ruObj, "starting replacement with lazy mode", "mode", mode) - err = r.DrainTerminate(ruObj, nodeName, targetInstanceID) - } else { - err = fmt.Errorf("%s: unhandled strategy mode: %s", ruObj.NamespacedName(), mode) - } - - if err != nil { - ch <- err - return - } - - unjoined, err := r.WaitForTermination(ruObj, nodeName, r.generatedClient.CoreV1().Nodes()) - if err != nil { - ch <- err - return - } - - if !unjoined { - r.info(ruObj, "termination waiter completed but node is still joined, will proceed with upgrade", "nodeName", nodeName) - } - - err = r.setStateTag(ruObj, targetInstanceID, "completed") - if err != nil { - r.info(ruObj, "Setting tag on the instance post termination failed.", "nodeName", nodeName) - } - ruObj.Status.NodesProcessed = ruObj.Status.NodesProcessed + 1 - if err := r.Status().Update(*ctx, ruObj); err != nil { - // Check if the err is "StorageError: invalid object". If so, the object was deleted... - if strings.Contains(err.Error(), "StorageError: invalid object") { - r.info(ruObj, "Object mostly deleted") - } else { - r.error(ruObj, err, "failed to update status") - } - } - - ch <- nil -} - -func (r *RollingUpgradeReconciler) getNodeCreationTimestamp(ec2Instance *autoscaling.Instance) (bool, time.Time) { - for _, node := range r.NodeList.Items { - tokens := strings.Split(node.Spec.ProviderID, "/") - instanceID := tokens[len(tokens)-1] - if instanceID == aws.StringValue(ec2Instance.InstanceId) { - return true, node.ObjectMeta.CreationTimestamp.Time - } - } - return false, time.Time{} -} - -func (r *RollingUpgradeReconciler) getTemplateLatestVersion(templateName string) string { - for _, t := range r.LaunchTemplates { - name := aws.StringValue(t.LaunchTemplateName) - if name == templateName { - versionInt := aws.Int64Value(t.LatestVersionNumber) - return strconv.FormatInt(versionInt, 10) - } - } - return "0" -} - -func (r *RollingUpgradeReconciler) requiresRefresh(ruObj *upgrademgrv1alpha1.RollingUpgrade, ec2Instance *autoscaling.Instance, - definition *launchDefinition) bool { - - instanceID := aws.StringValue(ec2Instance.InstanceId) - if ruObj.Spec.ForceRefresh { - if ok, nodeCreationTS := r.getNodeCreationTimestamp(ec2Instance); ok { - if nodeCreationTS.Before(ruObj.CreationTimestamp.Time) { - r.info(ruObj, "rolling upgrade configured for forced refresh") - return true - } - } - - r.info(ruObj, "node", instanceID, "created after rollingupgrade object. Ignoring forceRefresh") - return false - } - if definition.launchConfigurationName != nil { - if *(definition.launchConfigurationName) != aws.StringValue(ec2Instance.LaunchConfigurationName) { - r.info(ruObj, "node", instanceID, "launch configuration name differs") - return true - } - } else if definition.launchTemplate != nil { - instanceLaunchTemplate := ec2Instance.LaunchTemplate - targetLaunchTemplate := definition.launchTemplate - - if instanceLaunchTemplate == nil { - r.info(ruObj, "node", instanceID, "instance switching to launch template") - return true - } - - var ( - instanceTemplateId = aws.StringValue(instanceLaunchTemplate.LaunchTemplateId) - templateId = aws.StringValue(targetLaunchTemplate.LaunchTemplateId) - instanceTemplateName = aws.StringValue(instanceLaunchTemplate.LaunchTemplateName) - templateName = aws.StringValue(targetLaunchTemplate.LaunchTemplateName) - instanceVersion = aws.StringValue(instanceLaunchTemplate.Version) - templateVersion = r.getTemplateLatestVersion(templateName) - ) - - if instanceTemplateId != templateId { - r.info(ruObj, "node", instanceID, "launch template id differs", "instanceTemplateId", instanceTemplateId, "templateId", templateId) - return true - } - if instanceTemplateName != templateName { - r.info(ruObj, "node", instanceID, "launch template name differs", "instanceTemplateName", instanceTemplateName, "templateName", templateName) - return true - } - - if instanceVersion != templateVersion { - r.info(ruObj, "node", instanceID, "launch template version differs", "instanceVersion", instanceVersion, "templateVersion", templateVersion) - return true - } - } - - r.info(ruObj, "node", instanceID, "node refresh not required") - return false -} - -// logger creates logger for rolling upgrade. -func (r *RollingUpgradeReconciler) logger(ruObj *upgrademgrv1alpha1.RollingUpgrade) logr.Logger { - return r.Log.WithValues("rollingupgrade", ruObj.NamespacedName()) -} - -// info logs message with Info level for the specified rolling upgrade. -func (r *RollingUpgradeReconciler) info(ruObj *upgrademgrv1alpha1.RollingUpgrade, msg string, keysAndValues ...interface{}) { - r.logger(ruObj).Info(msg, keysAndValues...) -} - -// error logs message with Error level for the specified rolling upgrade. -func (r *RollingUpgradeReconciler) error(ruObj *upgrademgrv1alpha1.RollingUpgrade, err error, msg string, keysAndValues ...interface{}) { - r.logger(ruObj).Error(err, msg, keysAndValues...) -} diff --git a/controllers/rollingupgrade_controller_test.go b/controllers/rollingupgrade_controller_test.go deleted file mode 100644 index 869e11c9..00000000 --- a/controllers/rollingupgrade_controller_test.go +++ /dev/null @@ -1,2979 +0,0 @@ -package controllers - -import ( - "fmt" - "io/ioutil" - "os" - "path/filepath" - "sync" - "testing" - "time" - - "k8s.io/client-go/kubernetes/scheme" - log2 "sigs.k8s.io/controller-runtime/pkg/log" - - "github.com/keikoproj/aws-sdk-go-cache/cache" - "github.com/keikoproj/upgrade-manager/pkg/log" - - "k8s.io/apimachinery/pkg/util/intstr" - - "gopkg.in/yaml.v2" - - "github.com/aws/aws-sdk-go/aws" - "github.com/aws/aws-sdk-go/aws/awserr" - "github.com/aws/aws-sdk-go/service/autoscaling" - "github.com/aws/aws-sdk-go/service/autoscaling/autoscalingiface" - "github.com/aws/aws-sdk-go/service/ec2" - "github.com/aws/aws-sdk-go/service/ec2/ec2iface" - "github.com/onsi/gomega" - "golang.org/x/net/context" - corev1 "k8s.io/api/core/v1" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - "k8s.io/client-go/kubernetes" - "k8s.io/client-go/kubernetes/fake" - v1 "k8s.io/client-go/kubernetes/typed/core/v1" - "sigs.k8s.io/controller-runtime/pkg/envtest" - "sigs.k8s.io/controller-runtime/pkg/manager" - - upgrademgrv1alpha1 "github.com/keikoproj/upgrade-manager/api/v1alpha1" -) - -func TestMain(m *testing.M) { - testEnv = &envtest.Environment{ - CRDDirectoryPaths: []string{filepath.Join("..", "config", "crd", "bases")}, - } - - cfg, _ = testEnv.Start() - os.Exit(m.Run()) -} - -func TestErrorStatusMarkJanitor(t *testing.T) { - g := gomega.NewGomegaWithT(t) - someAsg := "someAsg" - instance := &upgrademgrv1alpha1.RollingUpgrade{ - ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{AsgName: someAsg}, - } - - mgr, err := buildManager() - g.Expect(err).NotTo(gomega.HaveOccurred()) - rcRollingUpgrade := &RollingUpgradeReconciler{Client: mgr.GetClient(), - generatedClient: kubernetes.NewForConfigOrDie(mgr.GetConfig()), - Log: log2.NullLogger{}, - ClusterState: NewClusterState(), - ScriptRunner: NewScriptRunner(log2.NullLogger{}), - } - - ctx := context.TODO() - err = fmt.Errorf("execution error") - rcRollingUpgrade.inProcessASGs.Store(someAsg, "processing") - rcRollingUpgrade.finishExecution(err, 3, &ctx, instance) - g.Expect(instance.ObjectMeta.Annotations[JanitorAnnotation]).To(gomega.Equal(ClearErrorFrequency)) - _, exists := rcRollingUpgrade.inProcessASGs.Load(someAsg) - g.Expect(exists).To(gomega.BeFalse()) -} - -func TestMarkObjForCleanupCompleted(t *testing.T) { - g := gomega.NewGomegaWithT(t) - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}} - ruObj.Status.CurrentStatus = upgrademgrv1alpha1.StatusComplete - - g.Expect(ruObj.ObjectMeta.Annotations).To(gomega.BeNil()) - MarkObjForCleanup(ruObj) - g.Expect(ruObj.ObjectMeta.Annotations[JanitorAnnotation]).To(gomega.Equal(ClearCompletedFrequency)) -} - -func TestMarkObjForCleanupError(t *testing.T) { - g := gomega.NewGomegaWithT(t) - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}} - ruObj.Status.CurrentStatus = upgrademgrv1alpha1.StatusError - - g.Expect(ruObj.ObjectMeta.Annotations).To(gomega.BeNil()) - MarkObjForCleanup(ruObj) - g.Expect(ruObj.ObjectMeta.Annotations[JanitorAnnotation]).To(gomega.Equal(ClearErrorFrequency)) -} - -func TestMarkObjForCleanupNothingHappens(t *testing.T) { - g := gomega.NewGomegaWithT(t) - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}} - ruObj.Status.CurrentStatus = "some other status" - - g.Expect(ruObj.ObjectMeta.Annotations).To(gomega.BeNil()) - MarkObjForCleanup(ruObj) - g.Expect(ruObj.ObjectMeta.Annotations).To(gomega.BeEmpty()) -} - -func TestPreDrainScriptSuccess(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - instance := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}} - instance.Spec.PreDrain.Script = "echo 'Predrain script ran without error'" - - rcRollingUpgrade := createReconciler() - err := rcRollingUpgrade.preDrainHelper("test-instance-id", "test", instance) - g.Expect(err).To(gomega.BeNil()) -} - -func TestPreDrainScriptError(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - instance := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}} - instance.Spec.PreDrain.Script = "exit 1" - - rcRollingUpgrade := createReconciler() - err := rcRollingUpgrade.preDrainHelper("test-instance-id", "test", instance) - g.Expect(err.Error()).To(gomega.ContainSubstring("Failed to run preDrain script")) -} - -func TestPostDrainHelperPostDrainScriptSuccess(t *testing.T) { - g := gomega.NewGomegaWithT(t) - mockNode := "some-node-name" - mockKubeCtlCall := "echo" - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}} - ruObj.Spec.PostDrain.Script = "echo Hello, postDrainScript!" - - rcRollingUpgrade := createReconciler() - rcRollingUpgrade.ScriptRunner.KubectlCall = mockKubeCtlCall - err := rcRollingUpgrade.postDrainHelper("test-instance-id", mockNode, ruObj) - - g.Expect(err).To(gomega.BeNil()) -} - -func TestPostDrainHelperPostDrainScriptError(t *testing.T) { - g := gomega.NewGomegaWithT(t) - mockNode := "some-node-name" - mockKubeCtlCall := "k() { echo $@ >> cmdlog.txt ; }; k" - os.Remove("cmdlog.txt") - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}} - ruObj.Spec.PostDrain.Script = "exit 1" - - rcRollingUpgrade := createReconciler() - rcRollingUpgrade.ScriptRunner.KubectlCall = mockKubeCtlCall - err := rcRollingUpgrade.postDrainHelper("test-instance-id", mockNode, ruObj) - - g.Expect(err).To(gomega.Not(gomega.BeNil())) - - // assert node was uncordoned - cmdlog, _ := ioutil.ReadFile("cmdlog.txt") - g.Expect(string(cmdlog)).To(gomega.Equal(fmt.Sprintf("uncordon %s\n", mockNode))) - os.Remove("cmdlog.txt") -} - -func TestPostDrainHelperPostDrainScriptErrorWithIgnoreDrainFailures(t *testing.T) { - g := gomega.NewGomegaWithT(t) - mockNode := "some-node-name" - mockKubeCtlCall := "k() { echo $@ >> cmdlog.txt ; }; k" - os.Remove("cmdlog.txt") - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ - ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{IgnoreDrainFailures: true}} - ruObj.Spec.PostDrain.Script = "exit 1" - - rcRollingUpgrade := createReconciler() - rcRollingUpgrade.ScriptRunner.KubectlCall = mockKubeCtlCall - err := rcRollingUpgrade.postDrainHelper("test-instance-id", mockNode, ruObj) - - g.Expect(err).To(gomega.Not(gomega.BeNil())) - - // assert node was not uncordoned - cmdlog, _ := ioutil.ReadFile("cmdlog.txt") - g.Expect(string(cmdlog)).To(gomega.Equal("")) - os.Remove("cmdlog.txt") -} - -func TestPostDrainHelperPostDrainWaitScriptSuccess(t *testing.T) { - g := gomega.NewGomegaWithT(t) - mockNode := "some-node-name" - mockKubeCtlCall := "echo" - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}} - ruObj.Spec.PostDrain.PostWaitScript = "echo Hello, postDrainWaitScript!" - ruObj.Spec.PostDrainDelaySeconds = 0 - - rcRollingUpgrade := createReconciler() - rcRollingUpgrade.ScriptRunner.KubectlCall = mockKubeCtlCall - err := rcRollingUpgrade.postDrainHelper("test-instance-id", mockNode, ruObj) - - g.Expect(err).To(gomega.BeNil()) -} - -func TestPostDrainHelperPostDrainWaitScriptError(t *testing.T) { - g := gomega.NewGomegaWithT(t) - mockNode := "some-node-name" - mockKubeCtlCall := "k() { echo $@ >> cmdlog.txt ; }; k" - os.Remove("cmdlog.txt") - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}} - ruObj.Spec.PostDrain.PostWaitScript = "exit 1" - ruObj.Spec.PostDrainDelaySeconds = 0 - - rcRollingUpgrade := createReconciler() - rcRollingUpgrade.ScriptRunner.KubectlCall = mockKubeCtlCall - err := rcRollingUpgrade.postDrainHelper("test-instance-id", mockNode, ruObj) - - g.Expect(err).To(gomega.Not(gomega.BeNil())) - - // assert node was uncordoned - cmdlog, _ := ioutil.ReadFile("cmdlog.txt") - g.Expect(string(cmdlog)).To(gomega.Equal(fmt.Sprintf("uncordon %s\n", mockNode))) - os.Remove("cmdlog.txt") -} - -func TestPostDrainHelperPostDrainWaitScriptErrorWithIgnoreDrainFailures(t *testing.T) { - g := gomega.NewGomegaWithT(t) - mockNode := "some-node-name" - mockKubeCtlCall := "k() { echo $@ >> cmdlog.txt ; }; k" - os.Remove("cmdlog.txt") - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ - ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{IgnoreDrainFailures: true}} - ruObj.Spec.PostDrain.PostWaitScript = "exit 1" - ruObj.Spec.PostDrainDelaySeconds = 0 - - rcRollingUpgrade := createReconciler() - rcRollingUpgrade.ScriptRunner.KubectlCall = mockKubeCtlCall - err := rcRollingUpgrade.postDrainHelper("test-instance-id", mockNode, ruObj) - - g.Expect(err).To(gomega.Not(gomega.BeNil())) - - // assert node was not uncordoned - cmdlog, _ := ioutil.ReadFile("cmdlog.txt") - g.Expect(string(cmdlog)).To(gomega.Equal("")) - os.Remove("cmdlog.txt") -} - -func TestDrainNodeSuccess(t *testing.T) { - g := gomega.NewGomegaWithT(t) - mockNode := "some-node-name" - mockKubeCtlCall := "echo" - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}} - rcRollingUpgrade := createReconciler() - rcRollingUpgrade.ScriptRunner.KubectlCall = mockKubeCtlCall - - err := rcRollingUpgrade.DrainNode(ruObj, mockNode, "test-id", ruObj.Spec.Strategy.DrainTimeout) - g.Expect(err).To(gomega.BeNil()) -} - -func TestDrainNodePreDrainError(t *testing.T) { - g := gomega.NewGomegaWithT(t) - mockNode := "some-node-name" - mockKubeCtlCall := "echo" - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}} - ruObj.Spec.PreDrain.Script = "exit 1" - rcRollingUpgrade := createReconciler() - rcRollingUpgrade.ScriptRunner.KubectlCall = mockKubeCtlCall - - err := rcRollingUpgrade.DrainNode(ruObj, mockNode, "test-id", ruObj.Spec.Strategy.DrainTimeout) - g.Expect(err).To(gomega.Not(gomega.BeNil())) -} - -func TestDrainNodePostDrainScriptError(t *testing.T) { - g := gomega.NewGomegaWithT(t) - mockNode := "some-node-name" - mockKubeCtlCall := "echo" - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}} - ruObj.Spec.PostDrain.Script = "exit 1" - rcRollingUpgrade := createReconciler() - rcRollingUpgrade.ScriptRunner.KubectlCall = mockKubeCtlCall - - err := rcRollingUpgrade.DrainNode(ruObj, mockNode, "test-id", ruObj.Spec.Strategy.DrainTimeout) - g.Expect(err).To(gomega.Not(gomega.BeNil())) -} - -func TestDrainNodePostDrainWaitScriptError(t *testing.T) { - g := gomega.NewGomegaWithT(t) - mockNode := "some-node-name" - mockKubeCtlCall := "echo" - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}} - ruObj.Spec.PostDrain.PostWaitScript = "exit 1" - rcRollingUpgrade := createReconciler() - rcRollingUpgrade.ScriptRunner.KubectlCall = mockKubeCtlCall - - err := rcRollingUpgrade.DrainNode(ruObj, mockNode, "test-id", ruObj.Spec.Strategy.DrainTimeout) - g.Expect(err).To(gomega.Not(gomega.BeNil())) -} - -func TestDrainNodePostDrainFailureToDrainNotFound(t *testing.T) { - g := gomega.NewGomegaWithT(t) - mockNode := "some-node-name" - - // Force quit from the rest of the command - mockKubeCtlCall := "echo 'Error from server (NotFound)'; exit 1;" - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}} - rcRollingUpgrade := createReconciler() - rcRollingUpgrade.ScriptRunner.KubectlCall = mockKubeCtlCall - - err := rcRollingUpgrade.DrainNode(ruObj, mockNode, "test-id", ruObj.Spec.Strategy.DrainTimeout) - g.Expect(err).To(gomega.BeNil()) -} - -func TestDrainNodePostDrainFailureToDrain(t *testing.T) { - g := gomega.NewGomegaWithT(t) - mockNode := "some-node-name" - - // Force quit from the rest of the command - mockKubeCtlCall := "exit 1;" - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ - ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{ - Strategy: upgrademgrv1alpha1.UpdateStrategy{DrainTimeout: -1}, - }, - } - rcRollingUpgrade := createReconciler() - rcRollingUpgrade.ScriptRunner.KubectlCall = mockKubeCtlCall - - err := rcRollingUpgrade.DrainNode(ruObj, mockNode, "test-id", ruObj.Spec.Strategy.DrainTimeout) - g.Expect(err).To(gomega.Not(gomega.BeNil())) -} - -func createReconciler() *RollingUpgradeReconciler { - return &RollingUpgradeReconciler{ - ClusterState: NewClusterState(), - Log: log2.NullLogger{}, - ScriptRunner: NewScriptRunner(log2.NullLogger{}), - } -} - -type MockEC2 struct { - ec2iface.EC2API - awsErr awserr.Error - reservations []*ec2.Reservation -} - -type MockAutoscalingGroup struct { - autoscalingiface.AutoScalingAPI - errorFlag bool - awsErr awserr.Error - errorInstanceId string - autoScalingGroups []*autoscaling.Group -} - -func (m MockEC2) CreateTags(_ *ec2.CreateTagsInput) (*ec2.CreateTagsOutput, error) { - if m.awsErr != nil { - return nil, m.awsErr - } - return &ec2.CreateTagsOutput{}, nil -} - -func (m MockEC2) DescribeInstances(_ *ec2.DescribeInstancesInput) (*ec2.DescribeInstancesOutput, error) { - return &ec2.DescribeInstancesOutput{Reservations: m.reservations}, nil -} - -func (m MockEC2) DescribeInstancesPages(input *ec2.DescribeInstancesInput, callback func(*ec2.DescribeInstancesOutput, bool) bool) error { - page, err := m.DescribeInstances(input) - if err != nil { - return err - } - callback(page, false) - return nil -} - -func (mockAutoscalingGroup MockAutoscalingGroup) EnterStandby(_ *autoscaling.EnterStandbyInput) (*autoscaling.EnterStandbyOutput, error) { - output := &autoscaling.EnterStandbyOutput{} - return output, nil -} - -func (mockAutoscalingGroup MockAutoscalingGroup) DescribeAutoScalingGroups(input *autoscaling.DescribeAutoScalingGroupsInput) (*autoscaling.DescribeAutoScalingGroupsOutput, error) { - var err error - output := autoscaling.DescribeAutoScalingGroupsOutput{ - AutoScalingGroups: []*autoscaling.Group{}, - } - //To support parallel ASG tracking. - asgA, asgB := "asg-a", "asg-b" - - if mockAutoscalingGroup.errorFlag { - err = mockAutoscalingGroup.awsErr - } - switch *input.AutoScalingGroupNames[0] { - case asgA: - output.AutoScalingGroups = []*autoscaling.Group{ - {AutoScalingGroupName: &asgA}, - } - case asgB: - output.AutoScalingGroups = []*autoscaling.Group{ - {AutoScalingGroupName: &asgB}, - } - default: - output.AutoScalingGroups = mockAutoscalingGroup.autoScalingGroups - } - return &output, err -} - -func (mockAutoscalingGroup MockAutoscalingGroup) TerminateInstanceInAutoScalingGroup(input *autoscaling.TerminateInstanceInAutoScalingGroupInput) (*autoscaling.TerminateInstanceInAutoScalingGroupOutput, error) { - output := &autoscaling.TerminateInstanceInAutoScalingGroupOutput{} - if mockAutoscalingGroup.errorFlag { - if mockAutoscalingGroup.awsErr != nil { - if len(mockAutoscalingGroup.errorInstanceId) <= 0 || - mockAutoscalingGroup.errorInstanceId == *input.InstanceId { - return output, mockAutoscalingGroup.awsErr - } - } - } - asgChange := autoscaling.Activity{ActivityId: aws.String("xxx"), AutoScalingGroupName: aws.String("sss"), Cause: aws.String("xxx"), StartTime: aws.Time(time.Now()), StatusCode: aws.String("200"), StatusMessage: aws.String("success")} - output.Activity = &asgChange - return output, nil -} - -func TestGetInProgressInstances(t *testing.T) { - g := gomega.NewGomegaWithT(t) - mockInstances := []*autoscaling.Instance{ - { - InstanceId: aws.String("i-0123foo"), - }, - { - InstanceId: aws.String("i-0123bar"), - }, - } - expectedInstance := &autoscaling.Instance{ - InstanceId: aws.String("i-0123foo"), - } - mockReservations := []*ec2.Reservation{ - { - Instances: []*ec2.Instance{ - { - InstanceId: aws.String("i-0123foo"), - }, - }, - }, - } - reconciler := &RollingUpgradeReconciler{ - ClusterState: NewClusterState(), - EC2Client: MockEC2{reservations: mockReservations}, - ASGClient: MockAutoscalingGroup{ - errorFlag: false, - awsErr: nil, - }, - ScriptRunner: NewScriptRunner(log2.NullLogger{}), - } - inProgressInstances, err := reconciler.getInProgressInstances(mockInstances) - g.Expect(err).To(gomega.BeNil()) - g.Expect(inProgressInstances).To(gomega.ContainElement(expectedInstance)) - g.Expect(inProgressInstances).To(gomega.HaveLen(1)) -} - -func TestTerminateNodeSuccess(t *testing.T) { - g := gomega.NewGomegaWithT(t) - mockNode := "some-node-id" - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}} - rcRollingUpgrade := &RollingUpgradeReconciler{ - ClusterState: NewClusterState(), - Log: log2.NullLogger{}, - EC2Client: MockEC2{}, - ASGClient: MockAutoscalingGroup{ - errorFlag: false, - awsErr: nil, - }, - ScriptRunner: NewScriptRunner(log2.NullLogger{}), - } - - err := rcRollingUpgrade.TerminateNode(ruObj, mockNode, "") - g.Expect(err).To(gomega.BeNil()) -} - -func TestTerminateNodeErrorNotFound(t *testing.T) { - g := gomega.NewGomegaWithT(t) - mockNode := "some-node-id" - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}} - mockAutoscalingGroup := MockAutoscalingGroup{errorFlag: true, awsErr: awserr.New("InvalidInstanceID.NotFound", - "ValidationError: Instance Id not found - No managed instance found for instance ID i-0bba", - nil)} - rcRollingUpgrade := &RollingUpgradeReconciler{ - ClusterState: NewClusterState(), - Log: log2.NullLogger{}, - ASGClient: mockAutoscalingGroup, - EC2Client: MockEC2{}, - ScriptRunner: NewScriptRunner(log2.NullLogger{}), - } - - err := rcRollingUpgrade.TerminateNode(ruObj, mockNode, "") - g.Expect(err).To(gomega.BeNil()) -} - -func init() { - WaiterMaxDelay = time.Second * 2 - WaiterMinDelay = time.Second * 1 - WaiterMaxAttempts = uint32(2) -} - -func TestTerminateNodeErrorScalingActivityInProgressWithRetry(t *testing.T) { - g := gomega.NewGomegaWithT(t) - mockNode := "some-node-id" - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}} - mockAutoscalingGroup := MockAutoscalingGroup{errorFlag: true, awsErr: awserr.New(autoscaling.ErrCodeScalingActivityInProgressFault, - "Scaling activities in progress", - nil)} - rcRollingUpgrade := &RollingUpgradeReconciler{ - ClusterState: NewClusterState(), - Log: log2.NullLogger{}, - ASGClient: mockAutoscalingGroup, - EC2Client: MockEC2{}, - ScriptRunner: NewScriptRunner(log2.NullLogger{}), - } - go func() { - time.Sleep(WaiterMaxDelay) - rcRollingUpgrade.ASGClient = MockAutoscalingGroup{ - errorFlag: false, - awsErr: nil, - } - }() - err := rcRollingUpgrade.TerminateNode(ruObj, mockNode, "") - g.Expect(err).To(gomega.BeNil()) -} - -func TestTerminateNodeErrorScalingActivityInProgress(t *testing.T) { - g := gomega.NewGomegaWithT(t) - mockNode := "some-node-id" - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}} - mockAutoscalingGroup := MockAutoscalingGroup{errorFlag: true, awsErr: awserr.New(autoscaling.ErrCodeScalingActivityInProgressFault, - "Scaling activities in progress", - nil)} - rcRollingUpgrade := &RollingUpgradeReconciler{ - ClusterState: NewClusterState(), - Log: log2.NullLogger{}, - ASGClient: mockAutoscalingGroup, - EC2Client: MockEC2{}, - ScriptRunner: NewScriptRunner(log2.NullLogger{}), - } - err := rcRollingUpgrade.TerminateNode(ruObj, mockNode, "") - g.Expect(err.Error()).To(gomega.ContainSubstring("no more retries left")) -} - -func TestTerminateNodeErrorResourceContention(t *testing.T) { - g := gomega.NewGomegaWithT(t) - mockNode := "some-node-id" - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}} - mockAutoscalingGroup := MockAutoscalingGroup{errorFlag: true, awsErr: awserr.New(autoscaling.ErrCodeResourceContentionFault, - "Have a pending update on resource", - nil)} - rcRollingUpgrade := &RollingUpgradeReconciler{ - ClusterState: NewClusterState(), - Log: log2.NullLogger{}, - ASGClient: mockAutoscalingGroup, - EC2Client: MockEC2{}, - ScriptRunner: NewScriptRunner(log2.NullLogger{}), - } - - err := rcRollingUpgrade.TerminateNode(ruObj, mockNode, "") - g.Expect(err.Error()).To(gomega.ContainSubstring("no more retries left")) -} - -func TestTerminateNodeErrorOtherError(t *testing.T) { - g := gomega.NewGomegaWithT(t) - mockNode := "some-node-id" - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}} - mockAutoscalingGroup := MockAutoscalingGroup{errorFlag: true, awsErr: awserr.New("some-other-aws-error", - "some message", - fmt.Errorf("some error"))} - - rcRollingUpgrade := &RollingUpgradeReconciler{ - ClusterState: NewClusterState(), - Log: log2.NullLogger{}, - ASGClient: mockAutoscalingGroup, - EC2Client: MockEC2{}, - ScriptRunner: NewScriptRunner(log2.NullLogger{}), - } - err := rcRollingUpgrade.TerminateNode(ruObj, mockNode, "") - g.Expect(err.Error()).To(gomega.ContainSubstring("some error")) -} - -func TestTerminateNodePostTerminateScriptSuccess(t *testing.T) { - g := gomega.NewGomegaWithT(t) - mockNode := "some-node-id" - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}} - ruObj.Spec.PostTerminate.Script = "echo hello!" - mockAutoscalingGroup := MockAutoscalingGroup{errorFlag: false, awsErr: nil} - rcRollingUpgrade := &RollingUpgradeReconciler{ - ClusterState: NewClusterState(), - Log: log2.NullLogger{}, - ASGClient: mockAutoscalingGroup, - EC2Client: MockEC2{}, - ScriptRunner: NewScriptRunner(log2.NullLogger{}), - } - err := rcRollingUpgrade.TerminateNode(ruObj, mockNode, "") - g.Expect(err).To(gomega.BeNil()) -} - -func TestTerminateNodePostTerminateScriptErrorNotFoundFromServer(t *testing.T) { - g := gomega.NewGomegaWithT(t) - mockNode := "some-node-id" - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}} - ruObj.Spec.PostTerminate.Script = "echo 'Error from server (NotFound)'; exit 1" - mockAutoscalingGroup := MockAutoscalingGroup{errorFlag: false, awsErr: nil} - rcRollingUpgrade := &RollingUpgradeReconciler{ - ClusterState: NewClusterState(), - Log: log2.NullLogger{}, - ASGClient: mockAutoscalingGroup, - EC2Client: MockEC2{}, - ScriptRunner: NewScriptRunner(log2.NullLogger{}), - } - err := rcRollingUpgrade.TerminateNode(ruObj, mockNode, "") - g.Expect(err).To(gomega.BeNil()) -} - -func TestTerminateNodePostTerminateScriptErrorOtherError(t *testing.T) { - g := gomega.NewGomegaWithT(t) - mockNode := "some-node-id" - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}} - ruObj.Spec.PostTerminate.Script = "exit 1" - mockAutoscalingGroup := MockAutoscalingGroup{errorFlag: false, awsErr: nil} - rcRollingUpgrade := &RollingUpgradeReconciler{ - ClusterState: NewClusterState(), - Log: log2.NullLogger{}, - ASGClient: mockAutoscalingGroup, - EC2Client: MockEC2{}, - ScriptRunner: NewScriptRunner(log2.NullLogger{}), - } - err := rcRollingUpgrade.TerminateNode(ruObj, mockNode, "") - g.Expect(err).To(gomega.Not(gomega.BeNil())) - g.Expect(err.Error()).To(gomega.ContainSubstring("Failed to run postTerminate script: ")) -} - -func TestLoadEnvironmentVariables(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - r := &ScriptRunner{} - - mockID := "fake-id-foo" - mockName := "instance-name-foo" - - env := r.buildEnv(&upgrademgrv1alpha1.RollingUpgrade{ - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{ - AsgName: "asg-foo", - }, - }, mockID, mockName) - g.Expect(env).To(gomega.HaveLen(3)) - -} - -func TestGetNodeNameFoundNode(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - mockInstanceID := "123456" - autoscalingInstance := autoscaling.Instance{InstanceId: &mockInstanceID} - - fooNode1 := corev1.Node{ObjectMeta: metav1.ObjectMeta{Name: "fooNode1"}, - Spec: corev1.NodeSpec{ProviderID: "foo-bar/9213851"}} - fooNode2 := corev1.Node{ObjectMeta: metav1.ObjectMeta{Name: "fooNode2"}, - Spec: corev1.NodeSpec{ProviderID: "foo-bar/1234501"}} - correctNode := corev1.Node{ObjectMeta: metav1.ObjectMeta{Name: "correctNode"}, - Spec: corev1.NodeSpec{ProviderID: "fake-separator/" + mockInstanceID}} - - nodeList := corev1.NodeList{Items: []corev1.Node{fooNode1, fooNode2, correctNode}} - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}} - rcRollingUpgrade := createReconciler() - name := rcRollingUpgrade.getNodeName(&autoscalingInstance, &nodeList, ruObj) - - g.Expect(name).To(gomega.Equal("correctNode")) -} - -func TestGetNodeNameMissingNode(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - mockInstanceID := "123456" - autoscalingInstance := autoscaling.Instance{InstanceId: &mockInstanceID} - - fooNode1 := corev1.Node{ObjectMeta: metav1.ObjectMeta{Name: "fooNode1"}, - Spec: corev1.NodeSpec{ProviderID: "foo-bar/9213851"}} - fooNode2 := corev1.Node{ObjectMeta: metav1.ObjectMeta{Name: "fooNode2"}, - Spec: corev1.NodeSpec{ProviderID: "foo-bar/1234501"}} - - nodeList := corev1.NodeList{Items: []corev1.Node{fooNode1, fooNode2}} - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}} - rcRollingUpgrade := createReconciler() - name := rcRollingUpgrade.getNodeName(&autoscalingInstance, &nodeList, ruObj) - - g.Expect(name).To(gomega.Equal("")) -} - -func TestGetNodeFromAsgFoundNode(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - mockInstanceID := "123456" - autoscalingInstance := autoscaling.Instance{InstanceId: &mockInstanceID} - - fooNode1 := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "foo-bar/9213851"}} - fooNode2 := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "foo-bar/1234501"}} - - correctNode := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "fake-separator/" + mockInstanceID}} - - nodeList := corev1.NodeList{Items: []corev1.Node{fooNode1, fooNode2, correctNode}} - rcRollingUpgrade := createReconciler() - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}} - node := rcRollingUpgrade.getNodeFromAsg(&autoscalingInstance, &nodeList, ruObj) - - g.Expect(node).To(gomega.Not(gomega.BeNil())) - g.Expect(node).To(gomega.Equal(&correctNode)) -} - -func TestGetNodeFromAsgMissingNode(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - mockInstanceID := "123456" - autoscalingInstance := autoscaling.Instance{InstanceId: &mockInstanceID} - - fooNode1 := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "foo-bar/9213851"}} - fooNode2 := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "foo-bar/1234501"}} - - nodeList := corev1.NodeList{Items: []corev1.Node{fooNode1, fooNode2}} - rcRollingUpgrade := createReconciler() - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}} - node := rcRollingUpgrade.getNodeFromAsg(&autoscalingInstance, &nodeList, ruObj) - - g.Expect(node).To(gomega.BeNil()) -} - -func TestPopulateAsgSuccess(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - correctAsg := "correct-asg" - mockAsg := &autoscaling.Group{ - AutoScalingGroupName: &correctAsg, - } - mockAsgClient := MockAutoscalingGroup{ - autoScalingGroups: []*autoscaling.Group{mockAsg}, - } - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - TypeMeta: metav1.TypeMeta{Kind: "RollingUpgrade", APIVersion: "v1alpha1"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{AsgName: "correct-asg"}} - - rcRollingUpgrade := &RollingUpgradeReconciler{ - Log: log2.NullLogger{}, - ClusterState: NewClusterState(), - ASGClient: mockAsgClient, - EC2Client: MockEC2{}, - ScriptRunner: NewScriptRunner(log2.NullLogger{}), - } - err := rcRollingUpgrade.populateAsg(ruObj) - - g.Expect(err).To(gomega.BeNil()) - - expectedAsg := autoscaling.Group{AutoScalingGroupName: &correctAsg} - - requestedAsg, ok := rcRollingUpgrade.ruObjNameToASG.Load(ruObj.NamespacedName()) - g.Expect(ok).To(gomega.BeTrue()) - g.Expect(requestedAsg.AutoScalingGroupName).To(gomega.Equal(expectedAsg.AutoScalingGroupName)) -} - -func TestPopulateAsgTooMany(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - mockAsg1 := &autoscaling.Group{ - AutoScalingGroupName: aws.String("too-many"), - } - mockAsg2 := &autoscaling.Group{ - AutoScalingGroupName: aws.String("too-many"), - } - mockAsgClient := MockAutoscalingGroup{ - autoScalingGroups: []*autoscaling.Group{mockAsg1, mockAsg2}, - } - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - TypeMeta: metav1.TypeMeta{Kind: "RollingUpgrade", APIVersion: "v1alpha1"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{AsgName: "too-many"}} - - rcRollingUpgrade := &RollingUpgradeReconciler{ - Log: log2.NullLogger{}, - ClusterState: NewClusterState(), - ASGClient: mockAsgClient, - EC2Client: MockEC2{}, - ScriptRunner: NewScriptRunner(log2.NullLogger{}), - } - err := rcRollingUpgrade.populateAsg(ruObj) - - g.Expect(err).To(gomega.Not(gomega.BeNil())) - g.Expect(err.Error()).To(gomega.ContainSubstring("Too many ASGs")) -} - -func TestPopulateAsgNone(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - TypeMeta: metav1.TypeMeta{Kind: "RollingUpgrade", APIVersion: "v1alpha1"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{AsgName: "no-asg-at-all"}} - - rcRollingUpgrade := &RollingUpgradeReconciler{ - Log: log2.NullLogger{}, - ClusterState: NewClusterState(), - ASGClient: &MockAutoscalingGroup{}, - EC2Client: MockEC2{}, - } - err := rcRollingUpgrade.populateAsg(ruObj) - - g.Expect(err).To(gomega.Not(gomega.BeNil())) - g.Expect(err.Error()).To(gomega.ContainSubstring("no ASG found")) -} - -func TestParallelAsgTracking(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - asgAName := "asg-a" - asgBName := "asg-b" - - ruObjA := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo-a", Namespace: "default"}, - TypeMeta: metav1.TypeMeta{Kind: "RollingUpgrade", APIVersion: "v1alpha1"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{AsgName: asgAName}} - ruObjB := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo-b", Namespace: "default"}, - TypeMeta: metav1.TypeMeta{Kind: "RollingUpgrade", APIVersion: "v1alpha1"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{AsgName: asgBName}} - - expectedAsgA := autoscaling.Group{AutoScalingGroupName: &asgAName} - expectedAsgB := autoscaling.Group{AutoScalingGroupName: &asgBName} - - rcRollingUpgrade := &RollingUpgradeReconciler{ - Log: log2.NullLogger{}, - ClusterState: NewClusterState(), - ASGClient: &MockAutoscalingGroup{}, - EC2Client: MockEC2{}, - } - - err := rcRollingUpgrade.populateAsg(ruObjA) - g.Expect(err).To(gomega.BeNil()) - - err = rcRollingUpgrade.populateAsg(ruObjB) - g.Expect(err).To(gomega.BeNil()) - - //This test ensures that we can lookup each of 2 separate ASGs after populating both - requestedAsgA, ok := rcRollingUpgrade.ruObjNameToASG.Load(ruObjA.NamespacedName()) - g.Expect(ok).To(gomega.BeTrue()) - - requestedAsgB, ok := rcRollingUpgrade.ruObjNameToASG.Load(ruObjB.NamespacedName()) - g.Expect(ok).To(gomega.BeTrue()) - - g.Expect(requestedAsgA.AutoScalingGroupName).To(gomega.Equal(expectedAsgA.AutoScalingGroupName)) - g.Expect(requestedAsgB.AutoScalingGroupName).To(gomega.Equal(expectedAsgB.AutoScalingGroupName)) -} - -type MockNodeList struct { - v1.NodeInterface - - // used to return errors if needed - errorFlag bool -} - -func (nodeInterface *MockNodeList) List(options metav1.ListOptions) (*corev1.NodeList, error) { - list := &corev1.NodeList{} - - if nodeInterface.errorFlag { - return list, fmt.Errorf("error flag raised") - } - - node1 := corev1.Node{TypeMeta: metav1.TypeMeta{Kind: "Node", APIVersion: "v1beta1"}, - ObjectMeta: metav1.ObjectMeta{Name: "node1"}} - node2 := corev1.Node{TypeMeta: metav1.TypeMeta{Kind: "Node", APIVersion: "v1beta1"}, - ObjectMeta: metav1.ObjectMeta{Name: "node2"}} - node3 := corev1.Node{TypeMeta: metav1.TypeMeta{Kind: "Node", APIVersion: "v1beta1"}, - ObjectMeta: metav1.ObjectMeta{Name: "node3"}} - - list.Items = []corev1.Node{node1, node2, node3} - return list, nil -} - -func TestPopulateNodeListSuccess(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - TypeMeta: metav1.TypeMeta{Kind: "RollingUpgrade", APIVersion: "v1alpha1"}} - rcRollingUpgrade := createReconciler() - - mockNodeListInterface := &MockNodeList{errorFlag: false} - err := rcRollingUpgrade.populateNodeList(ruObj, mockNodeListInterface) - - g.Expect(err).To(gomega.BeNil()) - g.Expect(rcRollingUpgrade.NodeList.Items[0].Name).To(gomega.Equal("node1")) - g.Expect(rcRollingUpgrade.NodeList.Items[1].Name).To(gomega.Equal("node2")) - g.Expect(rcRollingUpgrade.NodeList.Items[2].Name).To(gomega.Equal("node3")) -} - -func TestPopulateNodeListError(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - TypeMeta: metav1.TypeMeta{Kind: "RollingUpgrade", APIVersion: "v1alpha1"}} - rcRollingUpgrade := createReconciler() - - mockNodeListInterface := &MockNodeList{errorFlag: true} - err := rcRollingUpgrade.populateNodeList(ruObj, mockNodeListInterface) - - g.Expect(err).To(gomega.Not(gomega.BeNil())) - g.Expect(err.Error()).To(gomega.ContainSubstring("Failed to get all nodes in the cluster:")) -} - -func TestFinishExecutionCompleted(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - TypeMeta: metav1.TypeMeta{Kind: "RollingUpgrade", APIVersion: "v1alpha1"}} - startTime := time.Now() - ruObj.Status.StartTime = startTime.Format(time.RFC3339) - - mgr, err := buildManager() - g.Expect(err).NotTo(gomega.HaveOccurred()) - - rcRollingUpgrade := &RollingUpgradeReconciler{Client: mgr.GetClient(), - generatedClient: kubernetes.NewForConfigOrDie(mgr.GetConfig()), - Log: log2.NullLogger{}, - ClusterState: NewClusterState(), - } - ctx := context.TODO() - mockNodesProcessed := 3 - - rcRollingUpgrade.finishExecution(nil, mockNodesProcessed, &ctx, ruObj) - - g.Expect(ruObj.Status.CurrentStatus).To(gomega.Equal(upgrademgrv1alpha1.StatusComplete)) - g.Expect(ruObj.Status.NodesProcessed).To(gomega.Equal(mockNodesProcessed)) - g.Expect(ruObj.Status.EndTime).To(gomega.Not(gomega.BeNil())) - g.Expect(ruObj.Status.TotalProcessingTime).To(gomega.Not(gomega.BeNil())) - g.Expect(ruObj.Status.Conditions).To(gomega.Equal( - []upgrademgrv1alpha1.RollingUpgradeCondition{ - { - Type: upgrademgrv1alpha1.UpgradeComplete, - Status: corev1.ConditionTrue, - }, - }, - )) -} - -func TestFinishExecutionError(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}} - - mgr, err := buildManager() - g.Expect(err).NotTo(gomega.HaveOccurred()) - - rcRollingUpgrade := &RollingUpgradeReconciler{ - Client: mgr.GetClient(), - Log: log2.NullLogger{}, - generatedClient: kubernetes.NewForConfigOrDie(mgr.GetConfig()), - ClusterState: NewClusterState(), - } - startTime := time.Now() - ruObj.Status.StartTime = startTime.Format(time.RFC3339) - ctx := context.TODO() - mockNodesProcessed := 3 - - err = fmt.Errorf("execution error") - rcRollingUpgrade.finishExecution(err, mockNodesProcessed, &ctx, ruObj) - - g.Expect(ruObj.Status.CurrentStatus).To(gomega.Equal(upgrademgrv1alpha1.StatusError)) - g.Expect(ruObj.Status.NodesProcessed).To(gomega.Equal(mockNodesProcessed)) - g.Expect(ruObj.Status.EndTime).To(gomega.Not(gomega.BeNil())) - g.Expect(ruObj.Status.TotalProcessingTime).To(gomega.Not(gomega.BeNil())) - g.Expect(ruObj.Status.Conditions).To(gomega.Equal( - []upgrademgrv1alpha1.RollingUpgradeCondition{ - { - Type: upgrademgrv1alpha1.UpgradeComplete, - Status: corev1.ConditionTrue, - }, - }, - )) -} - -// RunRestack() goes through the entire process without errors -func TestRunRestackSuccessOneNode(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - someAsg := "some-asg" - mockID := "some-id" - someLaunchConfig := "some-launch-config" - diffLaunchConfig := "different-launch-config" - az := "az-1" - - mockInstance := autoscaling.Instance{InstanceId: &mockID, LaunchConfigurationName: &diffLaunchConfig, AvailabilityZone: &az} - mockAsg := autoscaling.Group{AutoScalingGroupName: &someAsg, - LaunchConfigurationName: &someLaunchConfig, - Instances: []*autoscaling.Instance{&mockInstance}} - - strategy := upgrademgrv1alpha1.UpdateStrategy{Mode: upgrademgrv1alpha1.UpdateStrategyModeLazy} - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ - ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{AsgName: someAsg, Strategy: strategy}, - } - - mgr, err := buildManager() - g.Expect(err).NotTo(gomega.HaveOccurred()) - - fooNode1 := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "foo-bar/9213851"}} - fooNode2 := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "foo-bar/1234501"}} - // correctNode has the same mockID as the mockInstance and a node name to be processed - correctNode := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "fake-separator/" + mockID}, - ObjectMeta: metav1.ObjectMeta{Name: "correct-node"}} - - nodeList := corev1.NodeList{Items: []corev1.Node{fooNode1, fooNode2, correctNode}} - rcRollingUpgrade := &RollingUpgradeReconciler{ - Client: mgr.GetClient(), - Log: log2.NullLogger{}, - ASGClient: MockAutoscalingGroup{}, - EC2Client: MockEC2{}, - generatedClient: kubernetes.NewForConfigOrDie(mgr.GetConfig()), - ScriptRunner: NewScriptRunner(log2.NullLogger{}), - NodeList: &nodeList, - ClusterState: NewClusterState(), - CacheConfig: cache.NewConfig(0*time.Second, 0, 0), - } - rcRollingUpgrade.admissionMap.Store(ruObj.NamespacedName(), "processing") - rcRollingUpgrade.ruObjNameToASG.Store(ruObj.NamespacedName(), &mockAsg) - - ctx := context.TODO() - - nodesProcessed, err := rcRollingUpgrade.runRestack(&ctx, ruObj) - g.Expect(nodesProcessed).To(gomega.Equal(1)) - g.Expect(err).To(gomega.BeNil()) - _, exists := rcRollingUpgrade.inProcessASGs.Load(someAsg) - g.Expect(exists).To(gomega.BeTrue()) -} - -func TestRunRestackSuccessMultipleNodes(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - someAsg := "some-asg" - mockID := "some-id" - mockID2 := "some-id-2" - someLaunchConfig := "some-launch-config" - diffLaunchConfig := "different-launch-config" - az := "az-1" - mockInstance := autoscaling.Instance{InstanceId: &mockID, LaunchConfigurationName: &diffLaunchConfig, AvailabilityZone: &az} - mockInstance2 := autoscaling.Instance{InstanceId: &mockID2, LaunchConfigurationName: &diffLaunchConfig, AvailabilityZone: &az} - mockAsg := autoscaling.Group{AutoScalingGroupName: &someAsg, - LaunchConfigurationName: &someLaunchConfig, - Instances: []*autoscaling.Instance{&mockInstance, &mockInstance2}} - - strategy := upgrademgrv1alpha1.UpdateStrategy{Mode: upgrademgrv1alpha1.UpdateStrategyModeLazy} - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{AsgName: someAsg, Strategy: strategy}} - - mgr, err := buildManager() - g.Expect(err).NotTo(gomega.HaveOccurred()) - - fooNode1 := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "foo-bar/9213851"}} - fooNode2 := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "foo-bar/1234501"}} - correctNode := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "fake-separator/" + mockID}, - ObjectMeta: metav1.ObjectMeta{Name: "correct-node"}} - correctNode2 := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "fake-separator/" + mockID2}, - ObjectMeta: metav1.ObjectMeta{Name: "correct-node2"}} - - nodeList := corev1.NodeList{Items: []corev1.Node{fooNode1, fooNode2, correctNode, correctNode2}} - rcRollingUpgrade := &RollingUpgradeReconciler{ - Client: mgr.GetClient(), - Log: log2.NullLogger{}, - ASGClient: MockAutoscalingGroup{}, - EC2Client: MockEC2{}, - generatedClient: kubernetes.NewForConfigOrDie(mgr.GetConfig()), - ScriptRunner: NewScriptRunner(log2.NullLogger{}), - NodeList: &nodeList, - ClusterState: NewClusterState(), - CacheConfig: cache.NewConfig(0*time.Second, 0, 0), - } - rcRollingUpgrade.admissionMap.Store(ruObj.NamespacedName(), "processing") - rcRollingUpgrade.ruObjNameToASG.Store(ruObj.NamespacedName(), &mockAsg) - - ctx := context.TODO() - - nodesProcessed, err := rcRollingUpgrade.runRestack(&ctx, ruObj) - g.Expect(nodesProcessed).To(gomega.Equal(2)) - g.Expect(err).To(gomega.BeNil()) -} - -func TestRunRestackSameLaunchConfig(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - someAsg := "some-asg" - mockID := "some-id" - someLaunchConfig := "some-launch-config" - az := "az-1" - mockInstance := autoscaling.Instance{InstanceId: &mockID, LaunchConfigurationName: &someLaunchConfig, AvailabilityZone: &az} - mockAsg := autoscaling.Group{AutoScalingGroupName: &someAsg, - LaunchConfigurationName: &someLaunchConfig, - Instances: []*autoscaling.Instance{&mockInstance}} - - strategy := upgrademgrv1alpha1.UpdateStrategy{Mode: upgrademgrv1alpha1.UpdateStrategyModeLazy} - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{AsgName: someAsg, Strategy: strategy}} - - mgr, err := buildManager() - g.Expect(err).NotTo(gomega.HaveOccurred()) - - rcRollingUpgrade := &RollingUpgradeReconciler{ - Client: mgr.GetClient(), - Log: log2.NullLogger{}, - ASGClient: MockAutoscalingGroup{}, - EC2Client: MockEC2{}, - generatedClient: kubernetes.NewForConfigOrDie(mgr.GetConfig()), - ClusterState: NewClusterState(), - CacheConfig: cache.NewConfig(0*time.Second, 0, 0), - } - rcRollingUpgrade.admissionMap.Store(ruObj.NamespacedName(), "processing") - rcRollingUpgrade.ruObjNameToASG.Store(ruObj.NamespacedName(), &mockAsg) - - ctx := context.TODO() - - // This execution should not perform drain or termination, but should pass - nodesProcessed, err := rcRollingUpgrade.runRestack(&ctx, ruObj) - g.Expect(nodesProcessed).To(gomega.Equal(1)) - g.Expect(err).To(gomega.BeNil()) -} - -func TestRunRestackRollingUpgradeNotInMap(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - strategy := upgrademgrv1alpha1.UpdateStrategy{Mode: upgrademgrv1alpha1.UpdateStrategyModeLazy} - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{Strategy: strategy}, - } - rcRollingUpgrade := &RollingUpgradeReconciler{ - Log: log2.NullLogger{}, - ClusterState: NewClusterState(), - ASGClient: MockAutoscalingGroup{}, - EC2Client: MockEC2{}, - } - ctx := context.TODO() - - g.Expect(rcRollingUpgrade.ruObjNameToASG.Load(ruObj.NamespacedName())).To(gomega.BeNil()) - int, err := rcRollingUpgrade.runRestack(&ctx, ruObj) - g.Expect(int).To(gomega.Equal(0)) - g.Expect(err).To(gomega.Not(gomega.BeNil())) - g.Expect(err.Error()).To(gomega.HavePrefix("Unable to load ASG with name: foo")) -} - -func TestRunRestackRollingUpgradeNodeNameNotFound(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - someAsg := "some-asg" - mockID := "some-id" - someLaunchConfig := "some-launch-config" - diffLaunchConfig := "different-launch-config" - az := "az-1" - mockInstance := autoscaling.Instance{InstanceId: &mockID, LaunchConfigurationName: &diffLaunchConfig, AvailabilityZone: &az} - mockAsg := autoscaling.Group{AutoScalingGroupName: &someAsg, - LaunchConfigurationName: &someLaunchConfig, - Instances: []*autoscaling.Instance{&mockInstance}} - - strategy := upgrademgrv1alpha1.UpdateStrategy{Mode: upgrademgrv1alpha1.UpdateStrategyModeLazy} - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{AsgName: someAsg, Strategy: strategy}} - - mgr, err := buildManager() - g.Expect(err).NotTo(gomega.HaveOccurred()) - - emptyNodeList := corev1.NodeList{} - rcRollingUpgrade := &RollingUpgradeReconciler{ - Client: mgr.GetClient(), - Log: log2.NullLogger{}, - ASGClient: MockAutoscalingGroup{}, - EC2Client: MockEC2{}, - generatedClient: kubernetes.NewForConfigOrDie(mgr.GetConfig()), - NodeList: &emptyNodeList, - ClusterState: NewClusterState(), - CacheConfig: cache.NewConfig(0*time.Second, 0, 0), - } - rcRollingUpgrade.admissionMap.Store(ruObj.NamespacedName(), "processing") - rcRollingUpgrade.ruObjNameToASG.Store(ruObj.NamespacedName(), &mockAsg) - - ctx := context.TODO() - - // This execution gets past the different launch config check, but fails to be found at the node level - nodesProcessed, err := rcRollingUpgrade.runRestack(&ctx, ruObj) - g.Expect(nodesProcessed).To(gomega.Equal(1)) - g.Expect(err).To(gomega.BeNil()) -} - -func TestRunRestackNoNodeName(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - someAsg := "some-asg" - mockID := "some-id" - someLaunchConfig := "some-launch-config" - diffLaunchConfig := "different-launch-config" - az := "az-1" - mockInstance := autoscaling.Instance{InstanceId: &mockID, LaunchConfigurationName: &diffLaunchConfig, AvailabilityZone: &az} - mockAsg := autoscaling.Group{AutoScalingGroupName: &someAsg, - LaunchConfigurationName: &someLaunchConfig, - Instances: []*autoscaling.Instance{&mockInstance}} - - strategy := upgrademgrv1alpha1.UpdateStrategy{Mode: upgrademgrv1alpha1.UpdateStrategyModeLazy} - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{AsgName: someAsg, Strategy: strategy}} - - mgr, err := buildManager() - g.Expect(err).NotTo(gomega.HaveOccurred()) - - fooNode1 := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "foo-bar/9213851"}} - fooNode2 := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "foo-bar/1234501"}} - // correctNode has the same mockID as the mockInstance - correctNode := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "fake-separator/" + mockID}} - - nodeList := corev1.NodeList{Items: []corev1.Node{fooNode1, fooNode2, correctNode}} - rcRollingUpgrade := &RollingUpgradeReconciler{ - Client: mgr.GetClient(), - Log: log2.NullLogger{}, - ASGClient: MockAutoscalingGroup{}, - EC2Client: MockEC2{}, - generatedClient: kubernetes.NewForConfigOrDie(mgr.GetConfig()), - NodeList: &nodeList, - ClusterState: NewClusterState(), - CacheConfig: cache.NewConfig(0*time.Second, 0, 0), - } - rcRollingUpgrade.admissionMap.Store(ruObj.NamespacedName(), "processing") - rcRollingUpgrade.ruObjNameToASG.Store(ruObj.NamespacedName(), &mockAsg) - - ctx := context.TODO() - - // This execution gets past the different launch config check, but since there is no node name, it is skipped - nodesProcessed, err := rcRollingUpgrade.runRestack(&ctx, ruObj) - g.Expect(nodesProcessed).To(gomega.Equal(1)) - g.Expect(err).To(gomega.BeNil()) -} - -func TestRunRestackDrainNodeFail(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - someAsg := "some-asg" - mockID := "some-id" - someLaunchConfig := "some-launch-config" - diffLaunchConfig := "different-launch-config" - az := "az-1" - mockInstance := autoscaling.Instance{InstanceId: &mockID, - LaunchConfigurationName: &diffLaunchConfig, - AvailabilityZone: &az, - } - mockAsg := autoscaling.Group{AutoScalingGroupName: &someAsg, - LaunchConfigurationName: &someLaunchConfig, - Instances: []*autoscaling.Instance{&mockInstance}, - } - - somePreDrain := upgrademgrv1alpha1.PreDrainSpec{ - Script: "exit 1", - } - - // Will fail upon running the preDrain() script - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{ - AsgName: someAsg, - PreDrain: somePreDrain, - Strategy: upgrademgrv1alpha1.UpdateStrategy{ - Mode: "eager", - }, - }, - } - - mgr, err := buildManager() - g.Expect(err).NotTo(gomega.HaveOccurred()) - - fooNode1 := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "foo-bar/9213851"}} - fooNode2 := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "foo-bar/1234501"}} - // correctNode has the same mockID as the mockInstance and a node name to be processed - correctNode := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "fake-separator/" + mockID}, - ObjectMeta: metav1.ObjectMeta{Name: "correct-node"}} - - nodeList := corev1.NodeList{Items: []corev1.Node{fooNode1, fooNode2, correctNode}} - rcRollingUpgrade := &RollingUpgradeReconciler{ - Client: mgr.GetClient(), - Log: log2.NullLogger{}, - ASGClient: MockAutoscalingGroup{}, - EC2Client: MockEC2{}, - generatedClient: kubernetes.NewForConfigOrDie(mgr.GetConfig()), - ScriptRunner: NewScriptRunner(log2.NullLogger{}), - NodeList: &nodeList, - ClusterState: NewClusterState(), - CacheConfig: cache.NewConfig(0*time.Second, 0, 0), - } - rcRollingUpgrade.admissionMap.Store(ruObj.NamespacedName(), "processing") - rcRollingUpgrade.ruObjNameToASG.Store(ruObj.NamespacedName(), &mockAsg) - - ctx := context.TODO() - - // This execution gets past the different launch config check, but fails to drain the node because of a predrain failing script - nodesProcessed, err := rcRollingUpgrade.runRestack(&ctx, ruObj) - - g.Expect(nodesProcessed).To(gomega.Equal(1)) - g.Expect(err.Error()).To(gomega.HavePrefix("Error updating instances, ErrorCount: 1, Errors: [")) -} - -func TestRunRestackTerminateNodeFail(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - someAsg := "some-asg" - mockID := "some-id" - someLaunchConfig := "some-launch-config" - diffLaunchConfig := "different-launch-config" - az := "az-1" - mockInstance := autoscaling.Instance{InstanceId: &mockID, LaunchConfigurationName: &diffLaunchConfig, AvailabilityZone: &az} - mockAsg := autoscaling.Group{AutoScalingGroupName: &someAsg, - LaunchConfigurationName: &someLaunchConfig, - Instances: []*autoscaling.Instance{&mockInstance}} - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ - ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{ - AsgName: someAsg, - Strategy: upgrademgrv1alpha1.UpdateStrategy{ - Mode: "lazy", - }, - }, - } - // Error flag set, should return error - mockAutoscalingGroup := MockAutoscalingGroup{errorFlag: true, awsErr: awserr.New("some-other-aws-error", - "some message", - fmt.Errorf("some error"))} - - mgr, err := buildManager() - g.Expect(err).NotTo(gomega.HaveOccurred()) - - fooNode1 := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "foo-bar/9213851"}} - fooNode2 := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "foo-bar/1234501"}} - // correctNode has the same mockID as the mockInstance and a node name to be processed - correctNode := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "fake-separator/" + mockID}, - ObjectMeta: metav1.ObjectMeta{Name: "correct-node"}} - - nodeList := corev1.NodeList{Items: []corev1.Node{fooNode1, fooNode2, correctNode}} - rcRollingUpgrade := &RollingUpgradeReconciler{ - Client: mgr.GetClient(), - Log: log2.NullLogger{}, - ASGClient: mockAutoscalingGroup, - EC2Client: MockEC2{}, - generatedClient: kubernetes.NewForConfigOrDie(mgr.GetConfig()), - NodeList: &nodeList, - ClusterState: NewClusterState(), - CacheConfig: cache.NewConfig(0*time.Second, 0, 0), - ScriptRunner: NewScriptRunner(log2.NullLogger{}), - } - rcRollingUpgrade.admissionMap.Store(ruObj.NamespacedName(), "processing") - rcRollingUpgrade.ruObjNameToASG.Store(ruObj.NamespacedName(), &mockAsg) - - ctx := context.TODO() - - // This execution gets past the different launch config check, but fails to terminate node - nodesProcessed, err := rcRollingUpgrade.runRestack(&ctx, ruObj) - g.Expect(nodesProcessed).To(gomega.Equal(1)) - g.Expect(err.Error()).To(gomega.HavePrefix("Error updating instances, ErrorCount: 1, Errors: [")) - g.Expect(err.Error()).To(gomega.ContainSubstring("some error")) -} - -func constructAutoScalingInstance(instanceId string, launchConfigName string, azName string) *autoscaling.Instance { - return &autoscaling.Instance{InstanceId: &instanceId, LaunchConfigurationName: &launchConfigName, AvailabilityZone: &azName} -} - -func TestUniformAcrossAzUpdateSuccessMultipleNodes(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - someAsg := "some-asg" - mockID := "some-id" - someLaunchConfig := "some-launch-config" - diffLaunchConfig := "different-launch-config" - az := "az-1" - az2 := "az-2" - az3 := "az-3" - mockAsg := autoscaling.Group{AutoScalingGroupName: &someAsg, - LaunchConfigurationName: &someLaunchConfig, - Instances: []*autoscaling.Instance{ - constructAutoScalingInstance(mockID+"1"+az, diffLaunchConfig, az), - constructAutoScalingInstance(mockID+"2"+az, diffLaunchConfig, az), - constructAutoScalingInstance(mockID+"1"+az2, diffLaunchConfig, az2), - constructAutoScalingInstance(mockID+"2"+az2, diffLaunchConfig, az2), - constructAutoScalingInstance(mockID+"3"+az2, diffLaunchConfig, az2), - constructAutoScalingInstance(mockID+"1"+az3, diffLaunchConfig, az3), - constructAutoScalingInstance(mockID+"2"+az3, diffLaunchConfig, az3), - constructAutoScalingInstance(mockID+"3"+az3, diffLaunchConfig, az3), - constructAutoScalingInstance(mockID+"4"+az3, diffLaunchConfig, az3), - }, - } - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ - ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{ - AsgName: someAsg, - Strategy: upgrademgrv1alpha1.UpdateStrategy{ - Mode: "lazy", - Type: upgrademgrv1alpha1.UniformAcrossAzUpdateStrategy, - }, - }, - } - - mgr, err := buildManager() - g.Expect(err).NotTo(gomega.HaveOccurred()) - - fooNode1 := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "foo-bar/9213851"}} - fooNode2 := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "foo-bar/1234501"}} - correctNode1az1 := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "fake-separator/" + mockID + "1" + az}, - ObjectMeta: metav1.ObjectMeta{Name: "correct-node"}} - correctNode2az1 := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "fake-separator/" + mockID + "2" + az}, - ObjectMeta: metav1.ObjectMeta{Name: "correct-node"}} - correctNode1az2 := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "fake-separator/" + mockID + "1" + az2}, - ObjectMeta: metav1.ObjectMeta{Name: "correct-node"}} - correctNode2az2 := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "fake-separator/" + mockID + "2" + az2}, - ObjectMeta: metav1.ObjectMeta{Name: "correct-node"}} - correctNode3az2 := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "fake-separator/" + mockID + "3" + az2}, - ObjectMeta: metav1.ObjectMeta{Name: "correct-node"}} - correctNode1az3 := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "fake-separator/" + mockID + "1" + az3}, - ObjectMeta: metav1.ObjectMeta{Name: "correct-node"}} - correctNode2az3 := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "fake-separator/" + mockID + "2" + az3}, - ObjectMeta: metav1.ObjectMeta{Name: "correct-node"}} - correctNode3az3 := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "fake-separator/" + mockID + "3" + az3}, - ObjectMeta: metav1.ObjectMeta{Name: "correct-node"}} - correctNode4az3 := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "fake-separator/" + mockID + "4" + az3}, - ObjectMeta: metav1.ObjectMeta{Name: "correct-node"}} - - nodeList := corev1.NodeList{Items: []corev1.Node{ - fooNode1, fooNode2, - correctNode1az1, correctNode2az1, - correctNode1az2, correctNode2az2, correctNode3az2, - correctNode1az3, correctNode2az3, correctNode3az3, correctNode4az3, - }} - rcRollingUpgrade := &RollingUpgradeReconciler{ - Client: mgr.GetClient(), - Log: log2.NullLogger{}, - ASGClient: MockAutoscalingGroup{}, - EC2Client: MockEC2{}, - generatedClient: kubernetes.NewForConfigOrDie(mgr.GetConfig()), - ScriptRunner: NewScriptRunner(log2.NullLogger{}), - NodeList: &nodeList, - ClusterState: NewClusterState(), - CacheConfig: cache.NewConfig(0*time.Second, 0, 0), - } - rcRollingUpgrade.admissionMap.Store(ruObj.NamespacedName(), "processing") - rcRollingUpgrade.ruObjNameToASG.Store(ruObj.NamespacedName(), &mockAsg) - - ctx := context.TODO() - - nodesProcessed, err := rcRollingUpgrade.runRestack(&ctx, ruObj) - g.Expect(nodesProcessed).To(gomega.Equal(9)) - g.Expect(err).To(gomega.BeNil()) -} - -func TestUpdateInstances(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - someAsg := "some-asg" - mockID := "some-id" - mockID2 := "some-id-2" - someLaunchConfig := "some-launch-config" - diffLaunchConfig := "different-launch-config" - az := "az-1" - mockInstance := autoscaling.Instance{InstanceId: &mockID, LaunchConfigurationName: &diffLaunchConfig, AvailabilityZone: &az} - mockInstance2 := autoscaling.Instance{InstanceId: &mockID2, LaunchConfigurationName: &diffLaunchConfig, AvailabilityZone: &az} - mockAsg := autoscaling.Group{AutoScalingGroupName: &someAsg, - LaunchConfigurationName: &someLaunchConfig, - Instances: []*autoscaling.Instance{&mockInstance, &mockInstance2}} - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ - ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{ - AsgName: someAsg, - Strategy: upgrademgrv1alpha1.UpdateStrategy{ - Mode: "lazy", - }, - }, - } - - mgr, err := buildManager() - g.Expect(err).NotTo(gomega.HaveOccurred()) - - fooNode1 := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "foo-bar/9213851"}} - fooNode2 := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "foo-bar/1234501"}} - correctNode := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "fake-separator/" + mockID}, - ObjectMeta: metav1.ObjectMeta{Name: "correct-node"}} - correctNode2 := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "fake-separator/" + mockID2}, - ObjectMeta: metav1.ObjectMeta{Name: "correct-node2"}} - - nodeList := corev1.NodeList{Items: []corev1.Node{fooNode1, fooNode2, correctNode, correctNode2}} - rcRollingUpgrade := &RollingUpgradeReconciler{ - Client: mgr.GetClient(), - Log: log2.NullLogger{}, - ASGClient: MockAutoscalingGroup{}, - EC2Client: MockEC2{}, - generatedClient: kubernetes.NewForConfigOrDie(mgr.GetConfig()), - NodeList: &nodeList, - ClusterState: NewClusterState(), - CacheConfig: cache.NewConfig(0*time.Second, 0, 0), - ScriptRunner: NewScriptRunner(log2.NullLogger{}), - } - rcRollingUpgrade.admissionMap.Store(ruObj.NamespacedName(), "processing") - rcRollingUpgrade.ruObjNameToASG.Store(ruObj.NamespacedName(), &mockAsg) - - ctx := context.TODO() - - lcName := "A" - rcRollingUpgrade.ScriptRunner.KubectlCall = "exit 0;" - - err = rcRollingUpgrade.UpdateInstances(&ctx, - ruObj, mockAsg.Instances, &launchDefinition{launchConfigurationName: &lcName}) - g.Expect(err).ShouldNot(gomega.HaveOccurred()) -} - -func TestUpdateInstancesError(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - someAsg := "some-asg" - mockID := "some-id" - mockID2 := "some-id-2" - someLaunchConfig := "some-launch-config" - diffLaunchConfig := "different-launch-config" - az := "az-1" - mockInstance := autoscaling.Instance{InstanceId: &mockID, LaunchConfigurationName: &diffLaunchConfig, AvailabilityZone: &az} - mockInstance2 := autoscaling.Instance{InstanceId: &mockID2, LaunchConfigurationName: &diffLaunchConfig, AvailabilityZone: &az} - mockAsg := autoscaling.Group{AutoScalingGroupName: &someAsg, - LaunchConfigurationName: &someLaunchConfig, - Instances: []*autoscaling.Instance{&mockInstance, &mockInstance2}} - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ - ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{ - AsgName: someAsg, - Strategy: upgrademgrv1alpha1.UpdateStrategy{ - Mode: "lazy", - }, - }, - } - - mockAutoScalingGroup := MockAutoscalingGroup{ - errorFlag: true, - awsErr: awserr.New("UnKnownError", - "some message", - nil)} - - mgr, err := buildManager() - g.Expect(err).NotTo(gomega.HaveOccurred()) - - fooNode1 := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "foo-bar/9213851"}} - fooNode2 := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "foo-bar/1234501"}} - correctNode := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "fake-separator/" + mockID}, - ObjectMeta: metav1.ObjectMeta{Name: "correct-node"}} - correctNode2 := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "fake-separator/" + mockID2}, - ObjectMeta: metav1.ObjectMeta{Name: "correct-node2"}} - - nodeList := corev1.NodeList{Items: []corev1.Node{fooNode1, fooNode2, correctNode, correctNode2}} - rcRollingUpgrade := &RollingUpgradeReconciler{ - Client: mgr.GetClient(), - Log: log2.NullLogger{}, - ASGClient: mockAutoScalingGroup, - EC2Client: MockEC2{}, - generatedClient: kubernetes.NewForConfigOrDie(mgr.GetConfig()), - NodeList: &nodeList, - ClusterState: NewClusterState(), - CacheConfig: cache.NewConfig(0*time.Second, 0, 0), - ScriptRunner: NewScriptRunner(log2.NullLogger{}), - } - rcRollingUpgrade.admissionMap.Store(ruObj.NamespacedName(), "processing") - rcRollingUpgrade.ruObjNameToASG.Store(ruObj.NamespacedName(), &mockAsg) - rcRollingUpgrade.ScriptRunner.KubectlCall = "exit 0;" - - ctx := context.TODO() - - lcName := "A" - err = rcRollingUpgrade.UpdateInstances(&ctx, - ruObj, mockAsg.Instances, &launchDefinition{launchConfigurationName: &lcName}) - g.Expect(err).Should(gomega.HaveOccurred()) - g.Expect(err).Should(gomega.BeAssignableToTypeOf(&UpdateInstancesError{})) - if updateInstancesError, ok := err.(*UpdateInstancesError); ok { - g.Expect(len(updateInstancesError.InstanceUpdateErrors)).Should(gomega.Equal(2)) - g.Expect(updateInstancesError.Error()).Should(gomega.ContainSubstring("Error updating instances, ErrorCount: 2")) - } -} - -func TestUpdateInstancesHandlesDeletedInstances(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - someAsg := "some-asg" - mockID := "some-id" - someLaunchConfig := "some-launch-config" - diffLaunchConfig := "different-launch-config" - az := "az-1" - mockInstance := autoscaling.Instance{InstanceId: &mockID, LaunchConfigurationName: &diffLaunchConfig, AvailabilityZone: &az} - mockAsg := autoscaling.Group{AutoScalingGroupName: &someAsg, - LaunchConfigurationName: &someLaunchConfig, - Instances: []*autoscaling.Instance{&mockInstance}} - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ - ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{ - AsgName: someAsg, - Strategy: upgrademgrv1alpha1.UpdateStrategy{ - Mode: "lazy", - }, - }, - } - - mgr, err := buildManager() - g.Expect(err).NotTo(gomega.HaveOccurred()) - - fooNode1 := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "foo-bar/9213851"}} - correctNode := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "fake-separator/" + mockID}, - ObjectMeta: metav1.ObjectMeta{Name: "correct-node"}} - - nodeList := corev1.NodeList{Items: []corev1.Node{fooNode1, correctNode}} - rcRollingUpgrade := &RollingUpgradeReconciler{ - Client: mgr.GetClient(), - Log: log2.NullLogger{}, - ASGClient: MockAutoscalingGroup{}, - EC2Client: MockEC2{ - awsErr: awserr.New("InvalidInstanceID.NotFound", "Instance not found", nil), - }, - generatedClient: kubernetes.NewForConfigOrDie(mgr.GetConfig()), - NodeList: &nodeList, - ClusterState: NewClusterState(), - CacheConfig: cache.NewConfig(0*time.Second, 0, 0), - ScriptRunner: NewScriptRunner(log2.NullLogger{}), - } - rcRollingUpgrade.admissionMap.Store(ruObj.NamespacedName(), "processing") - rcRollingUpgrade.ruObjNameToASG.Store(ruObj.NamespacedName(), &mockAsg) - rcRollingUpgrade.ScriptRunner.KubectlCall = "exit 0;" - - ctx := context.TODO() - - lcName := "A" - err = rcRollingUpgrade.UpdateInstances(&ctx, - ruObj, mockAsg.Instances, &launchDefinition{launchConfigurationName: &lcName}) - g.Expect(err).Should(gomega.BeNil()) -} - -func TestUpdateInstancesPartialError(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - someAsg := "some-asg" - mockID := "some-id" - mockID2 := "some-id-2" - someLaunchConfig := "some-launch-config" - diffLaunchConfig := "different-launch-config" - az := "az-1" - mockInstance := autoscaling.Instance{InstanceId: &mockID, LaunchConfigurationName: &diffLaunchConfig, AvailabilityZone: &az} - mockInstance2 := autoscaling.Instance{InstanceId: &mockID2, LaunchConfigurationName: &diffLaunchConfig, AvailabilityZone: &az} - mockAsg := autoscaling.Group{AutoScalingGroupName: &someAsg, - LaunchConfigurationName: &someLaunchConfig, - Instances: []*autoscaling.Instance{&mockInstance, &mockInstance2}} - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ - ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{ - AsgName: someAsg, - Strategy: upgrademgrv1alpha1.UpdateStrategy{ - Mode: "lazy", - }, - }, - } - - mockAutoScalingGroup := MockAutoscalingGroup{ - errorFlag: true, - awsErr: awserr.New("UnKnownError", - "some message", - nil), - errorInstanceId: mockID2, - } - - mgr, err := buildManager() - g.Expect(err).NotTo(gomega.HaveOccurred()) - - fooNode1 := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "foo-bar/9213851"}} - fooNode2 := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "foo-bar/1234501"}} - correctNode := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "fake-separator/" + mockID}, - ObjectMeta: metav1.ObjectMeta{Name: "correct-node"}} - correctNode2 := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "fake-separator/" + mockID2}, - ObjectMeta: metav1.ObjectMeta{Name: "correct-node2"}} - - nodeList := corev1.NodeList{Items: []corev1.Node{fooNode1, fooNode2, correctNode, correctNode2}} - rcRollingUpgrade := &RollingUpgradeReconciler{ - Client: mgr.GetClient(), - Log: log2.NullLogger{}, - ASGClient: mockAutoScalingGroup, - EC2Client: MockEC2{}, - generatedClient: kubernetes.NewForConfigOrDie(mgr.GetConfig()), - NodeList: &nodeList, - ClusterState: NewClusterState(), - CacheConfig: cache.NewConfig(0*time.Second, 0, 0), - ScriptRunner: NewScriptRunner(log2.NullLogger{}), - } - rcRollingUpgrade.admissionMap.Store(ruObj.NamespacedName(), "processing") - rcRollingUpgrade.ruObjNameToASG.Store(ruObj.NamespacedName(), &mockAsg) - rcRollingUpgrade.ScriptRunner.KubectlCall = "exit 0;" - ctx := context.TODO() - - lcName := "A" - err = rcRollingUpgrade.UpdateInstances(&ctx, - ruObj, mockAsg.Instances, &launchDefinition{launchConfigurationName: &lcName}) - g.Expect(err).Should(gomega.HaveOccurred()) - g.Expect(err).Should(gomega.BeAssignableToTypeOf(&UpdateInstancesError{})) - if updateInstancesError, ok := err.(*UpdateInstancesError); ok { - g.Expect(len(updateInstancesError.InstanceUpdateErrors)).Should(gomega.Equal(1)) - g.Expect(updateInstancesError.Error()).Should(gomega.Equal("Error updating instances, ErrorCount: 1, Errors: [UnKnownError: some message]")) - } -} - -func TestUpdateInstancesWithZeroInstances(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - mgr, err := buildManager() - g.Expect(err).NotTo(gomega.HaveOccurred()) - - rcRollingUpgrade := &RollingUpgradeReconciler{ - Client: mgr.GetClient(), - Log: log2.NullLogger{}, - generatedClient: kubernetes.NewForConfigOrDie(mgr.GetConfig()), - ClusterState: NewClusterState(), - CacheConfig: cache.NewConfig(0*time.Second, 0, 0), - ScriptRunner: NewScriptRunner(log2.NullLogger{}), - } - rcRollingUpgrade.ScriptRunner.KubectlCall = "exit 0;" - - ctx := context.TODO() - - lcName := "A" - err = rcRollingUpgrade.UpdateInstances(&ctx, - nil, nil, &launchDefinition{launchConfigurationName: &lcName}) - g.Expect(err).ShouldNot(gomega.HaveOccurred()) -} - -func TestTestCallKubectlDrainWithoutDrainTimeout(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - mockKubeCtlCall := "sleep 1; echo" - mockNodeName := "some-node-name" - mockAsgName := "some-asg" - rcRollingUpgrade := createReconciler() - rcRollingUpgrade.ScriptRunner.KubectlCall = mockKubeCtlCall - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{AsgName: mockAsgName}} - - errChan := make(chan error) - ctx := context.TODO() - - go rcRollingUpgrade.CallKubectlDrain(mockNodeName, ruObj, errChan) - - output := "" - select { - case <-ctx.Done(): - log.Printf("Kubectl drain timed out for node - %s", mockNodeName) - log.Print(ctx.Err()) - output = "timed-out" - break - case err := <-errChan: - if err != nil { - log.Printf("Kubectl drain errored for node - %s, error: %s", mockNodeName, err.Error()) - output = "error" - break - } - log.Printf("Kubectl drain completed for node - %s", mockNodeName) - output = "completed" - break - } - - g.Expect(output).To(gomega.ContainSubstring("completed")) -} - -func TestTestCallKubectlDrainWithDrainTimeout(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - mockKubeCtlCall := "sleep 1; echo" - mockNodeName := "some-node-name" - rcRollingUpgrade := createReconciler() - rcRollingUpgrade.ScriptRunner.KubectlCall = mockKubeCtlCall - - mockAsgName := "some-asg" - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{AsgName: mockAsgName}} - - errChan := make(chan error) - ctx := context.TODO() - ctx, cancel := context.WithTimeout(ctx, 2*time.Second) - defer cancel() - - go rcRollingUpgrade.CallKubectlDrain(mockNodeName, ruObj, errChan) - - output := "" - select { - case <-ctx.Done(): - log.Printf("Kubectl drain timed out for node - %s", mockNodeName) - log.Print(ctx.Err()) - output = "timed-out" - break - case err := <-errChan: - if err != nil { - log.Printf("Kubectl drain errored for node - %s, error: %s", mockNodeName, err.Error()) - output = "error" - break - } - log.Printf("Kubectl drain completed for node - %s", mockNodeName) - output = "completed" - break - } - - g.Expect(output).To(gomega.ContainSubstring("completed")) -} - -func TestTestCallKubectlDrainWithZeroDrainTimeout(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - mockKubeCtlCall := "sleep 1; echo" - mockNodeName := "some-node-name" - rcRollingUpgrade := createReconciler() - rcRollingUpgrade.ScriptRunner.KubectlCall = mockKubeCtlCall - - mockAsgName := "some-asg" - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{AsgName: mockAsgName}} - - errChan := make(chan error) - ctx := context.TODO() - ctx, cancel := context.WithTimeout(ctx, 2*time.Second) - defer cancel() - - go rcRollingUpgrade.CallKubectlDrain(mockNodeName, ruObj, errChan) - - output := "" - select { - case <-ctx.Done(): - log.Printf("Kubectl drain timed out for node - %s", mockNodeName) - log.Print(ctx.Err()) - output = "timed-out" - break - case err := <-errChan: - if err != nil { - log.Printf("Kubectl drain errored for node - %s, error: %s", mockNodeName, err.Error()) - output = "error" - break - } - log.Printf("Kubectl drain completed for node - %s", mockNodeName) - output = "completed" - break - } - - g.Expect(output).To(gomega.ContainSubstring("completed")) -} - -func TestTestCallKubectlDrainWithError(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - mockKubeCtlCall := "cat xyz" - mockNodeName := "some-node-name" - rcRollingUpgrade := createReconciler() - rcRollingUpgrade.ScriptRunner.KubectlCall = mockKubeCtlCall - - mockAsgName := "some-asg" - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{AsgName: mockAsgName}} - - errChan := make(chan error) - ctx := context.TODO() - - go rcRollingUpgrade.CallKubectlDrain(mockNodeName, ruObj, errChan) - - output := "" - select { - case <-ctx.Done(): - log.Printf("Kubectl drain timed out for node - %s", mockNodeName) - log.Print(ctx.Err()) - output = "timed-out" - break - case err := <-errChan: - if err != nil { - log.Printf("Kubectl drain errored for node - %s, error: %s", mockNodeName, err.Error()) - output = "error" - break - } - log.Printf("Kubectl drain completed for node - %s", mockNodeName) - output = "completed" - break - } - - g.Expect(output).To(gomega.ContainSubstring("error")) -} - -func TestTestCallKubectlDrainWithTimeoutOccurring(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - mockKubeCtlCall := "sleep 1; echo" - mockNodeName := "some-node-name" - rcRollingUpgrade := createReconciler() - rcRollingUpgrade.ScriptRunner.KubectlCall = mockKubeCtlCall - - mockAsgName := "some-asg" - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{AsgName: mockAsgName}} - - errChan := make(chan error) - ctx := context.TODO() - ctx, cancel := context.WithTimeout(ctx, 50*time.Millisecond) - defer cancel() - - go rcRollingUpgrade.CallKubectlDrain(mockNodeName, ruObj, errChan) - - output := "" - select { - case <-ctx.Done(): - log.Printf("Kubectl drain timed out for node - %s", mockNodeName) - log.Print(ctx.Err()) - output = "timed-out" - break - case err := <-errChan: - if err != nil { - log.Printf("Kubectl drain errored for node - %s, error: %s", mockNodeName, err.Error()) - output = "error" - break - } - log.Printf("Kubectl drain completed for node - %s", mockNodeName) - output = "completed" - break - } - - g.Expect(output).To(gomega.ContainSubstring("timed-out")) -} - -func TestTestCallKubectlDrainIgnoresNoiseInOutput(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - mockKubeCtlCall := "echo 'I0105 Throttling request took 1.097511969s\\nError from server (NotFound): nodes \\\"some-node\\\" not found'; exit 1" - mockNodeName := "some-node" - mockAsgName := "some-asg" - rcRollingUpgrade := createReconciler() - rcRollingUpgrade.ScriptRunner.KubectlCall = mockKubeCtlCall - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{AsgName: mockAsgName}} - - errChan := make(chan error) - ctx := context.TODO() - - go rcRollingUpgrade.CallKubectlDrain(mockNodeName, ruObj, errChan) - - output := "" - select { - case <-ctx.Done(): - log.Printf("Kubectl drain timed out for node - %s", mockNodeName) - log.Print(ctx.Err()) - output = "timed-out" - break - case err := <-errChan: - if err != nil { - log.Printf("Kubectl drain errored for node - %s, error: %s", mockNodeName, err.Error()) - output = "error" - break - } - log.Printf("Kubectl drain completed for node - %s", mockNodeName) - output = "completed" - break - } - - g.Expect(output).To(gomega.ContainSubstring("completed")) -} - -func TestValidateRuObj(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - mockAsgName := "some-asg" - strategy := upgrademgrv1alpha1.UpdateStrategy{ - Type: upgrademgrv1alpha1.RandomUpdateStrategy, - MaxUnavailable: intstr.Parse("75"), - } - - rcRollingUpgrade := createReconciler() - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ - ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{ - AsgName: mockAsgName, - Strategy: strategy, - }, - } - - err := rcRollingUpgrade.validateRollingUpgradeObj(ruObj) - g.Expect(err).To(gomega.BeNil()) -} - -func TestValidateruObjInvalidMaxUnavailable(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - mockAsgName := "some-asg" - strategy := upgrademgrv1alpha1.UpdateStrategy{ - Type: upgrademgrv1alpha1.RandomUpdateStrategy, - MaxUnavailable: intstr.Parse("150%"), - } - - rcRollingUpgrade := createReconciler() - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ - ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{ - AsgName: mockAsgName, - Strategy: strategy, - }, - } - - err := rcRollingUpgrade.validateRollingUpgradeObj(ruObj) - g.Expect(err.Error()).To(gomega.ContainSubstring("Invalid value for maxUnavailable")) -} - -func TestValidateruObjMaxUnavailableZeroPercent(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - mockAsgName := "some-asg" - strategy := upgrademgrv1alpha1.UpdateStrategy{ - Type: upgrademgrv1alpha1.RandomUpdateStrategy, - MaxUnavailable: intstr.Parse("0%"), - } - - rcRollingUpgrade := createReconciler() - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ - ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{ - AsgName: mockAsgName, - Strategy: strategy, - }, - } - - err := rcRollingUpgrade.validateRollingUpgradeObj(ruObj) - g.Expect(err.Error()).To(gomega.ContainSubstring("Invalid value for maxUnavailable")) -} - -func TestValidateruObjMaxUnavailableInt(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - mockAsgName := "some-asg" - strategy := upgrademgrv1alpha1.UpdateStrategy{ - Type: upgrademgrv1alpha1.RandomUpdateStrategy, - MaxUnavailable: intstr.Parse("10"), - } - - rcRollingUpgrade := createReconciler() - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ - ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{ - AsgName: mockAsgName, - Strategy: strategy, - }, - } - - err := rcRollingUpgrade.validateRollingUpgradeObj(ruObj) - g.Expect(err).To(gomega.BeNil()) -} - -func TestValidateruObjMaxUnavailableIntZero(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - mockAsgName := "some-asg" - strategy := upgrademgrv1alpha1.UpdateStrategy{ - Type: upgrademgrv1alpha1.RandomUpdateStrategy, - MaxUnavailable: intstr.Parse("0"), - } - - rcRollingUpgrade := createReconciler() - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ - ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{ - AsgName: mockAsgName, - Strategy: strategy, - }, - } - - err := rcRollingUpgrade.validateRollingUpgradeObj(ruObj) - g.Expect(err.Error()).To(gomega.ContainSubstring("Invalid value for maxUnavailable")) -} - -func TestValidateruObjMaxUnavailableIntNegativeValue(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - mockAsgName := "some-asg" - strategy := upgrademgrv1alpha1.UpdateStrategy{ - Type: upgrademgrv1alpha1.RandomUpdateStrategy, - MaxUnavailable: intstr.Parse("-1"), - } - - rcRollingUpgrade := createReconciler() - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ - ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{ - AsgName: mockAsgName, - Strategy: strategy, - }, - } - - err := rcRollingUpgrade.validateRollingUpgradeObj(ruObj) - g.Expect(err.Error()).To(gomega.ContainSubstring("Invalid value for maxUnavailable")) -} - -func TestValidateruObjWithStrategyAndDrainTimeoutOnly(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - mockAsgName := "some-asg" - strategy := upgrademgrv1alpha1.UpdateStrategy{ - Type: upgrademgrv1alpha1.RandomUpdateStrategy, - } - - rcRollingUpgrade := createReconciler() - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ - ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{ - AsgName: mockAsgName, - Strategy: strategy, - }, - } - - err := rcRollingUpgrade.validateRollingUpgradeObj(ruObj) - g.Expect(err.Error()).To(gomega.ContainSubstring("Invalid value for maxUnavailable")) -} - -func TestValidateruObjWithoutStrategyOnly(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - ruObj := upgrademgrv1alpha1.RollingUpgrade{} - - rcRollingUpgrade := createReconciler() - err := rcRollingUpgrade.validateRollingUpgradeObj(&ruObj) - - g.Expect(err).To(gomega.BeNil()) -} - -func TestValidateruObjStrategyType(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - mockAsgName := "some-asg" - strategy := upgrademgrv1alpha1.UpdateStrategy{ - Type: upgrademgrv1alpha1.RandomUpdateStrategy, - MaxUnavailable: intstr.Parse("10"), - } - - rcRollingUpgrade := createReconciler() - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ - ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{ - AsgName: mockAsgName, - Strategy: strategy, - }, - } - - err := rcRollingUpgrade.validateRollingUpgradeObj(ruObj) - g.Expect(err).To(gomega.BeNil()) -} - -func TestValidateruObjInvalidStrategyType(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - mockAsgName := "some-asg" - strategy := upgrademgrv1alpha1.UpdateStrategy{ - Type: "xyx", - MaxUnavailable: intstr.Parse("10"), - } - - rcRollingUpgrade := createReconciler() - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ - ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{ - AsgName: mockAsgName, - Strategy: strategy, - }, - } - - err := rcRollingUpgrade.validateRollingUpgradeObj(ruObj) - g.Expect(err.Error()).To(gomega.ContainSubstring("Invalid value for strategy type")) -} - -func TestValidateruObjWithYaml(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - strategyYaml := ` -drainTimeout: 30 -maxUnavailable: 100% -type: randomUpdate -` - - mockAsgName := "some-asg" - strategy := upgrademgrv1alpha1.UpdateStrategy{} - err := yaml.Unmarshal([]byte(strategyYaml), &strategy) - if err != nil { - fmt.Printf("Error occurred while unmarshalling strategy yaml object, error: %s", err.Error()) - } - - rcRollingUpgrade := createReconciler() - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ - ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{ - AsgName: mockAsgName, - Strategy: strategy, - }, - } - rcRollingUpgrade.setDefaultsForRollingUpdateStrategy(ruObj) - err = rcRollingUpgrade.validateRollingUpgradeObj(ruObj) - g.Expect(err).To(gomega.BeNil()) -} - -func TestSetDefaultsForRollingUpdateStrategy(t *testing.T) { - - g := gomega.NewGomegaWithT(t) - - mockAsgName := "some-asg" - strategy := upgrademgrv1alpha1.UpdateStrategy{} - - rcRollingUpgrade := createReconciler() - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ - ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{ - AsgName: mockAsgName, - Strategy: strategy, - }, - } - rcRollingUpgrade.setDefaultsForRollingUpdateStrategy(ruObj) - - g.Expect(string(ruObj.Spec.Strategy.Type)).To(gomega.ContainSubstring(string(upgrademgrv1alpha1.RandomUpdateStrategy))) - g.Expect(ruObj.Spec.Strategy.DrainTimeout).To(gomega.Equal(-1)) - g.Expect(ruObj.Spec.Strategy.MaxUnavailable).To(gomega.Equal(intstr.IntOrString{Type: 0, IntVal: 1})) -} - -func TestValidateruObjStrategyAfterSettingDefaults(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - mockAsgName := "some-asg" - strategy := upgrademgrv1alpha1.UpdateStrategy{ - Type: upgrademgrv1alpha1.RandomUpdateStrategy, - } - - rcRollingUpgrade := createReconciler() - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ - ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{ - AsgName: mockAsgName, - Strategy: strategy, - }, - } - rcRollingUpgrade.setDefaultsForRollingUpdateStrategy(ruObj) - err := rcRollingUpgrade.validateRollingUpgradeObj(ruObj) - - g.Expect(err).To(gomega.BeNil()) -} - -func TestValidateruObjStrategyAfterSettingDefaultsWithInvalidStrategyType(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - mockAsgName := "some-asg" - strategy := upgrademgrv1alpha1.UpdateStrategy{ - Type: "xyz", - } - - rcRollingUpgrade := createReconciler() - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ - ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{ - AsgName: mockAsgName, - Strategy: strategy, - }, - } - rcRollingUpgrade.setDefaultsForRollingUpdateStrategy(ruObj) - error := rcRollingUpgrade.validateRollingUpgradeObj(ruObj) - - g.Expect(error).To(gomega.Not(gomega.BeNil())) -} - -func TestValidateruObjStrategyAfterSettingDefaultsWithOnlyDrainTimeout(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - mockAsgName := "some-asg" - strategy := upgrademgrv1alpha1.UpdateStrategy{ - Type: upgrademgrv1alpha1.RandomUpdateStrategy, - DrainTimeout: 15, - } - - rcRollingUpgrade := createReconciler() - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ - ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{ - AsgName: mockAsgName, - Strategy: strategy, - }, - } - rcRollingUpgrade.setDefaultsForRollingUpdateStrategy(ruObj) - error := rcRollingUpgrade.validateRollingUpgradeObj(ruObj) - - g.Expect(error).To(gomega.BeNil()) -} - -func TestValidateruObjStrategyAfterSettingDefaultsWithOnlyMaxUnavailable(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - mockAsgName := "some-asg" - strategy := upgrademgrv1alpha1.UpdateStrategy{ - Type: upgrademgrv1alpha1.RandomUpdateStrategy, - MaxUnavailable: intstr.Parse("100%"), - } - - rcRollingUpgrade := createReconciler() - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ - ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{ - AsgName: mockAsgName, - Strategy: strategy, - }, - } - rcRollingUpgrade.setDefaultsForRollingUpdateStrategy(ruObj) - error := rcRollingUpgrade.validateRollingUpgradeObj(ruObj) - - g.Expect(error).To(gomega.BeNil()) -} - -func TestRunRestackNoNodeInAsg(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - someAsg := "some-asg" - someLaunchConfig := "some-launch-config" - mockAsg := autoscaling.Group{AutoScalingGroupName: &someAsg, - LaunchConfigurationName: &someLaunchConfig, - Instances: []*autoscaling.Instance{}} - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{ - AsgName: someAsg, - Strategy: upgrademgrv1alpha1.UpdateStrategy{Type: upgrademgrv1alpha1.RandomUpdateStrategy}, - }, - } - - mgr, err := buildManager() - g.Expect(err).NotTo(gomega.HaveOccurred()) - - nodeList := corev1.NodeList{Items: []corev1.Node{}} - rcRollingUpgrade := &RollingUpgradeReconciler{ - Client: mgr.GetClient(), - Log: log2.NullLogger{}, - ASGClient: MockAutoscalingGroup{}, - EC2Client: MockEC2{}, - generatedClient: kubernetes.NewForConfigOrDie(mgr.GetConfig()), - admissionMap: sync.Map{}, - NodeList: &nodeList, - ClusterState: NewClusterState(), - CacheConfig: cache.NewConfig(0*time.Second, 0, 0), - } - rcRollingUpgrade.admissionMap.Store(ruObj.NamespacedName(), "processing") - rcRollingUpgrade.ruObjNameToASG.Store(ruObj.NamespacedName(), &mockAsg) - - ctx := context.TODO() - - // This execution gets past the different launch config check, but since there is no node name, it is skipped - nodesProcessed, err := rcRollingUpgrade.runRestack(&ctx, ruObj) - g.Expect(nodesProcessed).To(gomega.Equal(0)) - g.Expect(err).To(gomega.BeNil()) -} - -func TestWaitForTermination(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - TerminationTimeoutSeconds = 1 - TerminationSleepIntervalSeconds = 1 - - mockNodeName := "node-123" - mockNode := &corev1.Node{ - ObjectMeta: metav1.ObjectMeta{ - Name: mockNodeName, - }, - } - kuberenetesClient := fake.NewSimpleClientset() - nodeInterface := kuberenetesClient.CoreV1().Nodes() - - mgr, err := buildManager() - g.Expect(err).NotTo(gomega.HaveOccurred()) - - rcRollingUpgrade := &RollingUpgradeReconciler{ - Client: mgr.GetClient(), - Log: log2.NullLogger{}, - generatedClient: kubernetes.NewForConfigOrDie(mgr.GetConfig()), - ClusterState: NewClusterState(), - } - _, err = nodeInterface.Create(mockNode) - g.Expect(err).NotTo(gomega.HaveOccurred()) - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ - ObjectMeta: metav1.ObjectMeta{ - Name: "foo", - Namespace: "default", - }, - } - - unjoined, err := rcRollingUpgrade.WaitForTermination(ruObj, mockNodeName, nodeInterface) - g.Expect(err).NotTo(gomega.HaveOccurred()) - g.Expect(unjoined).To(gomega.BeFalse()) - - err = nodeInterface.Delete(mockNodeName, &metav1.DeleteOptions{}) - g.Expect(err).NotTo(gomega.HaveOccurred()) - - unjoined, err = rcRollingUpgrade.WaitForTermination(ruObj, mockNodeName, nodeInterface) - g.Expect(err).NotTo(gomega.HaveOccurred()) - g.Expect(unjoined).To(gomega.BeTrue()) -} - -func TestWaitForTerminationWhenNodeIsNotFound(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - TerminationTimeoutSeconds = 1 - TerminationSleepIntervalSeconds = 1 - - // nodeName is empty when a node is not found. - mockNodeName := "" - mockNode := &corev1.Node{ - ObjectMeta: metav1.ObjectMeta{ - Name: mockNodeName, - }, - } - kuberenetesClient := fake.NewSimpleClientset() - nodeInterface := kuberenetesClient.CoreV1().Nodes() - - mgr, err := buildManager() - g.Expect(err).NotTo(gomega.HaveOccurred()) - - rcRollingUpgrade := &RollingUpgradeReconciler{ - Client: mgr.GetClient(), - Log: log2.NullLogger{}, - generatedClient: kubernetes.NewForConfigOrDie(mgr.GetConfig()), - ClusterState: NewClusterState(), - } - _, err = nodeInterface.Create(mockNode) - g.Expect(err).NotTo(gomega.HaveOccurred()) - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ - ObjectMeta: metav1.ObjectMeta{ - Name: "foo", - Namespace: "default", - }, - } - - unjoined, err := rcRollingUpgrade.WaitForTermination(ruObj, mockNodeName, nodeInterface) - g.Expect(unjoined).To(gomega.BeTrue()) - g.Expect(err).To(gomega.BeNil()) -} - -func buildManager() (manager.Manager, error) { - err := upgrademgrv1alpha1.AddToScheme(scheme.Scheme) - if err != nil { - return nil, err - } - return manager.New(cfg, manager.Options{MetricsBindAddress: "0"}) -} - -func TestRunRestackWithNodesLessThanMaxUnavailable(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - someAsg := "some-asg" - mockID := "some-id" - someLaunchConfig := "some-launch-config" - az := "az-1" - mockInstance := autoscaling.Instance{InstanceId: &mockID, LaunchConfigurationName: &someLaunchConfig, AvailabilityZone: &az} - mockAsg := autoscaling.Group{AutoScalingGroupName: &someAsg, - LaunchConfigurationName: &someLaunchConfig, - Instances: []*autoscaling.Instance{&mockInstance}} - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{ - AsgName: someAsg, - Strategy: upgrademgrv1alpha1.UpdateStrategy{ - MaxUnavailable: intstr.IntOrString{Type: 0, IntVal: 2}, - Type: upgrademgrv1alpha1.RandomUpdateStrategy, - }, - }, - } - - mgr, err := buildManager() - g.Expect(err).NotTo(gomega.HaveOccurred()) - - rcRollingUpgrade := &RollingUpgradeReconciler{ - Client: mgr.GetClient(), - Log: log2.NullLogger{}, - ASGClient: MockAutoscalingGroup{}, - EC2Client: MockEC2{}, - generatedClient: kubernetes.NewForConfigOrDie(mgr.GetConfig()), - ClusterState: NewClusterState(), - CacheConfig: cache.NewConfig(0*time.Second, 0, 0), - } - rcRollingUpgrade.ruObjNameToASG.Store(ruObj.NamespacedName(), &mockAsg) - rcRollingUpgrade.ClusterState.deleteAllInstancesInAsg(someAsg) - ctx := context.TODO() - - // This execution should not perform drain or termination, but should pass - nodesProcessed, err := rcRollingUpgrade.runRestack(&ctx, ruObj) - g.Expect(err).To(gomega.BeNil()) - g.Expect(nodesProcessed).To(gomega.Equal(1)) -} - -func TestRequiresRefreshHandlesLaunchConfiguration(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - r := &RollingUpgradeReconciler{Log: log2.NullLogger{}} - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}} - - mockID := "some-id" - someLaunchConfig := "some-launch-config-v1" - az := "az-1" - mockInstance := autoscaling.Instance{InstanceId: &mockID, LaunchConfigurationName: &someLaunchConfig, AvailabilityZone: &az} - - newLaunchConfig := "some-launch-config-v2" - definition := launchDefinition{ - launchConfigurationName: &newLaunchConfig, - } - - result := r.requiresRefresh(ruObj, &mockInstance, &definition) - g.Expect(result).To(gomega.Equal(true)) -} - -func TestRequiresRefreshHandlesLaunchTemplateNameVersionUpdate(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - mockID := "some-id" - oldLaunchTemplate := &autoscaling.LaunchTemplateSpecification{ - LaunchTemplateName: aws.String("launch-template"), - Version: aws.String("1"), - } - az := "az-1" - mockInstance := autoscaling.Instance{InstanceId: &mockID, LaunchTemplate: oldLaunchTemplate, AvailabilityZone: &az} - - newLaunchTemplate := &autoscaling.LaunchTemplateSpecification{ - LaunchTemplateName: aws.String("launch-template"), - Version: aws.String("2"), - } - definition := launchDefinition{ - launchTemplate: newLaunchTemplate, - } - - r := &RollingUpgradeReconciler{Log: log2.NullLogger{}} - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}} - result := r.requiresRefresh(ruObj, &mockInstance, &definition) - g.Expect(result).To(gomega.Equal(true)) -} - -func TestRequiresRefreshHandlesLaunchTemplateIDVersionUpdate(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - mockID := "some-id" - oldLaunchTemplate := &autoscaling.LaunchTemplateSpecification{ - LaunchTemplateId: aws.String("launch-template-id-v1"), - Version: aws.String("1"), - } - az := "az-1" - mockInstance := autoscaling.Instance{InstanceId: &mockID, LaunchTemplate: oldLaunchTemplate, AvailabilityZone: &az} - - newLaunchTemplate := &autoscaling.LaunchTemplateSpecification{ - LaunchTemplateId: aws.String("launch-template-id-v1"), - Version: aws.String("2"), - } - definition := launchDefinition{ - launchTemplate: newLaunchTemplate, - } - r := &RollingUpgradeReconciler{Log: log2.NullLogger{}} - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}} - result := r.requiresRefresh(ruObj, &mockInstance, &definition) - g.Expect(result).To(gomega.Equal(true)) -} - -func TestRequiresRefreshHandlesLaunchTemplateNameUpdate(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - mockID := "some-id" - oldLaunchTemplate := &autoscaling.LaunchTemplateSpecification{ - LaunchTemplateName: aws.String("launch-template"), - Version: aws.String("1"), - } - az := "az-1" - mockInstance := autoscaling.Instance{InstanceId: &mockID, LaunchTemplate: oldLaunchTemplate, AvailabilityZone: &az} - - newLaunchTemplate := &autoscaling.LaunchTemplateSpecification{ - LaunchTemplateName: aws.String("launch-template-v2"), - Version: aws.String("1"), - } - definition := launchDefinition{ - launchTemplate: newLaunchTemplate, - } - r := &RollingUpgradeReconciler{Log: log2.NullLogger{}} - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}} - result := r.requiresRefresh(ruObj, &mockInstance, &definition) - g.Expect(result).To(gomega.Equal(true)) -} - -func TestRequiresRefreshHandlesLaunchTemplateIDUpdate(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - mockID := "some-id" - oldLaunchTemplate := &autoscaling.LaunchTemplateSpecification{ - LaunchTemplateId: aws.String("launch-template-id-v1"), - Version: aws.String("1"), - } - az := "az-1" - mockInstance := autoscaling.Instance{InstanceId: &mockID, LaunchTemplate: oldLaunchTemplate, AvailabilityZone: &az} - - newLaunchTemplate := &autoscaling.LaunchTemplateSpecification{ - LaunchTemplateId: aws.String("launch-template-id-v2"), - Version: aws.String("1"), - } - definition := launchDefinition{ - launchTemplate: newLaunchTemplate, - } - r := &RollingUpgradeReconciler{Log: log2.NullLogger{}} - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}} - result := r.requiresRefresh(ruObj, &mockInstance, &definition) - g.Expect(result).To(gomega.Equal(true)) -} - -func TestRequiresRefreshNotUpdateIfNoVersionChange(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - mockID := "some-id" - instanceLaunchTemplate := &autoscaling.LaunchTemplateSpecification{ - LaunchTemplateId: aws.String("launch-template-id-v1"), - Version: aws.String("1"), - } - az := "az-1" - mockInstance := autoscaling.Instance{InstanceId: &mockID, LaunchTemplate: instanceLaunchTemplate, AvailabilityZone: &az} - - launchTemplate := &ec2.LaunchTemplate{ - LaunchTemplateId: aws.String("launch-template-id-v1"), - LatestVersionNumber: aws.Int64(1), - } - definition := launchDefinition{ - launchTemplate: instanceLaunchTemplate, - } - r := &RollingUpgradeReconciler{Log: log2.NullLogger{}} - r.LaunchTemplates = append(r.LaunchTemplates, launchTemplate) - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}} - result := r.requiresRefresh(ruObj, &mockInstance, &definition) - g.Expect(result).To(gomega.Equal(false)) -} - -func TestForceRefresh(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - // Even if launchtemplate is identical but forceRefresh is set, requiresRefresh should return true. - mockID := "some-id" - launchTemplate := &autoscaling.LaunchTemplateSpecification{ - LaunchTemplateId: aws.String("launch-template-id-v1"), - Version: aws.String("1"), - } - az := "az-1" - mockInstance := autoscaling.Instance{InstanceId: &mockID, LaunchTemplate: launchTemplate, AvailabilityZone: &az} - - definition := launchDefinition{ - launchTemplate: launchTemplate, - } - - ec2launchTemplate := &ec2.LaunchTemplate{ - LaunchTemplateId: aws.String("launch-template-id-v1"), - LatestVersionNumber: aws.Int64(1), - } - - r := &RollingUpgradeReconciler{Log: log2.NullLogger{}} - r.LaunchTemplates = append(r.LaunchTemplates, ec2launchTemplate) - currentTime := metav1.NewTime(metav1.Now().Time) - oldTime := metav1.NewTime(currentTime.Time.AddDate(0, 0, -1)) - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ - ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default", CreationTimestamp: currentTime}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{ - Strategy: upgrademgrv1alpha1.UpdateStrategy{DrainTimeout: -1}, - ForceRefresh: true, - }, - } - // If the node was created before the rollingupgrade object, requiresRefresh should return true - k8sNode := corev1.Node{ObjectMeta: metav1.ObjectMeta{Name: "k8sNode", CreationTimestamp: oldTime}, - Spec: corev1.NodeSpec{ProviderID: "fake-separator/" + mockID}} - nodeList := corev1.NodeList{Items: []corev1.Node{k8sNode}} - r.NodeList = &nodeList - result := r.requiresRefresh(ruObj, &mockInstance, &definition) - g.Expect(result).To(gomega.Equal(true)) - - // If the node was created at the same time as rollingupgrade object, requiresRefresh should return false - k8sNode.CreationTimestamp = currentTime - nodeList = corev1.NodeList{Items: []corev1.Node{k8sNode}} - r.NodeList = &nodeList - result = r.requiresRefresh(ruObj, &mockInstance, &definition) - g.Expect(result).To(gomega.Equal(false)) - - // Reset the timestamp on the k8s node - k8sNode.CreationTimestamp = oldTime - nodeList = corev1.NodeList{Items: []corev1.Node{k8sNode}} - r.NodeList = &nodeList - - // If launchTempaltes are different and forceRefresh is true, requiresRefresh should return true - newLaunchTemplate := &autoscaling.LaunchTemplateSpecification{ - LaunchTemplateId: aws.String("launch-template-id-v1"), - Version: aws.String("1"), - } - - definition = launchDefinition{ - launchTemplate: newLaunchTemplate, - } - result = r.requiresRefresh(ruObj, &mockInstance, &definition) - g.Expect(result).To(gomega.Equal(true)) - - // If launchTemplares are identical AND forceRefresh is false, requiresRefresh should return false - ruObj.Spec.ForceRefresh = false - result = r.requiresRefresh(ruObj, &mockInstance, &definition) - g.Expect(result).To(gomega.Equal(false)) - - // If launchConfigs are identical but forceRefresh is true, requiresRefresh should return true - ruObj.Spec.ForceRefresh = true - launchConfig := "launch-config" - mockInstance = autoscaling.Instance{InstanceId: &mockID, LaunchConfigurationName: &launchConfig, AvailabilityZone: &az} - definition = launchDefinition{ - launchConfigurationName: &launchConfig, - } - result = r.requiresRefresh(ruObj, &mockInstance, &definition) - g.Expect(result).To(gomega.Equal(true)) - - // If launchConfigs are identical AND forceRefresh is false, requiresRefresh should return false - ruObj.Spec.ForceRefresh = false - result = r.requiresRefresh(ruObj, &mockInstance, &definition) - g.Expect(result).To(gomega.Equal(false)) -} - -func TestDrainNodeTerminateTerminatesWhenIgnoreDrainFailuresSet(t *testing.T) { - g := gomega.NewGomegaWithT(t) - mockNode := "some-node-name" - - // Force quit from the rest of the command - mockKubeCtlCall := "exit 1;" - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ - ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{ - Strategy: upgrademgrv1alpha1.UpdateStrategy{DrainTimeout: -1}, - IgnoreDrainFailures: true, - PreDrain: upgrademgrv1alpha1.PreDrainSpec{ - Script: mockKubeCtlCall, - }, - }, - } - rcRollingUpgrade := &RollingUpgradeReconciler{ - ClusterState: NewClusterState(), - Log: log2.NullLogger{}, - EC2Client: MockEC2{}, - ASGClient: MockAutoscalingGroup{ - errorFlag: false, - awsErr: nil, - }, - ScriptRunner: NewScriptRunner(log2.NullLogger{}), - } - - err := rcRollingUpgrade.DrainTerminate(ruObj, mockNode, mockNode) - g.Expect(err).To(gomega.BeNil()) // don't expect errors. - - // nodeName is empty when node isn't part of the cluster. It must skip drain and terminate. - err = rcRollingUpgrade.DrainTerminate(ruObj, "", mockNode) - g.Expect(err).To(gomega.BeNil()) // don't expect errors. - -} - -func TestUpdateInstancesNotExists(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - someAsg := "some-asg" - mockID := "some-id" - someLaunchConfig := "some-launch-config" - az := "az-1" - mockInstance := autoscaling.Instance{InstanceId: &mockID, LaunchConfigurationName: &someLaunchConfig, AvailabilityZone: &az} - mockAsg := autoscaling.Group{AutoScalingGroupName: &someAsg, - LaunchConfigurationName: &someLaunchConfig, - Instances: []*autoscaling.Instance{&mockInstance}} - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ - ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{ - AsgName: someAsg, - Strategy: upgrademgrv1alpha1.UpdateStrategy{ - Mode: "lazy", - }, - }, - } - - mgr, _ := buildManager() - client := mgr.GetClient() - fooNode := corev1.Node{Spec: corev1.NodeSpec{ProviderID: "foo-bar/9213851"}} - - nodeList := corev1.NodeList{Items: []corev1.Node{fooNode}} - rcRollingUpgrade := &RollingUpgradeReconciler{ - Client: client, - Log: log2.NullLogger{}, - ASGClient: MockAutoscalingGroup{}, - EC2Client: MockEC2{}, - generatedClient: kubernetes.NewForConfigOrDie(mgr.GetConfig()), - NodeList: &nodeList, - ClusterState: NewClusterState(), - CacheConfig: cache.NewConfig(0*time.Second, 0, 0), - } - rcRollingUpgrade.ruObjNameToASG.Store(ruObj.NamespacedName(), &mockAsg) - rcRollingUpgrade.ScriptRunner.KubectlCall = "date" - - // Intentionally do not populate the admissionMap with the ruObj - - ctx := context.TODO() - lcName := "A" - instChan := make(chan error) - mockInstanceName1 := "foo1" - instance1 := autoscaling.Instance{InstanceId: &mockInstanceName1, AvailabilityZone: &az} - go rcRollingUpgrade.UpdateInstance(&ctx, ruObj, &instance1, &launchDefinition{launchConfigurationName: &lcName}, instChan) - processCount := 0 - select { - case <-ctx.Done(): - break - case err := <-instChan: - if err == nil { - processCount++ - } - break - } - - g.Expect(processCount).To(gomega.Equal(1)) -} - -func TestValidateNodesLaunchDefinitionSameLaunchConfig(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - someLaunchConfig := "some-launch-config" - az := "az-1" - mockInstance := autoscaling.Instance{InstanceId: aws.String("some-id"), LaunchConfigurationName: &someLaunchConfig, AvailabilityZone: &az} - mockAsg := &autoscaling.Group{ - AutoScalingGroupName: aws.String("my-asg"), - LaunchConfigurationName: &someLaunchConfig, - Instances: []*autoscaling.Instance{&mockInstance}, - } - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{AsgName: "my-asg"}} - - mgr, err := buildManager() - g.Expect(err).NotTo(gomega.HaveOccurred()) - - mockAsgClient := MockAutoscalingGroup{ - autoScalingGroups: []*autoscaling.Group{mockAsg}, - } - - rcRollingUpgrade := &RollingUpgradeReconciler{ - Client: mgr.GetClient(), - Log: log2.NullLogger{}, - ASGClient: mockAsgClient, - EC2Client: MockEC2{}, - generatedClient: kubernetes.NewForConfigOrDie(mgr.GetConfig()), - ClusterState: NewClusterState(), - CacheConfig: cache.NewConfig(0*time.Second, 0, 0), - } - rcRollingUpgrade.admissionMap.Store(ruObj.NamespacedName(), "processing") - rcRollingUpgrade.ruObjNameToASG.Store(ruObj.NamespacedName(), mockAsg) - - // This execution should not perform drain or termination, but should pass - err = rcRollingUpgrade.validateNodesLaunchDefinition(ruObj) - g.Expect(err).To(gomega.BeNil()) -} - -func TestValidateNodesLaunchDefinitionDifferentLaunchConfig(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - someLaunchConfig := "some-launch-config" - someOtherLaunchConfig := "some-other-launch-config" - az := "az-1" - mockInstance := autoscaling.Instance{InstanceId: aws.String("some-id"), LaunchConfigurationName: &someLaunchConfig, AvailabilityZone: &az} - mockAsg := &autoscaling.Group{ - AutoScalingGroupName: aws.String("my-asg"), - LaunchConfigurationName: &someOtherLaunchConfig, - Instances: []*autoscaling.Instance{&mockInstance}} - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{AsgName: "my-asg"}} - - mgr, err := buildManager() - g.Expect(err).NotTo(gomega.HaveOccurred()) - - mockAsgClient := MockAutoscalingGroup{ - autoScalingGroups: []*autoscaling.Group{mockAsg}, - } - - rcRollingUpgrade := &RollingUpgradeReconciler{ - Client: mgr.GetClient(), - Log: log2.NullLogger{}, - ASGClient: mockAsgClient, - EC2Client: MockEC2{}, - generatedClient: kubernetes.NewForConfigOrDie(mgr.GetConfig()), - ClusterState: NewClusterState(), - CacheConfig: cache.NewConfig(0*time.Second, 0, 0), - } - rcRollingUpgrade.admissionMap.Store(ruObj.NamespacedName(), "processing") - rcRollingUpgrade.ruObjNameToASG.Store(ruObj.NamespacedName(), mockAsg) - - // This execution should not perform drain or termination, but should pass - err = rcRollingUpgrade.validateNodesLaunchDefinition(ruObj) - g.Expect(err).To(gomega.Not(gomega.BeNil())) -} - -func TestValidateNodesLaunchDefinitionSameLaunchTemplate(t *testing.T) { - g := gomega.NewGomegaWithT(t) - someLaunchTemplate := &autoscaling.LaunchTemplateSpecification{LaunchTemplateId: aws.String("launch-template-id-v1")} - az := "az-1" - mockInstance := autoscaling.Instance{InstanceId: aws.String("some-id"), LaunchTemplate: someLaunchTemplate, AvailabilityZone: &az} - mockAsg := &autoscaling.Group{ - AutoScalingGroupName: aws.String("my-asg"), - LaunchTemplate: someLaunchTemplate, - Instances: []*autoscaling.Instance{&mockInstance}} - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{AsgName: "my-asg"}} - - mgr, err := buildManager() - g.Expect(err).NotTo(gomega.HaveOccurred()) - - mockAsgClient := MockAutoscalingGroup{ - autoScalingGroups: []*autoscaling.Group{mockAsg}, - } - - rcRollingUpgrade := &RollingUpgradeReconciler{ - Client: mgr.GetClient(), - Log: log2.NullLogger{}, - ASGClient: mockAsgClient, - EC2Client: MockEC2{}, - generatedClient: kubernetes.NewForConfigOrDie(mgr.GetConfig()), - ClusterState: NewClusterState(), - CacheConfig: cache.NewConfig(0*time.Second, 0, 0), - } - rcRollingUpgrade.admissionMap.Store(ruObj.NamespacedName(), "processing") - rcRollingUpgrade.ruObjNameToASG.Store(ruObj.NamespacedName(), mockAsg) - - // This execution should not perform drain or termination, but should pass - err = rcRollingUpgrade.validateNodesLaunchDefinition(ruObj) - g.Expect(err).To(gomega.BeNil()) -} - -func TestValidateNodesLaunchDefinitionDifferentLaunchTemplate(t *testing.T) { - g := gomega.NewGomegaWithT(t) - someLaunchTemplate := &autoscaling.LaunchTemplateSpecification{LaunchTemplateId: aws.String("launch-template-id-v1")} - someOtherLaunchTemplate := &autoscaling.LaunchTemplateSpecification{LaunchTemplateId: aws.String("launch-template-id-v2")} - az := "az-1" - mockInstance := autoscaling.Instance{InstanceId: aws.String("some-id"), LaunchTemplate: someLaunchTemplate, AvailabilityZone: &az} - mockAsg := &autoscaling.Group{ - AutoScalingGroupName: aws.String("my-asg"), - LaunchTemplate: someOtherLaunchTemplate, - Instances: []*autoscaling.Instance{&mockInstance}} - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{AsgName: "my-asg"}} - - mgr, err := buildManager() - g.Expect(err).NotTo(gomega.HaveOccurred()) - - mockAsgClient := MockAutoscalingGroup{ - autoScalingGroups: []*autoscaling.Group{mockAsg}, - } - - rcRollingUpgrade := &RollingUpgradeReconciler{ - Client: mgr.GetClient(), - Log: log2.NullLogger{}, - ASGClient: mockAsgClient, - EC2Client: MockEC2{}, - generatedClient: kubernetes.NewForConfigOrDie(mgr.GetConfig()), - ClusterState: NewClusterState(), - CacheConfig: cache.NewConfig(0*time.Second, 0, 0), - } - rcRollingUpgrade.admissionMap.Store(ruObj.NamespacedName(), "processing") - rcRollingUpgrade.ruObjNameToASG.Store(ruObj.NamespacedName(), mockAsg) - - // This execution should not perform drain or termination, but should pass - err = rcRollingUpgrade.validateNodesLaunchDefinition(ruObj) - g.Expect(err).To(gomega.Not(gomega.BeNil())) -} diff --git a/controllers/rollup_cluster_state.go b/controllers/rollup_cluster_state.go deleted file mode 100644 index 64167f55..00000000 --- a/controllers/rollup_cluster_state.go +++ /dev/null @@ -1,176 +0,0 @@ -/* -Copyright 2019 Intuit, Inc.. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package controllers - -import ( - "sync" - - "github.com/aws/aws-sdk-go/service/autoscaling" -) - -const ( - // updateStarted indicates that the update process has been started for an instance - updateInitialized = "new" - // updateInProgress indicates that update has been triggered for an instance - updateInProgress = "in-progress" - // updateCompleted indicates that update is completed for an instance - updateCompleted = "completed" -) - -// ClusterState contains the methods to store the instance states during a cluster update -type ClusterState interface { - markUpdateInitialized(instanceId string) - markUpdateInProgress(instanceId string) - markUpdateCompleted(instanceId string) - instanceUpdateInitialized(instanceId string) bool - instanceUpdateInProgress(instanceId string) bool - instanceUpdateCompleted(instanceId string) bool - deleteAllInstancesInAsg(asgName string) bool - getNextAvailableInstanceIdInAz(asgName string, azName string) string - initializeAsg(asgName string, instances []*autoscaling.Instance) - addInstanceState(instanceData *InstanceData) - updateInstanceState(instanceId, instanceState string) - getInstanceState(instanceId string) string -} - -type InstanceData struct { - Id string - AmiId string - AsgName string - AzName string - InstanceState string -} - -// ClusterStateImpl implements the ClusterState interface -type ClusterStateImpl struct { - mu sync.RWMutex - - // store stores the state of the instances running in different Azs for multiple ASGs - store sync.Map -} - -// newClusterState returns the object the struct implementing the ClusterState interface -func NewClusterState() ClusterState { - return &ClusterStateImpl{} -} - -// markUpdateInProgress updates the instance state to in-progress -func (c *ClusterStateImpl) markUpdateInProgress(instanceId string) { - c.updateInstanceState(instanceId, updateInProgress) -} - -// markUpdateCompleted updates the instance state to completed iff it is in-progress -func (c *ClusterStateImpl) markUpdateCompleted(instanceId string) { - c.mu.Lock() - defer c.mu.Unlock() - - if c.instanceUpdateInProgress(instanceId) { - c.updateInstanceState(instanceId, updateCompleted) - } -} - -// instanceUpdateInProgress returns true if the instance update is in progress -func (c *ClusterStateImpl) instanceUpdateInProgress(instanceId string) bool { - return c.getInstanceState(instanceId) == updateInProgress -} - -// instanceUpdateCompleted returns true if the instance update is completed -func (c *ClusterStateImpl) instanceUpdateCompleted(instanceId string) bool { - return c.getInstanceState(instanceId) == updateCompleted -} - -// deleteEntryOfAsg deletes the entry for an ASG in the cluster state map -func (c *ClusterStateImpl) deleteAllInstancesInAsg(asgName string) bool { - deleted := false - c.store.Range(func(key interface{}, value interface{}) bool { - instanceID, _ := key.(string) - instanceData, _ := value.(*InstanceData) - if instanceData.AsgName == asgName { - c.store.Delete(instanceID) - deleted = true - } - return true - }) - return deleted -} - -// markUpdateInitialized updates the instance state to in-progresss -func (c *ClusterStateImpl) markUpdateInitialized(instanceId string) { - c.updateInstanceState(instanceId, updateInitialized) -} - -// instanceUpdateInitialized returns true if the instance update is in progress -func (c *ClusterStateImpl) instanceUpdateInitialized(instanceId string) bool { - return c.getInstanceState(instanceId) == updateInitialized -} - -// initializeAsg adds an entry for all the instances in an ASG with updateInitialized state -func (c *ClusterStateImpl) initializeAsg(asgName string, instances []*autoscaling.Instance) { - for _, instance := range instances { - instanceData := &InstanceData{ - Id: *instance.InstanceId, - AzName: *instance.AvailabilityZone, - AsgName: asgName, - InstanceState: updateInitialized, - } - c.addInstanceState(instanceData) - } -} - -// getNextAvailableInstanceId returns the id of the next instance available for update in an ASG -// adding a mutex to avoid the race conditions and same instance returned for 2 go-routines -func (c *ClusterStateImpl) getNextAvailableInstanceIdInAz(asgName string, azName string) string { - c.mu.Lock() - defer c.mu.Unlock() - - instanceId := "" - c.store.Range(func(key interface{}, value interface{}) bool { - state, _ := value.(*InstanceData) - if state.AsgName == asgName && - (azName == "" || state.AzName == azName) && - state.InstanceState == updateInitialized { - c.markUpdateInProgress(state.Id) - instanceId = state.Id - return false - } - return true - }) - return instanceId -} - -// updateInstanceState updates the state of the instance in cluster store -func (c *ClusterStateImpl) addInstanceState(instanceData *InstanceData) { - c.store.Store(instanceData.Id, instanceData) -} - -// updateInstanceState updates the state of the instance in cluster store -func (c *ClusterStateImpl) updateInstanceState(instanceId, instanceState string) { - if val, ok := c.store.Load(instanceId); ok { - instanceData, _ := val.(*InstanceData) - instanceData.InstanceState = instanceState - c.store.Store(instanceId, instanceData) - } -} - -// getInstanceState returns the state of the instance from cluster store -func (c *ClusterStateImpl) getInstanceState(instanceId string) string { - if val, ok := c.store.Load(instanceId); ok { - state, _ := val.(*InstanceData) - return state.InstanceState - } - return "" -} diff --git a/controllers/rollup_cluster_state_test.go b/controllers/rollup_cluster_state_test.go deleted file mode 100644 index 6e3f4c05..00000000 --- a/controllers/rollup_cluster_state_test.go +++ /dev/null @@ -1,157 +0,0 @@ -/* -Copyright 2019 Intuit, Inc.. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package controllers - -import ( - "testing" - - "github.com/aws/aws-sdk-go/service/autoscaling" - "github.com/onsi/gomega" -) - -var clusterState = NewClusterState() - -func TestMarkUpdateInProgress(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - populateClusterState() - mockNodeName := "instance-1" - clusterState.markUpdateInProgress(mockNodeName) - - g.Expect(clusterState.instanceUpdateInProgress(mockNodeName)).To(gomega.BeTrue()) -} - -func TestMarkUpdateCompleted(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - populateClusterState() - mockNodeName := "instance-1" - clusterState.markUpdateInProgress(mockNodeName) - clusterState.markUpdateCompleted(mockNodeName) - - g.Expect(clusterState.instanceUpdateCompleted(mockNodeName)).To(gomega.BeTrue()) -} - -func TestMarkUpdateInitialized(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - populateClusterState() - mockNodeName := "instance-1" - clusterState.markUpdateInitialized(mockNodeName) - - g.Expect(clusterState.instanceUpdateInitialized(mockNodeName)).To(gomega.BeTrue()) -} - -func TestInstanceUpdateInitialized(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - mockNodeName := "instance-3" - g.Expect(clusterState.instanceUpdateInitialized(mockNodeName)).To(gomega.BeFalse()) -} - -func TestInstanceUpdateInProgress(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - populateClusterState() - mockNodeName := "instance-2" - - g.Expect(clusterState.instanceUpdateInProgress(mockNodeName)).To(gomega.BeFalse()) -} - -func TestInstanceUpdateCompleted(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - populateClusterState() - mockNodeName := "instance-1" - - g.Expect(clusterState.instanceUpdateCompleted(mockNodeName)).To(gomega.BeFalse()) -} - -func TestUpdateInstanceState(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - populateClusterState() - mockNodeName := "instance-1" - mockInstanceState := "to-be-updated" - clusterState.updateInstanceState(mockNodeName, mockInstanceState) - - g.Expect(clusterState.getInstanceState(mockNodeName)).To(gomega.ContainSubstring(mockInstanceState)) -} - -func TestInitializeAsg(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - populateClusterState() - - instanceIds := []string{"instance-1", "instance-2"} - for _, instance := range instanceIds { - g.Expect(clusterState.instanceUpdateInitialized(instance)).To(gomega.BeTrue()) - } -} - -func TestDeleteEntryOfAsg(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - populateClusterState() - mockAsgName := "asg-1" - - g.Expect(clusterState.deleteAllInstancesInAsg(mockAsgName)).To(gomega.BeTrue()) - g.Expect(clusterState.deleteAllInstancesInAsg(mockAsgName)).To(gomega.BeFalse()) -} - -func TestInstanceStateUpdateSequence(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - populateClusterState() - mockAsgName := "asg-1" - mockNodeName := "instance-1" - clusterState.markUpdateInitialized(mockNodeName) - g.Expect(clusterState.instanceUpdateInitialized(mockNodeName)).To(gomega.BeTrue()) - clusterState.markUpdateInProgress(mockNodeName) - g.Expect(clusterState.instanceUpdateInProgress(mockNodeName)).To(gomega.BeTrue()) - clusterState.markUpdateCompleted(mockNodeName) - g.Expect(clusterState.instanceUpdateCompleted(mockNodeName)).To(gomega.BeTrue()) - - g.Expect(clusterState.deleteAllInstancesInAsg(mockAsgName)).To(gomega.BeTrue()) - g.Expect(clusterState.deleteAllInstancesInAsg(mockAsgName)).To(gomega.BeFalse()) -} - -func TestGetNextAvailableInstanceIdInAz(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - populateClusterState() - mockAsgName := "asg-1" - mockInstance1 := "instance-1" - mockInstance2 := "instance-2" - - clusterState.markUpdateInProgress(mockInstance1) - - g.Expect(clusterState.getNextAvailableInstanceIdInAz(mockAsgName, "az-1")).To(gomega.ContainSubstring(mockInstance2)) - - g.Expect(clusterState.getNextAvailableInstanceIdInAz(mockAsgName, "az-2")).To(gomega.ContainSubstring("")) -} - -func populateClusterState() { - asgName := "asg-1" - instance1 := "instance-1" - instance2 := "instance-2" - az := "az-1" - instances := []*autoscaling.Instance{} - instances = append(instances, &autoscaling.Instance{InstanceId: &instance1, AvailabilityZone: &az}) - instances = append(instances, &autoscaling.Instance{InstanceId: &instance2, AvailabilityZone: &az}) - clusterState.initializeAsg(asgName, instances) -} diff --git a/controllers/script_runner.go b/controllers/script_runner.go deleted file mode 100644 index aced7ede..00000000 --- a/controllers/script_runner.go +++ /dev/null @@ -1,184 +0,0 @@ -/* -Copyright 2019 Intuit, Inc.. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package controllers - -import ( - "fmt" - "github.com/go-logr/logr" - upgrademgrv1alpha1 "github.com/keikoproj/upgrade-manager/api/v1alpha1" - "os" - "os/exec" - "strings" -) - -const ( - // KubeCtlBinary is the path to the kubectl executable - KubeCtlBinary = "/usr/local/bin/kubectl" - // ShellBinary is the path to the shell executable - ShellBinary = "/bin/sh" -) - -type ScriptRunner struct { - Log logr.Logger - KubectlCall string -} - -func NewScriptRunner(logger logr.Logger) ScriptRunner { - return ScriptRunner{ - Log: logger, - KubectlCall: KubeCtlBinary, - } -} - -func (r *ScriptRunner) uncordonNode(nodeName string, ruObj *upgrademgrv1alpha1.RollingUpgrade) (string, error) { - script := fmt.Sprintf("%s uncordon %s", r.KubectlCall, nodeName) - return r.runScript(script, false, ruObj) -} - -func (r *ScriptRunner) drainNode(nodeName string, ruObj *upgrademgrv1alpha1.RollingUpgrade) (string, error) { - // kops behavior implements the same behavior by using these flags when draining nodes - // https://github.com/kubernetes/kops/blob/7a629c77431dda02d02aadf00beb0bed87518cbf/pkg/instancegroups/instancegroups.go lines 337-340 - script := fmt.Sprintf("%s drain %s --ignore-daemonsets=true --delete-local-data=true --force --grace-period=-1", r.KubectlCall, nodeName) - return r.runScript(script, false, ruObj) -} - -func (r *ScriptRunner) runScriptWithEnv(script string, background bool, ruObj *upgrademgrv1alpha1.RollingUpgrade, env []string) (string, error) { - r.info(ruObj, "Running script", "script", script) - command := exec.Command(ShellBinary, "-c", script) - command.Env = append(os.Environ(), env...) - - if background { - r.info(ruObj, "Running script in background. Logs not available.") - err := command.Run() - if err != nil { - r.info(ruObj, fmt.Sprintf("Script finished with error: %s", err)) - } - - return "", nil - } - - out, err := command.CombinedOutput() - if err != nil { - r.error(ruObj, err, "Script finished", "output", string(out)) - } else { - r.info(ruObj, "Script finished", "output", string(out)) - } - return string(out), err -} - -func (r *ScriptRunner) runScript(script string, background bool, ruObj *upgrademgrv1alpha1.RollingUpgrade) (string, error) { - return r.runScriptWithEnv(script, background, ruObj, nil) - -} - -// logger creates logger for rolling upgrade. -func (r *ScriptRunner) logger(ruObj *upgrademgrv1alpha1.RollingUpgrade) logr.Logger { - return r.Log.WithValues("rollingupgrade", ruObj.Name) -} - -// info logs message with Info level for the specified rolling upgrade. -func (r *ScriptRunner) info(ruObj *upgrademgrv1alpha1.RollingUpgrade, msg string, keysAndValues ...interface{}) { - r.logger(ruObj).Info(msg, keysAndValues...) -} - -// error logs message with Error level for the specified rolling upgrade. -func (r *ScriptRunner) error(ruObj *upgrademgrv1alpha1.RollingUpgrade, err error, msg string, keysAndValues ...interface{}) { - r.logger(ruObj).Error(err, msg, keysAndValues...) -} - -func (r *ScriptRunner) PostTerminate(instanceID string, nodeName string, ruObj *upgrademgrv1alpha1.RollingUpgrade) error { - if ruObj.Spec.PostTerminate.Script != "" { - - out, err := r.runScriptWithEnv(ruObj.Spec.PostTerminate.Script, false, ruObj, r.buildEnv(ruObj, instanceID, nodeName)) - if err != nil { - if strings.HasPrefix(out, "Error from server (NotFound)") { - r.error(ruObj, err, "Node not found when running postTerminate. Ignoring ...", "output", out, "instanceID", instanceID) - return nil - } - msg := "Failed to run postTerminate script" - r.error(ruObj, err, msg, "instanceID", instanceID) - return fmt.Errorf("%s: %s: %w", ruObj.NamespacedName(), msg, err) - } - } - return nil - -} - -func (r *ScriptRunner) PreDrain(instanceID string, nodeName string, ruObj *upgrademgrv1alpha1.RollingUpgrade) error { - if ruObj.Spec.PreDrain.Script != "" { - script := ruObj.Spec.PreDrain.Script - _, err := r.runScriptWithEnv(script, false, ruObj, r.buildEnv(ruObj, instanceID, nodeName)) - if err != nil { - msg := "Failed to run preDrain script" - r.error(ruObj, err, msg) - return fmt.Errorf("%s: %s: %w", ruObj.NamespacedName(), msg, err) - } - } - return nil - -} -func (r *ScriptRunner) PostWait(instanceID string, nodeName string, ruObj *upgrademgrv1alpha1.RollingUpgrade) error { - if ruObj.Spec.PostDrain.PostWaitScript != "" { - _, err := r.runScriptWithEnv(ruObj.Spec.PostDrain.PostWaitScript, false, ruObj, r.buildEnv(ruObj, instanceID, nodeName)) - if err != nil { - msg := "Failed to run postDrainWait script: " + err.Error() - r.error(ruObj, err, msg) - result := fmt.Errorf("%s: %s: %w", ruObj.NamespacedName(), msg, err) - - if !ruObj.Spec.IgnoreDrainFailures { - r.info(ruObj, "Uncordoning the node since it failed to run postDrainWait Script", "nodeName", nodeName) - _, err = r.uncordonNode(nodeName, ruObj) - if err != nil { - r.error(ruObj, err, "Failed to uncordon", "nodeName", nodeName) - } - } - - return result - } - } - return nil -} - -func (r *ScriptRunner) PostDrain(instanceID string, nodeName string, ruObj *upgrademgrv1alpha1.RollingUpgrade) error { - if ruObj.Spec.PostDrain.Script != "" { - _, err := r.runScriptWithEnv(ruObj.Spec.PostDrain.Script, false, ruObj, r.buildEnv(ruObj, instanceID, nodeName)) - if err != nil { - msg := "Failed to run postDrain script: " - r.error(ruObj, err, msg) - result := fmt.Errorf("%s: %s: %w", ruObj.NamespacedName(), msg, err) - - if !ruObj.Spec.IgnoreDrainFailures { - r.info(ruObj, "Uncordoning the node since it failed to run postDrain Script", "nodeName", nodeName) - _, err = r.uncordonNode(nodeName, ruObj) - if err != nil { - r.error(ruObj, err, "Failed to uncordon", "nodeName", nodeName) - } - } - - return result - } - } - return nil -} - -func (r *ScriptRunner) buildEnv(ruObj *upgrademgrv1alpha1.RollingUpgrade, instanceID string, nodeName string) []string { - return []string{ - fmt.Sprintf("%s=%s", asgNameKey, ruObj.Spec.AsgName), - fmt.Sprintf("%s=%s", instanceIDKey, instanceID), - fmt.Sprintf("%s=%s", instanceNameKey, nodeName), - } -} diff --git a/controllers/script_runner_test.go b/controllers/script_runner_test.go deleted file mode 100644 index a09f4330..00000000 --- a/controllers/script_runner_test.go +++ /dev/null @@ -1,52 +0,0 @@ -package controllers - -import ( - upgrademgrv1alpha1 "github.com/keikoproj/upgrade-manager/api/v1alpha1" - "github.com/onsi/gomega" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - runtimelog "sigs.k8s.io/controller-runtime/pkg/log" - "testing" -) - -func TestEchoScript(t *testing.T) { - g := gomega.NewGomegaWithT(t) - ru := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}} - r := &ScriptRunner{Log: runtimelog.NullLogger{}} - out, err := r.runScript("echo hello", false, ru) - - g.Expect(err).To(gomega.BeNil()) - g.Expect(out).To(gomega.Equal("hello\n")) -} - -func TestEchoScriptWithEnv(t *testing.T) { - g := gomega.NewGomegaWithT(t) - ru := &upgrademgrv1alpha1.RollingUpgrade{ - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{AsgName: "my-asg"}, - } - r := &ScriptRunner{Log: runtimelog.NullLogger{}} - env := r.buildEnv(ru, "testInstanceID", "testNodeName") - out, err := r.runScriptWithEnv("echo $INSTANCE_ID:$ASG_NAME:$INSTANCE_NAME", false, ru, env) - - g.Expect(err).To(gomega.BeNil()) - g.Expect(out).To(gomega.Equal("testInstanceID:my-asg:testNodeName\n")) -} - -func TestEchoBackgroundScript(t *testing.T) { - g := gomega.NewGomegaWithT(t) - ru := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}} - r := &ScriptRunner{Log: runtimelog.NullLogger{}} - out, err := r.runScript("echo background", true, ru) - - g.Expect(err).To(gomega.BeNil()) - g.Expect(out).To(gomega.Equal("")) -} - -func TestRunScriptFailure(t *testing.T) { - g := gomega.NewGomegaWithT(t) - ru := &upgrademgrv1alpha1.RollingUpgrade{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}} - r := &ScriptRunner{Log: runtimelog.NullLogger{}} - out, err := r.runScript("echo this will fail; exit 1", false, ru) - - g.Expect(err).To(gomega.Not(gomega.BeNil())) - g.Expect(out).To(gomega.Not(gomega.Equal(""))) -} diff --git a/controllers/suite_test.go b/controllers/suite_test.go deleted file mode 100644 index ff03687a..00000000 --- a/controllers/suite_test.go +++ /dev/null @@ -1,79 +0,0 @@ -/* - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package controllers - -import ( - "path/filepath" - "testing" - - . "github.com/onsi/ginkgo" - . "github.com/onsi/gomega" - - upgrademgrv1alpha1 "github.com/keikoproj/upgrade-manager/api/v1alpha1" - "k8s.io/client-go/kubernetes/scheme" - "k8s.io/client-go/rest" - "sigs.k8s.io/controller-runtime/pkg/client" - "sigs.k8s.io/controller-runtime/pkg/envtest" - logf "sigs.k8s.io/controller-runtime/pkg/log" - "sigs.k8s.io/controller-runtime/pkg/log/zap" - // +kubebuilder:scaffold:imports -) - -// These tests use Ginkgo (BDD-style Go testing framework). Refer to -// http://onsi.github.io/ginkgo/ to learn more about Ginkgo. - -var cfg *rest.Config -var k8sClient client.Client -var testEnv *envtest.Environment - -func TestAPIs(t *testing.T) { - RegisterFailHandler(Fail) - - RunSpecsWithDefaultAndCustomReporters(t, - "Controller Suite", - []Reporter{envtest.NewlineReporter{}}) -} - -var _ = BeforeSuite(func(done Done) { - logf.SetLogger(zap.New(zap.UseDevMode(true), zap.WriteTo(GinkgoWriter))) - - By("bootstrapping test environment") - testEnv = &envtest.Environment{ - CRDDirectoryPaths: []string{filepath.Join("..", "config", "crd", "bases")}, - } - - var err error - cfg, err = testEnv.Start() - Expect(err).ToNot(HaveOccurred()) - Expect(cfg).ToNot(BeNil()) - - err = upgrademgrv1alpha1.AddToScheme(scheme.Scheme) - Expect(err).NotTo(HaveOccurred()) - - // +kubebuilder:scaffold:scheme - - k8sClient, err = client.New(cfg, client.Options{Scheme: scheme.Scheme}) - Expect(err).ToNot(HaveOccurred()) - Expect(k8sClient).ToNot(BeNil()) - - close(done) -}, 60) - -var _ = AfterSuite(func() { - By("tearing down the test environment") - err := testEnv.Stop() - Expect(err).ToNot(HaveOccurred()) -}) diff --git a/controllers/uniform_across_az_node_selector.go b/controllers/uniform_across_az_node_selector.go deleted file mode 100644 index bf2640a1..00000000 --- a/controllers/uniform_across_az_node_selector.go +++ /dev/null @@ -1,61 +0,0 @@ -package controllers - -import ( - "github.com/aws/aws-sdk-go/service/autoscaling" - upgrademgrv1alpha1 "github.com/keikoproj/upgrade-manager/api/v1alpha1" - "log" -) - -type azNodesCountState struct { - TotalNodes int - MaxUnavailableNodes int -} - -type UniformAcrossAzNodeSelector struct { - azNodeCounts map[string]*azNodesCountState - ruObj *upgrademgrv1alpha1.RollingUpgrade - asg *autoscaling.Group -} - -func NewUniformAcrossAzNodeSelector(asg *autoscaling.Group, ruObj *upgrademgrv1alpha1.RollingUpgrade) *UniformAcrossAzNodeSelector { - - // find total number of nodes in each AZ - azNodeCounts := make(map[string]*azNodesCountState) - for _, instance := range asg.Instances { - if _, ok := azNodeCounts[*instance.AvailabilityZone]; ok { - azNodeCounts[*instance.AvailabilityZone].TotalNodes += 1 - } else { - azNodeCounts[*instance.AvailabilityZone] = &azNodesCountState{TotalNodes: 1} - } - } - - // find max unavailable for each az - for az, azNodeCount := range azNodeCounts { - azNodeCount.MaxUnavailableNodes = getMaxUnavailable(ruObj.Spec.Strategy, azNodeCount.TotalNodes) - log.Printf("Max unavailable calculated for %s, AZ %s is %d", ruObj.Name, az, azNodeCount.MaxUnavailableNodes) - } - - return &UniformAcrossAzNodeSelector{ - azNodeCounts: azNodeCounts, - ruObj: ruObj, - asg: asg, - } -} - -func (selector *UniformAcrossAzNodeSelector) SelectNodesForRestack(state ClusterState) []*autoscaling.Instance { - var instances []*autoscaling.Instance - - // Fetch instances to update from each instance group - for az, processedState := range selector.azNodeCounts { - // Collect the needed number of instances to update - instancesForUpdate := getNextSetOfAvailableInstancesInAz(selector.ruObj.Spec.AsgName, - az, processedState.MaxUnavailableNodes, selector.asg.Instances, state) - if instancesForUpdate == nil { - log.Printf("No instances available for update in AZ: %s for %s", az, selector.ruObj.Name) - } else { - instances = append(instances, instancesForUpdate...) - } - } - - return instances -} diff --git a/controllers/uniform_across_az_node_selector_test.go b/controllers/uniform_across_az_node_selector_test.go deleted file mode 100644 index 6bf6ff1a..00000000 --- a/controllers/uniform_across_az_node_selector_test.go +++ /dev/null @@ -1,109 +0,0 @@ -package controllers - -import ( - "github.com/aws/aws-sdk-go/service/autoscaling" - upgrademgrv1alpha1 "github.com/keikoproj/upgrade-manager/api/v1alpha1" - "github.com/onsi/gomega" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - "testing" -) - -func TestUniformAcrossAzNodeSelectorSelectNodes(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - someAsg := "some-asg" - mockID := "some-id" - someLaunchConfig := "some-launch-config" - diffLaunchConfig := "different-launch-config" - az := "az-1" - az2 := "az-2" - az1Instance1 := constructAutoScalingInstance(mockID+"1-"+az, diffLaunchConfig, az) - az2Instance1 := constructAutoScalingInstance(mockID+"1-"+az2, diffLaunchConfig, az2) - az2Instance2 := constructAutoScalingInstance(mockID+"2-"+az2, diffLaunchConfig, az2) - mockAsg := autoscaling.Group{AutoScalingGroupName: &someAsg, - LaunchConfigurationName: &someLaunchConfig, - Instances: []*autoscaling.Instance{ - az1Instance1, - az2Instance1, - az2Instance2, - }, - } - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ - ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{ - AsgName: someAsg, - Strategy: upgrademgrv1alpha1.UpdateStrategy{ - Type: upgrademgrv1alpha1.UniformAcrossAzUpdateStrategy, - }, - }, - } - - clusterState := NewClusterState() - clusterState.initializeAsg(*mockAsg.AutoScalingGroupName, mockAsg.Instances) - - nodeSelector := NewUniformAcrossAzNodeSelector(&mockAsg, ruObj) - instances := nodeSelector.SelectNodesForRestack(clusterState) - - g.Expect(2).To(gomega.Equal(len(instances))) - - // group instances by AZ - instancesByAz := make(map[string][]*autoscaling.Instance) - for _, instance := range instances { - az := instance.AvailabilityZone - if _, ok := instancesByAz[*az]; !ok { - instancesInAz := make([]*autoscaling.Instance, 0, len(instances)) - instancesByAz[*az] = instancesInAz - } - instancesByAz[*az] = append(instancesByAz[*az], instance) - } - - // assert on number of instances in each az - g.Expect(1).To(gomega.Equal(len(instancesByAz[az]))) - g.Expect(1).To(gomega.Equal(len(instancesByAz[az2]))) -} - -func TestUniformAcrossAzNodeSelectorSelectNodesOneAzComplete(t *testing.T) { - g := gomega.NewGomegaWithT(t) - - someAsg := "some-asg" - mockID := "some-id" - someLaunchConfig := "some-launch-config" - diffLaunchConfig := "different-launch-config" - az := "az-1" - az2 := "az-2" - az1Instance1 := constructAutoScalingInstance(mockID+"1-"+az, diffLaunchConfig, az) - az2Instance1 := constructAutoScalingInstance(mockID+"1-"+az2, diffLaunchConfig, az2) - az2Instance2 := constructAutoScalingInstance(mockID+"2-"+az2, diffLaunchConfig, az2) - mockAsg := autoscaling.Group{AutoScalingGroupName: &someAsg, - LaunchConfigurationName: &someLaunchConfig, - Instances: []*autoscaling.Instance{ - az1Instance1, - az2Instance1, - az2Instance2, - }, - } - - ruObj := &upgrademgrv1alpha1.RollingUpgrade{ - ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, - Spec: upgrademgrv1alpha1.RollingUpgradeSpec{ - AsgName: someAsg, - Strategy: upgrademgrv1alpha1.UpdateStrategy{ - Type: upgrademgrv1alpha1.UniformAcrossAzUpdateStrategy, - }, - }, - } - - clusterState := NewClusterState() - clusterState.initializeAsg(*mockAsg.AutoScalingGroupName, mockAsg.Instances) - clusterState.markUpdateInProgress(mockID + "1-" + az) - clusterState.markUpdateInProgress(mockID + "1-" + az2) - clusterState.markUpdateCompleted(mockID + "1-" + az) - clusterState.markUpdateCompleted(mockID + "1-" + az2) - - nodeSelector := NewUniformAcrossAzNodeSelector(&mockAsg, ruObj) - instances := nodeSelector.SelectNodesForRestack(clusterState) - - g.Expect(1).To(gomega.Equal(len(instances))) - g.Expect(&az2).To(gomega.Equal(instances[0].AvailabilityZone)) -} diff --git a/deploy/rolling-upgrade-controller-deploy.yaml b/deploy/rolling-upgrade-controller-deploy.yaml deleted file mode 100644 index 00233d71..00000000 --- a/deploy/rolling-upgrade-controller-deploy.yaml +++ /dev/null @@ -1,67 +0,0 @@ -apiVersion: v1 -kind: ServiceAccount -metadata: - name: rolling-upgrade-sa - namespace: kube-system ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRoleBinding -metadata: - name: rolling-upgrade-sa-role -roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: cluster-admin -subjects: -- kind: ServiceAccount - name: rolling-upgrade-sa - namespace: kube-system ---- -apiVersion: apps/v1 -kind: Deployment -metadata: - labels: - app: rolling-upgrade-controller - name: rolling-upgrade-controller - namespace: kube-system -spec: - replicas: 1 - selector: - matchLabels: - app: rolling-upgrade-controller - strategy: - rollingUpdate: - maxSurge: 1 - maxUnavailable: 1 - type: RollingUpdate - template: - metadata: - creationTimestamp: null - labels: - app: rolling-upgrade-controller - spec: - containers: - - image: keikoproj/rolling-upgrade-controller:latest - imagePullPolicy: Always - name: rolling-upgrade-controller - resources: - limits: - cpu: 100m - memory: 300Mi - requests: - cpu: 100m - memory: 300Mi - terminationMessagePath: /dev/termination-log - terminationMessagePolicy: File - dnsPolicy: ClusterFirst - nodeSelector: - kubernetes.io/role: master - tolerations: - - effect: NoSchedule - key: node-role.kubernetes.io/master - restartPolicy: Always - schedulerName: default-scheduler - securityContext: {} - serviceAccount: rolling-upgrade-sa - serviceAccountName: rolling-upgrade-sa - terminationGracePeriodSeconds: 30 diff --git a/docs/RollingUpgradeDesign.png b/docs/RollingUpgradeDesign.png deleted file mode 100644 index a6386e86e3183acc772f555c8e36cc49572556ae..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 229125 zcmeFZWmJ`2*ES3YNGghefJiAIDU#CNjeKF<}i)||&N*Z!cUB1?!zfro~MMkp^QrGbWaO&tyG z7B>zy_)q@D!%Q?Z0z>y_FC491W$jgX$RB20fo6U_<7 zU58sI*G+8i)YHIkinq|3V_$T?*6}&J-@eP3oUx?JbIvO*J$5ynebZ&MYVaoAZy%b0 z18QM~d~d4Qz3V38I*9Q?tfn-Idgbf&o;%G0^N*2yyM{ZRp~i_`6)y_Lu46sCN^8BA z*;#yv6zAWW)#|Od_DJ!|BKeP){Q*CUsDLXmAGM^+%Yi$&F6lyF3a;MovGVF>{B|kw z+o{@(RmTs!ddA6WaHMl_G+E)67j#)oNg$l|2iEN#HVJe3vibazaAoe!J9`Tf5{D5# z_R%JWi|JIphv$mK=_c~neo>57S6GOQEkh#w_$`hTvVikutBpp8@z&BhT`uuo4tM!E0R>SRzM|i zD}m(KhH(b=_9&UN_=>Q>zVzC!tkngmG4<2=5-{W<74LKJtDN1-3Ui&P z%@2FkMai@pohgqLLvMJJF{>0Dk0$z;jc>o^dx_?Myh(w{kep_@wek7+| z4=iA^As)Fm(^L#oOa1k5cG}oMKFBxxB5_3bh)Z)q{k7Q{gqo$vxh9vof(bM1oe3(_IY_c-8x3lU zuo#rY3j#e;e5>fnLX1(|hI|I|v`+HZ z?t3XS7kR~k-cn9`<9wbcf_oI26q9)|Sg{-zuDi#FOkN+JA~QU-?o{9Bcp+(6)B8kn z52aQ-1=;nuoc(t|Uk}Wzhh?61*(XC1c z@HcI;PixWZ8|v7SJ!WEUUZu7eKhzN4u3X{a_%(*aH)X118q8k^!wWI@>3h!f*w>FK z{K;%IN%m=HYY}?Bfj@5_Hs74_u(2I>6fg1h0+%E7{SR-@I8MWO(gz1S{#@5?duBGG zaeLI*lo&U`vK4*Kwj!hVx&O>tn7OhTJ5gPO^JyXB&BIEgouw2~LCu|nKAX*Wm`fT) z%}PSVrA%dn!7X}2*Y1#G3V&zv0wuR-eQ6vz?AgG+8Zx1k?#@`wsnj3p4#X;@Rpyy| zQ=C7Dg?SwEU`_JI9}dNIBr95?4xWum{^+TH!x)HhMc_Qq=3?Rm-_UXTQ?`knk<{ZM z$r>BV<{d+$HR&>Qhne#XZaWd1V&0Q&mumm^;v$WV_h{B=O33Z_{;aLylNSs?SdWIm3I$tW-?aNOw8_attBv7 z>5p)14HfuLO(rM#>b2U?KKQaPWT+>US>W*$_mK^$zY^d3*LP;!Pgv_RgbXj9Wj_8v zQfA+K^E{HM^ycBw!CMAB8W!3EUs@blx(bFu$`(m#Y!XST;a>=Ef0~&*g4^!0WL+)A z2NS*{LEhgDUYxu)4C}Y$`}0_f8tV?d$9?(>BAY#Y1b;WhJeOPQ=vKFM=DEOx07S?k>u-)Q)Gq;w5I= z=$1LmttFA@{^oMG_jZ-i>Rg>T3zE*cVJ?sSkBHp1!sBIPekLjSf#aI1-;IxtmaL_o zthz|O#kgzy_C=xOnF--(Y>tNW@Z!DYqHyZ@JGYd-w8q%ikaXu#Q=1I9yr0IW8=c*_ zeofaoHg(z9&pdUq)eL)BFF?u9z-WA=)i?9?>o>AwCO)j0qSsG+7Z9c0(w?8{p5mdvDyLRj^s_6vCEcxO%>n-DXu@Wjc0Z{T zILa}9@_+t?I92SHZ8Zlum%{x^=e_vReLL(^WjuMFg91JWh$7 zq=EbG^&E}xgzDp!2fZ!5^+ilU_Gi{6U0b_qF{+Z@1KS5IE%LR-5nTgGGOi(5=3IW+ zx5|ZAs65aM<}((%au!Mr$_T5abcIRtF+d&Ie{-a;CMp^RZXo->^;i*@$Hu8vP zYjDdhy8S?>{{8zvaS7v_q&3?M>;vJq%UNjC8Rb9I|B*M?k^cOHA#(Y39xmZC&F>3r zwLFUFz5}thR;Bxs^Ru77mG1U7<*yj2YN2@Ra@OTMH^W&N7c67+Z0Bjr0c(eENkq?lTXX90f>bLL%sMzyD~`*26F=(i z6rz_EHD+uS6k%GhSQ6b=)Yh5HC4J8EC>8JF7yj$WFH{_T69dy+YhPYXo!fUD?&!6C z^Nrol%-Z&+6HT7%INH&cD#iI7^Ll*h0ok{(PSH@Fo8J*}!>1D&GsbZ^pD0|85IK;w5YHVC(M4m{z%f7Zqx-}3%sZQ3%Fk7*ChqPAcO?1;N z&$G|`Uul-MI?r!ENHLycc~rPWgRboEF+?0`mb#LEJd2zB>cMkGSEBIW)f}x{9Wyp& zexW4zD{46G{k@G_A6s@Bm+$LPaS^>L$e2X5-R7$*=y{mhw5+81Oe^3I}DwAkxq##xQ6xRW}15N?_+r+!C}PASjm z)cp*Gfb0-+tGdId;eo?-jWyF*?-M3xir!t@9SNPKUvkXfimwS|zWa=&Ak^`UQ7OADRKV6o8!KuT{x=;G{x~9<$*qck%PlK+~k^X_(GE&X@F&D)a&$BMq_IiKI znI;NUROtNlnsLa^2S1Y%=roK0SG>5KcVd%%^+AJz!u zL_P>OaP GQ3Z4FEZIm;N)V~tT=5#a4I^mg<~j^|po(pUBHEA9MYwwS8js1ocaYr1tVKI* z@|I0KPtA7wk|0!(*4vnwxF^0CGS0$@X zxm1e#+M`L=rLg6e>(mF{J;j8!UD1#Bj;3MVDuSDDyiI%J7pI8e9N?`;TtD@IT&0l& zaXMeIAgrzWeU$`W>n}Y?Pc^INy*(0y+{#a1S7?e!Zhcl2d8DtepWfoK_#@DMK>YgT zq>g(a8V1grXTC%+gFm?5ViI)(HP@`Buyn_=eb$9FJZ}pn{~RXn!QJ(|K&$ihdn(f( z+tOtEe+E15;(@6bs?cb=Hj#rKm_%Y}X};LGgBh~jS#bs*KP_RbCOL%Tw#4b#e^{db zdIoKd8Y6&7bghGlNDR%72lIfmacoShW+_}Sa#uEkLgNmR&`4#i_Yd9zi|@d4^!l25v0+x< zCS*T(KslnFIzBe`<*;!AYz;h03?4P>kAfb3ftjt6%p5|-5uRmDp={`j%r?B{?h8at zl+J%0db%G|CXC|Y#l=NR*vG{`n7JIQ7?4gI(P)`3C;c5S^HEd!Fo5E+>HUlT|%iv@p-9QDrp!F5Q#==o2XBCi? zK|^oC`reV<=#Bqa09uw87!Jvkb{(izEE9-R?rD)#DDtKDZGiwZ4DvfPQYNKv0&*Z< z(gnIiB$BNe4k`Ws42Kx^y-x##5QRsSPWOMx#_sEQX&38Ri16FYe2?N$ThN3x9L5&D ztW*#=DvRj!*lu84FKKFgTi(e)FM@qY#Ysgi1(_SrwX}>p5U;N<6~=3h@;DAd_UX43 z$}N4{Rns;anws>Csh;K0(a{r}9l;)UtH~~1U*F%ieCc(*+io(h?-FCu9ov~H|G6hs zc(;|fe7{fdy~x^rzbJ>{`ERn`)q}h5@7V{RUa5*+eA_7fdE@K!@Xdgt?cf_f-lJ@7 z?2%aTZ%lEp5+Pz7kEmp7(lP0;ei9s)?Ge=53Gp2qKBtOLSK^KFf^zjr@wj#(M38bv zvo3~iJ1y6jL%I4Q3yNzMOnbjPgdBAPc>YR%dnh?^wtNL_0vRNNMhzJ>(OoTHA|BRT zf{@Z|a041T?%zxA6WkWk0W#`IGjvBhJRL3Z=0Qt)y_A~~Zjb*!3>L?(&%C+0xl&a} zAI>i!ayHpt%t_4iiO6Z*>$ubVY|||Ka{lUK-c*alXH3;B`-aHL0$PKvLtN=0p%CYS?Hvp4$U)EnX+* z^3wZsg|F$6%Z%N7+~*W?2jqE=b%B^I61_mIK*#jQB}(^yT1E!;z7Q%l5wfu;@l0Dg z?rrs~v*|gRzGIZC3?v1LMqXg0&Fwec1q}!6Rz>l7T(EH(Ga*z5{kpok%4l6TsjFw~L!O3TZZE#477^Q>-S*ao z4zs6Q-h}*nt6eYVT2!yjNpV+jp0R89Yh1ow zSWwb970~|wE#@?i*nv`eL{?naM-rAdar(}j6>9E+DUEN9eiHI=SMMXN-I1#lZQS>g z-BuFz_EWEBCJ@#x?pBG7wu>x=rypxZmCeE&uMopIiMCm~W@b4Xjb~dcSL>l#TScz! z1q;^Iv!>qmevDT~KhF6!^c{M67+T>3W47(%Y7SQy$2Td{9eLT|-+5duiAHuU&cMlb zkKf0US8@I_r-hp@dxpzgT%{a1#t_39ib{78YLj zd{~QN;X0r@>KMyRmVe;m#H+yP<@c_S8H?+PcFf|C<>6iHThv`acN=m+i zvkt4#DoExANl}G}SY`LeARLUutHC0t*w1L8MPh=p2&>vi6`XIWtqTfnwY{LPrzE*Ju?C5k(v}ch?@`opQZ+Cm9&Tpvs z?N4FzIbWhS*_bF>sp3msrmz(CVo*$FbYBYN!=5wE>DR}?RY%4)(zG`rvJ-;{Oi#bE z_Iw96tN0Y8F};dU?5lZJnu;Eaa@z~0CSTncWC3D$#g8sAVk>bj4EcAw7$>a)$J$|R z!})ago?&(xm-zDX^0f*{h2fzgBZJ>N9rO_$wD0d3eRCX>n~%)lNHw>#eBzVcB5M|W zx$|C>U?Wb~@rY6M^h1)CWmLZUrRf2ut}CPfrKjcNM6p>~ES zdN(>cIy64c=&@1O9$BI^-j7gC5Ri#iDY##OJh!;q`OnvHeMC z)gy&iZ$ESvoZU0ak7d=Nn@5gSkKu?Qf8!87INNTzLW^e)KRYb(@mkh=c9M8jwSNlr`A z1Tk2?tpp7sWCMiZ-ugMch%<0Fxp&0LdY&x4Pv^)RsNO?Vfm5S90GBxEU}QDl(G)mq z!n>d_qw2>XHWvadguMp&wH(e?^np*CJ@cuZ6M+OZV z83G;C+y{xAQ1Q2qK*gC1&C<@62Ub-4$AvLL+1Ky8iZvofOTMVZ_|(0gnA4aqHQI;( z&q;BB>W6q@rH$O=_re8S$X`~PUVJS^$8_WaDK%Db*-UBBCAKcX&6SWhbQ7M2Hl z!FdE_(&->DNW&CDpyMPTL+?#a7xh_79&Dra9;EL7nQGWZZE7|AIEqCEkr?|xYjp=} zJ;?qR46T(JtTq3+^5x5*`2=B9 zLTLofmR~Z8KOyZ1V2tZIgh9m=d(-)zqd>E|F){yoVa1!FOnLh$i=1E11QaMAy=wsS z!}?nuknQ*3hEnpW%(<;nkqbH(I(X&$l*F$;p7SW5G)?TR+it|a1+7i23#=^~zj+bb z96zS_+4Ovu)3`pP%dC_3z20F{JPZ&;?2jhR6rVqTjwibs3En}&I0Y4C-*sRI&6<8X z6{el}&kNoFKS&9GEckP1=UWRlWk&>v+ZzO5qUgZmr1(VXtn4@Rq0;Ch3IEmkz6oL| z%YMD6y6a?jDa1OwpOH1KdWXt+;&u6Yaoz2-+f}Q!kl<>8w3Dd8U%smtF7sa1N{PJ6 zDo>a1OoGJK1f=k!UW5XP$rwlq2d}_W5HndER{LVWv1*T>nw*uc4RrJx*lAEm|G+IV z8eI(F{Y5qWIa& zJ=lQ(ZYTwh0`lyvfr9uQl;snZ>iQCyxOl;-P#S&=1fp^ayoKhM32a|J#6bx4K@NO~ zvLCvO5;YoF3)%zczg`EWvI)!@P7c0VC;-*_c-IqlXjXbKYv`RXuTgRXc_=G0wm#|& z9N_ABur@}Nb$kTYL9*JO1GI*YsX+>=O+B<5P(b)cNh;RN2ogrs(KoD+04c!=`PQU3 zP;cNtWiJnEq)*`GhirYjs1J0YxR&mcH%2KL9S9#iasmr%R;<|o4AoHi)eIV{1Po=! z()}IUZ73M&iH3O)2y{3vzzI>Fd)E>522DLEVC%_KQFRNGiZ9XgB_SGS?oR zUkGgdAGI(VdMcPu{L%!mw^Y)~GKG2we4_#ObJ7DYoEW$B8Z_`MfK2vV<2V4J6D;8{@ zp>hBvK4NLUXtWu>da*{l0e-^+5xC$V}o60x!@IiGj7Feb7hAeFMmSQkT*l z`sM`S1$&j{q4?mHj$j}}`asioNbJ%;>>(e;wNY>U050_-aTgcNf#&xc2=3SEt_A8t zG}!thGT)z&+5V&YC)$w*IF2R{IX;+tAqZ?eOkhwE^#%rHyo9Jf|7*PeD~)#%iXWR{7QC+0A(()VToac?ppJ;w+y06_wmEmf9oY zIE~vnK=Shkq-0PP`)tnV$_uKe3-xNAG=$p$gbUc266Eaarr40PpFFhP{Y9n-DiDCp zSiX8?+#Vj&)+Up{V-;)o&V(BkLB>G*DrCW>f;^81hc2x`jYX^Q?u66KcRQyTTZ`9*z!3x)IW{D+pW1z0ayo~7{NgDHrCb(BnY|` z!`Zyvq85qs0QjzgM1nCCc5w0FweoMh@ehuV&xFNf-h++4UN%UoJ$76D{D5&yT?wF?wv_QLi zqRcp}x>{R7L1An{b$ZewC;1PA37vx)JsL!{nQzq~2qx}0_B~yd)7MW8mY z$Q5pLLYIwis-oI!iB-hyP;K-Zz~;a56sMu6frr7a_onG+%4#o9)l1>N&$-EstjW_kHgKxr7? zoNHQGqD`$l1Z?yRA1C3ytl0!G^rvZ9lRh9yP#84?T3Pk@BZ_K6#3SXDsK4UgRab?rO@~2C(sy z7A3PJJZk~stk)Zb6@8xox7oTiWK=<3L7L1zBs%fdSf(n__3Q>36)SwLc&kUU`CG~W=)|D~10-D)M zQ9y@=`2__gb(>XOTsrSytHZvT)%(fbNCcMR6NeS08su8RIMRdAY8}szcVr?EIkg}~ z4Nb`p(j2?BAzmLy;Os{T=RVUos1* zu?pS7^D8PGip+AjX{^hBk(z2f@)zQLHPciaDyUr=ns zz#QlhD+xJ8F$jlx1$Qr`fsF*+vZa+eJUED-d`Kvce5+wIOeO;%_x6BM=mt7uIoqg+ z5CSliX_dtRd~hYq6QZhmH=QZcgT8#I1q?%Pg^GK(;U()RK>g`}ZDa`T_i)*FJ<2KM z(LMZu&k=9DZ-jeEg35L8f)tu3iO2^^^7$DSB!Y)lleyt=2Neo;2GtZ+Diqc#7Jlzz|^kdS{ni;%>+rf0(vq^40d?miP9a4MeTj^wPth+g}7Ia8(i}#v64d+dAgs?@-^-fx&gdpc{$F| zU_G{RxjU{KqF_vu?Gv%tFG6jA#VxX2)jT4!BOxIf8XBUzPVH%%^pH87$gqboC=7+ALSHMg$?f|6aNK7gfcd@sz8GXhsGS<;ezsZNTjU1v8pEaS!!U2; zz2{#yw|%ba;B5Z23?mEuK~9w>KI3avA(R(TRgn(7T{=%*7kIlb7C5JM)dKF;iFp~f zWPPCqa<=^ffOcD4Dt zb{0^nzG9^6aH5AZsCP0!={uS>49UKH=G||lf9yaNT z=X~<|2d3M8FQ3V^45obi#8vIqk>w z7<$TY1U!`&=X(Rf25>^9Xnc4->f}b@?2)6sh5BY>nA=yO@h|~y(JRA=ALITA4lOA1 z5jbD7yLw+iD{8`Ye1zX{}K6j)@(F2Gjx=ULpI0-c#VcphWjs8;-2mqcPKQAZ}Tg3i94&9(4W2)7+G0aL!+Y) zq4a5bdVPZ*&X4*+7u5dMYtk^3a4sK~j%w+48b=5-n3$Lh4G+@;g7E!0VCR1Q3yzrp zn=T`EKtt^@7t3+Q0zgO_1_mMnjxH|M_qac{tBF;~^ zAxa5s28fpbCQ7hLM7eo+oYm`ye%r*kwH#4vYwMqzUGx9aJHUfUzJ%UHDd+)Pnw2=h z+Ox8<((mT%)Rd*ox|M|m?m4+L3Pu*Af#B5O+kcVwnG9-%h8#!+k%lmijqK_}1m)S6 z#KfQ8(SO+x5fjKme*b|qOH`_rmX{|F(EqHmGP_o2WbpNME_}q%(NQ*UG+;}Hrk41Up^O-@5;+Df;Q^%P}mcY$60E4TA) z9>Cxg35i~*&`DxMt-}UXW;6L@4Nzs4L{@rQaNyMc31r4J??ApvK$=Vo8I1+* z_fiqfPwUUKo?j1#DjOO#AjW)tQ4uJ21!-eiZS53mw(j%iBs;{2&9TA3*Q86>Um=>rR}_PXhsTst|H95>IMDSlr>LkmhJ6_7wm|a>0A(?*xWip& z1qB#RPEMwrz$#y^;=I(=?KP@rV`J+(Oum6K(`dkvd4cp z?{6TnUz9^oC!=pO4LY?M%rpM^k`yE(qF@=|IbrlgD9<@pY+hXr*7I%*A-;o}L;y_U z82b&fhBpQ@_^@vdyUOnpL^;tRgQf$Vku-sV1Eex7tnzPFpA@YOOb$KY%s~H100zS%Es49UhuBL#yG&@bdC9rJbFdn=_|y{2l{Y{aAEDQNcTsz#U|%$g!Z2 z>(JKeuo(M(*Xp!5{nf7kBFeHiAvvlUv`);)sZvL&IzONJ$oSOWH$41C#a)<(M;(Wx z(LI#-t^gTj$Bfctp=Mfkb{lJUf#};sOh;OS3kxeN{A2MWln_fnyfkrggWm&_Miz?! zAPfv!RmD3S&hL2u`hdZ(ii#mWGt$;k!~RY5qWRHct*x*38r8Z}QtR4b062%eZF*)# zPfzkMnyeWQI0cDG8Zmg!S06)JSs4f&5CU}qE4Hu^4YEJh=H|ApcO0ou+5>i(1U0Op zVOZfjwz@r!+p=NcS@jJ%(;k3&T0tYx7|I7?@psNu&ar+!tvCsL3Ry8xHBhb1>nSMO z+`F})Y~M-pk?%f(S#K2NZ~VZV>~e=7a&I9X&5Gio;p4G6g}Eep=)_S6%1Hl|Nd+lG z{oj$2kWaT91BrC?!Y(Q=e=>Q^fg5~{=9dlzlHT2Y4Lxi9cc_e;zGnCR7#yyq;&Xd@ z_U~oI$02up3kEKK4wHdG2?5Rl0E7Y^oi6CA9I+gQ*^_ZuSy&XS>F=TTO%n;0Ped#T z?adqa;P5aZ(9y%g@raw zW8_z{`v4_)APo%m(E*|vJL1u}?jMmRkxO`3N%lh<@Px7gsQn_67Coh*tZ721z|?0v5?x_+>>&(ze^%ssS*zD(ani%-2Uzrv=)7^3Cl(_YY&O!M>a7&%42a>Dr^ zd!)W*bY~8Kr`EekQOQdhByOooD87Xm9N{q1;nC@niF2ZXX{00P96*{yv0ItyL9_Gp z`X1ilL#b1*ATH1--H|D#srgw)uDYy@^YhZN+rUfB23HGn^BuSN(B5kRJrHL4Ef6H; zg$qVsM2*ha{5J6Fp;_$U6~)plYDO^nHhLd}slB9h5YDJU?B4PngA&YIquEiNw3yxF_UN}T)eLtb}`tM>DA zhyyAX`j2#Ww9@_+P_fHs(=ck;Om&{XmsYscC^|sXh%X+CHs_5&-!1jMgv;NM4sDVK&Pr10Aw-jj9md3HYPwKbY zESm=;;Wr%34w?Ctp>G0qU(h0EeBM2k;3icG-;j+5(Zhn+V4+zqM<^PSCKIi{J^-_O zXG@4HZt=J76Gs62Jd#qHO=}FujaOm&0-bK@Bnz8XgYKqZdFJNg1DU4baagWiJ*Et%Frb38Jbew8BstRdtxx7c-PNx&eGKX#e1grcVx^eE% zr{V9ese(RS&fd&R9YU1?t)fq-`>Oyj-XjA&Jl11)>H?ZsSAFpLRM1w8YMX5rz1$r+ zOR2m8P)e6ykYu;Hk{~9s0>iy5w|aMUY)n89Tfinx%*v`jC$u^%>sN7jAiVdM>^<7~ zv$g3ox0FqZ&s$YUSBE-a@ ztXrf?gv3|A&G)r^y;H(RN%26UG(B#@>Q@3eqqdKCKmwI)+`os(A=m!4# z=U`YhuEhJWz&E6pZ~A(G)H0-_up32a>3L~-t4)lPVIFNRb#RZ zn#)nd5@>!@WCWei!Zi#2_)45m%e#XIHJ5s4YNFm7Mp(+9TR;)ao%>;^BMl47)72nw zulC`>ZwD99z+K71@3k9Zcn?RtUuZw{p65=>-h0vWmh+mqs*}&D=dBxY9uI4EpW9bg z1Nc+7jz%~KSTM~hE~-z0u)=~bddr`4s2c;I$ePFF8cDLk6UWZ*jmMUkCxiP@V~p9*PVVRodk^9 zLqImByan`1eGJE?rg;awZPbM-=|#%Ww&wiY4S6u%Bd-|&>1+9Np zfgb*SHE$bKUwW*rP1FS_^(VH30gs}grNscKK0T}Xn<6J-?2RWz`?ZLf?DZ2p(5Pm5 zkplMCQZH`3lg%+X5QpQrVN9*NzD>%m#{}36$`c3u&=ZK_g6mGdos9(-L#|UfMndT( z`y62MR#L4e4xErkB?A+aEW+8ZH*esuzWgQ!u6Q;13oHMo(>1(T4h%D7t}USnfqW%; zzIfB5o2^^4tpOkw=Ft6^J+ANq}_dItfoUVWZd5~^6J&1oNas1j4&FM0Sj}Fu@vb0+_ zf6*7J3KOSfcFnd^fwMFpjt%YY?Q!4e=cFPYn0%A`e6(74P51V7n7rPb)w#x8Kl!E% z$86m*$IH{z#BD4uL~tJl$rPR!O!_%q!^Jb$!;8z~Cc{$s3meyiLD^oH$!?A&R*%if zxaafufdVN~UK#n4dYgZz$Z5V}s;Ut>a@ynTWv{#S7uG|tRYXqz@YH6u$oVL`gD&PT zqT?Y~(A_WwoC~ksg(^7veO z67xGzuJs@OQ1LOo1dZeI^N-hm&4burJ*&Iq==CI-d{jr87jx|4W7wR~)6H+gQZV$s zh9{z;N)PR#imvV>nj)&_+zXUagxu2VC=?Fj?m+jAAYRL_vHHNnEq6N%=D9@q#A=Ln zUyT1vSdC2iO+d)fB0gLFF9}6>tOyk5!R@4LujX&r;+lYVyf5%*^uS( z5Pq!2E`>_dG2u70zY*U@%*bc| zUxf^kbwn)@`(cCa$#Ci=_haJ2rmM5272dlIr2_-8P0Oa~j$@#WweM51^OU^EaZNvA zy_a&t=~_WamrK)`{(jkvXn#EC&jZ7VEG})Nzy`SdHQw_I!8*7gI*$LC@Fg*W(&OqS zxY8GcTyvUd1@sX@_B3gm(B-M>IpoYwHJ`?Qkuw4UaeW@lIlFeMaz}*5`W!_L7Xii2 z{c!$!^FpQZ4aMHP`T*pChDXrT^n4`F(EXmExAlz9+14r zb7ql?yx8`ErQnWkD_kspzAR_Fa&FWM#+yn7H)ndzB`kBd^He>56PN80r+-i8sQMfd z1XUGZp$m(a7=g-(m+giedxw(`E{ZH{tf?BZRQMViRLy)2NQJ!A3}MNG%I#k(v<2uL zf8a89xUD9eVJKf_bBVb7L04MH>WgF~N5HkVF}lH5Ib`=iM?vRHpPc?Da4xkeXW#%t zR`J95jL8XI!tWx_%r2C7z;!9?wmc5%0o|x@!7cuM-QP1O4nD=zvj|wVq1!SEv8Pj; zif9SC9L!~8USXXr0;&7zVgtfyJ3l>?ysYsSslHhCxy-5Q7>*=el~&fK*N@MO z`2!DL@aK>W7rAYGo7OVdEd5)*Kt_IbM>1f%fq8?t0pDzeZiaX*vlx=UXx3@+Ihyl; z5zjdC>iX1t0~b__Te21zjvI2D&QH(guh<2*>nq%7!!I>tVd5i(QixT@75R5M-oX@W z9&c@Z?B`P@y;Kn{E_#%-ABjlDYozNy&H&DG-aaXWPp;@`r)sfXck(5vwogT2b&=4) zfFy$yqA;Rw?hmX+T18d(@Y(+D;Ff-9+(k`7f>T(|kl0>`;rS0tZeYJ5cL-@}%+ehh zq!y@TO{VY$4dg{=l6zxq7*A?S2hc|5_xAW&3DgSs_*#>uc!OV5tyVJa7&EbOy#d`3; zRdR>wJ1dtW^XxwQT*(PT35*xNG;HctzF1+r&y~2)89$D`{KQfmMkF2laeXEo0xHtd ze67KrVH$_O7cxJqJgHzhO%z#5t1dI&88GuckdRJ&H$arRDC<*~U;Ts70F!52W^1et zT=lXHn;%-^!gp8fffCd~HSQ5EbtL5<(1_578GWwoQ&&*k^vAUIVB67Dj(TBCErTO} zz?r8W+=k+`FdxFFZlLKBR?PC#l1!ibkz1kl?n=+6yU=R{Ew{SLf>I#7-}|Uy#YkE5 zYvWTj(VWeEKMKOb)fyd6r(vEjZd*&DZ`Ml*Ow-IpFduuc{HMwIsx@Yw-^RgFMTYm7 z;u$wT;%!Zz%JtK^p~xpA$1=}o6`J}-6q+I&o%LaaVW!#W2KP= zGdylqi|-WY&W$sPn1>{b$evGqTjtb`*mBbY_&JjmX5G}6JJm21@_@Y@KalBSYly5KZrv*0 zRDB?5o4ZYzj2mIZpkNkw;34C4nlbN()X(KIM=It?RPSRUBD}A!Zw3I#{W{I=eXwSA z)?^yMG|eJ)1wa&C^u)0bL!}udeEmKj-ifHDUELZtLw5vY6>^`z*`CxQ=B*+Jb$7TU z;YG|ft3-Pt*|6`!A*>e>o+o4}T>tI`uo*V?z{YIIU69k$IXF&!5jxBcw#+FMM(NgrFvF0gnI5u_%{jUKjMikp-R44Vc9q%=mrE0B7{boEZTJ&0W^)!N#=#aPm+^p;%UO=XSXlQrYCwm}uJrQr-`^cr#E7$b4*MS1uM{eAR@^w0Y~51VZ*JJ7lB8k z+jlQ+jBKkM4(c2b@=m0^8h(w`gnKZ5^u9S7WPKQBm9%adM-&qz&<;^Adzc58UKfMo z%jL9@nZ9%Rc_AX}Hz`5?254^0%J#BWH#A^Ccs~7Q12O^}3_BdqCY*5SaiC``Ny84%kfl$R8Tz2O zEMHQ}B!h~}6toJ=q#Jg)kEh82XLSR@leQiF!PMs>KcRM8ewOBa=Dygzsqp=o(Phu?cbzx0cc0x6il;sMqmat-`1m9o$wp} z7`gsx`sJIvcLRP>06S3$cv|@m08P&<8L3g!i9a6PD)&$)7ThyVqkb)l8TxNnT_lyz z?*Vl0OGEI~7)<569P3Y2reF&*`(lhv76SVD@`U+!uA8xjgHqm2v#1-L~nql*$$fN#BB zl7nP~2{Dc8N5TWnD?_Cf+i0z zPa?G4kJ#@e?Eeg8KAo5ClWek5`cDa3f9F+pOLw`<6DCt;`q!CJI1prpHu#$Xg3Nvo ze=dd~vp_7@mB0$)QndaaJC%tC00C1T4Q}Ls5U>j!Y1EZA353?u$9enx(p{VY zV0MDs37XUZ++xa6c_DxTnHkdnU1?O`+uI8fp@RQFL5WJAgKjKc7GmWlATH)FTpzT6 z%+W(IDJUtWuda;`47^e=fh*jB@cE(nd5)LHyqIqx(_UG-GtURq#C*w`js?YGC47klw6?y5B7}Sk0Je^XD<0d`q#h^;`YKOC-~Ly_zPC2bbneoX2ptvWu-&ztAD;usc{gHqXUq=RQyvM0JFb}Lo3OtEVy4Yr$#2YE>Qxltp%*J4}}@N!{~9H&~`M%*(|^n zhUiW_KqBz-K@WxE?QDbz2d?Vycvy<7LjR!wqtYmByo0cHW&qj`aj?Ram71^*zxc#+yUejvrI%kgsrCF{EbTrlSf_r%q=eF4MAu< ze?B>{8oSJB9ADjOT;07+H7TLwAw|LABA zR!%D`Q+He(4+;vJ2#=}^O8oK#>QolFyeVtb&IsZkdLp2uc(va* zLJNw;Da}?tktjXVa^(_%*!l0^>X?0&#sPGhSWZXGu)C=L=$V#Q_6#w&S~EFmG1r_^ z5@Lpu(GDadLuGR4h9!^@xF@U=SzTW*RAu~q&K9&#=txx;6m%bc9e}RVqWS&f{*@X= zZC_MhX>?3XAX1Oic?*Hy#7C?N=>Bz80<(#HwE@BPfG0-R57j1%F!}k;0kWEoFMx=$A~CCnL@0&PQ-AIw zXSzZskY^6vtFe^=>zDfqN8OUkK{ok zR{c;)`b_}cetQ8({$KK_b_TWohrPE9t8(kwKovm=1CdY=1yQ6yS%QGT0_hT@B~`ja zQaThxky1h$7u`xqqoN|+(pYpV4FYG(g}V29UEew9-?_eX&ijYUy?J<^nPZMQ#(m$T z5KH|@^2T5`?g||(3@>3U03if)1O-u(dd{@|NvxM--8{FL`Zwu*c?yFFMhu4Kuy~e9 z+eDE$V*HamIzH+6D!2Q0QVG8t7-CSOtUFExCDLD#07)U5hO@A+u-55uA#H9}7H_~r zcXxMc7tv0#A0VfU^}eTC@m7xi@0wgTF#~47sG>XMH}O^lTMVnAH&-+X^HIna<4lOp<3q8Z>Ix;3oKqz#p18=|K>*~pZWaLUP454 zIJ|qU<~CYzCn3)$q854%B|;;}Uad-%b5xkO{_MCvih>)nhjaRRwR|&85uV92 zH9ye*`?u(dH+DY(U*oP`3={R9bBf4#5!XWA^3m>a=AB2$lGiw%1cG!yt~vXOaeedSoQ@- zMWQzw5D5wjHr}WJ!JL-|UjAK?{pZY%!qP@^O_2P~G+16jv} z@uq<@F^~0=X&f6n`zO0U>C2>ooiQShgl=AF2|HV@G@XzQqMbnb;G1;?OQ9|2IMj05d)+Nm8<2UZPz1z7d}W#-7~c=soxz z)BEO#VQn0-n?EOmsri`SW{w{^K&3pg-vKuMmlcdh zo&T58j#kY7SF!iRbU3cE@l(Jx3{6Y$yv8?RdztqINN)*%{sTU#4nhs}&(7VPrsomH zyG=4c=M8ry**P$<=LoJ}Rs*kQy0AmZ?)@>-I)o)XxH+B_To0T=bKkCmLet6$&Z``F zE4vqL+A)^%m(ktVof3*cUlcnM*XU~9~MST2X{P4!sr zl|JF}{qRCW6#_#*vd7Y94kXl?_S$T8L@JE;{~}gVN)W`4;=d)Rur+&q0Am2N1%p?Q zS3tu#bX-VYOgU0!UDzh|^Yh(9(UDicdz91D8>|V5iHRA3)YMle5jrJ;b#(Te<^c15 zH1;jt{7bMySJ^tBIuE%>1IYXg#6c`MDzdz`hTP^pLN=snc;{MUqzY@gsf^bGb%D<3 zqIekp#r^mWF;D}|Em8gg8Wwz1Xj%dqzHl|{J+)u}-ygqjBz#Xse%x{eeGtV833EdBP^ewD#0G#f@ zyKzdS-&=sEF=9kAfmnk74pHp#8L*-(P>w~R0-#>N{&>r<_Ck$>(2JpjfI_09bW(eH ztdd?I-U#^XVohU#%}WYlHr=clxJU{B_NHk+)$bd~zwpdRd>44zis+1F>v4h@DTy}R znnF(|?A_^?GD4st@Upk}5fnihbrRk@X49dwxd;$>>sNYfNUInIKpcX6;!l4dLxr@H z95`7j1M>ioJpUY!Z%x)70Fw6+eH(<&7*n-p$H;K^6Rr77rD@Om$D>@Qz_>)-$R?#`RPLo+U( z#J4D$(%+6c<5@2U`|WWQ0!Hd+_tMqY1n`%h8fAjpms&7S#xm0n?tJH8_v>s5|Ai*k z>qGOVRHSiu7xx9QJB-x@@NA9{YoF8IcS!TQ)rzxf#__{+A58s$v$Z>9meZ$Nf5(x| z1iN*7N}1x-3=a>#ee%&V{@F_qjC9dbyN{OuWNsZCw5D#GnVgi%p85kx{#hIJ(Ffm$ zUvL9lQ<0WJe#%Am){YKzQ(B6BGUTwy8n*^FGWqq^9Z{khP{;~<`z~n;I_f?34_LnQ z{#6Kf66ngx%364QMo@YrIr|UCqJpAT($?|ZU?IQGdT8+;ylMhDG2DnrlIZbd;tqP=Q}5~LuUzBg4(e_z*kMh9ausr zXrR0oE#rU7S}}0$rs+17v#ZQ3oZJbc5dMc876B=UwZ^=2$jZzl@H|yH`1eXb;poeR zHxpVAGtDo?I3Om$y!0Q)Wv$3@j5;N6@uW}jAjSXscfX4n48-Sv=AMgspDdQ z;gh9R9oO-zaTP4gkKXq4uqHxWeAd@pAs{gHeryski}0cP+=D)z_^(UAj(>DCY!p9H zTdb{}okrB{)>c*y@e&^r@gb&*qSW|aGbn=>n0e>&wH-@w`t!Hxz(WNVvFZ5nZ3Ai; z?soFdVdX`#`s;XTLyPlmUwpho0`WXXn!$t z1UaqC#71;$*qJwFWyc5O=@b6ovQVXQS);}tJN)P3?qh1)@u_&KzyOw{_ObKdkfTk$ z3b*)2s`xu?BFkT#yr#;`tJN=$9+4I>+;VX#4H!&o!8e!*7H}YWP%z$s^8XE#r%%bY zDLWX^Tm$bl4)?x|z8VF)!Z`Q)eiU@^m--uLr&GLpumr72Un5!BhkrQ=VzAG}QhM+b zTZRXA>$z*tbQo`>?K%-lg8J$N6VsbeakMY+spS4*m^&t{BTa#vcuz4bYK` zVv=yEkp2P9bH;je)n*~a+ZU0#A^afzn>5%?mHs!p*?;&!@X()J((OM)i+}Obc#^Rm z9+lb(nzz2)WLH9I(|32k$qlgdzp}F_AfSQi7C`N=<02W5vMDeQ-)pnw@52ih6iLG?bgwvHb$R80C#_P$T& zV~o87nhdtx;;%i*i~nE&)YP67qK$cx?T^*`Vj{1BH8`2qk_gavW;es2oeLYd)|o6K zc%S4I=U=mF=f-ewrRq=EX?C7vTo7$tutI9ZlTBt|aea3GA-);ZM&x?uen{cNJk-Rt&>PBR`$+co36%0a% zLa_U%JCQUaRtg>BKMo4L45%yON@%t~clG?X z0sl-3-2K=A-Yd$;_9JuE5$nNp22z4mRU{F>261WlKV^gDHE*3R9T5)`xv~Nn?gk`w z15i%`b_q`1SIVi1F(zf*0|OkqCXFGWXm+|+J!m9-SKg9yq3!PS8GA^)?!H2>qs2jy zo|nfTFfnQEZskclZtYQaH1B>`aE(k05=gqU6jaBbPBMSEtyMD?VD+GZLi>|PF63PEJMP^_rd^0IIy$;hybRn`CkKCh z3FgELT>(#ijXgbO>F$3YM&#AEpnRi`6d+}7`W-<1`4w;v#$)U6l_DH`kjT2(kysS4 z=y-#qEkOe|v6RR*DUk0`fm`8>k9P}A?}@OqamH0$$osdY?K^;0&oXiHIi2e*;{|O` zPn#YCtj{pz#oJ#ckQ(xVpMdRa1W=am!;eqT1`IW)U6(IGs`nsGex^lqyJBrW9l;dE z{MxCQ`hc~RLoE`wnb9kWwKzk%?Rhnl?(~%=$2-#<$2m~Ty=|wvRl&I4ju;6sH5k+8 z`IsE0onzX#mW5-~86Rv0m|7x@$!#d7wFEK(FB&Sni6r5i|>6eN(X zaJg;k-yS+yn);zT0U=gBLF~B!q8gSaDer?%a)SWO3zvXlsALg+aKE&p92)5~(!;8z z_>oS1dfx=?(`3SSeR-CQ+jg-ntVH`8l)_ke=`IxrR(A!?93VBaE~ zdKXTb(d}CRjt<)oO117)btjzWkM+DBgS7v7TLNA%HuIX;1obxl`H88#-h9@?yqBL$ zzhEkoOD8xz%TN@=-0AL3A?D8ulEpVu{8P%Id*oCNte2C}dGm{4z40LzE~_Q+g|QCy z)9v>DX>h#@4!j&?x8aZAPJM+0t}dUcfPjbg-sq{Aud*GI6E^FnB>yupas{4PQs}tPbd&2t>84Y z%PC?~%PbqU>)kPAe?T#4!7J=IyI=U=$gdaq91fY@yT-DH)9$19u6fL!2ZGeR0+4T7 zNNL~NKFfFvsg|3lxk7xH2eiwjfIfJjBvA&jieBo^QIS@kL>tFlOt>CXZ=|xK&qe1J zAMZy4XeH1vz7GuPi-Ws|s@_i=*6=Rj2hHGZ3?YoO(%X*Wo9nP#!>mo|Hj#h4m!v2E zAi3Y|E8dQ^16kVLuG8?cZVQ3VJ7|U<@gjYz7g~P_xyHA-C!(7O{#Yrem~z?S^K7UA zQu1Z*U;OKD)c)F`{~*%LYeQiY)GfR0Q*^p7Z}@>7y9Sp^9V7YIqO-_Ayl&aEj4Ug~ zk-ujvGti>Hm;6n2znk+pFPKKCTT$ov#gFDY$k6|yZHC2L`U`3bgmQaK$ZM<{*nK?n z^WhbDalWBDH&H-P*Y-8h!^Mtw>wEneC5Xarprj@P<6l4V!+l7m{NYp{H!djjtMMm- zFX$IrMS)ftmmR9s0IT>mV!HrjO^2&uUVLJ5^8D~HpLIhdcO;4gZv7e)ZGeC(QX|I4 zH39c~V=gsd>A)lLKNMFi0fK&viq?TyA~_y-*+>PEW?ldfuI}snu>xa)+XhBcQ(I8k z;94(~nfVU7^kJ~EHBsdTq}tY)V@=DGTeWVn&~oU~y!R!zAEL`Ul6@=HHG{MnH z>TU;w0Z@S)jF{xz2Ych6i+6Th5#>Z$ZSFvG_z9rzs1}PmuHQqY8@E5TChwkr@bQz& zhlJHYPMRaex;W~%T^*wxLTf|tk{u?qoWuxUq{p82XQAE!H54ELro_MibGnaQzbB|{ zjsTS)%x$(*+Tr>P(6UrdxX+6nsk^_vpyxc;MJN-;y77jl%JBa45+)^3t%m5hKrg7W zKVVl;@jB`35iqG6dXia!ilpuX>NpGG9sM(;l6}NXHTKTe3w!vvd|q+q+m{TP+qN>K zY+phYsh=OuHYuoWiE`_{YHe&4fgm=s;ly-E^Wr$ili8SlyVq7PJ|ZBpbq}H|m`lG6 zDNgz#bywBqy^Lu&CfVFfXgu+*LV7P0m{90xkNSG#%B8jeyA@FTd#1BT6x0Hv&Kp09 z_1aovei3*3{jpK&7=Y`H7Jy(EKcg*xqal~ftK}MV-KJaNZRA!|kp0)%hquGK3+bFa zZqEGT2|R*p?;PtvKBWKJp$oT^zNyJv_U6Z?K5ss^K=ie$u6fWkjaP~G=-vk`2N(r! zCO@AXjky&8!i0m3q=58q5Oi~d$WtB_ZV@2AxBDDA6q`7m1dkNHqt?k@Dh0P>xRKBt z+Yg+9rs1*mFV7gH`f$~EIuuPmE^2ecsnAZgY(r9^L1bh4rpbVv zkL+-J+GYa(t@v>T^;n&OrNDx@w?&Hnv_NUZg9#{Cy`a8Tg}O1SZ}n*Z!9+ie`liNQ zRbKI>6P+}kPz=Tj z90<0TrqgA|2wWpBSFd#w=?(cghmM5bs602=Wk$&qB7OrgZntZ%|1B9uwGY@#I?3g`^PaOzj|Am7OOq(`xu$bdUgu3qhnWRLXR zTrYhP>$qFFbev49V}sg}y^h{_;ERn{+n(O=l18-?sx-TzXZ2_V>2o@$nKrB;+cMO* zPWO4h&?hJTK`Xl#Sx(>N9I2WMb=(z@)H{?=z@VI51FL-6d(Xjf+sMF!Dzu@p1W$+UOni((8R1+IVXF>(ClOx{fo-&+>jKVmBj+l zn7}Jf%~`Y@JGV|v88~?aSXmZroRUyyf4`2Ljok!f)Eu^lPE5(a?2~NCA20E@KIt@T zCb~XjGF;^`_|CVgaY@&^>e9Z0$H%K`mvppdR9BiU6Uw&r1AGI4!0*{MISh(@*PUPM z)ymT?>FDU$jz6CY;#%*zkM?X+d2E6NO&=158)yX(o5@4x^5k%Vhg(X zmDu@9Ha7V|lw*wa2=4HSCcc!YXKa^)t7II%B=##{$ih8&Gcs%oBulo;M_Gv~>&Yzj z$URodhaHp55bv-2sEQk=Jtbh*=CXSAgmw9^S?6kd8g5d4TJ?4gRZ_=j7dE$9>}D(R zBd*a46w7Zz;P>kA69){IlcAWRcXddVSc^F@rYyE z69EP^>}9R5fX6ZIOU7kOBiL8MrN4-MWkb@A^UT;wdXDsYnO2z22rhc6q|sGNSIO1( z*S#Z3wn<)-S*gs;`q|k$8J-qit|qem)}Cim$nm)%54+KXMXzf##OV{oKwA5U4#;Bj&yRK zDx;G8BF4nHK4aD2Kf9=7I7pE0rTL<*u%g%hnZ8yAdCE!zi|%AeUbLUw7xB5K_QEPu zeoQEc>;2-`KG9VlGAf{yWtUeF(W z)*iWx1a@w;Rpzp=JT=Z?cV!b_|1;*aRVX?%#&vb^jkI53eTv=p%%pD?i_%GS)-;t# z$=0ix?kw6LIx9C}4f{QL^Mv4Pf4}bHi^4ZDPW$~%4!Yu&^2t+b5_z7tCa)xo2N;wj zun%##)`$pnJy+Zzjw~jF>zohvx6tvG|=J@~d~wH>;H@SzzT*Q{qq&kequUvdG9^ zmK2(0Rr<4@U1+~bnc-C3%oBO8+bt5~8Z>?l4!5OW?GH`A#3=XSDo=xNtdu~$ zbEFZYT+6u$fvjo1r_cd^S%bQj<9T!@+F z%d3w{&hK4YY<$Wr+e#duKcB8+PD#0FrRY}jOMP!#z4OCP3LIq^z*Kl>NINwGIp8m+ zokRvlodI)F>ZLl~F&^N1cRhUx6Xj>)*kLb*q8)T+ItY)HaVj(Mx=(9(6dwZCt;NZ;I_UwK5Sz6IadoG@LXX#` z(u|YP$E1FZ0+R^=p27T#P`*K~^fnennddjn8fz(x;{~Qs z)K4n_n=|+D>gP62`)F%fF)^ldy*OEB%1eD*v|lcM54k)LX;-Y)%##+%;&*7UIg+z% zP=Tg98YA|cS*P!QdVT+NzD{FbkgK`Q%-Qm|D>Un67ZeM=u>LacAhtA0`>92Gs7&!( z%ri#IyUbthR(w-(exN7MtUr!<%nSD1`d@0672HPZBUU1Y^h&`7M8CK)7} z;f{H5T{gIZqJN78PnVgzv8@Erl;TH@jSn&uagj2Wd3Wbng?D@|E)mVi>hDWo z@V0JU_)%My$~9Z3$h$q^EA{%!9B%m3)bi!h?(+0tix`s+swca+h41C{uYJcQW@S8K zPvgrSdqzvuX&n~L7f<9PMYie`I@p$VbaCNob!S^$WyG}c$a=+OB-eUx>X_=1Ko>hf zpPTw!m~Iv4LD%eZqWtOvr#^EV2_0t^v-@fuCzc+kZE02??XYg7UFyEQ7(f{)G8JLG zInXiF016Eb{g&11?;@`R+*;ynzfZ27%)EgRDHi0dktthdSN*hT2IFzgisoH;%yMp~e&lRS-aIFs(ubg=c|bd8#{ zfa#*jM`BF$gx^hL+IbOD^}-!JpJqxHguiz3B{zWCs*=W;<~GzZ@8o`oCc$ z%Weg5y`o&ie3U#Kz*Xx|uDtwY_|)=85%Qge2kE~{G$24lAh|A1=-FPw2qr5hB_@rr zP$fczrG)P7p|&jAKBd~Hn|dvt7C(;(~dyk93Cj zr6m|xJYY`Es5n#B=So>wb!CaO;udEkyLKV#2Era#P_$ZmxCTG4IdWYrQDL_tNnX^d zLpeDaK@;Wt#HsQm&2!`Q&aN6NKFk|uq?UiBi3dx?VSJMfs?6GtZf*zZcPSskQARkp zCE8eabJq4&t8aM)_L%azWGa!WxJup~(n%xla%$tvUnhFV9RMHiyBC^{lk7#cIFuiF zGas%?-JN@r-|7+e0RcobY+G(k=-w1d>?IJ_9J!!mEs# zC0D}ALsyQ-7;*PD>pw{R#&*d#@1FAK_MCgYIMbBHNBa_}hbMKC4=`rRC@v!zZ%US$ zMsuPY%vD*GCf*7awXdCqVYgh$yP@3EhFdAjvDWU-q8+;P?Ep=DU)6oFjucB@|N2k- z*;y|@O4Z8aj0BT`&dXdUTU zG@8c;^K5&A>Ch}HHr)z{{N$?8><-i&c(jVCGUN-Z9 zX!l7X_wu^sP4)dE-L}@&KXa-i*AL?!bs8C)y)HgYXxwj;PRVPR;1lhS1F@8M}JeS z#uWcIPPf$h@qeRYdiVp1I+KzEF7ZJ*Pq&+x7P;9KLnuZzD_IuJ97VKynCM1T-SS}7 z;h6vkdZ$apis7L?zo|N(kTMBI0-v z_a3Ej49_w(p+xoh-6jU#bn6J8RfgI#s-a$7&XEnB2|{f;)k2A%Zj6@q%B~AX2j0H@ zapK&e0P&}GWVP<0adeDdsqn4(dL$CKl2}@T0@D^J?r(7yEH!B9QzLT-?(RF}#N9!U zF(YRqCS)at=y?bc>#MeNkS$Q3|M+GW89Ni=amk8xUWmz>9FM=rCuTU_`h3X72 zbOLaY9$b5Rs-GSO%q8Plr;*l~8Lp<#yknr@PMT6&U#*7ixD@f@qvzgK0z`NIOW?)h zAFzoS)Dij1WM&`Gx2Rel%BZ4 zC7)!W=qThD9$mcLgYpZXLym23g)al1jG%;_Q#pq5a23SSB zB61+JdrZd2_CB*6C5IEmy`4$?+yv(jzOc2FZa8dqg<&)_s*_F0UoPkCBF%2XCzP!Y zJ+7&#UfWx;%c8YPEiwEpO*LueQ`(RtI0(tr>%X~Axcx8{Z_ zLx1WTf;jK&nX-kkmX(SW44@z~0VJiKG_q0#OxLN)Q#%b^^kfXQTAb|4YTe8)_=;eB z-}RHTRyII4;j_KMh9Zuw4m1aHnIcPl~H8 zSYpXgGwze|d~LJ+>3AaK@Eb>pmjnc_0VCndecf{3KfO~YnV(iv*G6 z0o`#?eN;v#?w=VGLXSq^3n>qouHUCOJ_$VzC z){uoPp0CgL@(yDy2`4RR%=ih2 z!=UUS8%Z%q`5B1|zw47Ql8p;X(_V+Dpl015OUg_DHK!eSdzV|W7FveIXuUU+{L`Pe zgszL@Uwp+^W(Vn<6owBKf>6-PsX>T1tq!U6!2zfU=!^>_RrlUf;N-Bzk!TD^33T^o z>l@lgA*}(z(=~Ms+f)IerQS-q(T}i40Fj|s%e~rq zFWpzH7nO|+c>;i=sB%QDyec>~!A?*F+rgq_!CGEC>=xB?Ve}b80@fC(4_1CYaoHFe zA7*$UTGyN4&=r3+@~3*p;m9COUe!N66caw#t5khB9Vis)<#D%kC~)ps#4~6u5l}sY zqG)c%<^Wz#FeA2Qa_GdR9c?~!MG*3mI$*P+z;>mm=m11HYixaSr% zJOAbFry{m(lzz@eWT~-aSVzCtz#XPUh=!jZ0`WhWj#O8=^RA0#{6`XOzTHGkwtyAa zg&Q5?yT>9rC7y-m>@yMI0dAEU9_6>uo%e|1x$aw3Somzc7L9Z#a27Mwo2zGtRkphB zQKM05Bd+CfJ@PI68s_X#dNBs1D{V-mBX9rDp3x`0_v^SyV~ewkOMWX0WQ9@V4F9eP z?-ByWX0ymQ3U`O%c`*Bd#131UGUD&D{9J^UnbVM)h||nWsNg=+cIh1P8{onhF(nRK5>@ zr~O@{(@)8}jyne-e!*#OWYKA~%ADth90F40Bg<$exU}<=C#;|EhS%qzJ=aI^*ZlY% znIG|e6X`^#iKPW7H<;vtR5q?WO=bxfy%V&^A*|*6{FXT63e%0-nlVE#;AqbxqqJl1 z$dj?W36ej5^ky>z$Rq*c2$B70RLR*7<9u0;1ful%S8B8O*>^-*D2_T8P0$le0M`0z zhA_P5TKf@Xk(``d}`0jpa7!|;K(4-Ro#B}^me;Qz1_o>2WY+mn1pg1bdGT12@2_;5yo>Q?;;V}fzeKhz+DL|ummjZu>Lu8jwyz3PQ!J62Lc? zj)_9n=b5*JyWVMucbclM2sbQoR=Ht8!T6FU6+H)(W%Ow$Ds!5#h!b?ow;3$!1&C9Q zX-j-Dlm*i%vIx)7dWl>7X76nWYy0;}xy#3wC^JesukqLx=`3|QXnocZA;ON0tbVDT zvE>yZ=hIB6uwsmm$-qQwUlo``*)o1XT`US!*kJ(4n`aV2O@)j*6&)CyUWzM zGKY!$zF~Rci$dIzU|UGph47qE7I|z(*pznK*5*duZGWg`S$b?0BP@u<=5qjc#EddY zxk@di%848|xp31Gmdq|6W@4x^@j*Cbu-tjBJBeHCMNOrI^VY95MYF>X(33zP1RY~dIMNTKRwZFNPhn` zH^AtGdH{bES+Qpq@jaHr73P<?{Tyik8l6)CD6V!n zL8WhvEC7joKqyG0Zf6DQ=?3GTwyHjx3UFvP0AqN=V`I6gXxK5w0V*~2fUK)|$$*{w z18WZeGFibj4FC=UZ#D{Ym%}&b&TYFhON4tYzwp9uuB${hww`cN1;sIa_NgfoR|BVP z8YEInAhlmvRK#oATnIwPeV!m?`~4t2NEPSnIrY9vy^>JXSJg+sf~5yx+vVwkmgqi> zIb5;+?RQkmK)CAzcIEKwY`oW&i_nkSBQufJzl>*ZvQZYBs@`%77!nyxXo0X!3-vNO za~N8ceV$b((~*9BO>X9ktz?w@@_iyLp%^h=xUyq`iRbj1h^7ZHNM?BzKOdnQf;RWY z0LN1#&sAD`!fpB-4X$v||1i_^I~s#RI05>g5S~qN7@Ic1!=B%?*x3+22n?R{>%J&F_wE>9MKH62Fj(YW^t0XDB z&NQg@#3yu}25w7t$aI>DEFAk3k5o%@51<0%9*apQi{PFbBj}h(@Ad1)>@&eF0BsF; z&9O+ZKBfnlQR=riV5Zu{2-uJI09?WxO6xj`&2(uR6ro&pZs{oP*ah3q*pm45+^Q3R z)tJ3A8M%Ds?2^~7)gQ}%LlIqxqF88wT;FVmHx=Goxl{VD$@fX$>br*xC0M$jUOrpa zQ1xqsyB`pty5s`C=HJc$ps5gwF%AztDS-@U4{KWNSAsVne{EX5y*XzMdbbW<2%OoE zxVI=etCx<%?gr8$-vn==ghmliDuw#g^p|OQdGfsj1rFZ!r$_Jj~M!jZLyN&;~iaRe6Ix{yA#PS{tx{^NZ`#@8@D)}iD z(wmJAF<(lmmN?y@I-@l}TX?MbZ@%9T`(-RPyc=mvPpqeqr#V(F7v+_Dn3q?`H;N{Z zsJOHcbR!>^eEuo~fXumHn^O`Hk#^GID4j5w4uB0VJsbf*wl{kZov;GPN8W*B=WG4K z&{+z@c5CDNS<#6%p#%X^T1eeA%4VS)FmTh-qU)jkEyG*O{QRUzDs;jP$u&Hn9ZM6f z@405&Z;_QVklqMoC_6H35uAnx7s|s)DhiU#!xoHkVv3B+mcdd_L|{nERpJpEr%3y? z@!bP_w$__HA4EodHCy6^X8;mig1}+0HM~mO7y^^c3$rRi(}8haR@?-B)6$hMG9GF# zu9CO=_k)<}RM2Kuo@du)o4qORS|8y32Fpl;NYnB=?M>IWU6&QSG+zk;?nw`yb?s=| zSzSJᥗExobdkZrK3V3w9hd`V!r@yq13^mVBgUgH2M6(rB=ESA@HTCbH|*X*NB zU`LJ{>He~L&0TfO5v{x`LNkR=%ZsisarN4@itQ_{N$j7etSZ#wtMAe{U4DjVV~$ju zlw)}HJ^z4Ly|3%=D#IR zu}ux-4y-b7BaXKD)R7*$p3RW1vrg$wd6X&_WBvifBZ<{V3N*^%UgjJ0{-C0DbW4r^ zOv{}dCRM!QyRd9ZB#obW7A=_h3Ezw!dV(R6?3smfz{PKLrimD%YzNcXUL4>E27(A4 zN=0DKhYj9H)tSPy<3bZqjxZ-lCj5xJS)o)(C+oe5Ql>$+w6VvF(_!Lj)Mk{em^Udp zKMOYj060bP#HL{ic!QR1luBU7Cu_Yl>wrm^YFsaNKpJ=6WZ_^cLw4-9Qy zd=8foxK(u#*Zw-A7r1BY6gjS@Or*NZ`%`)cTq_5NpZEq#*&wy$m5SguH2`li4Hj8X zM*t}&06J=Y%GDcOkaC@sO_C*LiN5g*s?6Fken#m@vb`X*8s>IFwnf>zR(cB3GjqY0 ze)_v@$zGRfIWwVw9b8WP$VB@V17DbLw&gfRL=h8<` znI+4W6qvVJo0E!?%0=tJ-ec0bhWQq|1(F24GxE>HOUlhVWtaqdtz8j}luXoBBx45m zktTm>ePwRW=3J+}I_Wv!%oYgcN;y#`=1XoX&U6rm0#ilp#91<(b^fIpqz|64UA@+H zOQB>>vB!gm3d$E4-*C=oBXJ+a029K_^k;7(Y(^}(C5-1ruNro|q&$w#irQMYV2d2J zH9hu-I0lka*)5p~p)KuQ9@3t34x&fo@z|l3yj1p2GLNL9*wT)EBJKdFv-#Ns_xzZ4 zo!5y!Hb^u2H#=BfuG1dDsWZy#*h)yT&hK{^mQ=3@Mo51o+dZmS zJS0`?Q;wZrv6RnZqw?}44P$kO^0>aNZ-6jRn~OH=iD5QhJ4C2d);8>Nl8`L?577yB%r3`!-tkCaVH1) zLZm|zi&tvGvQ+_b&$QGo&v9l$lWl)|Q(`=zx(oS?#b6iau4TB44>=ttMnCaF8*R`a&_fCXmldp~K`>h&eIA z8It)mPLl4YgDCtTcJ}PCCUi3-Pgg`A>XYap^|2*f{dHe8>)JJ16m2hxJ#2;eLJXe+ z(Fz=5=pZMQqxig_Ly>*9JP_gZuPB|5Mz|>k=bg+|Z1}Yaln^FVf{spQ^0`IC;D(6aB z8ZcYQFxA2O`q-wEQ+o_ck9F@m4H;!V=Lg4IioE>@nocsAFXX~wd z>8&kFefVn-7n+r?DJdHy#P_z=y2_s|+Fh%ZpMZnDV4@?foMuoTfbv}Xkz>Slk>0UW zW!AU|&Z81Hl^5a>-Kbt*B~*mAss90T(9O-F+wL^T1!7zgAAm!b$bbdaN>NltP*ZX2p%?!Y=IB??9s|%P)5Y#V*iWvRi7=3D;sB5_RP&^NkQI89J zJy6<<_zolVuaKq&L*PR}B)ICAY8+ptwGI*O9D9{p#J-?OFYbGrX#?Hs(88Q=AjO=r&8(pq3M7^?~Dh+Q8#%P zl*KLwk415t?k9ySEG)(Zk3xKnBEuYlb6&Elu0Uxd-du>`mjK7Zvlv}o;e0d$xsen- z_aE<&C)clfMc5d@0mDdTr4rqH|6PeJ;%@8DbNtV34<^hcvymtf`NiTDBOWP4+K#c=A2d(4;WX70ORAO;>nxl*++GP$|axJQdKQCA=13jp8EFIv3<~m-ROG#`V zjlhnPbj)l=I4cMge0PkM-uUr z>nt<6r2BqAsl9gptuYD^`0UyD72y=1(dN*D zcC%h2BI=FRCeSN)Dgx??c8SfP)sL_5x(iIN=m6X)=UP#{Mz)qp_w`+W#+oEo6oc36W8i>pYc1i5@PE6_o~V)zMdwRg z9VXh1i>mkhZi_awCEsZ03lNtKPf~d-9xm6^jK4Ye!Gv5qgC8nisyeBpqkg2t_b`Z- zMvO|g8}*tYMxM7fA%^yX)3jC$&vl3v-j>6=AQ+J3>Ef~2DXFPk=xy?i4l*N(QRm+| z$e%=p0y+{1LJb$hqi&*qAE0b?d#VMIadY#B<#!hmJAWPuc8|FDn4x)B&cQDK?6oMl zxyeJAe(R_sE*wSy1d`Zh*V~04lBo=m&aVQLJig^}qasW^5VgJD`A`F$OQj$Y-KU4EMxOd>RmN?U9KNmzjD@xa)wrNHtk^vO5SAU~C}*N_0hccebduO7mFb(o<+ z3I+1d!+;tQr3j(6)AZg>*we|s^L1h%I8#Z*nVo?bhUh>`2GqoIWjyY3&;#5iZ|89#9Hj z5y1YpbN}y~xqA+wl~7BqA1V${L%dEu#}ENAjfqW3aqp5+QqoMP#M~M#iy)(({_dm^t&kiCCJ+|e13_Qq3P|F>4fN?HFPx%F1=nG%O%y;+||GG2kja-mm$(NH2Bl` zlb&-g1eR3j&Kvn3I4Lr)Irpi0%aM-H>KTfXIRV|R+!_(JH0iCgsuNr%Gi(awuTMl< z=Dyr>=u#_+Uj@)`7ejUor$8I z40w4~2UONKKAugO77run1lrs;@#rm27}q}V-DZ6IA+S6r-q$QNH*Fx+U&xZY?iwt+ z2MrA^q+u_-d5O-q&sk__v=b34bwH*Q!YG!=7bO!px!(G~3Q{nDT21Sv($sxPTVnI2 zqppqQY^H=QJ~*Myr{OzKX$U0)Cw!ku#W^&@6Lia-jWe37SO`LDcEz?|#QQDt zDhC#>X!~C^a|H?~AU4+{vKk?_Jjt8_fEHI=I}*rXfb!jHHSdxp;JsKkf^+v+0@)Pf z1iMfiy$6DLa|EF_b12kF=iv&$zyTw2SL(@-%Li{?pML7)MR*nDPoW>) zNCv2SQVhWhVP)+6sIdoWtT7^-%b%yvqn`m&QJ?49O8e(2^!&fw#lLS|1?a)gtP_R! zkk6mD{ZaWAv{~aOBl}-evyBgGhq< z@2l|7?;sm@P}~1JS}IOC_wcL4f2>!+Eop{NZnzqB|NU1($F&RtBWHYfE>nN|3GF%l zpY0(%YP?-}zq*ZOpn{gOg{SUB{(iqf*&hza{|@EOX#B4p z$}KWo1a&|aeeq%lpfq7UJ5ajCFZh!B;8C8MA@>Evss56@4g1NAG-hV3Gx9$EeRID+ zHUQyoCy-EE<{T4+(ggng}6D1PTn%KGw*@mFlo7GArhe; zSH|wPfD~TAoaah83lWzsC{>k_yF&?8!l5J4C@xfsaGk%AR9cGIl6l1#z9vYipD;Rg zxRuibWr;k#kQw+VOFN+G7%_yWsBC7;1Yj$)KCjtJXJrTlvAi z4T5Zn!cF~GRtwJQ87&VEotzz7dAHM^OVFN$X#u3~sQjn~ZlIP>iP(^J!z8VVCb0^S zJqN?`M~}^jV9srLBO9WI`TA{f3f}8>Txjz*fpc)7dT${zK`d(E5sC{`_8(hX$~}z% zNFfF2Bc9Q6C<6rIZSW)&J-$&Vfwwv&lgDHB0^O^G1#1*=X?g`WkV9dI=#kq z7(-MC0xwpRbS1m+o=7z4##wkCFhe4&%IRP+l$GY3q&zCwNe%$=j}YETLLJ8i-X`*P zRfk^@jU$9vt2^rGZ@|7dJmhERTEaUyr6+4f4luTV0)by4I25z~o)ZgUoLxSnC|YQ+ z=Nhtblz_)&3+pJ4Q!$v66Tb_$95HyOuVh*SK43-D+sSXU8~M(}7z3Dq|AW2n4ytPD z_7ns~FrXv_Nur1#L83?$ktA7y5=F^52uKDML;(Rol$<0;&N&$nMUV`VQIH%|@OR1H z9yxnK7ZONikibf+!sV(#Yw$Tbu5?t%KFz!Xa`y9FkJQZS_{mbkz;j}1Tt3wMim~ZWd8`RnO zaB~zi0Qiqe_qIwU?E;){QKA9WgZe=bXhJ}7U$%vrSw6}DATcB%Ai3(?!F@U8Wr}he zkm5_R4db5H#wV#nD_ar(0Wa(7U4SN^Ar7|-^#FaH%8x{fm)EA*LFRpr$PM2zy$G}#Ia@eA|LELCj&b4PsKy0PQZTRl=8dF zD-a%LAF<`wh5!h*0K`xttqnE+{b2=)8Om(mf2}WO1NirYcP+#Ke%OAPB($YBO@d^7 zAwowXpp6@F5C^S1FS8qUmZq2F(?du6ZS@HW6hB7Q`;>7SPVGg294qf~0Kh6gY+qJa zQ5}`v@I9%sP(KLrWdoGCMv z2og_vc1$`5EwIsoTfKMi!uy)Zq+d5GqPH&x}T@&)OD9eSJ!vKgk4K-ZTpgTN2C9u|tsM8#GV9Y@n2wTm21iZ%)b5 zKR*`6t^5E3=QjwUSM9o70JaEyko4jyo$=b<7|z~9Y)bs>vIwV!ltI^V* zE{4paz5Dme34sB#jif1U&syBac~+n?$Whz!bd2rYW&i>^>L+vTJ#Ongm)}6gp{B+` zOeQd3dk@(mXK!N*NS?u)SrFqdh~zP;=n`AW4=&dp=?re%G)>y0aMKQOQKj=M0E8;H zs+=pyZef_BuYrZ3O#1-HU;XM{=Dm>8mII19Z9(oHkmxD}qw!8Xk7aeCpYNHprQaa& zvC7f$>q@1QXXIJvG2otabk}S~@ z(v+!1+aT-r7T6nYIQOSg`STSs90JoznfjZJA`f0%-q;%1D`tw!S_g;uz9x$&Urv;7 zXa(}F!WKf~metOb=X59vHT!ZC9``-t`x>!+Zd{vl1zzoNx(<{yFHmt67tvN{PRO54 zu30X6V1<@vatMlNG-5>|a(1Ezp4g{Sp}qjJU&W}n;KxGvtlVLuq=r*Aamqq}BWf|% z-A~4=hb1DFs3+;=D~fjowtxWKG%fC5hT6uL?yKqTiyE|pNzW*>7$?B>-E5Wrl5Z{I z`3w_H?o^E&?nNe2QmRQEA;c7w#FJ7iOF#YuiHo}s4G97i!N3TV1Pd;8 zdTsW2m23Y5SAuU0j?q!ZV>5llnHz3K{^lFP%0_v25C9Icg;wl-*3BK+-dX_Cw1$ee zfd!EZjN}0pmHw2UCFRBHd?bP-R2;Wkgv_?**pXj`#~2fRzVOVM_lJ|dZk=tzLNODP z`9|s(q>zbc`M9eL;-SMP?F2?8LVaQD0ji{=8jM&@rBOo4Sdz zc(Od-i*+?y*v-^rTSSl!7*@$q$!I(r4<+aRX8iQaoOK?c(=@w25%1h0?6LNS-pN3z zp&X^AcvQ5=e}YzAuFj0v_^a6xv7nfsqvvk&5lbbV*tpS#GA=;8Bdwo8E0QF$vXj47 zPqvu_i3$im`dNW&PmC8ZOX>c6`-Uh>`P_A$g)T?sL zqsAm!xQ&>uxxSO_c!q={t>4ul8;|U?iDZPZ>xCINvy~IM#(N^AjYFX@PNe0<3QNql z>q<!RkA66G1?n3QSRd*2?#tqRW=k!c~-)@&YPr0%pXd*l~F%jsDlj-jX&cn)2Lw(Hh3n7nYz#;_&V`0_obMfs73 za4b7=hIddBk?SPNFKwzi&V5ZGJxMyv9Yk1pX`+Dj0)VGp3c0io9 z2$;oFUvD>6ar+3+z?-hda!gWMHksKa8D=8IY;Kz2ca8b4B;1WR{>6sV5ar_z$E7Yl zOiqJ+RTAv0>N7e_U|)rtXWvDvLblv2mQJ@t)N!i49=9?#qxU+KYI{23?FlL%V*ge; zfSOg;_&W{Qw*iMp$55bz*EKiXc*zYnl{Of7ucH1tY+apOWT(2NgKkrlD-WOG>!m_F zv}o9&@e7oo({&Za$cf4yDQ!hFg~rYH3r&4U#T2}XYqz(onO`Tt&5Bqsj{o1Px86N) zGdJD13L0sT8D_)1Xx}&pk)dK=FWLY?x`?XRvnzSBTY2~oOTQZKh!}e4j>rRa=4a>N z`DtwdMj`Z1)39mjQh0w5oknXRSGtl(ce&ILNBLU~VH%=fS9Crhx^^*KTkl!g{5zH5 z-?vkSx@3m6gcF*4Yp1{?RezbKg0HjGR0V@_tHIVZVDAB(J9%k8^-YnloX z;Y46Pab%IYHScL~p@^KbSXpjPYg{EU^!|E*c7H=3TGgFq~Y zu<%uu^g)fB3Eu&ZN7DnmPA zkVL4yZ<4YGR@o+}VGn0~HFf(e#H3!CoXVnN!r$AM9BdBKSjE9Jm?Wk(VppOOTJ+mJk%m|wnjIKp24$_V%!IZ;jEI)L8> zbw6vZU3g1Qb>4U-psY-F8-VEoh*+@Oln;Ycw8P{~(R6S{Pa=8hJG_GyNZlq)lkJ!_ zd`l^$LppH|E$ma5%3-h{<(B z+b;rrlMm;To7rbK(R@AviJeobrWL5Beh1j9Khux$?i6VLHk28Z!B#UvRICKE;xH|M z4<86ezN2~kuN8im&vSUHbLwF==&}A@dmx>J1Ho@&s{b$0A_r0TQvcJv|A)Xh|7E+k z-!Y2m4;JA64({o{Y*+uUcW<2#o-Z*dWc+suW@>C9eed^gX0Plw@W9qHSBR*~#04T~ zml3%u@pa){@`0GnX>Bfg;GW_U{5@mtMW9;wKDBl$#;pjY`TtfhILRcQ?qAvYvP)pz z93iEMkY$wlH>4fF(CQ;@PSv~gEEpA&y~UH^enFHIchLWTz|Vj12yw7N7%y3ZS}VWh zXRePpA<##r;WH0!juGtGbMW}J0{cZLdN+^g(;q|8;qVu1xqfM6p-4}_81nkE6Hp&e z%C$RNTR`$Y?b7f|1$6FjX4@n1 zi^%#|SP7g7`ysWT_pSmViU6L{M+drB-CohWW=~=Pc+~y!>S-YmJf8*WPF*V@+6_Q3 z)gY>pLb}%0B|p{SUm{4BLYx7m-0$!u*TdcGv>FS)dk@nVq(|G}qq)X-)b{Fzc`iU@x)VpgC`wK%!(4=aLHP zg4aQ=XVlJwn3cmbD_peDZLnUMqLTB7o_mWtseQO%QW;Gb*r3U<51&!TuoiBKQk#8% zWiy78gzRH)aLrZB0&$V-6BUP)R4 z6u^@CQSj)@NAMklwz{iP@@rbNwjlo3*l8$nJnQ|@KyL%MsRr}p8x_k7n+smG&GBNx zt^k#}-}p_)kZ8HNhpZdzEMG%4z*k#H3so?R*hpVrUO`!8X&u*ep;vqcu3&5&YP1TZbhS_MzS z|0}186EVtd_B6Ro#Cz4~0$nKkIF*jAPjvADl6)`i6Ip7#v=Z6kp1M4|eDKU*V_N^h zL8wPyg$YP0wo{ibp~LV&!7);S>2P*poj$1msU-3m*7#^u{~3_ELSVRq7bX+9uc-CZ zTzYAv8*6wAjjq2zV#JH_6Lk%1p<_9e5pk>pNIJP~x+0v*Fy|_?7lHUu(N}9BB_<-> z)%Ft@=h4=OJB1SathKW`8SR88y5H^srnS|28+JJzSHyQ?TQ2j%>Boa^l4np}+#ztM zLk2-T^LRg&0_N3apg~_wr6hounzy(rT>nL*f(bf7lmRbnd2qqA2T?ELd>>^NEn%g6 zEl4ywx|kkE4^-EOV9H*|<^0d6Pk zAae_ZRi%xE?X88RLvN5oHT2vy;9=FvVw^!z5OR-OQXK>t&(abP05PlpZlX6x9wa|e zM<|0XWQM%%3fkKsh?P_fN9`|rpmJiS%e^xS12FA^p9{-nVFdVA zUL=YcZTz*v%Ds!DB3p@PMInm!WuLJ9zEYL`ngx%A zLO|GdRUP8r!JZ}nov(f!d*+c8@z7+MQ$s&(O@88%R>bH6S~zcFWUl! z#8N^*@XVF(j|ud@(NHGwAsgVx*$zSsvK!e?s<|H1c?9G`725kO zHdiiThu1pL=LUHe7-`*)L$<^a>NWSKIFzl zBzCE<#+2lDp)c>kL?OIafD(`l>OSE^+2rj)xvC=cQz$|D_b_qt1eeW_u)yy4)ZI!( zZAEjgE16!~TQ6@%?f{;DfsW)KH)swYLK4cdXOo}e#cFdl@0Z#ACE`kZbJ2_kYe5h+ zhx*HB#7+{g`=h=WteO!pbT8#_8H~>qIq_Po58>%|h=9BB>xs_lR&-X&IDdF@s(zMh zq#Ce4ZwNnt?8x2$k|>l1VF&?O=Tpr`%UGXkkhOj^S?1P_*gjl$j|Ww5Wu)P2#q6mw zsCCbGF>zX`a&v2bo~nP0{$K$*W9XUx9yGW;_kG{5+^{@y>|E_9buAQi!r|=AdL?2L zeS4Ugi)OGQAnK>3?@YJB0n~dtV$Cn8(94>_-;S>q!pnV;-Xo?*{5s*ud)6zGb{A#H z374r46B5Wu8mR=HAeOuP=yd{~isb2idWR)XQ00<5(V;!ux$8vELxuqSJ<1wnPRiSO zNNe z4NB}kYyFl`Xnn7nP*~W(k=ozZ7P?U`A^7U^=jXbC&bM##hI6zXf(PH_;|2BbGIFfP zD9Iz%e)}cX{GQJYzMKqXwfE~S8R+s`hhtOA&_*HJBp7}1C&70J1642qWM)EOwwC*CT)ZOoH#7Wj=21 zuO@pV1lN+KD|@uOPkNj+1>Cb?tEzt@FzavHa5As@|wXv@_iW-8gNBN7e+>I9Ho7f zMKxgfK-YGf#tEcResn#fgLd7=-@SV`botiS7Lgmh(=+v}gIUYV%f5qcyWAQSoV+h* zPaau5PzehCYtvVd&o(!n6x(ag^T~mp-Or;f#bzQ(x;h22)y{AHIIn*mlhP(F-?h;E z)&cK(O|)0>ICFn(R9E}WY3%@wqZ>zHN6u6YGxI%&%(K^!)cHnIaRQR65R9{;kl>>w zB)tnSw=UlSZa_5P_g-0o0L%}HG`ZmN4+l566KV{4O0^b1Ft;lPJ^c;*tW)IIE?P_~ z_F8yP4OW7U^;-lD8~+->$q!uB5(NHh0>Gtt1gE|=O1V+HlC9(n05Uv}@aqgMBWq1m z2c+JJWNx^-4}7Xio67C|3L<7Ju2ebI)z#$Q2y9}RUr#v>m|`9qmyg;_fCX*_4y({PxL7fZbX=7Med#lfLwP7&%&(fYPP|;8=P?d4i;3ZC~%>x11V$nO)M? zXqoqFk8f#pI+=Z#Zuo7%iNu<3WzSqRB-A2tG00vi{n8C%<99<`N=iySKg>jr6o@-G z6dvdXA75YRMR;S}P%3x;KwE9gqpJSxGE)O{h}IpvguspC?`Uj_hP!WhfVau>>i}sa zwsBTDJ;q$gGG+hilE+6o&c&?z>D{8FBci1weG2=>J z!ehrk+y#vg{HRO*th*wwqWFG&U$|gQNsTk%j}>~ z`|0BwRq+T2NXp8u?=8OY2Mh3O5SJSlQm;jpe%)TC5!^TgSM5u83kbb*^DT=&3)7sH-%jE3L{QP4wZ#@$ZE^GnsGCX8iPhuOU zTjdgJ@}lOld8HkPL^zFO0tGAyby@!knT#PR-gGk^H>>X5yLX)~;#Laj2(u93lZHxV z{onvila75Vj<0{5gH?m!kdKcKULi;}bPNy2H++F5cBQZVfWE#yJvAXQ+=;K{=UXBT&KLy&aJNG_zj&YJ4y{FS-VKqBpsxJN99 z5=Rh|QW&*cJWy%36g(MaOe)1pH`pm^r=95{%N zgH3r_IpF54YU6)~r*(EcOA_Vf;kktrFA9;m*HuxFfi(szoWjRZK1yC130jsNw{fng zbE~U8@^#m1&W*n&;pBzRnXxGNKEj9^T_C#P3HIlu zucpZ2*Y2-Fro{v$ND2YBoig1r5Q{W?lMj=}Y8xn7sfVakQdp7`Op7>uD}Umh{*Ypc zr5{{w4QiINZ6cv=I)JEl-EuxwuC9H&e0*J~i(c$129>mE5Nj=UDyfFDZ}*-zxgK*C z<69dG0zwN_h2$#UgLE0j~6BLyK{%pKCX2w>QKV-wD*T?TKt8U52Z*&m)nN=Sp6<4u0Fo1z0cz z+yWQeCxnb)cfsQ-Yz?HBVgYSLnl7TQlpEX)hwkry;a!mLY3@hI_>0BIQBBBaVcr(Iilx}H=u;yF_brT2UmmB%C<@B*)XlHeVjF!BryZm`Fs!&#_Sax6t-gY#S);$iY)n=SLm71TON4=?-bJ|SD<=fM77=fkbAdK7ub@kgD3v~lt zG)p!CIgXpr_HCD`dgJj3Idy~kmK;*VW+wDr=`*=jScCkH{KjHcYwaNp{VPnu zQsVw=f4b6+=BHb$NIzyH+NF3EXj=JgL}Aur6kDL8a(d%AAgD!6>EkQc$D^&3(=Bfm zt8fu!jD02+A`U~nXBFLbkdTD11Vx8&81@eiT-4v3*34W{_@Z8WU9*z>u^xPko&Xi( z@nbD_{Ma8@6H~ljav5y;AkFyWwAlK?b-@0cI)Tf&(9h&HRuPVuZIXjrH9)c|B%PD(J2O6E-(bWWo1s|BBKQpA*rqL({g`0)j0Ou=j z1L^UUZfYd7kE&81XqCBGeE7`*5Uq@nXS5e^NoJQdKBju7dN1H)bl5!_Z|QekaqK7o z4$&vSm&uPWWK1U~hz=T_W>Udj?`R41EKjTz`f{Iawk64v8QQ6@xmkK~TdHbCccI{m zuGke^udJu1(bHeTGgPd&RqBeJsSv9XAO}RBQt~LStLfr6&8eDV9-A8n~9sYq^Sq#^*Z4lXT)<0^fR+XqT zJ(z#6b&B>x(2eTJ%beUjIu1bx4kIVN1ANr77{_R=+c6Z^2YhNND3E^uTv|>(vdwv|iM(zueFO{fYcFJUX+Hnon z9)Kz9HfNeK)N^Iasc@L#GJgc6kG~5msx*wC>6eBE&nZO$(>P7fY3WSdD5NpSp{OR3Vi-B8EF0Z3C$_6dM4vl6P0=%=s6XG&?{|jTEui+_#HPH z<$^2Ed{3Y2)BM7=PxvXGO07t4w}9_yk-I6j$qI>5ak#nizLzUA&t+*Sl9u|H437|d zNKJH1P~3QOSX`taeI&U=edXeYS&_VB$9;wmK}q-;M*gaII&3y6WQxtmWs@yj4bA5X z7;!bPvtzof6~a{nd;O(eEv>pXWyA%Arr$P4p1zTtE%8#+%ZmwO8+1wJrb}1$ZJwPn z8BcsbSMW+Yczzn!F`B!$*`J`Fdfa2FZG}qI;ZnG#1}}HYEp<`;m|w!2OAD7O8p8|0 zt>*8?G4XKxpenMIP+}4D6z7Dcex0E2lht?2gXh%UPO~1XZyj~4z(e4Zxn!G)>+K3v z%&e_J-cpiJqNB%{<(TwwzkrrUCG_yV1oYyP3E_h=DOsOUUrzT#kqB$<;!{j2#9X-M z+f2}h<);Kp9Be9{kS^b&|5aDhpptDvd>kFcPiET_+iq&;8eA4|lI7|M%CqFXmn22C zcAEcy;08)aI$d zx^39&p;R9y#@7B0Qib3LmX%$%`+ zB&V~|BoeG8`Rr5sOm8bkTTfH8b_RnWT2g8}gYm?bmXvrx-C9=@>-_KG$NBD)rsz{n znp|hfq{fS17~ZqVeR<*9bW~R@nX8Rzas}?PPmW2D-;AB1R8qUX!|W8)RvN3LT2H`U zNHSr_F^N|$(fQsJg@vD{ekJ9>i_ISU<--^z2brtGk`>5!SY*z$nq3(pX%D(npcuZ= zH9Rw}A8q}KVvN7flu2lx@7-t2!m#GyhtV0SCR}Ba9TI3i5MHzNxMhHp&sikHJh<{y z@b=_q*BB)-!O3%p;*IO;-3?>eWUJX-I>}=)#(cxYQstL z5@LtIlbE;zcvH4o;sTu&Ut0IO6(I zX8oXopt-Tq_>^tVU*=n~UEjq8ClxR&M~fnx2sjzjUT%{+YZ{D?5lqSNi?m*7M_cdjB09*>ZN(4ePdflUA9C^+ZHg0z6iPWS?<5;)# zJ`vw==#S5^+a8ilwxqroLvz+baL)z@2kA6K`E7o+t^HFM6uR}IH*D&^+I!9(Huu)x z?Jj}db(&``d#1HhJq0Jg31#%tb#%bvBUri?Z#>i?q-aSrb|%4Dm6t4@ZuAzLls6nb zV?JIWS!TadQ?+;`sj+aPY3FWHb8L~^$LI)tlD3jBM78VnP;1-a|EFhAfD)442=@*+ z!HFj&wD?5#SJ{9er}fw(U1!h4bXSj)gqnx7J2A@Wt9!4i+kXrCIa1}UY26Fysvfb0 z!mGnozJo%de!HI;-0e$%eVr-qviUzjUBh3G|PJv`D?ORP>x8qIuYL&<-6 zIQgEw`ibR`#)6ii%FS<|&8!O?-a3x7cJ}z3ARi!*PtI0&b281cVCjYF1}WeG!rp9JG-Mt+@X4=--)-1Tb9;DkUNp0{b^3P+ z|5rR;&szz69eQg$`p)|13d1he>FB2~#2OeS*RQf^A&K8>cv945q~l~aUbGKd+>5q! zwg|iYG2Mppktgjv(r=p8rtHwr8-#Zzd`S#)b}clOdnmv&I(blA(SrDUYi$cz(yz>8 z?1oN0man<2Iix=_`g@Oqi7eBu-lv`+NFQ4T0`SeM#^0_r+(hWscb1UI* zGz-E1(ZiyR_!ekSA9X}EDbjLp%CB8L@88Re9rzAHYu%@lke1c}HKf5o+B-^brsxEEzjPL8DW#*7 zy4E7#GO{1#_M=o6fcV9Om(9)%y?XVEOGqYH{;wSSIukTltT-brrL&ZduU71hO!)9} z^776s+1%We+I}$9)AK;_2FkQc)o!Q-_5M0C!rY(h+SR8ut)2ONdjmQ$mr(>2OJSxI zou~@9t0Glv(Wp=|ysVwI-06HS@q3MuKXGpA65=-G;a3Teee$i_u-@u<5^cF{QKozd zd*Hv-077^1l&$Mq8tX09`c3;-7xccSyb@rL4%Xo$4AF zZLapRjMgRHfh>}21S1~@oPQZJnhr#SM}OGnP6m=zXp(6moI%rRkn-w79e#phWRS+0 zxTVs3zVmJbx{?{xkj%nABT+;mHSg;qjhtr-nInFy!0fm(25zFDSx0a!IKAp_fv2X+ zN(>T_HU;&pE`PdsRJz7Le3XW!FyP1I$B*~4Gp$y(_4b-!B_i9}{348ZdxH;D2B-*N zj(K}iv`SQ=#OB45CO$bi9F@;=YCdCq_X~>d>zX`Ts0cX5ypzg=T4dwg@qGHSzlN?5 z;U-GwRzXL*XkVa4_2Mz6zbZeXr~TK`p_&0zCy{pz7!YU#irS6CeQowAk161@wdI-Y ztG|K6*$YWq?xj0=lnVwvcYzVAo00DDM>kCoT`44bS2A;N8+XOC*i+J|k97c!QLaqb z2mMBLp0#d6D2U2$wTW6_j0iQ$0!o@atP*ahDGdbAWplRfa*Y~nI=LT#>)KuixRAZ0 zDGgkA#2+e?U3x^_Lm7jCO6V=rAmr3b@;QWKk)v!a8gMO5gEovslzHKP{IC;(a(#da z_IyOh6H!lL3Wep6=)b};U#Y{I=g5Z-vX%bxZ{OZu&`sQgN7{wI2p2Bn#X1+v~tUf_Vz%(FpvTZp0(sr2f`UKFyTq*ZSDT>UPp?G;sL@n z2m_DSMU^EA)wj!7>V(AKiN0HeqvoBNp=y6lMyS3Y8l%VVu*?eto;Of<@PA}wVwDYkz$B-8St)AJD;wSeR zA2+7C78MmGC+PJ}*5prAOeyuWP`r>D;Ao0bgs@_qn=^-zFR7>b?B zSPAH66_kL=9rORfaljJ71>NB-{uxjUd6<|wfL92LNONnXq@=iQ-2QDW5_JOaK11_y z3)V}250A6QMMOkkgwM~`Z4*N5R5r*VvKDK2ITzKz; z{cRNlZx>B&Y81TRo>p$zkawxlq=*+5L6hh2gXOzKLPX^9&zLVQBxk8n9AF4RJ_nk5 zm@F_uNQ!tAU)-Z>7OWpd?U2CoF1&46q_(;ODba>Tg~T+QJwei(a~|`_+Jb4z^9Kp&r6C8GxTiYbyx7RH(nf0gK`JYPFSo^-3b6h zV<1J*sQr8t-5S=U)`0;_={R0)J`s`FewUrsw+sI%0<-MJ9yON`>2NuKRgBx zq(K0J_)&Tkbu}kUUp-7^7@p}Qx;G(q=_L5~%K;8Zycrz`h*To&?YXcI?3$xz5 zDOd^XknppB)G6N;gK&`k$}qzXFwr&!(kJjDUYNI2!9D7rN#XRE)dBS3-xj1t2sg+6 zX&oC+f8RC{sKTnH%8-xwD|lBz1D!1{eMF5?hDIf7M0lbp5DGs!f$7QM5gZ4^p`hCS zD6E0Xm4J$M;t}0(-9R+ib7G=95Lu+iFotjIikdC~O}~!`+Ifr1K+{7qWcAUbh(Xgm zf1554H8DYq+nMex0QgleIzNGh@b?hTo}M|q7yBRs5{hy@Z#ke7cj3>&lJVy-BY~)W zTGw%iek%!bp^sj)-bQcM7QnUNl+T$GW3zt_Zr<0thV6~SZ^4Imzt%8E3ZO6FPk%yE zHwAA5c6CH5hcU6gUHFWMxR$}}fd8@^S!whe)VP zem==)zW(zYf-a1Mecd?#d6~S*{(2h=E9W4_+|1zj*g0rk18>3Wfc$C($uJW-_wN@A zfh!t~Bm@zv$rz9xK1)xmcVZhr{_X$RJ>Jms0iZzJ((-cTQ#Q>{^MJOPc@9TG2jFO* zK<5Bh0dpYU>bMA=$3EzRy}2s@m`o1ULOn5OWuofSM>}k%0}si8| zoq7RQ0e`UQ(r>VzJ!xZ+N-U$qmbXW+9iw>JT#kL6 zpoB%t&SU4gYze(kp!Tf>e7 z5Jk(;ZeSk@sCYJn04QyS!=Rak1rH<=hF0$RVT<;Tn7m#Qf~$&;4CA-Bb{96~dM>kd zfOUOpYHCxa-ZJYVQek_8(gJYR0a$y=$sF*p4_Sn~KW~E@$n);*?%qpEii%OC)pJ%( zhL5nU1`s^p$1c zLOyJf&dt)Y$uFy|t*xo01#hR@|F&TxL5YKnvbY+`{-yMIv|Xm`f4YD*k3qQ*R(WG? zQRpx@YZQ5R-)+C;J~%k|Qb`2BQS>u3)YNjD$DDRJBMd>X&IKLWz`8x0xwXIF0?P(x zI(7gmF5d?5&E3HTVmtl62y1)UdF&m=P#pc8IDn*=W2s8KM|)VDAr6O|7TrVtH#*7= za(d*2cto;HP@u8kA3BO0Yz>|^wtT=^ma%W_hv-$21$7mqc?hd#1`PQ!j-4AFdGBCc zt!2FCKR zj*pM;Y4;vQbWHf;fPig{8CdYx?`GED{y3y#Cf$252BQEm#KP-|C3C~hJcegS+KEC) zm~I|hv_Fpxuj4S*1i#vY8y1YxX?%{F0JZVT>pG{gC*E;vpvp#Zql(JbMWc$_c;8Y@ z0%kw>_uVQSpAXIOP?YPbUw0g-!v6N(_BCfPK;O*@7?)?nREDf?-MZy;ZU~0vPwH2Z z*b9t-w<)MiO^pui_nBI|x{Rngx}I@H@8oLzITcc4vC9&|dnLX*&I=#Jv+)0%+N>9d z{2n}*yIFPV(xsu|*gZRE4LG+pwnQ+Y4{an%D=U5Vx0jE`=SKg5>(w~!f23*9+05z;28M`>;((v@UWYet6+4KI%Wx_h8%h^miHz><DE1GioiU|)jE z-P3a{572J-CWXJ5)v5UIsz=MNFEG`SI2e$^j)LgqPx@DO)u*Artk0=>qibl0?#iUutH?lo;c+m%^&ikZ?z4LzHg=TLV45qQ7sV1KH<~&-!*yjeXISbh z%xKRB)fE>Nz5Pu+nS>2}8|Fc^3%{4Bsi_HQ32A9*g@OqU(q<^pRq*oVhf%%O+t?^7 zf~PG~oyArJgu0JMsQc|Bb@$j1@#Jqnc!rS3-^7z=z2EA6&}5x}=l}GZcw)Wp`@z;< z%OkYsN^^FICu1;g8?QuQsw{Yjc=GjfkW0zyddJV0usTBfOVLgtHO7$2UH?!){-J{W zLk0PV3i1yX**1b@wz+Y*UaTFO`$HCJzM}W@$YOY! zu+h+)7lXM&PMwS8yAPhPJMPeQZk%R#&CjK$a!9>V-a6{Zh7qS*q#&q{Wq)yxHhz!1 zW}3lYM)|5HXW0lo51gJ9Mb`}%^MgH_wlsUZnGNrNlY&l1{`h>>sQEcPg_SyuDgfQo zM^m{-ftq#+&BO`Ga+SSWC==U6{AUlVU!Weno<+x<-IGkHP^`QlK~j)9m>b96xe-|k zKI`L1LVO*x)Xd4bKVG{k!zdT#?!H<%%m*eCA1CG>;dg>-f|Vz5b*Kk*QU=MB6a~ML zJG?t)(&U#81hC#|bKQ7;Ya^<58AbSECXhrrfI~H*X@&}MUW#V{% z>JsLEKgMNx6md(M@fR9>jEDqZ^e)Sasnqdbi!A~|b44BI8#70#?d}Y344G`}SuC;K zKW_h?aU5Af#_Hh^!u`5Mb}?Xj7mqmrg{}8+POkemHmBNE5Gc-^OB+d2Ztd$c4|e|g z@N}P;P3fSMMa%@?;Iq!0f5GS45j|x1SZwPK4FPiV={I3-%{ZKa*dNEiQoh>?Xe_)4 zx9726)hk!jYu+Kgkf+YQs^LrJnqCVVo&g_~x)dNBS4=HZq`jHcN=7FiIBjxR(fl|p=ypd)I2>4<#6e{xbK7u4L_{fGv+YykUR z*iiBH5nGec(2s*O#Ngzv0YjxWN{p_Icyx(pn6q zoPv}WdC`)Ig0AkIu$E!5J(B&t@#f3G9MUec+HfDv)De&|5DzF}We|pDl>{@@j1Y3> zG0s2Dbx%>b)8S@lzUmOap|i0HjJ3)GV#L~HWMurxEIOD~M#B@IyOTh^GXmz$K^hhc zPu^qqRyzD!fn8@ggzGMi5~%qR#Q*jbRG|crfk>}; zW_A228aYIt+rnLiNm9=hisi**jHR4QnT z+w%FYCJmjwn*JnoMGQLI@_ff+!r8I(Hcl&I$-Zr1rbav0D!G3plVVaPR_*IF_7yAk zN^DrI0T+jUm1199`b?6&Zfg08!*Jzj1Ghd5#>Hj!jmgf^Sz>Z8PYJaj<1t8zp^#nHKX)XKHzOfO<77J@jXkgN~Ci22F~CC~OfrwLDH)4ym}R8^KE4Riw5kOsm;q7imrkb)RC?}8 zB~lWr1T3q3vChAB@&=&)R@HIwqYh!ZF^5Fyu?3%^wae?+7)e=i(F~)~Gy^75UWardnRHL6Hga_U?J1NH~!!NtR`oeK| ze!;7i5}J<4&yRr`BtvF5X(#NVU&uO5n{KF44T6&W>Z|>B4YYIV5kuuqD)|XHHeK>V zS(dLa$f(aG!|0<)x~ZHeOnlJpopu{bdo;_k7R zzv>w^r8ZNsg{+ZO%pscP4J+YWdZ#BXJiri7%lCbXCnTneN71=^E2u5sYwlaRdcIC(0o zfUmg1jl@NbjX0_2af>k%O;?gvYlgo234Ctj75S9j@Rj`M5?v>_PaAKnMy*LqXwXv> zkbiE!&*FPjK_>CmSFN4Pd(Be8W*!7%-S3eOvMT-oPS+X=IxoIw~g!JthrTB z2OO9gldH97Wuz*@tS3z*@brSSIf{_-Yidf0QL%KO#p|E8Ch^lVjJaV7%#woFDXF1m znMh(>)Es2SV$@(;-txa!V$~}Bmgg8Cn1&v3w zb{GMj!sEOl*O}%|H`2VrW(A<0e(x^-rlx=<^>UkCW~{VV{A%jqbA%@qY!eljqY^F_ z4o!KS6RAm82&ObIngpY){I>LvAze9~&iZ-%Z$c`>Do=w)rj8O{p?1mCIW6~v&m~5= zkcPF4#k*0T$&<#WD!=%NL-FQPO$2^EeqzaL)r5%CeU(?{iOxx#tLwOQS$6V-Fz5RA z?u)_3({`^7_q;2L53myx$$2@F`|vui!>y=hMDII%jgVR4TW+YvvZlqesVLpE2{TmG zuB&;0MnvtMCxSiPGi>NKgMU1~pMEq`G5)Z4f*Ky-k-Jl5^O_*@(sJPlUL9jVsv>TFUPeBtJq?5t1cd-Zk)RvN|2vW|}?pOy1Ka(Q9FeCqvq# z_!N)d^fN(f#vcDT(X>#VYWnnJJQQ_HBn&~W7Gm`LFl_Fn9Re)!y7H+$|6Ta6h$f9C z1TtLd{hM|12AihNRgJ|mIcrCGd?epP@tsO5P;jFV=PbE-^Xsd<^hF0Wl@(uyO9}FA zZVjLpVR%w*p?iu-l=@-&u-Vd8rgGe=dvxErn7pWeblxhBlDWKqH(jfBgB4TysWy{k6NSDc6Fi>wPs3H1Pij{Z zsnnBu&%FZ*PZt=Oy}DU)!4Ey0Swr@2ba!4y zWe~Bo)&m8`<_GbYK#5z7jd51NpE&W+5+oC=4&ON07_EA_Um183PiyQ?%7vAxJ(g7G z5C}Prdg5ejjD6!7r;~EQ?9+B?g!I*#FE6JThZHh%y&PB~iuc4ze0c52e`D`GqpHfH zE>S^H3@9i;!9Y+z5KsY;B!~nNNfPCf1xW&eB*`cs-b+x)8Ob0)l$;G06v!HP#Kl~OxQ z%*&kI|Mt{>Ln@Q?X5L%k#nIV?w7`C$7Jm&&9!FFAtK~s2=!*1Nm&(Gy@_(t9DL5#Fy-8qq`LuS#0C~@@X&a z#J)+1cD3<1FxR!GDb~C=ARgSW-r|cHPPEl*!^~Xo0~-$eEH=3VFQnJ=#zoSR|F3<4 zVwOP4(KV`9IxFvJBLe~sl%9GRO4M}w=~}E?gi^7VSbohad!;VU?&vz`3NN^b3$rq^ zf5>P7i>IGbdJzBWzvmYa;M6)IMV}F$5Z2s2gL1K4-nzfs=g{*rVZdOr`iAnc$vZbH zXDc9)<`mUe&0clb^#lmr+0OJfT1?)NBsVE#db0ezl7Q8%b7X4|+We9t zXLCD-cg?-LRv>*KzoLuX+m2UUMPo6EA7VK#Jo!crFf^_9n(AC~WxMwZFbxQR$_ zS?&z;x4NWf<+!t-F-^aew4#2#u!+p8UvA!FycS44ew=*M2ZY!QL}hn_DT=8iaytz$6d)CU{^WU6GsDeLP4kaqN{r=x+0Cr}*G;VzTou+?+;!mW zHf7tgSKjHYLXRAOpCM!=FkdLuT%YAcksnRq>I$-wv>)E3Q)n$gNpQg0S6#Y!bCBsfP>*x{+wzby0dKBRts@!UDRRNKz{YPD!|dcPHB z%`$=UaVj1?B@r^&*!AmdVL*RcQLo!5OuKG4OxIF66#09s%LjEtH%lDU3>yI>_?Y8Q zGB)1y$FHH8@;)BV$|an^{+Haxoa65pT671iEF$c6yu#x zS`!Z`+*oo#DHG|7PM6;uGBsLjPKJ2NX2+~qiy7Tr9)>`puwG!!z@BP*I5rrxW=57&Gu)K z7_R;N7PV)M)F4V=8U&OVAS*+j`NeiY^4BS1ohf^mzI0IeTI{+;n*-lfJM+4|MM6c_ ztu>od;=$Y@2@u7jNZw(-B5`Z!hS2dLO~VM{JGSR6U-LY)T{zd)+5GPNm$S&ySlP~fpPl0G?-yGO~g!&?mn{LINrv`_l8<>lqYKdO32 zjHt9&$1I`h5g&MnWgBCTn&Lp2VV?-s-<&@^aw55sc>1A~6c$^GH1_zT`>2cZm#CwJ zpqrH{kE*hl+Li>N>1 z{(2NyW zU#TVZNHuo*?(?+!uu-(D=+jc7{9?iLwGNKanNgI$DIgR94r*u;<*F7E6vSRLB>tNQ z{?X}&u0!#srJOtYdEsjv3DnhK_;haK8aJ>4ZT_-L`*?bA-0d#{?ho)%nRfH1Gi6+~ zkZ5ddJe07JNuTv=3?f!I@Vv>%UmPy_5BsX0u|BOL;3AW5lpm`&@M;p1KLRQPO={i3 z@$)DUn)0{pioX3*od_U^gA4*hX!Ft@QFq+Ako z2g`GE4#rfG?Ao=<&uuxK?#=3HkmTqtX5{C$6{?-zUWON`d({b=PnOA8Wg_WFOA7W? zoxxFeD<=yzCevpO#J{;h>dD_C0+2jeu|)6lT<+V*InCd-)p_s52||UlKs&QyCXZaO zn27G&o3H!=P=Pyn`_4YYH{P68ygT(f9UyY8WZfRs4_gZ_FsJ%%)qHDb-7lpD86vsc zyArIc^+lo&{+ePrG8ohUB)8{xG?;a;xADb z?j->-%rv!`1K?s;`W=>e(HX7=2a05Vr>LK)&IMJ^Lu zG@1TZijgNt|8PkBi$ng+?Y1c+T z^aCH|^c)?(z~JEJ4ek*z5WV-e^P2tMHN;zX1ZL=5fi3Sbbg}f`axfQpidiKfkxUzu zIQa$sQZnfRKeRGGaTrb9mc&Q}wy$z&aS>z^Uk(Vv;sPDt$nf(oEe_cC)y}Ln6PSzq zL{f6E6OC-Sg=7PeP^pLkm@O~%=AlUYJr$2-2RiQkGZB22A1P6Ey%Xa2*{WVk=oSU~5xC$80+lZx_J(uBD~rX6vZImj`BMwOmOr?FxQLn|@{az&N3Y{GyRR5+`)V zpVL);*#MdjM9Mo8iZuheSUXwu4+0fp8;qdJG=)RS4`#!A5o16(eeP1HR-(5pMeJ9$ zvHSV6Vm~hS1IXRd5opG9Ns7D#nz53HgNL81SD?cOtujH=!|-!_X9j1=>C zZR^yI;7`l5{+eKbenCZeCi%V(MO=p_@9?>F-P?g)DF!}qVRGU(_^Sto_%8;GQXyUF zRdZx_MPSDW9E%|&)*vea#<)$ncpf4j|GXx)Z+61m(C|Z|T4zVwKLlTcf2b6GDH>Qc zj}Qc3M#ujad}*s4zQm3CxW`WDrzH2=B)ryW&BBijz3Ej zHvnF&o7u0kcN1LXyr8H4KE_ep)2gaDI&Bbc+BEywX9Nwj2H;gibCN@XC1i&C)6^n4 zyrXRIBzq!klRd=F1H;51THqJE{k*-?z4F_jX84g=^IOf3_gl@NC zNTV@w5V$rcX#82w-1%a!-(N12_a_a2LkPTq#6)ILQvP;3hHLdZ!}`&;>EC0*sN?GI z=V#cz^pCzskp`Z#syWBv2yf}n>9I=ysF+g*CBPw!I3Bckv}kCl}BL-y|Hf;7=*9(txahBzS0n61hwMN zZWCq`xbh?0&CtYK#dc4367F?GlXtcuFG2jJ5Ffi7G3S`We)$0*6DGqSo#m^y7$?u< zu)zB`S`FS`DkpWAGP28#p{ZPqLDrMkT$nnyLnYL8Py;&T4&p7KrTrytPM~Sx9d_zW z2g@+>$G?|q>0e8=7y~zyqx1S0LGAX(>^M0k@b^Mp9254dz6_A}oT`(EFmS_&HJ4?W z{suSbb9N9pK+Avp&UpF)8YTWa<0%s?ZIZt;o?Ze4ht1y^ zPwnC1(|245Zt3yo;g{QPE@&Kx#T&vee|i6TLDR_yIn8%?K82<;{?2&n20NmSw8L2h zl598_%i@0xcf~xI3rwsV2+X^uXgX7tKR{TMpF$Gw5aK7=gej&M_F=^sdQJ2fFBn^U z^ItFQgNOgAhx`gm;o27lO(+H44$p3AV1Cprffemp4K?v!{bg9}`5gXon%1KgeRN7_ zI@4J}ZS>y7fr;&{BDaP*&!@P8IPmGnuo`I=(eNR#UBxR zY73bO)dYCR#`M`iz(aew(KO)gWw;9ZY8!~?GNk<*3BLq$5hN2TeVsIQKfzVkc1!%>PO|X--sM-3h8}WUrPez4m*B zVT0L2)$772+Mn#DYATTdKh zB?A0IoC}gcIMH(nB45}A@KX6xdY424VR^ogFrk~e+@GZq;n0j^oqD9nLg*dvcj@cy zY~!C=Ly!FVhXn4aCNNXlRCIn^b1f&VIN-BDrsPQ-`~$r#FtBzdM~89Q8K~$VI!ZA! zsEM-jgJR~z#n*RvVe%!0iGoy3u>@@f?*$!S;uCXypUqtkdznA=jOlFJ$Ji>z2cOE;CmjDWuC4=LMrVRqtO~AnJ}*B%0&JT5V`t7NJh(nFBn%ncM-bCswY#(8 zjwS}9>~2|c+^S;rj=A%YKYwI?hvv(XA?eMn7PD59T+kLtEdCBu zyxx%T%RP+V3kZ)n0--Dtw(yA*vsqTIpN~PZ!x5l{@qrQPYFUi3M>j+irsXOc&JTbv zEHDC@n%)7C*$9L?SIgG>fMjq7SxCd79)jS>cT^T7zl|eLT5<)Db;QPA1Ph|{z~rWc z9S=Hg87FVmi3_nG0khk1_3{Z-yW`xsLfaejK%lv+ym-Vbt;A%N0>;pv`<7ttS50yS z4?pkI`T4M&IOw-CDzL92MuDG&wy*`;82&lwW7b}1q74+D^Bo@>d*wL_#HCjP!@bv$0zXsZrqa-M9g<+kAaXfOLKgz!WI` zalt_sqE2-*n7jJRN|zo0UCy#GY~ulnlQCW=8k5ysdrtlc%m9|w5w+}Z`}6>Tkt02i zfi0H)l3F^D(h`g|N08kpxjunYW>MI8M|2)g{c0Cri}9;#S$hNDssA>lhb5YaU~;#Q zod3D>N<^&KoT{qw01I{J%okE`sU|MZJqy%0>ADS{>ZJ%+^v5r@4Lw?$iFN*A+H+d@ z)(nuY>_tgSPlDmDy=^AAr?PJw`_Z6T5ccS(2QZxT^tOD>*Yb=c3OUP;wCA|xKL9SA zdF-s8u?A^5Yd*xlB7xJ^;u8%6p??<%X%Natx^;{3g^iaCk#yk7cK> zvGk=qAhl`^IpOYl%F`&^dN;paZRA#?Q|P9U|6nv^xg7~~-$-;lrKn4HdVNe<)T}41 zDM^&W_Y@c#K0s`=dX$q?s3iHP-7KEA=NX?jxA*Jfz>nkS-*r}QEnd6?md{QF?pDq{ zJ$J{Hw!v#%dmh}7TXww|6&@8>I$!?dhh?IY@Mih`yn)@KD_7_XsE8LWI|&T9LBsfX zhfk-&!cl(KfWb3*F!-96tZR0kbZH8B2!im4bP=Z{r7I4=Bz)tZ^Gd*hS%P$Fr>q04 z6SApt_b~QhJAdV>9KBe@e9X)jvE=4l zFf?qWTTOR~o{{8sdPaT$2%5Teu+y+Wj_0#fYPL7L^J?X0e9lmSrC%FRS@cIBocPKW z99##iVpr>oyT7K%Su|hkg6OI7qtFxPDk~XNR>&D2vsY=a-ClWMG{$dk{1uQ}FNbJc z`>qOPg<4>-H}aEjxOnuMKT;we4?Vi8h#-k7l7Q)`T|tbcoavVecO zq^mRK8LR)7v1(mL@i%Rzymx-2+S$Yl`gr{|fRc;swv8Tu}o5iiQrFMitAeqx1>DA;Pz_@$PxNwgyCnj|FlEoDRy4X7ZMRQ~tEGzRbCnD=^CJ-=Lh! z6d4rB1?;C(D|(Wfv#wW~%>+{>?A^2;G(?*dRXBUlqO%7@c$GK=XL;oVn@%&!$#n8f zBs|#<6A{nf2LC>s<&}x|lBB12AH|+9Zgo}&VVpqJN!Q(M0(XW#9Dz^)yM(xVHG(ftA~#k>j0%Sq2`W)aobdCQFP`W30fYjhT^WRWcISAh6rnO!z_ zUFf-r!0p(6k(S$!WK1II6Bgt{*0O3Ax4K(coK!^}jQ6=z}B-M;8u*IC1Zhb&C2Hi9baeiNBm(=<& z%SmR(J2YL{U4OX&23JqllG(#uq?wQlx6g-~^h|lw0?L(bpLEu=P|AnO?0L-`6gL;_ zB_U#&T9vY8M)d>H$&MGdnuG+rEGYj*ZuQEd)$P9YQ2CnT zj%H@d$0wZ9HkTkBuGLj<7V{=%yOPxba9!`z*ew%SFrF)5;dD^u3d4h7i;AH$A5Jl2 zoP-thIc}9^R8|g*O+uy)o*Adk(!0*oH``9gwZuC_EOZ%)u$Ny=+OZ5)x$golo3>5( z@ic@_C!Dy|W=I}_x$rP)YuxpWNNC#8tu{-6OpV-k6&#_*qxp^^^u_^mIR~O3-$O=w`DzscYE}JM!r>%T1yNqiF!3)g`*yJgIj98m*E{5oDP7?d%WTeC zr4f}^)^*61(nN@W7eIe|-()Xk}#Q9afe!CV(AK;|@ z=EwMEMy*43OW~lXvss?pWd5H_0KK>!ejE1}oJx?xt?;x9q zqUYR=K#ZG!R^3$2Wrnf=6yWc2YfJA}yE))Ayn<2|K1IUX-(i>a4&aR5358n0zo}p_Jl4uL%KnhZ7M+ zNF??+LOeuC{)p_>j3O{+3Z{y#TQ)%A5v$q*6^miJYUv+eQQT~Upk9W2XcZr|5Q5m; zkjJES%!I}1#jM}1NnL=ty?*ok8mib~sF;wFPcPjS*ZYaXRj6WS&?+-_f-rmoz0W|w z_MYn#nBS1_YdB#U8Gm|)PP?p7G5du5#e5IfMx+RSLY{5Vsy3A=;J*Q61n={tT}EL@ zTcM7?Wdvv$A~kQrb7q9a$)k!%9JMhgS*-UDC4i;r07Rvye?h!tI2}Twp$Qc$j&!SX zoBw@2v^~>=j2lNGoU}yK+!jp)EhDOiOhvV0w{(nQ~8tCf(y4C#u;8p|q z`$Rnp64~dgw>QP>eqNQNqM$JK7E3HJApXd3@d}0JGZGylNfwNl;ob{*6s9r{`zREv zq#pTWElzom)j$2Ae)o!x{f;XUSA0m`5(P8xHi_+G++J*(`|f_&=0$_ZXsmMnq_TOj z_>Zja{N>T~%>2q9bY#2|5c0Y9f;?yk5i#{4iUf9*{o0IT9y>gjNWIdQ>W^LVfIr0S zAybN36^Mi%N!HFXNGulbkXL|fcI_ooB0pQgc$h(qft84nJ=KhEA6zoQ2&izPyps&N z_YoPH5*Il!2^qy$K12)5tw)(r@!4>z_Hf$~xSTAQ0=$?Qy1Y^G85|4}s)ZMwq~IDY zsDc)Uxfa}O;tTga6{lx`OW04rC22=W#8L6n`^c20qXZtJdj$_uBnZym=S5{M+!FJh zxK9)v3x6P|0^=tQ+p8>M9%Ayu;wtj(yfko0+f@Yh#VkBW_wJ&kNYJ8#Cowdc6ES9V z4mL=m;Io0cM>oBuuuy4RM4LBeii)C83|_6FKDN_O~%l!M$&su@CRN+=rMZD-0q7V4s;wI6u?OLj6Lu}K=-w4vm8!aS2qcqO zwncyhXx&8f#oID(y=nRWDX0}ss~c_KYb~vV$`;f60*%#AY(Gb6U|7{yb7HG6C38#o z`P(gUIF41*bzNKlXD35{(JUw_xNhr+g{x?k4BFRh5n^=2-R5;^ zhhgCKV(J*Wv0$;5m2uP>b85(CcWn| z%Y5pPI(8HxlgRE&}&=Fb=SLj4L|_jrhjYRU-rPXx5)a@4~Ms; zv`Hwp)HvW6lHDeW{-Hnk-M2RKAVFCw=r?V~8&U49WTePDQ+={0zf&iwAF{}BZ*H3F z%tzV+hLv>Op?CK5=sEy)zh*B$U6Vo}X=BrtmfPyEDV(wajcf^PIbU;3If1TkAUSMt z+HEHP8jNvan8t)BOsJp{8thFkW$ZJB<-*S~;>{Y7R5%Yn>l7p-G=)h~4mjLuB~!0I z>s)*);SqVs+5lX;N<(+Wc^ci#-v&V=FG`0#o|YJ(xiF67?YfP_>}5z3VWIQZ%El|kG%0_YH++zK+rXuw+D7-ff(*#a`_&<)~B zou%s#WGinG8dz;uC~kDkwo=#1ndm7b58O%?)YmZs6qB5PJEvN;t##}4p6n+ zOUNDB!XcLS!4s&p-QDmxs3rnEM*J*2Y6heQ~!9M^@T|i=1jbbe3 z`z}j|{s+d*@>A}$XHu#3%^}sbDI|zDg|uyQx{!4(^p@!wVC|s*$CW4a0Td~C-0`dD zxH3Rh7c8VOD1y*!9OoLk`oOCanB!?=W{?+14tt+ltLT-6Q4h$=S`doKFS4$e^Ogz6r} zGT~z*w@Lbx*`D_m+5TgEt+j0iCxhvZRNmTGWN`T^-=ax`?%UpD6L(4yCk;?|dH6E-3$P|T8 zFWRBNuT-MAtb7}qm*&>}-@2WY2EI*{+Cz(;;^3Gqd)ogIERyb&W9_GeEOT*+RE6z$ znnr4o`8po_wEgXx08UGH90m1D!OvE+rH-=_97Si{IR+phtgf&X-#wghHq)06oDWel zHp&bG+*2yiz<6rz=e|t^pr1%xpQ)_Wc!A|yJNiW|Ztm%~q5w$?Pg>x1saXoA{S{p&jH&{@q z!6*c&6~pv>dv@GTPHUy0wxaSI%^tj!jbKi^fI{6Bl`Eq-zD{M&?U^hFF zQ~3t=@$r+y<$INKi5Fs*%TiaQlr#H3Oem0KKy+b9@~JNcb)z&s%T9FllL9s*s)&~9!k!OH9YtqJRKsrA>?&K0o^$C@a`Wn>loHheNICsP0Li*}Anu*eZTn&f0 z%S`%gzAV1mN`9-;?ohOrqYYL64)fKpqU3GZWmWC))RBM9T=oG+lU*{Aku2idiG>J_ zD3{v_Zod^sOIW;6oUZZWNl5DKPz`Ni9X{(Gq$OIrFkphsBR$Z(}X^|cG!!ATGV3NR3<7Tx~CNZkbZW|7TmnVe2NcMeRTYD^tHhdhgWz#we z%f-hV@h5|(9Hq|mC($+vWQ~zK&St8VF>rl@6odCj7tO3et*f=JK%5;ShxTduu6)mAVVF}$u)Atk&8rXw| zDD(NFqx&M9$|jUp%TJ+W+!Ycpwy<6BB(eofm@@N==Ai~2V(M^fwv8DA17ObzJiVt= z1aP7qj@%g-dJ#C^fnO`oT0{7_;NE{Rmwz#r|M{595kT^6ObOjR2l}H9PUs}+sJ5!m zQQ1?g3FFBN&Q`>qxM0a)P}By@q9nhW2d?=J@X@g5tsy4pr&nQk_)vA9L(2jqTq2Hd zqJ~Q-OmqSE8K|Q~_clcUY84Te2-k?kH$umb+xK+=T@w%gUX?(90Um4vV4_V%N-nzO z1Y9CAkramRJqJkMa0_81>Hv0EPUCPH6+e~%x2nqf6UNgXcrM9vZ9yUt@Dkd zaj?OQrs2}Mq?Ovvg+2}i8%!v)#OTL1dA(}CqoxyW>88yIlCtZUZN7ixwdl^{HL4Bn z_LqK4c-K16((;MqLiC)Uw4NH?ny>WU-V9Vql;WIsP(=SKrdI?X@Z0lWQ4SDDzGK4;2L8|#uN$rIhM*KLM{W#(E{95~AN+mJioA_{8;YMq!!t&W* zSpvA~1gPbrU-uLAj_Exei^Y`-;3b@|kKiHX`!*d8QMaspg`+k3LU{G=P}A+H7z7f& zXQCL8!}51_%~AQP7M_+3{Vs4R5p^{=sA(kCb$)1?q8dW9Ua$A$=nz8`ivZF+fJy#A zM|&Fdt5+T}_WgMEfwD<++kXPL$)q=4%PP0qNH1m?b=gMl@T`ME>-L&mg4Uk<)UZwm zlG6XQH`#PsO8vzJ_)lKd|D&p>#{0?bn>TF>U;-g(nx@GMC7tEjwix@Vm?}64MycI0 z3Q-q+L9vLHu!rGa^$O;(_wrLD(B;Oms^)^G0A9IRj%=~IpCJCvI5fKp1=bYCJlfA2X&9ps*+ zcbsZf5RCz9owy9pg3|v zHG$j@n@v+=mR>Us0d}vT01O>LIF<_8-zsyorW3F+%&nzyuNXQ$bB$KpwwWWqoQ*Yr z!VUp@@OrXUe6b$OkdgMRMe%T`qk*hDjFQ>pJP(c&+o|LaI&puH|;_KAnah=2rTn7;1z90Yyt#De? z-Z=a23j)QGE%MY7n1q2v<0?sZY4v+h>uk+Q-OONNpqPzw=&oKzHgy^#0iN!a3oWv( zz%#2~=qRk<+nlW2GG7Bc1CyzP&$`x-ybzd z0Va|G603Cc4`4`rVj`&k$_|?wfNP`40D@hV4Br5XV3btg3GGY?{L1c0<9%lu=q--h zwJD9eZ;iUQfJOhaYJd8$B>)rN@xJ>4beuSQNYiZ#vv&$u*S*mgF9EQ0_q0MDr^Mny zTu>0KZ+hOu;=%h6_!6-vkELsVw2_--a(<+qgJm+8Q!_7(OfFH%UsG@bS>>5_=b4U+ zZqJLhCI()PWRZ&u&IVj8uS3(En$diFb6Ej#Cc}#BGpQ4{#^biq!#8SXZhNQa;B5ko=VP!oeP%v+SqyhWd9xHf^*7#3 zgnH7B24Xnh1LEXYMYCw;e}8q$@5uQC6HVXRsgs%CU_JXIBD9~YDbld!lq?uPKp zVa$y!#98DjKR)fb`@}WipgLniGBlphtt=Pyam6NMo2W3tveb>51FDp< za+{A*&P+l$G$7m7zlW^CKJGjW@N4R zeFfN4E03CyTdhlz=v=Gq0fG1_ zqJbNnW_@_&?yUG@4R7oxr_76zz6h)i^?9KtZGbb5E#)hOer4`2@OrXO>)fVjt%?_+*f#^ozj&Kyr6XeKk)ot7S5s!<)5V*?oNB z`MJ)$?70{65{wf<-SEUOXn)20#HHYpl8taL|!Eft+GYVt-z)q6emuq-4nY`J8!t$&lQma=K)Zb34dFZTnccuUe z`%;4v8xje&r}1P`xh01I5_~}4%}A$T8m~DF3jo! zkHM^#dN0cn49SO;^|}b2Rx}yBbE{@=RlKS~Qm3)0spM{8RCis)Zo7UQg2?r-&y7^S z&&(pmaNpCDgL-yDzBhozlhKk7w}bNS88y#cmj@1C`5vq+f2QAtwydN<=t%;n_{^zvZ_f~+I8H+*z zwjUWECqbAepX{1-lvc0ikx|@jW)dmq*^8Bq%@UL0MfF^k^*GGtVE)8xw5ss^wub8} z;Swf$!%XKi44;y&o3G)65l&_{!5k6ik6Vr8UUZQm?QsUy9PSFB^h7@(oI zDJcw$*bB%sC|L#z$~x%REQ`MR43ax?(uyVOxu&+qo? zj6~L$eMQG|5|alWLpmhqo=fg>SYP<$;){c>X=BQFDcLcMyuxa2jh?D>j{QYO+-hT* zInq{r?~b_pxKBDLmA+kTHNyZ$-Rf;Dbk09Q+@6>BWSTrbo^!x`B*GZW@Mh!jeN3Mp zfT8m@bJ=ut&tyDh&R+SGO1&)wN*@28eG^LbI)cS5Ce7Om3&{7m#J?nC1w z820tawatqSX;z&wd?@Br0Oyc`Psh+c#0-Mf>ZO-smgHT9sBMK|hzSCVLXf`=otM~8 z>pj;_x3Mkfndq^*Nsf+z7Vz0DF6>4J&Te9suJ&w&)NHuZm(uSyX#W!jpeS{mfUZ|Ipe z4Z&?6m$IFN^Y}lyhTJ|yq??F5r!SstcCI(ZRMY=)kMcU|Ul3n0Q?J{C9~l(S5ET}x zW*id)Y~51|Pz-;zH&>whGUx)1InYgqs5%*d$EfCCSR+IUlHovm^M*Yfn0I(`A=cD~ zHUd{=K?lZBc`*;_`r~lgZq3;-^k0Pn6|X|q^^t5fLTCS#ZP+Vk??ONR%f$LWhKcnr zDu@Q%|14Bc5BPnx$plVhIZXUcoIhL$#jCmuL#h#~ z@bo3b6wHnMaj;a71KE)7rpbGVDTsN)@ZL*(Rh@vExk5RvJko@FjiDT4w*Vk5?(n<} zL)VqdqZKVjp78HGah4z9!5?J+5KWJ)hG2uI6HFzGX+b(L9e7kRz==&3o!I_6!zpB1 zcI~RBVAZH#9Al@}?Y_OK%pp-9$;Sc50|9N9S-dImU^3+&FA_pDuVEMXnj@VE&V~%` zp3RBUsg%r8N#}1@C)J#E7tX3!n;2gmjZ?tUa?9_d<*GYbMMjt`bD-k7t1$#Boe$GF z07rT4Cfb0mmyrDU55Areo1CBzsybOPyyEN1UC#)M>nVztiu-`D*PTp2JD1rPRdvxj z1D_8@_8ccVa=3$znLyZ6xm{}+53eT;!QD?Tj1gFu`z7P^>5Qn&*qj0gA*2^Dm0k_G ze+)fnL7PD>Q66-kysm4ujpB>tbb{!ukH9OfPtSjL2zZ_UBA)+?VXOaN0hS-_o^O5+ zQb++@I|-4H{JCth7cFd+$Ya6H17eW*91#a#*nS-*wTUtu02^|`Fgtj8MeqDi0uD9f zJRl2z-t=d2VW8BR{${h?(!NmK&i``o$bKSXe>7bvo9#R;7Jr6VK<+i|S&gR`!4TUP zGCrmPYQuwAU#3e_UE@FBA5;Ks?lnz7n09lTlcYU4%!-bb9Nscq82QEeWR~hH9%Az_ z<&3oV`a+yviy>8JskiukE3!_#p@(I#1BfvBc7Onj|8b#^jyUdxelBTUOiW_B6TzD$_II8)a~ZXO(jXB$QdsveIF%j|y zltv>LC9%j`SWO2|jSpmV7M7!*HzN6k(M<3kVO_m&Qk%U>2*ty;r@<-IZoIY?KA3Wu z36omnpoJ^oa5b_Lpx+)D4FAHdbK}|PESK|5(Gx;-+f=;h$ni%|Xe>}nK4@wmR}q6; z+-2AC>&mhV;BO&MZ;_{@2~@xqfzjxoY?wEh?w0?2w@=apK_4xU%OaY~(+%tCaN%KU8+%Vc%K@m{0~F z<^B-Fx3YyAryw03nGym%OTPKgV>=T@Uk|GcXqg!R`I1)w_NhAc{^1ZtLERPQd^p>m z$A2$M;QpO8IIn7Fo^u67^$kQTo1H}1HomrXWp=0)VZeExSfY~J)pbP>=8WP&4GDA5 zhj9sn9sz3xox06V=n)Q{$5^{?G@RE_XpGWpu-*mXhbexI;L?h99TZ4&d<$f?o^=Fu zR~l{$>h2WiC~R14av&y&JOcs#H$SB0JjANuD0Y;#6L)%!D;r49iuAB_UQBLLLhA-R zt5%@hj(ai7e0U-1A8h(pH=k|tfjCOnVCN~Z_+8M6$gxQ{SS(DWzlN}M&A#nksc2z; znfY7?ST=>N3-RmK%h9mxC<;4MO{*C>1HBcOTaDSU>C#EBDE4c;1Z$9@5j?xLy_p}L zz5j~dE14xQ$c?av&xXD-m&GE?5Aw*48Ap3j3&@G%(9vaTAg7IN!x~VHF1dm-To&+- z#S4|Vu!}&BWJYN`bl2uhtRNgXzP-M^Rk?uF{wr>+ki?}U?6^$VX06SK!i2%ArCGVv z91WE)pr~`M23{V3-i(i0M~FsmJ{XC>b94>)djO47bA0LZ4>pOA zkk=?e>;oU2?lOiG?Jc;8IU)@Qz!GIVX2!)BvDk6yk1Ch)bL2pKElbJE#NawaKforg zlG%$a47kUa5$7y#Yi%->uS4T@!~h`*F_su8KT9{a65Cq#V7}#q^VBmWs80g+;`1N6;h3a-)W51_TvyYD4TNuZqs;rYApM)T1(1Z zm>R&t&w&lIJNvs)B>B$+I|HhdOaAw3%sbxef95PayEL=JB4LM7Au#0sBU+u7he9$% zF`N(Uf%h7NaMr;``B_=i**_}Qf7rqRry9`0swTQ`cDi$_r7_v!ryqbG;$Dhx2g;YOs2)AxURN@1|KdTJ9*@)#cyL({= zWeSu?OvwP48a%6*3&~01cALfF9pVSy7iU%0Si>+h0g~EFtd+E6B|u+WW)YX`V}uIF zj7d_C+`mhi1le_7*+S?i1p)V5;;LTJz+isuXy@d&2=g(=+6GXR`N~H{X3nTJH(Xwy z8f30G-!OJNQ!GCKXI*0t(FNLdiS=B`DnUpH)S@JPJOpuesj3x3H*IqT9Uuo$63XA= z{dQw@9ygCV5NE(T;9=1yLP90F4Ur2)uL^3{p6k9D3!O*PSTNoz0s00?WPkaZMEZ9BRn`=vt}+wD;CLOxym9xR{rpXU}rw_GTW4$7zGM z$t024KntdSF4Q25j1|DTI*B9nB2KoNum#0Z=K$N7-4)^8mwoA{YupQ$a1@6;In-*I zBzl3`j302H#pxopcCN;7B(LdB>$-z(KOcp;T;o^sGl(V-ldGs0PpdU+zF>V|ph1u8cya8Ci&2 zIbhqJY4Joc%jjEJEk-gd84&WKP)4t%?T1UL$r5Y97S{Z7Eq+Qj?98Ou(r5cQWKLzd` zPViZ1e1|~tKNuhJ+r*EuBp_Lnsg8IWx2vjiF5xa$TT|9?6@PzjFq}zkV_>pBU#e?LV+U9#mvx{k3DF zW{x`*C}rDLHx9zW09vhG@V}=KReFs;1vR#&zm@qTuOJSLP{tpf+|ZlVx|!*=Kl-P% z;}eYe{s-EBaRI*V+NT8blNH1$@t7IS9@q(Jm3?5^^6S~ue0!F-4%S!o%!Ut~=GH%u zFaiZk$&|7-nY2D&NH8z?a}B5%r@#wC9%KhrIOBMxd-%_12Uo@hs^YDYL}I)(HB|$D z^t{1`7B~DGS!-t<;ZiDnwr+pbjG{#pS(M3Tr-HE-rNbqGw>DBwFxbetcjWk!o(vY$*pfEvD^48^Vzw_(}e>Sj&Pm~kodUzomKSKh6~sv%0513a}7Er$ElsH+t!oX zUZF+pRuE0ZXSq;%G5LM?zKx;^@07j59S?9=n&a+&u%0ExLL5_szyQZK!A43oQ;ocr z`=ORFhPtrB;?>(-_=#I2sbn6MItC#evHEmJ=9r#JltSYg)$mcoL(%7be){&1F^@@7 z!2apOyv!`q6GvyAXbAl^ns2s>ZYt-Nv`-9Gg`{<4Xdk1kDJ%%XJYlDY*D=fFUu1N? zZfNNfnXPrU=gYg|!S1W*#OVVkPJh8MQdn0@Sk8U9F6ot6w3xN%rx<;%znb>#nSdu0 z)ZAB0EM_nGke#%0d`};`N6A`NGxO;O3wKJNK~CG*4JC6Nrrm(gWa>-qnf^9@{dZo2 z_`RSA_?TDGM@4jYi*41zjRdbw&gd{2ge`V>US0Q5#yn1i~IqB_gYdP^eiZqU5jXgu%C7qHPFHCC{PDY&x z(npQc04zs;xcw4*S|{$NX&L)oCRz^3S(m1{#?b{g+lDXGYllEs@~3jv6Y1^LON5U* zKnR1S4dEDqfN2Q+`Y{8AFtVI%OZg{yER>L9ApSq_pZFFOYT#0+iBTR1*u#g)4S)*# z{?QDDFkYVM`{BQ4|KHf~ps32{R_oEW!rS8HC~k2ZHmJXU)I_0?4CzAW#EL$e_7hO0 zKeVarD1rtUWCSh{vso1LtY-uMOP{RA;W(!9dIlnMJ_5d78QiGYry$l3Q5W$?kpO?n zBSiq+XPh_USA+>t1v++y#8F!UE!l^uNz?!JUOp<38Wc6DQiMhFMXf*pde9md#Y?5x z{cWFEE4Pf1xXAqD)b0VgraAs!{T6+x{08~_z zxQhCqnCA=m*C>T9;_f5=3r>q49y!HvtE5rc?UH1*( z=Sd@4a5q<*-gH5q;_lu57s+|+71=%k+qA{$hOT_Vm<3Cc;sa>7L{urVbmh+R!f{~% z>FrT=Qd2-612wBS>4S*;hb6@zn#;vvQP!xco-kdhe*9NwV|f1IZPALZzYbOfk?Bpxq_IRnHeU zUW1ZyB5%3hITg?uhY0FKvhycFDc%AW&KesVaNf4dr91p$!k0PVnsTbVT%+i=3&@FM zS|EH8ba_6ak%~jzdJ=ON;)OR-W9)%VnwHqIz0tUx1OqcsBdZ(4uPtCC8j%grj0ND$ zaiUY-I1(x7LSkiHob>u65PxO6%K?jQ1%71s383=jfum;%*k$r$!Tfh-%^gJHG^4oo zJdvd@%&wc0x(diFvIQ|!k-*y)oH1#q@d6iti%rF0g)B*1{EqUpfSmsA0_6HUUk+?# zjS*}Sz;-Q=<0~9I13HuTahIL*KOrL62aw-5XoF=dH-@>DGjHOz5$H)(gI{@f`rR13 z-L3^tmx~?;h3q)FmE+asP6Ql!cY)!fPUc-aXVA3e)pV=U(O06VJC&7rgs`cT!H*XX zAE`aH4#<<=A~hnJco8#J;ivha6N?h zULIASVFS%YfyN^>y$is~E+o5| zD|q;&5^J1`4T`bho*w2<%5`ct<9BlhGXv_ELMYM&-Ao73=9Njp!T#XlVAo4Bx7-P; zN>T91#cmF1=NvNl;guUnI6WUUQsc!6=)aSo^0;d#&#b&8pzc2fzzEba1wpY90VOC%5>zs%Bte2i0h=5IBp6U= zl1gr}iXe#KRzN{AG&zXmEQ$dG&?L!<0h%aCY|^dS?)`oHJ>#AqcZ@sE4deXTLw2vZ z)(kak)~tH!DeAKeq-^C4$GLT5nujk`P1cpn{4DdYGVi$*JB^&k(z7moAa-?h;59D0o6bfT^=oXR;Re8a~w z4z0Wrwpc%P)BJY651W&Qh~eCC1g=Mn6)l@{H^O|_L|!vhcD76oUdj@(*~{E^LxTis})0kJDb$1uMjEDoB`Jf+)*@^2`uzNijTcbwmkVo#MQ*nAP5=OU01j`>DA!l zg`(aXO%bvAJ}+&YnSVhDfQO}$*rH=rz*b&C7Yb9_rdt&4b3}N!guU`NgR`tz@kM)) z^-m5GH`@#(`94Q@pM#5&Espk{r|L) ziZKK_zYeY#Tsze!^{cOd&E%BwBAhU+#S+oxS!1QrSq;@)?;*1J))??b&xmzO z-gt7^7IKY9z)s^qMy0Pmgk2wNOG!bQky&o9B8`F|qei&Fy&LcN3c_Jlb?=JHTM}u$ zeZ&`5E_AQXzs+j!EDC7=$M8RP!-YkDETN{%OU|?k)P zbly^#2UbB9B^r9t{bwc|jp;dEkUNy3r?}V!i36?1G#Q9Ie#Lrlf}N);N!HEg`g!L$ zu^b;ST+T>wUjL5=Y61CI@}LUBzagVJRVqVA``VP&oSB0u=PU4RXPz^~g8`EQy)9E! zQfl>sbTtCXU>E`*MN3=7;Enelt3TqiefnNi47esj_EaJS*3GO6qrbkM4ghWu=ky}0 zp-C%q?+pN$HL1X07Lkwf1I>iey9ThKBFEBpxS~Y08n_}7Q6@$*WL6w9X<_3N3gkX5 z;ySDF_4Ym@im1-Oy%JbJDzvzLGtY&Rmns{R(p~t3pVJQ`TCw(>aNlAa*F!n^Us-b8 z!y^V}5RLOcj3(@7P60PP1r!6cCT9_kOLGUx5*n?j@9`pK#S>7tB4ulRti>e=;SNw1 zS^S0j1KVb+$GG`$auP}|D0Kbbly*{Dsh{{zc@HcU6ws@gZCH!*EnIyhWK2|JRD6W4 zzmOS{xDZ!acQ{}%-&U+)i$!?^aR_BCH8VBRPG-IQ}~So$~ruMr04m zQsUOs#s$J)19>I@Q{7za?{gnP!rypjKt{j`&(XU_2|kf21FK}8>+;gb>;uftaiPo$ zVBMr1HW*{-)?@u<&y1`KV1CBoq2C*Ssi8Z@!iN)WnGg82*xHPK{7-i7+0x(Q4dkEJ zwiyt_{Z}rwAoBJmnn~8=JV} zknta9^l`2v6nFT~1#2(chDz3Cfy#We^zOnNLbqI4!U6F9`hxSux4Y8;?KnB`L`xTa z-+_oE8xV3-B4iJ;l2wj^JNYR7J|#+#1}aj`&>i4q;8D>8|HWA$4}|XgZ?IXgN-{j3 z>?w_0945f0nRml;AqpO$u`yH$$7>u?pu995JTBt5a1@UV9vs|@Jp+B{cF22}uhkTE z3Xn5G%mZhwl=MI5nWr9xLyiqS`~PY(%m3}f{D0Jm3F7tt1qd}u%Jq)FmSG5-e( z@c%z{njmJ12L)=%BqJ~ug`LqXJVDyYEa8;zjT*qRq6+G75l|k^*hDyEoy#~VF!--~ z_X9wPqe#;tl*0@c1?ER^(+vcA>Cz6fDL8X0f}}S)M{T z#P=_cv|v01n7)8`b`O;A16?1iQb{}N;*XMcJJEkwDiJM1rWk8X7pvI{boRqd(Kyy;_M${dwk-m zo;*T)4Xl9YprQeCR(Y{c<(>^>Qv%C~2NU|UpVA*0>R#KR07Md515B7Zrhfxc(R}Iz z_$&MG>1t7;uK!{;aVTg4cb;Xz@gZ?5g+&q(sRIudZrfUt!bxFOM`AmP714QSB z3uQL!Y4CGtaHX3?M1bh}_2l{_av#!l>B7NnUcV3ree1)VO_0i6c|7y#?!K^qdjJa& zQhM#z!fL*It@WytNN)D3y9xF?_eK8#IlHQi*NWWo-1W!BMo zXuxj6WN1>B_MQrtjo9#e^TPOJ&l%CLCV+*_v%xNe!;392=YuvD5h0I|l9$z@OY`kr zEjxWtnUy?m(=AGSMi>xJdlDiL$L^~Fmro=kX&3~y`~Vq1(UFYv_c>*YWCD5+D4A(I z3j95sSFa+bp-Z$sKPH;{p8>aGb@_??3SQ20r-zZedIj`eKo9R%f{nJ~T#ih**YrqS zS7K&BT+OT^0fYX0g?AQZ+7BEXM;eCC1MxQ&W%r;?0dWz?tEf_Dt=)^8{fYOu6>H#W z2(ICj*|t+e^}kbLfr60ft&q=m7CP$!Bw+2;(8cOBib-a7GUT9J$osE7>g>XlZ)m|c z?q`_&{f(22_tHK|yt@{H#9w6@)@sHAcNZoiirt~v>BZzq&C+Yz$J@|L5jy&PY62ob zGp2>xXzl2wnz^+8!O9i8YyS6^;R^XNAm=rOZe0Y(*3Pq0y7H_Eu45n|dZ16fRZI|@ zo}P3E5wQWi=SeCJqcgL3V6FwwHk!!4w5(c_WVANasvnngrjh$FxC{U^aD^Oy?+v?geBdE z>)>(su}@i2>LJfAy>xEKacLSP6da$`4dIL06b@yw%L+ATfhAkA(qTB(s#<9oL z1%2TPH8@_aK?$y?6cox*&wU+T**^>Vbs%w{Jf#wV2RJ?_II`}*L>TCYXg@r{%F~$y za9PEj2H$Mjo}9>r=UL6q8-6n!mxp&J9EA*q;-8vmV`u$K!G`Ii&>-wPYzP4JP8u^l zMGBjy9~cDtS)U^>{&8lxsgjpSW#Y%vp4`i6cqs?&&B*9EaN$00xYh2CgdA!P;0&HI zphOX|5U_7pcsdmUhMQ)hj@ZTi7TSn~GqPyi>?A=^mLuhTXV>Eb^U5#HE}l-Ej%}N5 zTdT@RGGA9B=tV6`Ge;G>>i-2$5*Ghb@hpN}FzdU_`-xr$LB)Ga_J1j|RO1h3G%V&A z%lZkadm0;CAz3I1X5Vj8CHLO*Bfav3aek?y^gD48)3%MS%4E;U%-(-4qi6+mYLA3+ zv}22ti)}yJ6uklGWw-lkM~>U@qIJjkKj}&ikDCymFMzz2W^noOD*o$oN^c=#GtiU& z4ZpS*QI$$yP;4DigC?9HHAw3G7a)=lQ_A#%1Och@=?yP=lmq=i*Ixhjx*IlnGPp(> z=(=QQ$rw0;i{T7zw+r$8{gai4n55wCdW&^_Kl$^sOQ6YJ!MpFfJ0~OqNX2T$ky|ws za;SPChsq)6UlPy2`qu3;B6@~zKt=?A;|pm!br=l0MZoW(Vy5moG6%41Gq9Yix^`iX zZwQbNQr8xI8GRSVJK%89f;jWC%87$X+Fo}wxFiMsV<-kq>Ix(2^lujn3$?uhoXiPh z6IPl+#Q{J~^C$h0f&JQh^G)_(E9ps#OAn{;J%!H2-IWCC3aNmr34fD66fk%4nWM`>=k;(Y(L4jJEMOYfnh#7g|uyQT_OPPZWgoUIT_8X;Di~YqJrVlH+}*;jNi5cOd8N1TByO znfRae2Evk$G?EiiS3f-+y;KK98EK<12qjr1XFZ-k_M&=r1xsfrE%JBg zsaANNp4b9%sz``@@e*S{V-aFAG~zGWx9%5u&4uQG#+%pe0cC zbfQ`tz-`ofHFV(QMU?eKAgwa=Z<|6vY&`IWQ0U4UAn45=A`;jL zMi8U#Q%|{ef!aN$9C)5??{P0rMWgV%KbcsXbg!n3j?LYR&g-Y!t-Fc&yYOYMRo2Y_->yJ2g}Hg&;a;ICErO z&maA`8$X58kSc%+`p{I$-z5AxCCrO|~e|OoF-x z8lNMsW+UF3GDzQJ(A~;w&(Zc`&6D&GFc|szyb$+Z)x=aJ!xprXC2#VvyNskyknhv6 zXM^TxAM^&YZ1&29Y2XVd!QELfkf(By3-SRSA&08iC%I+@gub^mcCwNj=icfl z@lT~53%XagRzVF_)jNKv0CTYnG6LE5wWX~qFHvA4qb-q?gtLZ|6h3|wUWS6TzP4oG zbZ2DA_j`7+l&B+8pUD^vnS67{Z&PY4%wDuVoIG~7`%#)R0?;svb3>-s!h74z(sR$M z=U#J$nY>JU8uaBDAl(agGrVzezfcRQT%rty6h}J*X^9XNv_9`GQTIzVqPrXNfbyye zC5HRidp*wikE`+M$PRfJj}%4fS-(GMy9iYlc55b?+4%k#Rq&`-qRF!vS<<{TdmIdd z1Bcg;aiT@NrMxt=y+2~Ye`{rP7v>x@m7nNmdYxzN5jrZg_b&4Da?tYgdeLywF_(zY z8Q#>q{aKPNy@Q7JSv=j+O*=O#X|D*()?(ee!L6@1-aqyG^z?*Zn|IF*A<3&(X50L( zeRN+=u8$YBxX37puPX3g2e!MV&9^96ke%|`sa#62E*Y-VhbKO^R-Pn}XAB`_R1s#c zIKDjI@sdrcT}^)`tt5lwiYVxz5@u7HVXR(~eD#Z4I|rzDd7hL?$RT;S)y>_(UPZ}= zLy%5qn9b4iV>4E{v*d!>K#X~YbIU2=FhSZQ^4Dwg8UPkL(Npa6 zFH>p;8xN*vI|aPI@F3Q2Z+fYbBl+$&Y8$$)h2Z>PPXH73tsSuq+9F@&-d&J#v@(;hI0olD&$)qQ;#Rx8oCPV}4^ z>?3zqy4h9&wxbHt7E^u_9_%aJzYMg6MNHSqs@~Ku4?9KjId~W_Zca36cDmM(j#PsG z<7CvlscAG+pCz)H8YD(T(P2PM_4d6afdZNV=KQDNY0^Zv{h+X@y^~hzX|*Pvp_0e9H9K+2XNU5H3Y`=bC1sD` z!_5=RFFxh&T8tj-QWnEs1GP24qM$f9`Lx1iz)WFOU@K<}Fk3~mLWTJ3K%B6N>7s$8 ziJxck64~erQBf)s`}Hi9?yL|g#MEV?zdWDzG&VP4PoJoDjYrmOv8E_HMCGSbgci)a zO=Rt*?luaP8m@H@4+=fNxRWmg2NoCcb8DB$<~kAP_^g7ig1=(t+MdZ-U61iCS~At` zhDb6uA^}|5V&UpoFpH8jT~xaV{<1E_xV98vZqUQXyRenrAGl&X#HhQlb+1Y}Q#;S= zdQswQ@@q%gu`X+X=zA+D!4{!o!Sa66-NJ|WF6>tf7LgNFaiGG~kEC~GgEQp~p?!RV zGfLg@whrSrj!89^C4=YU_)bC7jS}_cj;+qo z^_U}Oe0$}Eq870^`RA8{ph9B<29?HqLOUw1Yz^CPh&yz<;TK_jHSDHc^5^J>@<+ zv!_h4`k>*&LX!kdaA^#yf}*&Hr-WglUmC{m(q(cAJMymK9X@zIY(eF1`Aw?BM5UyS z(zHcB{aCV%yzJniZA_D&AI5z!L%B2jUVlMqrjS3zmt4LliUntBYCj^-m@JxMD{CFV zliqmgdU4F_u&MD`uODBy7qv%xv1;90%2G!8{NxKaD?kV6=8^gCs6O1ZaX74roy?yO zQzl<{iO3R!gU+SRkC4?MH4l1=+*{A==&4*jClEvVP&&k2=4Lylv{6%Q-cOIX~bJ%>nm-}bwN0n$dwINuMbhfyjdhuO{{M0|PT1TcoNQTb#(>{EFGPd$#wL=)TUnIJTc`zX`6 z%OT>PruRY&(^&YvbSX@hkSEwbRiTLv3AQZh3r!h@92+#2RBDX+_^TSTW{&S16-9;0 zOjy$w);MxI7Pym&4blw73mXVwN1`$X1#%e@j%Sx-rSRIqbjpx346+^wmFos^*N{eX@A3;;{Kg5ND&a2N zKgtNI*K>I!sBuHUhEM;rHs{N{Esk>ImG?!CCX9={kdRME;O?A@3#p8b+s zNo_PEC0|ch)cu8}VBz7VdfkkYjGg-w%f{RF;Uu{k(|k))F5Mf)!n?p*9GqpixA0E3 znlbI-my9oI4@4CWG{T0{j7HK^^d{?%6kf@W9cSUb!KnC@M8*i1mTUDsh~<8}AReb# z`3d+Z=u>0W9u0T2muZOQ76wiRV7Txvtm$pW6^kYYrO26KI-vs&`_2W(c(*9iCJ!#x zT~2S|6P{I$wPK{r98d1>P*U{4bstysaWZ6F)V)v&B+eDomrm~KR*5{H<8`o{e%vo7 zH$lGaygv(JVOcX@)J%!E62n#a33k-RM*9e7)RoEEJk6xVt?e7v)cE@`{gyGBBS9OA zowMf4YP$=)a-B%C_XA`0dJE*iY&v{DH!E+psK&9Oa#&0Y1eGn(VBmB(ro4249*Qc9 zq&hlqc(&`8K9q+d;;e-W?GJ0O6SzmES@n#)7mPiyv&KH{YN=-U&X=|>iLDbg7%eInM5&1#gl2w zr$Vy_z0*xn1XsEjYeN0?W*ZzcPg7Z>CGaa#ZESSB%_7*5nbNy715%_-&G`(sXt00NIHNHEWTZE3Q8()6HRk-N!Qg*w=5aY20?>dU;#CLDR z9BG^o@RQ*awiuDJNn(n9o!5SZhKbHhg;4}AS?n^QU$V`kKviW{#qklgbG&TpQ3Gc} zX=cio_}}d|wwMCTs0SJNkiiFnV#zTId$Vl8hD!Gm7A5TziVga;ccZ!?L!PDN8+XFb zgf_w_X*g+=?F7GvNnIZ}-}{@+&@=Y;viohpASbQ&%qmT4RH~1{G;~YeCG=!3>q~K@ z3ME@IWe)D-%Ui;z&#F68-4Q%Rn)Sl&Wb7-iCHwDKT&%`>Iw|nwxw`s8Ic}z$=PE*k z-`&;M{Utl8J3JcT7J9Wmt>TlDl!dXS^p9s1zgVmz!>0O8Ha6WSCcQT%h#mjE zr~r4WiZxrD;IBCUQ(7vBa{4w&VN>6u-uv#Hu+rO@A53I2<@TP50JG?^ic;beIFZf!56Ts-0;GNtVCz>A$+bl=(Gv=ZP2!Y`ydBHl z6~#$y0m1WFSx0ey)<&HUm(tab3HECT9Y;j&f2Z+ztM^HU!JN?Gn4{`A05wqyWxds) z*0|YThoPGC!gj(&oQ|i%zF~CHQWEdq_Y*G9?!XaLE4EX~kthIWys}fo(rz-~;+-TQ zUt#@s#d^ifzv6_<#@lU!S~1RatO?-t&n7P8-nn7jA_EIeN6SVYI+s-! zneFI&*|3z>k+zY|V&^fgm<;#)z9HYSBsLw787P7_Rn~S7Ug*@*>fIkENx{hUz_WhS-`uAz-a?e8Gh0PL@CoXYX7&^48H3Zu7WzE0U#`597^~@$yHWqnP z6Kpz{&hswo`A(C=9HAZ3>9$U`2V8n`9bNHvZI^o$yN_T`ce2z#9LDl(r)D#UX@}7V zj9mXD0fG-R5bCb{7{(|+&dV8)=U6rd_ceE@K$k`P4g$|j=UKSRz5gOik!t#xOsKR^ z8~2)Wb}4O;p@?E>$s)io2d-pcbX*21JR!j(VVa)MAmTw-;4e&{EDWe%|4NT>o0Ofb7Q=tk2s$YxF7sFN@qv z>-p8~i+?WchXY15wJBZ5dDYp#B7Ww(-mT8D48y}i-tDQUW$F#WZhoR=KX1n`#HHZn z5wNyYKGG{C88!bMEGGGk2S-Dr$J?cv`&CV%xfQ1Z^9xEN{2*;&5EjBIL@z@Ja|Uz1 zI*cu@-J}lcri=tyEq4-!I9gg6E?1`*G7$w+o6ou`OA8a)1aAXtPH(eenVp7*HSJlf zP?w1oHtL@0BkBj#4~q&b@{6K4Wo<49=)BP3x6OL`I-3;Uv3JOGb>`QTnq9wQtYu|w zl{5kZa!uy?wZnyz51Xrkiomc5wrnrbbeF10U%oCibT(4b!Cs#B(2(uy9=Nk*T*L=y zbxup+vx3s?(%thDIM}gcosY^PD&$AYSYMp!xU4l`Rm$EkC`vtz&yyH{A9<<%HxPGbBZhw>w$-O3QW(}a7a$?qn z9r-ya)qG=+7tt!B?%0}1Skd6WXd0#d?h@Pw*_xy*O(QZ)PS>*p%#pZHU0yROTp!R0 z_$~W;W8ZDT5s@=%U|PwAWM*3mJ$1OSBdW)_tV6Tnc#@{0RMOem1CEwYsow#<_e$g; zK{)IXt<#cT^l&s$uxlrezujpqN=gBn9*c?;b_$9Yl%{6Jq^k9{D_|`RPDdIF(m3J< zAfwDN^7MG)V$lOai({D%znpZa;Kq~wg0)kC4w1nc@N47Eb@*+%qxPe;k_t?6Q8CGj z{8Tl8?WRHE!lrR=k@a-Kabh}~Wr^x?=S|ktd+MYliB()#fo{Bbk239BFfU-LW}6-5 z?IB+)Pc4^lZ^v?cDIM1+UoBm@Py@{(_WJd`I1WBxzH{P{=>B~u=DY3|hYRt{z2?96 zVD&qdv@JvQI@cGH4a>YAEEy2#qBaX#*wRV<9QKCWI>zs*MiM>?IXFmZWLR+oyi=5N z(f_KWTcgGJ8_dztnDj5{%UNZDxAqL#0tv5S!c6wHM+!;$on)ztgf@rU+F?ufM2}P0 z=W9?DSA;O+*m7#o^nxa%d_vy-MYWX9i_Ewb>N{+{?b1@A$8g!h4~X+WVwY1|>s?u( z)T{p3JIxFqOj2n_n)iz`OfoKC$X~vT5zB7Ei^V(sHWIfndh4y+)ixi)g|pupkeJ}~ z3do<};uH3h&VED3`{k|*@xA`!Y1<4eM4kv%H6FR0c>xI%i$H?&`R%h6FY#O0UhT_`&&5RsbL{F&}mni9;lIS3y0PWdPU;MBwjTB_rp|=t4 zm%~+)qLf`>qU3>1dJe#Ft8@A9)11Zwgrb}sA8F63yVQ^fhj*Qg9k9A@zc6``ZT>3g zit3u%IVJg4p0jj@76tMFn%k({Xr!n!HbTqpd7oq~Je%E3mO6~(=jeZ~%`a}bCn*CL zG9Dl-cDOhn3(rgS^!VkBV_-z96kD5&8C`NIZ3xzSyp ziW;x4ZQNcKZE%t?N|R(Q^x3oI+c^u}x7ql>C=%H~EoJ5)rA^i7t|@)cx$*^ehf;e0yl}_O_QUSU$bpKHEE?+|NW=QDS_WfIeNh{NyD={82dbeB!+ATP0$!PlzH;elbulm{Yygl3RVTMD6Tzq~N#`z5q-61Lo4 zJ4r(r

5;f}%qGF;TXX!8cLbiOryRT&Jk1Ih5XoA4Ood-Ga{W8XTcgHF@%-S`7Xj z^GN089R?S*;T;Sd0-p3mR!2gB-t)6&4dCy^jp?hv`;j8gQ6g+$H%#gzcgE(PO5a`8PVw!FA<)FcYg!2#FhpV;s6p-9*f-p=dH&5qPDX^3?wRQy4y!p!ZF2!1v*%m zbEj1}pW#j*$jYH_PgKJ8a5FWyJ)lP&+SR^1YfZ82 zFoKVzI#8b~V2>(wU_Ug-7WEfB`sQ7Fru4l>f9^m_D^L4aCN8-~^5b#E^3=oV{~UyG z9X$N$G|VA2s~y#?dGZ+^?c%$xkWDBQivk z5cE@Q8+yAxE3f*h<;ngi=&*o5d!4U4>=Vd@-8IBl8*g28EzO03@0vQ)?v9pY8}xdc zMkYOa`7>G$8D|I1c^QZH)X(vsrWWHMHpNL2x2cnRzQf}n8kh#ipUz>@PP8uSRNO0( z;j4|h%x5FrT3OMo+yLoqJ=_+6{{4tvaa`tC?3T}m=ZeG zub~&7-EU^7rbfTYz_0zns(wiD9_=bQ=siT=Tsr4j`RyUUy#jE3)@riq)6uYpXsH8>18!jBr%}3K5hb7qsxh503=F0^SgcH>&#H? zz?GVz8c+Hx-G%LFf_b%qnOb_ODF3Wzf^t;wH~?T?ZJzDX`)~Oz@J!l78uAGhX+?f*-Bw@xWT|#Z_wOnUJnA1bkeQsLubq>ok`WBzI|t_O*5yU(>$CJ{W^?cO0N^+V$gyY z!-29OJ@$C`+~zEw7+QE8F}SIg%PpfBqTcs*UKJx+#{4<{$qX;FJOd$C&U3tSId>64g+Kg7y9z8^ozy0BE>7Ke^2#a+xQ}iX(&y# zY08cj%6v-#LR{?`^S4-e2y7;jaN95JL4%>95eCJyqK6iX?$KF0@&c*iz$}E6%uD@# z`epo=)V^QNcp8lc6vGUi*2G|C(OxT~JDVim4KF`#uVa~D(s6&d-VdZ43{V~G0S6|^ z_wlfHq+D7tB$dbcB`-A7fSv||ynJO@dK-^n{urv3o%e zAluh6-?#um-?s++{#$76-RK!m&za6N!pdz(r-vdLDWP_xlW@eizhFzak6MR-PM{VM z975mq-zPXGRU@S*vQQXU#ttptrbEj})9g0XvX!RwS*ltBd({~WUxz)EP9LUHWwG5w zS#1-#M>ia=^Tb)|YZyBGnbKHvNbiTH+o*Gy+kaN{$5dL^dfhE|{9E5{{h)MDB~=_o zQA5JmPbgF=W+vg|a5$D24q07uXY zHVV|%?C3ugwCO)^#v7=%HENHrqZ=R7TLKIr(KEXj4Pk;dLDtO`RH#!=!Iuj0^HuFK zMC%f}(=U2Q(Rg6tV3GGw{*h^xdg0}r3O=`6I~XPgU;mexQ1n}Rh4;_H)o6T-f5yiu zsJTs8lSx;Mi2?0!Cc#J$Xwj)bq=!^^i&XfG=ORuXmUQ(y$w>J>{vaMJ(u6Q*9lT8K zX~A;nZi`S|Smt)h>oh=%v4%c~r1V5UET(P#ySDhlWDn(PCCa>$VQZ0!nZ+n_y^<0v z^S1j+9b{P+^kn|6Y-j$STi#vpp}l$M_ffujECg~c2d>n?ZD2%ad3?CO7Eb>{tIAh1 zZK0G6!^EpT$3BaILr7z}ZUkVw>T`upRtprdAHCgwN;Ol%W$< zY+v|VBXFDQCBSS1(sM%G_Hl_+y_mu`JzYa_*~!Qb~#@6U9~ z|D!X3&W!p|9{*Af7RKa!hYu1DUGU|`+h=r8o_YYgeHI;N@J5)xEVi*g%Kz7ZGnqB^ zVI)Pvzk(@fQ5j`HJr7v`jZU=Uv_zKFZDFv{{y5;7-iCuv@rM!;(Yqp`cS6EDu$1%U ztt|*=fAy6{zaR7dy(}ph^Nt5W| zBVWQmJfh`0bSTq`gSGRQru5|B2Q-r)bR{q6>3KAdD!g}~Upw|@|IrKPu)xFp?Z?p% zCh+x~*eqMLq?+)m142?mXipl!4CxpibHGq$WfQu!{&D4xQv8qe>IsN}z4*3DWHB5U zgoEC$gJLB!!NDY)i6gSZN17NxhIu^GV=2?xjWiUF(vwG)FsZofGhvj)LR~RDS-qd~ z#oh47ZUH@tM4l;zch#4t*`rS8KvxVzbQ_Uy--A~jH*%;!sD{ilFsk&6?SI669lEue zIQK`fH^aC>SLYll^K~4=u=V^lS9Hc*Hh?xS(mUz(<04Knj#6)I}&7 zB@a~DT=$>t06R0UYjlvJ0aH;|(r?^2MtSdk_+$T>r+*~U2Hu5tQhSd&X#ibGGmE2G zUk8wFm)Y)tblDl$%)%Nisei^(O@%hM zG_v2LnK%PqjLFlsMQacEcEiA~yJnQ*#e^SJN&cVhfPWL=RqU>aC?1}B6*m7 zpnrb8$)?Pgjws+$clTy!G0J*Po$Jlk=V1@cdH3*W?de-Psv)_p8ZczBRi=Y`D3YlQ zqYyPsY(PB^p_6j{+}>SmJ0gwidK&kb_RBhdmaOz35hL8qtX!mJ{a3EZf+_Kx(nF>*Ca~6m66{D&na=2_WEjAW zxv0t?4o9;UPow1@HI!NT@8i=)C`N&!MetGl;Y_4_gDXKM+;p{GQpT9xA*h+QBK}8; zjX;WZTxLT5mSS-Ba6wXw=A5{Rq5a?Dj74gN2d$bNwdH&wzjMfs_wivvr}0SiGvU%5 zJ5Swl3W=dDl}yS<^AE+pbBk>k)5}4ppyU5T4N(M^zhoO@-v#y*@*Qhi)G`oyyPmVJ z5N+6t;LPAkY<<5kxA@O-!33I6xa2003*HJ&o07)uU9wYTqQZY_JSHpn?X!tZ@|l2o zQO+n$@6LztgepP01oa{$|w<_M_Nf>e*4 zFumZhnVQ|U$WYs`0E@g1=h3R+QA@N-U$Mb&GO_VTjcg^}Yi5A^7pXrx+AIY$`!Kkm zojGow7qh(oQ_b;Ks0i_t2Ce>JUxwgA6(CVtq+JK)yf{xjo1xAsa2v9EcNEqoqyz-~ zPiNtL(pijAbpLs_7qUVyUxD1>JwW5Ei-6**Jg*T@Ypzh-_&U%?%wWbEAaiU62)jI@ z6_*~mxjy0efnlDC05+X@C=`{P0CF}%cK4D3~*XKQ+e*(cbpP}fohMHGr&W9 z4I)2<%xe&p@dV$FJIWf+;mgyC=5__Y69hm)>Dg-_DXSLfvtCUDL~jOC{5*jvBtKE! zGtW3k_xd`xcD>qQi}m(53Je%WkdO2Q7NHaHHZLqnTEqgnc)VpkL*`opB&!z!=qivh zAD(Xv+%PLKm+S|}w{@{i{nRo-zIa0-kwR!HWTMsp?yCl7uLgL^I#nMue+IIW^Vb=o z)HV2YUYzaiq%&Wh8A`g@3$V)RiQZy2KHV`OWpmP-t{zX19E4=BOTBdfnJWmWegiPi zDPVjQ?#(k5;e0{#<(TdV1o(6r1T_ax9Cv9%UY`%Uyb7s#gMeLh9ZLpW+YC`xqLA(` z#{mUk%a{t3w{@`w8kD_T;zz7UUMVyoV=QDFNGlb~1IkXUrq))ztd*e(Wq`d=@NEcB zB4)X+&i^O^&BKcI-CP|*7}X0@_5kjyY|qHWlPgutpy~<2bD2LTE(1ck+F0*oOLMI1 zdnmB|1L%@8vU5FvWF-xzd(wFUxAN<25%BD>y%n#WIxg$C6c(2CIxAknWPtf^%zX81znQdpT4zwpuYhg9k>J z!N$!+iOFSvJ&pl&iqvxA$ogxb@@ozNXzdJu&&z+set+N*@X%SImtu3lQ4(sTO-5>d zU7wc}<{TOg?x1Q&5$O}`U{qeWOH+NJ5k^=Zg5_qr(*qf<^!w#mj374ToxozWgOS^r z@9=W_nJ$QHO+r$FB)uHQfa0Vty?hr|bigJp8h(2OcgU7UX}Q43tz z<|4OGSewtD5Lyuu%$tD!G}1>}GA9!qzE#V?X~TpJ)EHGFG9G#NBGH9t zTV+Z5wX|G6d6q*`7*F_MEljq>whk#d+N43X&T*FJK&XsS3(?-%R*1@6%v5`EmQN>t z6JbebZqkcigNhnYo%+jiX^%&73$X+MqL9rC@Y$?uPYH`c`{5pjW3Bv@bR>sXB)x<% zDZ+#+6SWjQL_UEXss)W9GTascaX$q^j3E%peAlt9k(<56O2%Sq8bT|g+mOv>0lLmU zt#yGJj?Tj|3u+Z?=h~Z@QH=3H>LBu<&j9OS2~*aCB-knM7iULChOH2SR@8qyAPa(R z5FqZZcE6hVI@>Ww-n?@W>Sepk9(5!q?Oo`FLrQ#e$)tgCq0@Eq+%t# zJ0d1j9}VXcSsDT5PSmUcIH9(S!G1_Z`7NxOc!NEzz1Bi@@Y*e`Lf5QQu6KQQ*5=PU zl>!Yu+P$)OW!$l7hNVCaJFsRc!!hq0Vsz@V(-XH)0jx{sf+Fo>L>pMq)C>ej_WWA- z*)y|3h~dyR$+PEKM>eqEC|5AuqRshn;U6r(a47N2U(F^n6Znq3V;mo$_(c49b{cKI%Gk)=y&dIq5hI?~455AETS_b3qxtnM|)AwE)|O`j;@#P)SIdP2dR zV>2&y@)S5^Z(4QIbu)?`OjGl_C?ud^yajLI$X1)ZBlJaR;c&pC^akzakVzyf4Pf&5 z7xPXdi$r+Wqj^3ner3z9)5f2L-JWtv>vpbqNI4QOhsNQag%5VGkFpr?ThwIVJm-3b zSPnEuPFCO3ir(MTqQ5j=UpT0?(K&I;)LNE&BC70gy|8T;!Kg=84=aaGyMX6DvSJBdQB5|VM#~X@Zd|dmTwW4xz!YU&0IwJ^d(^0rn=y#_VBAaS=$cfJuk*SX+)w%swweu%Uhf~d8{DAYM!oD@~nPFiZpDl!D2H@nm1BX^Su&y8ID=p} zOz6pz1S+q6RUJ@vObJ+e#`?8P_AiGX3OS>~-n37zL3UCPf^&y2uf35<(x`l|bbgfQy; zCv*}ga2W#|%OfM2kzL7y41VQ?#m$!S;?~Y7d0BQ@jETzr}r26iq#6h(Ay1Y|W%Ny%Z$Lg*x?>W`*tvWsf=tajZ7$ z8}AOYsU!z1S*$(JPR6%7+GZvz`dRuwE_MBRFfz`=c{{btaWu9CnYtWxgS`TAxI^&lhzIbP7|H4aK=@bVN3RUf((YijmP2?5P#AVbJw zZMrgL!<|gZt_4;8N@l(LDuPj24p>xgVItuu4|D0M1t3c;w<5l3pZ=*1#5>^|P&r@d zX2U*q8D?Qy$T`>>=gvMwBD2B|Vp5^{s5x|48!+*7Mqn6;U3S1dZhDHM*S5olgPi6(h(k|S+Y=lYmfX}>c93xO^sV0#XfsqAKiR=*p`?RrJ2k^Wbt9~ z|Jjx@3k)Pz_#j43H4c?Vu4{35K2iYMFX1A;Z&9+ARfa2f_k;?D9xT`GHR3g5BAV2L z1^G$!9y!Y}74i(vx>qhT_OeiZhQ~^u|u^0y_-P0P8C%$vX&Sp$H z2!Gkmc&bD-oJjW(66!_wYb=`wcd#@JPTuEI=pisd;Jm*3j$rxeuV?x6=+8}6eY>w` zOx>-Ar8lALOjGdzBG}J<2GuF=lH#&|16n&Yr)v7%ZjVODr!BPGYb}vND-Elc*H|Eh zE}p;OI{3r!tAh*-#vn}7#LT<(ot;(JFVXXM zT$M^k$$zyFA*_NvS?;^~;qg}vhzBT<$N4$=B4!@yhZ9-kL)oxmU5lPMLwK!x!h;+* zo6YtX*fusqNjeG$Vs4Tvl`h>V7o3gJYc(&;}mXmtwZoWn_m@}phFa_ zwQ1f=QjUYXttTH2l38Uw8c^xy&pC$d#`3Ih%m%q_5dF-^0b&kmhBZ+#{cks*JY0!) znN8rTJ@Um`L;(Uxgv|^Bd zLSHHmA!BCGkOJ|tB%r-6(KzJca9PKtSHaM-UVl~<;kEjUWG33_5>yA_Qgk&RgFAgM zuRoW$D382_Fe^DpO8IIc+k}}W@T&(`dt_6b+{G!|;S|_Ur;dI?K9f4;GxL0F$fTh? z1PGet4nHFxaJkb53G1(J>UlUk($I}5hLTSj&x{R>eIka-i3)~*B`WuRHgQ9+%O5Tj zONLRZKko1Nv80jV$PsI#?9pidKCZ=*F1==R5-)%GTs;3fa!;0`|BnChbXUn?>`U;< z>5vfHk+vKwMXR52iDR{u$czhkZlXp8lL&ZTGPLNAtfwAAo?UU;ADc@kRvT@@hb*xNh_s^OJE~TDc*-QN>e$WLDlss%W zWrLjB_Y2a%_;f-X5QJoI8XzQtP)P66=?OU3uJ@-_fVQ52tW0k>Ve_28d!PUuu+t~{ zWpQxB@qo))Md)E`wZw19ko)Tjm{Q8Q8ocyRIuszId!CmhDwa=;hCIFJT{F&!Z@&6x{tg{LGqitYh@+I+BjSu@^}ZK?A*BK zz5QR2Ro9Cyr$A3$1M$f>poYV6;DK=mw$JkA9OMB=+_M77Q?&>%`+VYfQd{lK4mx>=S7Hhew?gjV<<8YE z<@%%LuQ{B)n^tG(#@7@#ZuS!vq(T0R+X$rZOp}R6Z2Qr5=M{JcS`0@I!W1|0gGH|> zY{r$K6EGS4S__>Xhd>2H>~8&m{(S@XwDk-|z%OVbu|UEc>E==JDUJ$W-vc}0%LBMA zaI@zBAMCwnP?X#DHz+6y%~4SsR6r$&5(HF&5=F8gp~)EoO3q1=1Vs@90SQWmCM%%i zAVG{^AT+VbDmf_N5)}9kGv4+Z`cv9HYqBj;}x!v$+?neLGWQKs4psF7(FV z!p_BZ!$}1FyBF>gu~n1}{mG;wlkeJ5`tCIR`m|C4B0XI@Tqjei@G4hEGspfw^ABn? zMa>{yBFlN44+a5ED7oum1l+ zg1r$KAKcCq$m&Ut)1whkQKYMUUrg=S2veG+=Up2KpqF;Wm?nWOalzft)pv zgYUx9-^fG{r2fJWSi!(uHS!NwSk1WK-%~?DK^>(JH>wdz2QPRg%MJLe9JKd=Sj@XPTHrJrtJXg00ZO|013LMLPeBbpFpEooh$`GjcaXhPnKtF2OW@X;E>U zJViASIm!hNWbR14ff5ZU%ZhlgE0BHZ1-4XVC2uF%kzr4QMap{@dBJ(M2N5Zlgl!%G zGu%?HNB0ldZk8l-gABF(1Tq7LBsD_t_$UT5fUAsCDUAWxADS`z3a)sJ0z7@7xB zE9f|rVUzJ*>_w2hqO8=)5zArGu;WxH!BghFJa=47p#U{k4Tc~I8+;XIRvf9)KA3+_ zYVx>Fqi$L|A!iMk*m+pfAGl`d!t#C*_PG~)lK+olX|VY{&3PssLGsiWW`r!)p9CtT zrV&GG8vbfn`TvBKpTce*rZPNyr>*l{VT%QvfPfo9uXlDF9Tk9k-Zemh*#F}P{*?m~ z#0$T(;pSIMuUQ{xKtOM1*I|_0(MUk&)^?~5JegFUg~<4WCm>}5zq_pI;lZ}BP-r6v z>xCr8ft8x} z#PVz3E|tAG=v5B4hPneFe3^l)z-_Ow)Z4uPHccRN+OEzskWcIblb`XnQt5ke=HW@aI*9N^RR0Bc%yrs((lDDonS2-5gU$I>8JZo` z=UE1BPl1h511d^HMj0U~R%H2ep}^Nw*Vr{0-GF*Qi)a|Gj+MkF?GX7w2jabZ6_7rd zodI~JiANR{5waU=PPK#lB5$??=?SG4z+JtN{2KwX?L4QpSih5)d5srN+nAgT_}antIS_*c6JK9o?PHk|IB9u z{_Dc75at3dT#jJx@ian+iv0je@Dey@J1h#{j(ueuo-f1;D{}UWLQNb z-2yo5bc<9b6Wsw5soZOF?YR%2SayJtA?OeZFI_;Dbgv2$HVB-Zk_0@ssID{R>}Gc; zr-G7Bl?Z%e(1L;yX1@6=lg_sfxapagrg%@rcfJxq%Z?OF44h;c_8djTh&AdjE8%$# zh0s|8Jv~t0gvvv%#%BiZe`VWN5eCyH7RY|U7mAs4UzfO0w+U+cZgB@&!ETF~`a9C{ zGeIBchTlymK%s??DnI~TnCa`uueE4lb8o{bbVpSV{2VzogA&%0TJ8mi-T}$BZkIa9 zR_N8I%w_2mBVuC^ z2TDF+Rk=-#{AFj4aS0BqR=-;B1j+Wc-lI3&J;mtGbaL@C;lO%OdgteQnPCz_kVB+s6FHv*2`7=r*n%M3{ex zE|Kzng1Op#E|P&vJ~|iAQ>d_8@fwyE4=zB54L5m0`p0;cv@qRi>a!4S4Fu>F`2V&q z&z;p^)nva0pr4yukcQV%kNugfaeph6dNcl9r5C;Zd!GGc{_W(qPW=VAgpDB~_tBkT zo)(3G-_utNhFzO$<*#9-=lh$$Xop8W%}GW}&>-DU+vDjrV`coQN*$Hk(oynGE(uUh z!w$^wgn0{`f*PxJvGQqw94^~cnnI+thF3tY+XAv1fXjr^ zGMw8=$NKoXEejRo_pS z(OjNSYK26A#gNLZsP^8p;vcW}n1#E3U zqljhY{#xtp0tB#=5JrvXbH+L0P9&GI4;qsLsT77Z>=C}oF*caSO%*`gAZnfFjY*Ni z=3;Od3}&a$0Y=Bx`BRBmRQrpmWo(yPi)E^JW_-<9$`RbIJ}bn<0-8tpk8KBe=TmZA zFlRi^D0S-gxYh!rUC+S)vhue6cg&MKUI*xEvrETWl>;c+tr(NJmh-_gS$Wo*OLlzq zn6YxI$`J1P`RSHm-?&AT*dUHUt9g9Cj~#`k!dWOf!-}8_xZq8x8h~3lJ;$071F~N2 z#2%aPN9Fje{pE(vKKg8#V;8tN(bPjUIX`3Etwl_+-G5Y-KqwaG?N140D=wFFUCRAA z6GPbr0%Kbq3ahi*N(4lqn&u~z^?0;s`3=KMohlP@8GOnc8Qj&o(239sC>YZc`MHaC zDq%j`Uc?;0SbHv=OI0FJEL@LXgN)ZjCHzHEL~&l~(2_Pb{1p?+5R)ywo8H_vxA7Jv zT=$rIilW4Klpb6uFFDvD{uI!QU&<_kk=CVmp4M&hnmv)?R#lxaam_VHxo1r?EIp7B zSjkghL-PpYluHW$!L$1y)cN>_PoF|(%Y04Tjy|!w=YAGi+D#4Ae2<*~$mGXTBOmK4KzRF+;d#v1+M z5d2~NBQ6PPpi$DM3&~1=%i|2Z%956Yw)tLwMpgoGh4}W~^;tNgxWiVJIxUxhW3|2Y zNG87D>KEtqE0=njqvfzSwt*W!P-|uxkNLdbpb2gUn^O zWxBHRL+dk~(_GN#yag3Jy)DX^|8aWvKS`6mz`6qPYT+Q%x!GHRJOfasS9nL{YGVql}w<&2lCX6AT}6>wUK+7z}7#YC-o27FIZaOFuC}yl~D~DFd+U zNy7#po(LTGC1_QEZ{QdwUk2_26(Dh|L~DB28qgrBkaHVDxv1;J7pQ7934Ad#!n8rD z&G=0D=N(MEq5v@VO%~MJrH&KS&bY$y;Hej^-+c^*HaNbum$UI5q$B-~cJOZgh0L0= zkJ8fTt&ScEHf8!A(GhdOs#NVKKSj} zLESZmG{Hjc<)BOw{fm>Kor=~z1Pi%%bhb)|tH`+qCu4NKIgcKc&nJs^dRuVgDt#aW z@|^R$jCaUR6RA9mwvEPmpCHTahUg@ODjm|Q>GkDT^^k&{9tO^3$;t7fzCFbqrLMYOJKQxlF#|jUg179!rVMx8~ABY4S?ZjL=^yreNYhK zs^%}xr5RgVWecSXDA5amxuxb$L7}fe5G&oqt+E;tmh8LMZ44{>;xep`u;>H1uM>Ge z&Mc26yTf8oSRK!C>QGp5N9Q&%A~KA?f^19s*tvXkfk>BS$=lg&;HXByMf5$Vxd>KC ze(CN>*dT^udKZE{_bUD3Y7l70^A=R$yY4O|36-_{%F9jd7!bD>;w#}NEA#$@JgCzR{Kjo__;!Y(EN%ek38?gSM9Ep)KNLtT zyf90PwF2D8iQ6_fAZQ;ImPE7$7IiGd+=X1#edZlC`a~RHO2omHO2p*!5%7__YY3IK-hCA zP=FT332)?Bt>83c2R=S_Kqo?EeyYA3JIUGGKOlTRKllHbkbX9~b30kp_NQ3h&(GNo z9*Jm#cEBB{KL~~naR~os5_RE}C$D4Fhe5ig6crmne&}J26kImQfOU;($oXi|oE7p} z3IZLn+}mKK1U?9soQ`gMx4RXxrc!l=z+5cLzz>nXuEhe2SNGma{zD}$z|1m)m&mG5 z&Rjov{QM3`M&qyT&CM=c7jmV56$Eww5g1;Q=MEy`CcN;e$jxWSIT@j>00XhLICM$CGl( zmXRXIrcv?QE!qFopZqPeV=4U~J{9~A=giyY!zJwd#mm4VzrZCAX-yz`{4NhD7f)w3 zJNE!#-+T$g7!}5D-LLZC_WYEu=-(x%6>vQU6)HEj?Q{d~+%m?t3Ov>Oz}{?DNB>H+ zo^6;a*&7U4Z=XIwV%h2p$~0DhF1Z|Z234TfuY$DmBcd}6&f}Xivf2no8h~9oeCYTQ zrIwO;dkR>XZ%|G588~P=BQMc+F}#l~#(>|ySF8=&T7?qIh{ivIcruW5c6+SMC5t$r z^#s=2%$KJ5uMR%y%|t0~P?%-2y?+KA&lxa}FE!;cYB)Xf4RUQ&z>%wfJ5CMp9`N3M z9g!EA<^vAdSJ7np-MTdPpFg$*>FkkK2(3fNvds=MyiZof zIx%b*R#9_A@7XE&BgGKN)4j-rxwB)`9?Zb2OlMO4Qp1-(2vh|u%4Srt+YVTKGvI&a zI9t7x#B|CofeEh9Rfv=Xf`b|4t9{C%ndOu%QLqdsDcwTuM9ZUqQaPR4uo$fR3Bu`K zP@dSXezYY(=6c$o#A+C*tLiO(g(i(J!5!C{!WwE%3+Ahx*APp*mcg`4DibsgnuLA9EI~ zo-1mx`P!C4W{o907D9lZPZcfIt^DeSL9L2y;CuZgPS8^0#O{p2ry$*`f{R)ws#O0p zJcH}%o};QR;8hmq1uQUI(LP9!P74osJ0m@!JJhnRgby^;2WNJxBun-oiq@iaXeOr_ z*Imw+W9oV>1@sJ_i&NwIwQHWVSd7q%+dJJsvr-Gg*>vnB*R`qIUu*PTm@^n&j0;Ar z+H0v)k{KnX)qF|!HIIt}DfgH+673Lk2gD!+$g5}!71tTAx8+|B6M0=9FZ~IzxbOyX zX|X+-pftLYK}R2Yi|eZeF_D>`)RqsZNLRqYN)UWKzLt_iv?|1|1=~38S>}D?$?I}_}7Q^ zQ{-~XqS_6@-05{0mOdB;xV5`(V|MT}HeC7vv!xFJCY{T}SiQC|kV!o!N|=9S?gRyk zbx42)2ylcgv15P`m3+lJWNpgab+Vd0b5{kbCYGiWcPYC@`{12m>9#Ko_ig9TPj`&c zr(@rmwLHD7kd@=lIMg~8M|1~HoncCAc8nzcYdK7%ZVyJ|kf~4CRcLUI8K;p9S5Xxb znhCtRC_oz7P=HXYsDvm!TV3o~bSS_vt2`&9ws5FCd?$fD2sXL6J%M*X3uqm9=AL{~ zS_Eg{10-&fe}*j=es(0zvAL;myMIYzK(JjMZ-5DG z8KtMB?-hjFE*25g2S`)MOK|q8oNa5Bj&>j?*!;~)1hNG_HG0#_6^c8Pb|Lq=EGzfY zLCh6QHn90`hnl^T0Fu2f6b3Hl@pqqS_MmhohFuN zmi0Bw-MzHy60O&)0^zG*9KMMsWvn^_xfCrky0*+(N6F)raWG3j5TtRu1uGHSIwr1C zp?S#4An5E9%u&x@oCXW3XY;{ip~J2EP{xJK$;mjzhpiS-^0`REH>*+&PHMuU=})qJ zc?PppS`?$|6Z>gG!jOhhXKNcZ%Bz-12O)Iln2RLT8xhW%&B;nCT(iz)0lx^}?d4o( z+GfMF#hH-t_b8K;Qo0OxkAvXeY?!F#&nXHEG>@UUw7*Qq;j!w2tA?W!)f~Lk-@sF_ zORE$ahL`Hq1InTc%1bd?#kki?V6WNZ%cGSDMNb%X1p8EJIYHu-Y6IIx$zb|`W(qj) zV{5_mrYG2Ex~HrxkiNKCR^@g;NajIFn1yz*VH62lS|w#}hH~k*gq1Q|Sg7^~@>Gvz zSbAe@paSQUoSPWY4{<_8P4c#_wL-hw0dbqWad%$z0nr!GyDqN_E!nz)X<4`veM?|% zS?ZVJUWDZq(pFl!D=Au09RnGTgeUHXW;%y4= zP|1J=v@jRlh^lE@b^qrpX994WC&G& zc=&QkuOqJWWMgtpZ|3RgppPxC!}yh)LqsN}c$!Xzt=}tde^e=~SuPFyjz;t`P}K?z zJV-!Tc#bg}W;NuYz7Enz`dL{j!)B!e@05IsG}K6d4P1vm{P~zE53LBf^luwwO8XUB z-{K@ShopZH%`$3O+kzz_H&G5*4+m@ICOAst$;td{|3}IEMc*87;aDQ!ga zowidOR*ql5xuS7%h$c$O9opjU^ET{z*!>u$&ceyK`Ybsnr+d9k)_CK**X~frvK$UK z?`{{qvcnfFg!Y|PMG+$oazPLE!*M7*`GtLOQl?aRwmsNQ$hL3=QpDYBss=Pq=B_@k zNbVe|Bi>MYS906!hrvMegp@C|x(7O7)0x~4+AKfHVb8Z6M~e0uyF^T9rAx#YKb!$7 zfo(leOzFA&8>sl$2N`gi_(7=a(;0KL6FhRebcV8lbzVx7FCUM@pvf&IZUa8`ZZngBVQ(Z%t3P*`?vtuwe<4RkcVY`Xu>dKNks*V$_Ty}Idnp@ZB{>{fCL%s5t0B^G<(?T`b0OP zkej%RE2mV9D)eOfSfTzIc3dT4siOIENMHyN|!td z-Z6w*l`Y@kWTSsUqZ|p8j(z3ATE6H3C2m2-!N3fJJxCTj$*6M!;)Q75)@&6gBzrQ{ z0Z9z#*H?6gbM5aI93F4SlXi_Q!z8*cG-l{sR=QJTGH zw;eF^m%S|?iIo?0wTIiLTM~n>I<-bh%SZ(r6!D>g=#Yz~%Vc^2ti9z_@zB=>bYS68 z84Zyva8JRe3kCx>{twh?TMX8W;s7-^-DVN~h)Bc3fOzEK`)=s@hPF{XmUdB9f^F z=*>hhrG-h7EZkA`C=^41&VsOdUd1T4w7}6ME09xAG=pw0uWUq*WN5x+lrmlmPWL4^ z0jify%rMF#q(%v}2o%HIZZ+6XL^;ehT#6Am{o|klmA}dh&9^O4r@@m)s~Q>jjok~| zFL1RSOcKj|K@u!|jy&&E9&=_{o6>$=2d(+kJvccmWeNtW&MI)# zamrdAR@5AsSB;WxgM&HFF%Rj8b!Jx}W^Eb=!HabU?l(8+Gw0=!@Acmi1@W&gK%{|O z#1d=s5UnwFMZy^bK3RHJ`hwtvUA1Cblddhh9g%)e?d&%b*tMrl?PXtMoT*S75mEbe zN%~Qge4vT?&T2Tz>;p5ZRQ)ArZpx(Pihq{t5YQfB{WvOfGqs@0 zyqCG9W4KvJh%M76(`7raL#kWO2cZLCaGVzpa41#2)4^e=-mvna8pVqsq^VoAwv&1B z*t2y@x2KR6EuaJLr{m}fu_dCOFSRbS z1ZMHL6T2-i3o4tUl~MlR1LnZomd`F|D5uevS4#($+#@lp`>*&T7I$#!TVx63j|Cs; z#I#F?o3Rt|o48|91kb@CWtismIQnuTQ2JE80x{Ny7I-(f+OJ?oqL+!VMhPS?V9|@o zy;4l!YCA=t#fM<`FOZ^Y+;f*eo{PvnbrD0~Avs~VVG$J!`yux&woa7x(f|g(TX1lM zlxDU)WwsIWgL(_f+UTSEvQX;GaxZ-OG(GoYpSH3Mza}i$Drx^EmyawBz|Q9nquGkX zXsEJaT~a%)igt}wKKODXkCxs_w*7{A66dYDUzcZ9>mGB?EY9u?@Wbs7y=3ITdPn~G zu;Uy0mscF#*cvJJZVlzZ*f}Pi-E#wnYq4m1$mw`UcK4n=K}?MJ>^s32M*1=q(mo-| zJ8VHe3#yXJMkNH^A0K@`PB4B~RuJ;;bizzN=_jMJU2bJCUGzprnRgE=m&&}#MjfS6 zCpT3cT<3CSk4qx@X*pkAtW776mI*Y~<9Y&86{o{6DQyh$4w^A#Hy@vB8QwH5OH6ht znB1KE9%%$cG9Zp*VW!-<;EkLy9gHCfB?RWD0&r>Xd>3i8a_Ny{?lp2+Q1L8`DSJK+_$g=o!{Epi zz2WglrwE=`(Z5Z)IHlp`Y}tK??8xK*7E;4@|HkubH$3m~x~rI<4CEDWOJ6cS|C@*fsf2f-`-cPFo+rPj_ZTs0bQT_c^a)|QY_~1 zc0O+Sx7~zaQ2!_0hy`*k#iRMi5)Rf^+}0ajUE#>*Onbu(P5mZ}5A|V5vL0qIADYtl zZ_LFG!}Ao^-6Sb)L-g(VEgOjfhfD6hy8`6iHUh~vdh|^0Ktv_CVJ&tV zywIF}o42L7z#xuaO&|}7dJ=S9{LDu(knk3Ww9y-~9J*8(p!)yhEJlQfe6t|u>djyd zHvTr7OGl@S-*$djw5C$txkvu9axj~7`Zw-)ba&u+k?X3QI-sva$^14J%YZ>tI`#Q4 z@}SH?*N2xKk?7e^>La|7gbi~%Y8ZxH|4Ee73>v9mg?0NaYZ!%&4I?xbZTKN~!)h*t z8h+TNNY-3Q)fq!ms{W00n<6wMV#6D62dn_hzugBpQK7oZ{pRG}MuDj`chkKh*lj4b zIKg}w-Y{;H1*f^-|7eZp1`iQm_wv(+Q7HWFmY4%8i^gvoG&!_gN_|4&woQ~X@SEO$ zBYKhrFOlF4-!=y@nj2RC?Tj}LgV_4v^t!YP@h&!&c$RQ5OXMEi&^$@QFK+%P+qPn4 z+t%{GW*Pqf=+fK#v<@zJHajHd)dk@23@$5g9vqepJ#?o3E+kpcKREt;3QAiXFb!H4 zov}2ZgMmIewC;_|a>gH=C*q)Z;6r*Hz2_i>J$e;#!lxj2k)aRLlx0Zm%~|hrCr{oW z*xu;n9*(1f23-uYQ9f^MoTv}Au$|vjy3g88Oiq3PV$mTTbF%F-0EMvBj4DarATL|N zyP7%M0;p)?&Je*koZ*+rdm(Z$BnYWQnB%VUq0Z#LbK1+vjFc%$PR*Opg3Jxg)f&z$ zLeo)?vH@X%7u%!zQ7Gm^#2OXQO_LomEp$Pe0+oL6?3eHqoR$8TK%O3Eps{Pb8^?s` zeAMZ@DeXUQ?L1^>SG}Nv=kj4D2Q;-AtR9Wh5534A*!YjsaR}NAO6S%$7;?j)Ews=U zhQ?lUB%^Z#3d8_tJe8)BD0yi|PeI?H4>P84y~x4F|A8U93(@{SqZO$7qFP`aLC2$#iKqL~C2I3~y!QW=&f_;%ZdA>?t08m1{HZCsC zptvkE^EES6(_Gw3);9m|T7QM+JQSq0U>txHbkDPo;jA)K#WfQX6NNJ`8rH`T3ytQM zzq8(~FGr38TEY}A7;3M78yur7w(mR6Qda78nY_53hULbw?V$YlBZputZ=J z`Q@^xgPmsXVLnNeiDL;5tOG&#)Z8#s9-3~O&dtrW`2iPv-K>wJhu7a*3^J2lCXa@L zY6_%z+ywfZ!UXeiA6G??E3y=e5xnj4k_>a`rC%(iWhh&;SllqjK`;PBM@Xdcq!Bnp zKI?JHvrI2N6kRdVK4Cj7KpvHEuuOi+laN3s@r2#kZH=B8`Cmt|n9%jZAtPT$@139JZRo=+Em`7Ft_w{k8wnDMDU;zUr{&w6PiZ`nw7A zgO#1`yrdC2_e!yNLsNen4Hr=g2?@Cg7_T;dYLT%^R(1xkqW`H<{7dXo_(j@huA5v*iklLk*fn8@oB@qlq zs#xtocJj>XR47@i_ZDF*i9+s(kdU_PfrM9SpQ5hUiREh+J9$Er5Fg{s2;nN~x<3X+ ziTkhj!{}nzwFMAl^HZ&0=9AxPT#%bv$od3ePLrh@ugQ}&9fZP48)P;dt;J?T+f1hV zT^gp++1q?j%o@SVsnDPyq9fK87CPdX3Mbx*LfpUw*%&$~L3C+}SfE;Sg9TFAv%8kp zuE*A@VR_LD`t!7Wjh&7Z>{gR6IQC()tPC2dCU`$d^mj*4=ksY^7M4ZYdzn-$CNv*J zarTD+tki2tsQBF=L~W#YT#CU^4SIKYlEEW4lV6h4?NrJ@qr zcV4&ll4$X+g6j~Zui9*p;Rd9Mz=QFVBkC6&r~U;n5QMo1OIKvm_Mw}wtX1u|mywa7 z?n_;_;pTv;rL^VqXB070l*n@rEZiAQ-@|faDbNpQjIZBI4WL2hglU9}NmNB6n)pvx z);RB_o;oICA=6)3Sy|^P?@kW)Kh^_ICBKN+(ueyV@=Sail0lDAhrmo}JZFwxV($EnU=34ZoY&6 zPDxZ&#$WRR(Br4vX|ij09UKdy?q5m8p$1D*f~=&736;1%mArL}TW4-AHc&j27*%k?hu4!7 z>*isn+3HrrqItK>KGfc@NBH9y>UWv)@!?pNvwGi-GnvnzY5wO{9fo<5g)0rbIGblv zUBQH+hznN146_P8`1fFW9ong`R<^gdhm{FF9zj*r?Ouz4j_!%5Jo!xoL8R}ZkIzQ~ zd*^p-o-W+`d(m9|P{9u0tpBSXh|&;TjVo z19s@R|wJ~VTWDjBTH}QW!k@&&9%@}D)?TYd&7bdocZYn{NOi038$h` zrtyx>&XvB1I)ehwe~C|j2Z-gNj+i90*L|S8d#!x+_;KkmHy;bolCm^IO5&9Oc`e|p zt2%WZ`@p^Zx9Ke`W}igR)S_g#K!3V6_+9Ru^+H2Z&!At|tks%;4eda&$Uaw8H#IP9 z-e(~MCfbzeCc(`Nz=|j$n) zc`4~b`Z4fO$}5HLRqT{h$fH-Av+c`Y@fMkb?>B|W0kpu$T>$rgH}<{?=h4d6!^DUH zDQ)fO?XOU+g&`{=Hd~4svn<|;j)m+3!v%*Np_`nsxn~X0^gFrb$C;IrUzyKD6Dti_ ztUlY>wp#38P?0(D-Wq1%W1eczdc57TFORxv@b1Y|wD#Lg|sL@1F|LZ&B&Ftfo zN(HbEvQhNM-@$i1n;p5DxEwis)wtD)enCa~}Y6(d1g^q%vkU1GYd=h0H_`JJPu zIxfANZ6ZGNs$M6XY@oK9Cy?xS(L9M&mwm-I#_i=-q3sS&)4o@1DC0w5*gonB*qhZk zHS{!p=Q!9DqBE8bm`RRL~)OkoFcSeImAXOwSFwNmDSfBG)YAFDPZ8Sr!>&>NUT{MOrJA6 zveHuF{4hd~Fcl*c)AZG6+qZA09zI+bJC!gfP|3USftvWD2X^~oj4EJ+e3}XsgN26( zO77UQn*y+ftIr_KO~tVd-oU6|(hzTs{jIK~zF@jx{pS`iF<){W_nfyVGHckaX=-}t z=%#9uOur;rpKj=y4EJUDkojLp-=?$eUk@Ja3!@vxJ+*)p<0V&}qD+D-e@W+Etr+f@ zkE6?xiDCFhx~Vn!I!AuF?Mj;~pdgoDA<_>f?mZZ$v1RU>xg7ulBj6l-994JL2N>1X z8Bhe{A|O4)BQ5M>;AmyIJg;h+(CKdIc+;pplnOhJR!xYwL4|y z=@;~8)44+SJsy0t+Bsj2Ujsqv2Qd0QALKOpG=(D(oh^3r1rT68!9L_;^YinioZ9@w z^n;O=W6Vkr*J)lTK>;3x2T)mDVUTqgL%u#;NqKH^3EFn;dx+`J`p1Cx^F_E;@_IMS z2qx}D-p`Fu%Iq|_iYLyf1V)OFKiGVC)Bh-T5!yzXNZ+jS!!y|qPA!Smvba&CejYh` zX8BHE0g{f4fPGTlc4ZKX-al{$X<|OO*RB{>{RP62Svw@O{|=x|cat0Fda}LBf3iZ@ z870p3$osrI_16OCw|9J)hfQ+(zTc-S0h84S z_P#4UBS3%>Lei9O04HpNG_pNGcJv-@ELRFn0A;Rd=oV8P2({m5`_vrrZdj}icxQJ8 zsr85F!|1fb4wBsyxiIg+7053Ak%aGQIz?qEuMkVEBG9W^8W+BD#io6fT(Kzvi7v4& z6PJVhqnN?Uas)260?6KJKUg-3Kw-Qo4_by}Pbid=TSRIO9q?&b`*zRQX64~C{>-(J zlCScYP=UO7-qNmm8q%|4!PWK(3$9?q_XRQ+je&{rXd~dInlJlKDim{{F|wdjG)~C{ zN!@poYR`=8)Ae)c#LP+^ly#yedsaQ3C3=n3HQpEr^ zUEM8N5O@ldH)d<2zH4vp!j&A=4jO^0Q zCm*i5eaWRO8ICa|?yC60`41&G`T5Y+@=sFL-8LiZlj^Ogl35Wf$3^2_VRL7^oqTT(W7 zJJYmgu90U1knO@w0`@bwNxymw)>piL#Vl0ZlO(;h7?uxJwK_c0S-V!!?LttQUiLgX zb64Qwlj-&e^mD3p@sEU^dZ`qL0!^bG^ImHC6b*bLDWD+q zI&Ibn(m;)X0J&oI(qsO9WLk*qw;yLn2h@1mmv&!0+vPUTQH7^l00uyp``g7ncT-rN z>|X@;Vm(1Af7ZGA?6z)+a+hQPtREgtx>nEE;Rp~RF&UJyJZ7Vm|Q z3ZLgxC-cGXa_WiCH_ME`9Etbw&8wz5Gk-8XfWQ%JbO|utLrgkt_v%{!Vg-(v{decM z${e5i;`ggiVc7ge+gJ|PXOw9KJnSnW*(@VnB|x4J|)OeTw2;; zn}b~wX#(E%N9}Tz9-Il9}Uu8jxMlb_U8W7}HMA!})f( zCN6jh$O;cpy&#Yq{(3s3%5IG@D*V}AaQzAd*k1g22n_4f?N?X`! zf5iaxp{D!JCZ6LQVkCTcx_W9jUh2M4l>=^P=oU0hq+>HbBUqL}nRNpE5sG>E8E$dr zm3MWFVy_DB*4kGoxE)7WhJVk+z^AP6ln?t;dBh{#Cowm|)=o;Xt6Awf^*X%RE-c%i z)FV6geWGAZjAy6q*1mop8eHOvz^)bjo7r8-n@DhJuAnw6xvh30kszN#>M>dkEy{fD zprs=<>+cP_S0=dSM_0F;%>QU&mO&T*g25tf&6@r`D^U!N(YPgmiFo0(h#;mIR_d;n zy($vcJhEIi7Qq9kTv}I4s-+FdIoI06XQAD#SnjsGRq^ve`Ltqwz1?YopYRf+krEXc zaM7McU9Q&?VO5$vI$qO29W9?0cKQ>w7-NQ1)`h5Jf^)EKb_u`x1RkwVd7>tgNdjpt zM?ysr*^(ENDU~@;3v0@Bu`HEbXtO2bMK4w*cFXC$SUPP|^0tr1Z_&NdghiU#>{trF z7%2b;>Bn!0u{Fj+@5V{SCFe9R&Pz%^y&aR=v9h%xHm&p{Vedho6NJ6YrhE7NfKub; zaH5HgYb`#O)xqOu>8UXnsHYtEYq`nS&O|CjZ?;(SpDQe((yY1D16aZDcV7RxdF7{7 zpjfE)n8yt80OsZWEkXJ3=D$7k*;sPi^;3{_74s>`qsvfFUBVZ$++Y}1Px;ZmRAW96 z-Rgz&besPem02Iq#idPn$Fns%Vj1$k9$iQylOIK#$#&)on4Nl zoGFWA%etHqty6drSV~DJ87=iYQo~ceMlx51$rtMwIm$8R3U251$d4SiC1v#=YV{Q; z-gh(1`J>1izSdOpbL?WiBtr63)556kFGGK&i@6SmI8|~8pRl6X!G%%nRHf&5SX~iY z8fdx+BV6tiMWmGPnB>(fKC0}-81;r!=k^`?mK331s*hHTHTkW@<6HEr2hwxg&(*Gd zy?y2Z2$l1w>~}ZL%Y^c+!v83y)wZ~DMxQ}-R9U=jU~#7E@vkGU*hsPKRzGC_DX&u} zLa{oXQzJh4>x=Q}`C%mAt$XDn?OfQ}fuvkL|QyrdN$jMcesRLN&w|%KAu5UI*e-)4yh~OwcCuzf-I!w#0<% zh!d3Oz~kFgV>r{Dmsn-u_3o54F=hsTJd^+GuODyDOi>5+;|x58qn*7D3sz>sAtTrmcOke;*u|4hwa7eT-|>@;ALTc)|tz3r9Gl| zv89i8(_ROIg!fntgEC`w>uD9%j@j{Mt?I=c#)iHg_@=j7?A9cE9fdcAGd&#@H?zZJ zKd<^oYdtH;8ii~?s-Hoo zyG-NoEZzwGgFJZ;urT+9&d8L={{9tV+hlvOF@ueYuM7L~c3bQx;4NUAx`t?pFJ+w3WtEN@(* zi#x%Ndq4s%4AdglFOH*&4m`O87w|DEBj!6AKbkFqGqWwzhkbu7`A$cP1=8|TrhX+r zKj{k>oZbYoDw%TA-q0v~>^P}iggHd5l11RWr!&`@%Kx2sNMVWVFI>aQUQ~(0hoX8R zNytjU@JweF2gWsTXSI8(WuR?!a^{i|_Ght$eBm9N*4e|dC9V<+)%Yt>CEM=F7tSTU z*6ms|JFGMt?NH2{BDV{hMDMwfFqtSE=v)Kkd|o0cxS=YPzaAcQMA;}_5ud7E{lv*f+DE#bxc{2&LwDv}ReEY+}{vDmW2+_n_sx!bJrhA4^8l$z*F{0x@M5UuXdM zwcIfnKTp`yuB#%^_GvHXzCnO~{530BwJ}x%jRTk{(vs=gT&!Eh)6)DULh@Siiz`tJ z{W1MQgtulx!IwG2m2$2Ze@)ER;%Dx9HGh|Da55RA*6=)bst(haD~75Q#1O=!wM0A< zkT^MtbFg*78?z%$C$qg^{$sew`;JeTIFqd5TS9pSdBf2QHIenQg>iZpgTRIvO2HE* zi}m~if*91Bu_Ae48IzOA4XKvu24iQRpId5w_sPIphA=Y!F_IynS>r7j0o9&|gt^UN z3$^}ifAw^u8&N?4&uM@$cqeHsdI5T^bxxC}B%!!If#F5-Toc9+cyrrT7WS9$=qpOk zhHw?<*n6re4{2!bfor=@Rc;)ywPV4I&~mboalht3+vqbH?4$4WzqaYo3Rna-bSS(v zEtKhsOZkB*TWtuRG|X3-8r_q2$(M5M3>+Nz1{)=DnB$DN)=xVdQr+(geoCuLks@@= zHO9&7miIcI{Gf`LfeP`NG0X6zp%X^*a^y0y|G^9RBzY^if`NXm^bw{7VkUBe# zaY&=Nr9u>4dc+JeFGO#7+0kj(G54XN>H>z3@oDQNMh{-we77PsEm7ZjnYG9R^&MM= zbws|9VyyS9EdO!PvnZBh=L{TTjlH$+@?KZ<3(dhE816g;1)`xzg{v3+acmm)I5tku zR+Z^0!$HR7mZ-Xp+t>4>o91iI4V`>nsaja3HA%G(JLjTCRah@vw&Wao`0zX4Nr@-* z?D~`j)3!c`LisGmzjVAi%0KCu!G}+9offjy7}6>E#Hm(S#@|gU%FLq{#N^dVwRqiR z)VHlRs}*M8>y4B0^+?MpAfS(ky zV|Dl?Dt2+uZHM`a&H!^zS~UkNH?frfrDWe4O7V1CYDt@3kUlMVK;7%PH&h{b*E_&* zvxd~WEfi~YV!Mj@7fw^ukOot?F%e|nJ~gjbUDeeQxfXHhrgk-OMCxKPX=Qp(S(5pgxhfr}?@uBUO8nI`cU7k3|kK8XQI)Bi}F) zZOhwTkn^UYV~gjZ^C6XoY2S>AHq`RST4D`Co^vXW$B~v0X}raD)f*gG3eV;`aNCR0 zXHRK+KeE3fRPG7XnBY-zMJjzM&9_}+f}?D*4qhtVS{9vcJYn-&yjLRgzT$Cgpf#Z6 zc}Jv_n%#cn>^XMrcfidZJx+73awPTkK`Y(Lwqlc2uc6tjEYoG4Wu1Y>5Qp1No4+>jZQjQm%{;MP<2uSOhkf#!PiqsX^OeJJu;|8;k6ZZmWTry*wiGE# z%up%BDl(6zxGbTRr8%{IRnSv=U)uc4efa`GpMuiTQV-TUW}?Y!505RF3Oal|G@n-^ z;wFmuK4wW0ev?uCR-EHlZnt*>UvNXuPMoQ|vhKHAd(ybqtDX9{ zQjrs~PZPvFoW_qj%g<99#2Ju*F$H^^<)#`lU4zpDlbQqGtHw2Atc}AH&B9=-W+27% zZ9-R5Q?qS{ON}5!by3;}mxU1g9Hn8PHCR9DE9S4dsHjPJW! zmYT}yL{n*ccLRy=%t>ZvLW$GQ37y8-_9k;P1F>|&*F!Bp*(S@EV(CM?>ps=>G(x#~ z^$LwegJs^!EPx}rX&`MZHYq+M>+@ZG>2uQWT8=DE{v?to{+B?(qeme|0W)PBzw@>D z(VOgs=sA;#3w_!49iP^E@6ipb1v|n~giS?PMpBxws;O|~0>5^vJ%3AB! zOp@t_?WZlEB8Dtyof!Wv2A}=LBZ(D1lea;J*z|@{sUJy4qjT1JPukD?LwO3D{Hdw* ze0b~A%37}I*t>id0pci=Yql0owta@#YAqjY<1fb6%`klGem;I|$2`x*b}PJ3Btd>X z(LgJWqpfUr?nR^fboBBE#;MRFQ!*YeNWKTaf^Ev@nQ_tvx;Ehag&$JdKdsdp7}N)p zCP+TE(oq8o_yzJf|5{s}zV!?o=0c3v!e*L&1HgAri%y;8hxGZ`h(-&x1>Vj4Uo~lP z{(B$Hu=+{;pd>OW;&%h2MpIj~YsmpL6>jXt)B9f!g>5n|?8}kkFp&1)No9O3M1#B7 z76JBCfOczZYcIZY1%FhwSn8sce=xWAJ3M=NYRtThp0UNL`h-XY&$9zFkXEgXY+SR} zB>61vh(ucHOh*i{{NKS7(iHMM^~XftF%!G*$!jDZnA1+W#v@lUz{8GH!w#AY5xRT)7hH4x-3q8wzaTO z9QHIyKS>En)Ux>Tj*bqUk*!ALs^Tm{00~+}8KK(d4!TkVQt|#-F!t-SHG9d+BxDr@ za5Uk^>GiJ$;I6`y-2ef1>1^td{wt2uGd4En?)=O)PJQ>F1kqxFzx_6u`|~d@tlWUc zrPn_o%I#x49~w!Ea!3yWp*}f0auuP6Yn0Et?)z?UqwLpq#_uRbCZd1xTCVZcaXTzC zglq5lzA!q2J^%%$vz@3!&_F^3mers+F6I^Y;m;q95u_~$FrcNC6E9kz1SoiO@|K)H zZ9J&|Zq;U{*pe`X@iuTuT5qE%0J^m$_+j*&+>{_LDr1^kob2R}o^*^TrMe34lBZe{ zPh8$A3x5FC|A0mQrf{^X=ZkPxOP-6DdAh<>a)JTq2udremZ161{MHO?%+E+QJcQ~j?mz}sX&SxA+F=I_ z+eR-#B?ufI;On^8Z_v20&_O|C?l4h;GHNe9d5!*Xd2(VxqatZ`FEHMU`x5l{*5A%I z7_v_}LK29Wpt0-1m*HRg_CHh}cKpN!j{zdnP0sbQqMQ7^sV@0$3vz!s=d)HW1`i2G zsQxU*0qHa9H2w8-@s~@a)eh;!kqP<0Cso>YUm1_{1%m3IYH9n6EM-g!De7zANWlaz>vMRgB)-gdr(MT!~>ru$5^0= z7M0v02V}2!{BtCEflvYtwTKQ~9P&B)!G~g!!j@V=(T67WAh{5zgB%c7lsGI9Yl<#N ziW7e-H8huM?pg~XPS?>u*zLKJvJ3S=kFus#0Xwl`d{N@cJh|8mu-~JTCCHS?0EVxw za#hW`jDL|*@y_Va<)J%wQ!2Kj@5TGsd-1#~9M!(t#O8yr^i}{8GjbYA$i0Z!{ov%4 z;J&-=VCJGa`PcrFFYPi7jg6;W#4J&DhmGn|8&*?;zY$)M?#{S4Mj&ckcU)%)HbV*J z+>Lb^Mb6)J8tzWzNw&XN%uLsH(>E5na)^PilmrRXQI5Y6UDq_^QA;M;9NJ(BzJf(A zIQkRn)+Wjv>bA~KCfiGfy1KEA3!+{TNNKF%-+pihd=GR@2^N9qv{&Zj(bw4Z)@#e# zl3h==)GMgTeYpz5@;|nXCP#Kt>l!^`Zu_x(1gBeL7Vah z&+!9j|Dgo#Kb3)AB~rWnU<+?wJ6|)UJFMJiaTE*Oo8=D(>l8Wko^|HNg)gzHAN;(y2!nI?hz$Zy28Qco%_tXu@1_g z2!CoX$jr=SFb6*3KkMZt|6{QF>vK9IyB;7%ZTXvXu;}?uY&!Yyt+Q*TPC%SoidyFl z`w6VCn64Hak}97+qbP&5Q5n{k7-BD38wBBg_%xUL#!UDZb%uIr9YWZ^b^pANk56~! zCp0_|LQxz_#fo=d6-EGXD2Zfx$Z)eg7kBx+7ZlvjwUVJJJXVVm2h8 zEX^P{30^<9u&BY8CXvX(J*gB;%0wERn)tVy`von+m!a6_MPZ@^eq==zIO7Q*8omkv zeHA=5rnI~^8psD9H(JOnL!m^6J`GB&6UgxUA%KD&Q%~QZy1nmFk=`OG!t(%pzT?4s zLzJ3Hl)QWBUGJy>jC1V6y5V5of&K_x|Cz7sF-@$?I3aFZDU_a31`-e3K<>mp|skDKXf@DKq^-~ha?0^SgrP7D&Y3>3zf0tRL zwb%F8>IkZh>aRrj33R>79s1@6zAyIJ$0KTYn7lC{EW$S9eK{ymizq3Qds^MdbKxYs z+be^Ph%ko;dBZn3OMXCePN2ez9suXr?$ZbooqF;jX5u~I_V!ux+YgAgJ^MNLZgEK{ zZx)P8g!`)8o3g(5t{;TmVd!T#QYgq_o`H#CN<+TiMS$9#x}6OsM(=vPzke8sP&jxZ zgn{=Y)O{C5z8c-&Gdfi)nMW+U@SIWrNV8Bkc;x|D*nYGDNIF9}etn{Bf%7{iJ)B&2 zKIk0FG+joWAo%utha%h$fiuxdCcgk_K^V&Ih8g=wKR*|u4yG@Ah4q941)-ph=inF* z^s^c66|dOIp)!~#MPyalgpog*;1E!v2O!0?m?pTVMtcBq{sFFYaiMTUn2H>O;$rh3 zL$|K4+dxEDdK2-8{C>KH}k&r@) z22*AsW9BJy1G0@7D^rvmB|@l>d7d+7mQoQi%eYmhZ5}i8yVq9F)93qp&L8Jo=eo}C zI@jmFcJ|)y_g?R{?sc#Gehp`Xjv%h_3Ls)8zbpjM5VjYIURbnm##N@#qCG>(NgG%8 zJ_fnjLmCe`@p<->ET(m1$>>``}p|aY}j)@ApiH<*ai`s+6z|b;G)y-4-Szut+tc zT?s$T@s9%+l^l|%-hcc#0TcZv{201=Fn$`p#t%YNX)to(l3?b5>!}VUAYi)05xCh4 zW0%jMU#kNlD(e>!nk*k^Q{TT4Zk)hDO%_`s0NqF7&eJrk2p*oyfq=2+JR!%8@+RiM z?S{q_fq;;c4^bw0+du?DSIF%X@o&-f{zT6HiT=-m;WWtnYZC3@ROYbz*1RL!`2R`a z@o#*Dr-T9wTL^P8LR5Q7IKzvHjN$`?2i^h?Fp3<_3ZiM475$-42$S(5h-g_4ATk^9iMSphJavHw6tZ9mPrm?Y!nx43TZmf%7Zd_+rql8F(}c-$ zf#obX`Z<75G=bxj0V-03O4F5F6)T`x-2mD|W}r0Y#@;Q@O8D%b09F##m>mUI-`=O& zz&FQ#R2Qd6dgq9Y!X*Ag3-2Jd-eb<2mX+0b8gOo#&VY$ga(5SvR~6xg!TOKGFi=^^ zX#!-(aU34(rGoF0GnQ!W&L`rMeM79L zD&;gxcn85`^WI)_BKS4{56umQpXoBy5V}i5YT{p}X3YLsDjwF)Kh!iTATx57h131? zrK0`i;{xk_F54`h&{c* zeUB(gBW{9M0O_GeV_!dwsDRaV)=_JyfKkD^WYG>4gR+t%Otl<{r65cd)84GOVockqr6L9zO|}0FF85 z`?MBzm`V_g3mj_?o-wjQh&&`>KEyko8I@q=o*2Bh;~>6pt;u>#^KFF|&Vum=x=z)Chsin_q@;`!Q;c@SP$4x{R$_n3S}Q2LO*4^6hAf>(*a_qF0>&rjehVXvk4DuZRbtv- zptzAy{TWY+Y-khlHIo1i^1c31yKewd4PKb!Tf+)75OGiYK(PgE}tqG=)Dvw0&X7sJA`W zFA$4)r-08VuC2Yj3-5$-#s+ZRz#!|Xd4Se@ddRYI*-MU9aqJ5q$t*Vk_z;fZiGR`a ze%ds+hzGBfv&f0J+58#tOUY1~+~6O6XDZGjOebPL)?<=inf1Gja|!1z!YxOD1#C}q zIx>4P0PTH#-ab$y83rqr!p`;u@q?7V>rsE4JjrCLd;mQbhZ3PO0Ac=om5d;$6}bEs zg;OuLHgH=7({jj`X#M-a%CnGPW$1#VPQB3ha6nz?Z!AE@jeE98Uj3TdJ!$@2GPpYE{xg|e$1(s>nIXURw@R#6kLV!?oKQqt^%7`2koZ+d8j(gy3{Dv1`w9xj~QIW45EH8&|!@;S$nVnJZq6(4R zN2Gfsn{o8+AoV=BQN2`)m+5L|&#Xw^kr2ABH%;mUz@Zm{4#Qt9&2=69%}-Y04}nkW z29ADq6@i@#`<(R|k_^Pi)VOtg3L&CAfnR?pT{#*;?*$@bNzIcppdD8^$t$Y#tu?kh z37}UsbkDDp-FdwNzC3>26biA5vg2y$>fIroH$h~y*5Np`@K&p`$|G;WM9h3*(v{UX}mS|BS*ygQ5eCvA|=gic}`M==!y?ZC${LbNi3 z1O6bSpmm*z)tZp>!+w0m|!PX%D@oNMPh(&q~*d{>>p?xk?q z4w`Ark3BP&URbzKZ7ktz*_0#A^Q~NDqpjVuwD+_HPwMSajo8V4YF}8J{Efd@(lpPB zAzGrQHWZ3XJfGd0cPjBbD);` z5_GLwWE!@OKL{+`1PQ^~neTCMe*+pt^o9uRbg z;h6})vGWZcWQ3ZnTX)lz<)_7HlN^^0mHz01lAdG;gv!r-KA^m@W)Ga|XzcS(PV#jc z+)x%n?RuCB`1h`HF5oIp4JCZxXhV==Gnj_F$UFoE`U$y!ETo9a z@E)`ScjbY0)k{Rx#dU6~mlVmA@y`e}eo0p+chNDwS*h(V^r8@l^@ zVHaRzNOPoIl(jatE@*!u+2$|MZ@yIM0_v8}EquD;gW{|UyFPqU`-oHzscC)IzQu~4Yd}Hik#@BV(o5H`DdA#hU+5mp7zlJsTlDsJIC8seJ0Gtby z4R=YzUY$YPB*IP~0apxXlcd|z{lj2-eirHpk%IVyXA_mD>N7wKQzl*?0waID`Qw3e zACKG!0D$#8!irNZ!V~JjyVu4x#Wr(Td#8Kyg6NyDWMYFeNTS}$`^;TR5YK%0684m@ zn-!8F)twY)%C5e9I^~2!6V;LqsE9N_WG9U)lnmit=>z`6B+Z9upc0t<@E71mg4N~G z#EJa7w2d7Nu^fx8aAQ_^V31S-T#g9YbjM2ACVL7;qPNzQ84X)UpJxa;P{j$we-K2u z-Vm(3(7d5K^lvk?cslmBb2LM9!^*sc_ky)sdf=>&`1)G|@YUe|>n{fW`lzI4CM+E` z8ey9d>l;O=hbNTU^?`goIT|7p%sUrvSwtHbOdaiW)a6qE|49i1bO*L33YP&3V3~(I zkHo|#H)yLJDa#Km!p_*;d=6VaxWQpK4JVefO~%i z46@>H(cfL|!EKsJg{*>kxJp4+01&*wE8N$@kzH7!8c1Aa+}qkr6@lDV^AMqS{3i>3 zkX`i=RpLOb8WUGiAj!URp2V6%G*^%kXwSFfG?;`-6erEeqxSian!lI(b}3#RIfcy& zt65_?{D;lQqN`uX)bJ{)yp{im5`_m>?9IE4$Egxs?|~Oi!?Y-^@;iV;;|))MA(96m zgC|nd#(#j`fk621^UEWaP>#rB3_qVTkb-xi_iz0aK|BiQtLQp74*YIyra~}Nj~|po z&%=W_luQj2MhoIE%lo#ZJg#ot!yb7x3qa|@nMyt~w!9XJ)C^cI{{eGqEJzPGh(KQC z{5T?MwqTzc7O-gf7!Y`OmR6ir!5hi@B2lo3PXRJsg*XpaVa`H;!-O4)bIj_FI^rTe zj>MZa4$;!l4lMBaM}aly*8m&L+s7y|;FrcP-J5y9l;5>aNl}xm?IvHSng$C6u99=9 z7Ap3qKCPgeErBKFC|XR9;nz#am^$6en`>~#)}L~t+dhVs!1QD&e2jp`#4_+2Z_uzk6AzCuXtO%iHlwd zXQeUkG0&9vUu%pnL^Z=ktxq~ulINXx_qRg^Te<^F98n$H-?`fL#iOJ5 zgV)a8-TW9wh1MOktGo>=8ixu1g7^)jg1cYCfw?=?%_4dq_S3Y-QfaCv#s9_5qnb*k z_4QQwfqYO&JIr+|tL6yrr1D27G)-(7VG={@PbNoR->GBIaR_pA8|QX$|FcM#&N;E9 zV@(>v`HIe=$yKMe?IKrO%*h9OpuWo23p|dWxv-}%wK{#@<)8&wymjdGR8bbI@u1{-&lT5%ok|CtnfBzn0cTWyJ?3VZ5vhq@vr;<4gKM zztCnn^>%uT5Ojg%?ICTKUKi#4rx z8r?{!{sG0iyo;l-z?}UYXdd#?w{=N%PJ+baUQ3FD+`Ks08q}@FCJLU-^X4X6>S$>d zFx>C9^a;as0*ak|B2a4I$)b~7jqJZwn7ndf^D+HpI(g44Un+#|QB&IOT7}yyOHGq> z=G{2tl=kjMZrrtSF1FvDEw4HF)DVL`LFVo@;ZwFvYx3~;TiY&#am6HdECzf%>N;^v zfKz!5`e9ski%^&F`h(>&;0?9Gx<|^*zGhGsr$p7{0Z^f<%XDd?M;pL4vE{DAewMlj z^=4H#+x(*gz$YIeO+Nj>pC?$WLwg8p;I4A&M z4ekH!8?U0u|5r-V-)M^&Xd4b0H<+J;UjrAKI1IE}?OP=P&uoM$WaJs@y{ND+M0fRg z+6~rd@r%B5#V+~4g_VS!A@+*aDiq}J@ZY{M2bCWueD42S)dDW`^Dih*oVw5l zE#nb*I*reLcOfzyBgNVM?XQErL_-bVnOlahTE?f^PQ!vbR0)ym)7lKf)v6mPLBfNl&ujT=~ z1EyIbOo?r0UkPR;;zVVdZ~)z{8{F-(h4NFBlKqKiAPWKdHRmHg%I*5}A1!n;6DSHs}P( zk-vX`N4VLOFe2*j2h|BLy5KLldBe(t7hEuIBB*DLq*tK$aGJp%6?pGMagtO`hR~sT z5BKyqdw2mAvu^#-0{4`VhaVXHW_jDn`Z2tqIaiFtOa8MMDgoU$8WnW6{|Pi;hUa%N zBuE)udlv5Ld%B<~y7r$k0slk41yyb?UFIe@Y5zx|xe26GY7T4{x&Qk(&=dIYO3hbt~r1LayYpAzC=a_$5|}amuJm@gl+~0IWurukferH1v}Ye2sFQtqX7BX zKiZX-jPJ;d(7lGGt%o%Fy0)>q(>zr6!qKr2Sc;sPJ-5(K59=iiV#9YLLals8<|?>f zByCsSbBY_F*(b2LxHt^Gt)(O;US!-WQDqBZVjI7^@g9Y{Sr+$7)ky9(etiTI4gbSX>N_lGpMVawG<`>* z0o{Ct02T3ChcLC$=}CuTg(-!(lMt4p!sNLUevmZk&i6e$&{ixZXW8!DZ9*Jc|Am0b zB@I5)2>{)QmRbq~)&prRZ;)7WLJ~{wkrdwRu?6scz7IPxsZI+WBSpN$paL~2wkWN)T4tu54dkTeC8_p zn&Z%A`ursbS-&6uVG-mI10fd*v;Ihe0{y?`lsPA9qonk8pZwHb1j z7YL<5NpdMJz_s+m5g!D z1GBAvSDHZoop*?!d=4x@Vln3CXHiicRnJpXQ_uMvvG)~(x>5tG6=biW8pFMNaIaQp zOTh%E!F{RkX2nwZ^h+p_e0_MTPgZ;jj(GKO6|)kj@!?*Fi@x?*$9VztdJHnvOPf2I z+@by`g$n#L3DmJ29}!>xXzvX3(8qXz>e8FeP)lXRwk(vQ+Hcv`=9+U|bP~}%sF6p4 zpU^O<85FQ;Q;N)ZGTOO7KY8w-*l z5!btf(1Y&*zVmovPs|o<58j@XEDEhtdOlGkTSzqmz}li*w)`^7?2aj9vc3}RcC@<{8Kh5pvIR1It27LMneOQK7? zH-OS^HvftB=?jS1U8z`dJ>iL*2Y5JGIeZ~usVfd{Vumw_0i<#(J0+H&5c?MqlcZ*@ zknx;y$RMma(>$;fLDKc9cR8c_wOZt!g&phMItp$??%8;#1(m*9O%6&!WfkXGU!)a} zTDWFlKFb4*CqF?b&DR5w=DK)#P63Dtc8^<*MfWFmZx^gxw&aN>k@rX|*)%xpIskg2 z2g|XrFU)c`OI^WZr0V4ha-y_r2PC%O&d7*5r8HgFq$k0{XSix7Wj32VF+mpMF*uba$b>$o7J@b4y?9UhKGtIG~Tlq8I+MO^yu0m<)>64#=CEJj# zc`^-DHO77#>Fqsx_^`%{Rx;q)S`Q6E^UXf@G%qAWvENCM1kk2c$UCEM^`2HBX9UzWM%`%OeZp8onz#df-W`6{XnSb6q{}l>85du@zO>gK8gFp+;F$$81R11Bc+I!03J0wcQ#XM?HcAC&V(ubXO^(*Ppi^u_BI3BK&W;)8AS-ssJQBxK)Xnw-+l=RQ|we99!uX$U8{tS;pEhs z<{iMA|1;Heu9tA*=F*_jH_6)s+nw-)6%sa{ms1SELC&&?v0Ql@YSjJrF z0r}%+viL^r)aL|$I~CEzw&8si z2@2w^yNbxKSorZ5i2iuOQ;*uXvbi)BsQU@Tof}IJ^*~v~zgzIm%+*q_DYV}pV=4w4 zgsumd1CxjQ7`sL45-bsNgV0h%aJ#*iR}*(6t(hO52SJs%OH7WLNO)z_$UYnOGt9uw z(9EB&<1F$plRC$Oshq`z_{sd=_^FmgP($i z!Ah=(RUP51iXo~rsCdjBaDaf_Gd=-zs}A+WU_n;3(6zjGp=en&!n6NXEC#JLVBNO% zAP)U@+w>I^ki_}PFpvA74R~-Ke9yZ)W2$lh{MC)O>~8uAK_Ks5|$ zo)-7xnveR|nt}^9dQjZ^-^9gRp;{tP{0dM&m7M;&21DKe*yiM zrd|AZIj;a|Tjv^X@vxyr{8e%+n8>{-hMs@s7P7&Ut>hI|kF{@%6dVp#cb*gfR9ILWBK%JKCVoZGd3h!7R+;mT1j1 zICNu@UxHp|gvZ9}ONiZ{1I(b=$*T1Qa5wXg6&>~YF&}~Z8gv+Y#NX;f>sXsI_L#d3Y7C&A*!`d)>rG@%W}Q5>0kn9AKE>(%sSUW5 z#@iEL@{6Znm6{QGOI|*a~Yf?Ble)7rS3;m5#?9ntxh=UZ4oV* zl%zU#6WNxP8@JppPY=EPx(w1Kt7veM2s-_W$K(?g0}Y>E;*QjcQ7AdhB!?h}AUR0E z1JmSrSRYfx1ocsnoM(sk=+9mout-Qkir%%Yvi9M^nyg!b7I##s< zfZ6c_v0@zc=7VEsgYuK;g?_M%pA<(7>SO$*cUh8SHK4z%&h+dDu(hvnO4II2GZmE% zkLnD2ltjr%Qp}zFqFnanvXZe&Q<)12DTy26{57|mtL_ZC6lln!v^#ty6? z)vHVAcrU4clb;qZ5ZIr%X%V#VReb6r`*uh9J}DV`yoSE1if`I76N?S1r2ScYvmtk+ ziQ_#Me`cgMmUOR8=_G&N_s6H`GnJEfgdSP7zlQ{#@Fp%#6Dwiw7AooR&`ElNEwF0tA*8dc50 zYMIIN_;OBYv44SL`uD_vitw10*G_?Ia%PmL$iUz~qk1PbSy{8ayw~RIaxuTb~kH$ohiX z2)Nsv2L8$vJcwESGoBvHc_gv9REgeetUl=Ua?Bc@-r^bmlv1_o&8)6|@Xl$uSzxR& zO>zP@%;kyo!VDS4*ruhq#!VzO<@4Q8-7y6SC~4o{_rk}i zQpzM`L1#i>w4BA-{PVYmp8X>&qdMnBgQ$jN49yW9UKL41A37n10I3(a;RoB7t?*{= z1lZXfX;d;Pf>WEigfzC!vo_7KZ3*GrME0Nxhdu;_qOpsX=*KAUSw0e%Z4ghQFZzoyd?>9-CUM)R)_p*J?Ifma7Qj`Xr%oUk$5*906i*>T?zM?Z z-$=}Vnf=qbMe=a>DFudgi=bUxKeEc7=d*`ptpP5*OKVR-e(}#cLMZ70l`qS@I?i;* zI>GE`;F9ydV>CR!uqdpZXn4uH85!`;~v+8y@yj>NT*Wzdy zE`?)i{#*HL7z7v^Ib{WHe*|GLr!CXcF>FTwc4xs^z`<#nh^Hxt$#|zIjM-bmPyEy6 z;R5@BtFJn<(0E3kRiF^~2{wf3+jLc5iTH}l1Tr!FKt@)q7j8VHS8PvT6gkJfaIMOB<+Vk6c4y2_s{jY7fZN828Msp%E>F zZRfY@E47!ZWEo9wrWch1>V1i)$Ly++V%mO0EXS?)bZ?pNo+F zvc{2wXoQcvyuiZz_0`keYlvRyTA%nHO*AOLv&7+}D;k5QKtr>Kr$-qH_ z*4_yZg16GF0}j3|j$mD>*#C$g%zD*ZR}lLM*}CZrx5jRcb-uCtF~M6t=(Z6h6FF5j zugs7G&qc75K3XOdl)SWBOK;nri3YgjbJKW$yi?^sW;5`;g%9G0ELsUxp>%+k1V2`u z)^wdS2s~I?)CNZv+&2Z+PQ%-;P;qFlLB&yT4&zg@rEq@Kgfu-RG|?EoD^vnNGi~=R z&OL@`)y~V$AAWEB2a{R6KR@T<{zf;@DZPBs9I?NyqP_$fda66%sV4d@L-nX9bd|sN zmK3NZ_S`Wvl?>?|il2q=jDh>BifcIm_Xq7VTIYuUyOl2ZeC|#7r0!ch=(=10k!|}5 zX^ohWMp{N)q|KvS5 z@cHs?K=&SiNW+eedkG^5P;3fEd`k@!KS_{YH_YMlQ1CRBY49h!-~^|GG;hvM)Jmkf z0zK2mBimeI(Bu6z#{bc{Ap%WT5b7m{r{pQ|U+*9qG^C*R@L%ts|9S`g_72*5TE6C) z7gw^iwWUUO^#)(uYbyWMR^~&lx)ZcHH>5MvSS{OpIn1AYb#8tS*4gy5hWj5eTAgFN zf8B2{AKm*)6ZD-naaZG0)|sZ~H}-9PF8SQpC%n|}&{#G+J3x&+IU*wd;N?=(ONYl$ z&Z4#bu-(AjhDH7`Crn6A*j~~Mc5tJ{_aInog}(b`s>BA-_;P!~8MJzl-QL+T=~Jjf zP0|fLUpq?_FUT~3Go*#F%R(%9=0Id?siCa=knjSU&!NPR3<;5A9WuFM5%aAe zcL7mYJ|R;j_FryhvXK2e{cWT3-~0XV`2HU>AE=iBy*~+5seFVW=1V&(Z3ym}h0you z0rtYE$^*Ka3S@1B_*YfWEG|eK?k#rD-FflggB|MOA^NdTDgaNJZ{nQG47P{4a7EoOPu;W^B&`EwY7~pIV zKb|w0dxXGR{A3`(stl#k*X=@G(j`DAdl%9Duu zIJ6Ke8yr-^d26v5j}EY#dAKVhh!>pwAQ9``O}PDwWC#>no|SAoY3!5F8!Yen4purjG&zz`%;=XogV`1~r?cPB72t9MAe)tmF;CXUl*Gc2{d3J$mn;e$0MLkwjff zPMlT@QuNqCJPC88`@nbblz{K%JchJ?1a?Lzoj9|Ugl}1F(6Y9+E%Qt(xr3|$+RK9CmC&O8MmZtf~r%;gr=1J#5Z z{&P~t?Qc5vM(=V_cf^^=l){~zgz=Gs++Shjy%FKvhbfkMi3i{LUinZK2{alw1S5XM z`S12f3Sfs<2dUdSU1~Va%q|)Y2AMaa=%pLwBfAzdqpA_z*|c+tS7%5QfqUgCu@4r# z;4k;iv=da-tCvF!veWdO21*Fm`j0(6gm~w!zi#>Ae@^4Xhp5|otU(Fg=3GC_QtfPN z0{}Ka$w42I_@czvYv{X0h@n^N?1vxI;fSCE=>?$T`?Q#JBL*t7DwrQ?Vq*wo)<7LE z5N#-WF$Jjb`>cq7rT?Vp_3LwQ3R|xde$pH~(7KzSU&0gf>ZL%Ia$HI2oy(Ga3wqIX zKNv+_98(Nq`hE=#(4}_`ITBuMokEULMS(Wp+%t20>;c_iW_C7Z#Tmr5`&~PD7>OkA zz(7XgNxZT%TAb@zh;9B7SglW$K8BJypZtCvW< z`N8j(9~4)fBSJIoJTeUZs}7GIMj81a#yT;uqpWUxL#`6tzdZ!Oy426b=--iK2;kmO zm5i^N&&6~*yRT{K>qk82ZdD=t>@5gpcBcq;5OVyVsVP6QXLbPiJ8wwuf+8~G^?3{r z;iiz-1Ecqkv*@!#dA}y7THSb(Qw}e2Crr+={Y|#;M`loONa+oY+=uXhU1as)e4n`z zgaP|*rF>{c7a-&YffK+mt16E{=EL9s4Bv}M{7Q+2=_(5mmsom@W7nq64$3JK^lhMw zlCs{m?H%`W&)K;-cLkZ@^QGk%5c6pO`V@6*+eFN&Gei|c7NtksScf~y6$m9Pd-PyW ze7=hYgN*crf7V9a@ZG}dpH5%VkPGyM{;UQ97!r69XMjt|VcTPl03n#|P4p;{n%)p08gI-8zFTPGB}5 zp~SWAS)}C!n2a|yj*Ik6`WRxDOclS`fTN-%`3LL>8;R#Z2y18RBnT|$N)qrb8KP?e z?Xa2jYeYpdVS0hcux!An33)~n)P&wpqJNOk?Z4WV;3hy4?>dw>RFAHC0v<|e+pr$| z+jE52tx{u?c=nae1B&<>tjtt{30J68lPvrrfdLH5o2v>MSm zBw?!xmkB*{$kJA;e_IDl7amD&@Sq5~NDMLTS#~@#7)?V z1CJ~iwxkoYb_8vK!YY8s_Z=&N?e^O3;{zQ_;5x?jsi*Yzw5RJuf)BMDeDj=^tqW3n zJxyh9W8+MLx7avVI7V(R%I!}AVs&{Xd%umrg+a;(zB%>9S0p^*tN2g!{Luc|(-org z4`ORzc<-DdRYKu$4tXcazQQ9&Z%&{moqPNB5-X2Ass}57RZ0ud*2#c=Q6tHr*_on& zCseDy&Ji@Q9N#dZA9G~S%%uNT&9FY-mZdZW%t)v@EWxcZ2Oc0Y^P)&@st-S;Oi&se zME2NUN?I-9=<#yLI5k%1R>w5?Hvf7K#SN8AJP0uiYN8$8=3L+A#@dJ`bFU5B)VV$g z+kJn3OZXx|z-HUd6|ru?{;6u9P%*xWEov>58u9(cFy;MeWSE3SIdRc1MvKMZ!TCZ3iN=Tc2@z ztLptW+arCfwk(28{Zeeh!oNlPQPFzr#RE-#(7?06d(L%d&aM~61tTmy*)Ar1^kWeP z=3hs^!DJhLEqD+~S4DWKv>N9PX-Kqw9oK&Yc5ZIOciP~}qMR~^!zJB6d(zLG+@nTx z@M#j@swzm6RHCQonyoD?>I6v2i~f$W{xY=Sz@Bx?gOppz{gmgCrV~Jx{^iR3NW#G( zPhU=FeAjChW}-}rBNxU?yX_B4YBF!v$0UxOwd^>(R3!q~ zFIB25a!3T9bbP9>lvRTiea6%1ts;Ep`tDmNB{g?kihG1cH+ILCY~w}_2hr5iu|B4+ zI$tXG>m*f#s_q*BKd4*Vx>Q6r61*Z}M2TcLdk%2oqUQq*#q^mPL0Tx~zPL4vGYtlMsa8urScLUeaE&tV?&)$)JyO1DS>ifqRjKbeyQas<=FCZT z_2-8=+&&Q#o&K_5orH?ewx3&iK!qo)66O)&jGZC#3^CqQc>!1U++n;B;xn-?hSZU4 zac{~MT60d8G~!PN1i`@+SPeUSua|%d;PmLp3)~}r1nH6eL(^iET|)v{!?Y>3e2W{U z{D}9^bPVE5)gF}X!j5ljMHX@_2771G5>Ty)kq>Y5P=h!9l)m@s2SeO-S6N?9?I$Qv z^&%%jNpFp#&tHetz0&O-iI5>F975E7fw}Di_yxpp7F`~A@bdxFdey77%%c8W#D_+j zJb>#nNa~zlF%dCYGkDbkjzo$APw;+*fTi#s- zAt<&Hom=5x$%x!me}Y@bS9;5gSs)k(ah8XriHm>tWt{+kZ4sbz#h#6?IR^hcnv2X< zdcfC*&ReOVTrb`kHv2*){;u~nWwSCp&?hLnnh|!Xd%RG`_zV?Bx(4orHhx$VCRSA( zY27&mPeJ0$Jtl(P#eli%0g}=8iXun@auFg0n}}RY{H5Iiv{eX1rjkyw+uay;VL+lx z0Hn))&3UR1yKn5HU)5Qd5l1{oQg&U1mQA;LQZ2Q2oo|1!U9uEPGyqa@XC@^HXz^7xvK`? zGZsa?_I{ZljO1Lshy9HM5Yj92LGk1ZI-MSj>!#<@oJ>-RFoP7zZlu-sJD^mofpj?` znW*39fyA8f1u9B5aY1j8bVDJ~VT*szrl4+RrxB^G|C1BA7E!Q7a$f%C(}S>TfSD*9 z_gU3{gBdg;+R&+74PCSL&q;gLJ=g@w4NIzcEk7fMb!sJu zS9~#;=>ewLBw$yIOBvg4`ZwV(kV=y}_Nj!Mj{s@Ya5M~Yf(>xM;X0R|n)>>|KA}&% zEop1&8=ZW-3Tdut?j6P8#MZ~3U7GSpwL9JR^3r*C40yY?S5Ol?m`xANVD>V(ch1!u zG=3AkFm}y5vy6DPb`2)a4Y1qdn#Ohux~5tN(LPLp3pJMKCi{5!$BlYvJyRZy5}M3! zHBjX3j@#`l6SrA(7Mnef_cEs3%&y!lp^yd`emDcNEgF7t!+=bIZTby>r25PQD=ul= z5`vi2x58_YkP{ztf|QGtgOt5p^5dzp)34v}&)SE8HVm;X=dAtd*uMP$U%6XOnq)0} zv{`hJ;1pUjFGriJA;WJvPFf+Splw z-3v$v6kBo}Pn&kxn-BbJimY^<*-d~3cSF>f*j!7zl^n!J%B^bzn3h7QA-ym%o@h^h zPGJhFT-PHyPe^MmipyR!1p1X`u{nS`-XH&c)P9xqQ+rt`usg2CeQ_o>dPi~}Z*ENV z3>)R=OvdD<83zgLUSa=Mp7k908JB_9biFy-m2z1h`mb*v zEO@5wZiYkML8o2*9HT6SGwo%H+{N#a?aZ5)gyvub5mgr7aJ+3KN@m$6m2HPD#4if@ zJC3&Vv}X=wI0GcpKg&E{Iqv{lgp8MS|J|E66N6GZfJ*+*XjT{=!a;A_@()RCmqyw) zQkv0_GFg}O)jEhR;znnS7>oWaw$fGtf3BN-KzYZ^LvYqmMr)`GE!9UTM!EBU1*&wZ0zjVJ@2C0;Y9Ocp=kp) zu2LU!(s9!;-K4Z4?+*6XPbrc{9X+5uXffA$N{{7BT{6fazJn5JcY}@2p`>mKHTrV3 zOz>yp7Z6GgI=GQ-REZ$^edDf~SPm>X=>?@vG1Q`@j}e}V^sf8C#GuQhc74xCN-A>#-VFb=bA<~2d5^d zTLEp{LbtI##17L90rCEMYVao~=!jC1((GD@Nz3{6IqZov=J-XP+ZeN)t);C^+YgGQ#s!wrqc3Qf7rzx5(YAdYz-BvO zlFn`{7*HxB;fggg$^h zKdfSRF;E63ky##v1o{0{Zh5fZQ8iwaNYXjLr!Bs01I&12bw3COu(X zoY;#qZlJspLro&ul*%P$<#JU^^)IJ{oZg4w`wgC%a@;XjtPYV3keTIe$xTsWx%llw z*TU^l&>hNKyM~nvHIh=45q#F!vrtJMRWi+c*qeXDy@@RPeyWV~!aGvyT1pO@6uqpS zq3ebn9xm=G_m*hWXhb0CC*CpM61LuFCLmHl=3Pd%3H!F%zIjtpO3^l1uXWHV)~)Z5(c zbkGS`bTIg{`?Q@$B_xX;1n?3tC06%j9Nq8Mv!D=k-@n31j^Dq6Lu8aFa|jj_v5S*R zJRdo`OO;~9qo$on9-ty6z-$H zQbFd&LQ_g~)Tw2B@(+411+m`FMZUybum;i-j;>bIXQ~Hlplqkw0zRLa_v$P7Lq&HAJpi|=-0i>j5xZ^_+%AJ>>%c_Ump zsh}p{v!EUaL&Ncu$|bIzte}EIw*2O3Ol!J}wY5;urfY(`Nt$Q6YIlEO%6OskX_hq6 zX15JsKN}MSN>xpXm}KJ?z@3F=V^EKkiwU%>*5vc5u@ye5%b1mmrHAxAt{Q=n1ijKV zRRyu*wkxkBuL&*)y^^`RFzL--9-e%-*6~#}Rp|PASMV*f&tA`u9Jn}XKDo5m>V_=| zi|}a4wB_JE)@1Jra3aH*Q;`9}_}lyAU=UXO`+dg47caBwij{%U&~BMHx(%jB9Wrr_ zm6pd9#6{!|#HFoAvvA`sLaAu?w_~DDsAw_Q@E}XRd}^RaK+VNk*)N(cyHHL!{`l(5 zJ)rs7w0MRw<&SGIU^Arx+aU$9&L|@$J#mEw#)w(KKYxFk7Qg0SF@$(Sh|sD z+wLXuXRD)zt!(2366KsJn7D$Is$)35cabu5`ocH_7Iqdl=oH}86Vb|6T|K|5(P$gD zzQLet@R_{wQHyAbZ5(fPs=v>~_tR265vvciU(XeSai5WKBht#r=NxFA0FNP;%*!~2 z(Fm4u^7yAki!QIkW;!h0Gu@lAI(=x^f~N(@r9vu1n~Kz3%9WWOp7J#(CM`dfJWcKB zv0&{HEE1mVG;c@!$M5|3k z**go@-TgW7T9WQn*=Lh$&n~D@qy1m+wMZ|uI1txSpDB{oaJcB|9P?0DE`!XmB`&~cOmk}wMkvMyspOQ z&%G1hT2T=Y*7 z4C(QGNIWVH(&OZK&&G(OAB2Qo?@7VecB>$z6KL3v#I zlCHK8u&-muE<@?eKW6k>Wp{+w1)Zx+vg72OTwpY`8dGjs`s|VjcnyUwbVpX?o$Lj9 z7#$F+Ty=MM%aW%f7kBHt8(fg6r(6{im~6=zNLyH(iR+YItld4ex-a9FqF9NJtGmh2%_?!v`}IJ^camL+TF*QA6c_y_??`G|(I$#R zX*JSQ=J@H-mUAYMMHm)?%8)lv0|z6X;`?GU+Fu3QE`Hs5+-E zN!KrOwX;3_cpz@>iqk5HGWPF5wTW{2B%L}*+`<9Na^++vb3F!Y^74DH^NsyN#`wfC z6W@jTOPEu3(XX^ue_dr$&z`r?R&*cfv?szxZ!RQLaD`;*+~sTblWATYOK*(tv|OB~ zKi4t-cC}D-JPYG>&0Qn4)2d+I)2OsPJs0R%HvFTsB8ujVnaoPuXD<2HnRel!JiW{j zvzv;`J`H=Ui;5%Jv*KjyWj7c9r(D8bO-FN|>DVrAe22{(&t}AQ_|dhVa&t8@XOMze&9;{z|jE+~rjc--*WZr%Rt4}S&{T#U*;B;z*^7BCl>y}o=u zSSF2*Hja0v+5TY1M)2D$cIxDbVilrsIJW6~=wJ}p_N zp_H#{Vb6`4DN;mc%AcutMv@;}?~*hAeAvZTp<30FLeA0)ZxTC!61a^#y0>EJlvrX- z-+;d1W=obmP-yC6>jY8l>j}w~ckHa6jzoV*F8Fl)jFc~n0`3$N2($^2*Vm^|x9n(u z%7I?4>d(R2XMM+dyLn5aWwKGkb8i_9HNv(26<901zjoYh6=(7duR7(|G{*4QGG+3( zG*(Zr8yft@)J?o?t6_;?(JtVg6yHTQ`tBZTVa<2douT?H^)rqxklTPm-%kS>UE}mG zm#(+87fYUA%nUGc(>Wnsqod&hT?lIO(I2+u!HF1b7qp;bFzE$^wrhA_7yC}Ag7Qc# zEYq}g{`jsLi}6n3jB${M(TQOt*6WmIB?r0XiBI2-JY)Ucfn~gr9=U_ED>N_bLnu@a z+cwJCSn6=yP3Pf51Fpo-aGiR4A>Vcnhv^Ur04AX4@(R!ZL)>+t=wo&`vi})GHFZ6= zShMYkL@Gs=WWkNxeR1NHo#~kVo}pV=psbp#osYQfw}_wc-eS!^Q6*>UG`$x}x8ufB z=^8kD3LhsANzE&ZaRpF|@jNgPanQy)jM$AwXT$Zr&%{_~Oe9xK#ZcSc>0a-Y84DRd z)N5d`ny~C8d$c1@VAbx-OyHy}V_iG>m_B#c*)MRm$+O9)CtNUN=X?Juq3fHyJE1zM z;b))ky&5LP*Xyt!^!ORMP}w%JDnb5P@AWbOR7)Sf`A=I+{HEwqUMYM^_j!iiCHK7G zCqG1x3NvaHKoM8ZR|mleq8%QsJKp=<8=IH%^sy2jQ_a#=#o5`NAo_9n}q& zNInEGj??|Hwz86DxBsVqdu$qWr4LHV87{OvdSm(-S9;eD#C<2owR%PF`MX#exhNvu z#|T&h{mVMpv#(c9R7n*YFMcZFYZI_+&iNt>NLP=10%Jc)LQ3iZersZUe7yP5<*UDH zrW*cT>jwr{dMd+Bx{i%SB$_RQIT_Yi$^{xJWNs>+&5xID$Yl<1WS^w$+U!gTp;O1H zFMJmcqoOrd!V?;kB+!Y7V(^;ot4=vV*>$ZgH9j+;f-%%@{(_`tq*I~};n7{)RWxHu zVrKML@q zisJhIft{Gjv7r2iCCj=8^^gb(Qi!orA>1U*)%Wiz<9H-Tjg`#Ncs3y*uPf@>eoEbe z>tEKmQ=v7VOqY+8n*V^9NF)%~6YFeLpSQfe_)9e5I)uNP$FB*!O4Lo$%8tpvX^FaK zA;sblxFOCGGc9eO5{L;`>LYsJ9H0o z|1{C&446OO{AUUiP#@l>0Bcd?oPPvw%z(rDWJ(GqyibR1Ar)BeK>v{CAhFMJ>;m2$X@gyxEzAO9~VE6XkFDrLl;_l&- z6rcUO+Z%fu`;(w7ln-s#?;U76q~cJJ7(asXPQ3Xg>JsB~pH~qP5q**+dN=*{!XG7= zjbQ$k6(rt7quWJ-5C^KGP@L+@qV9fl^Y8dd~1>s}@7HS6g?L)3u zb0%0SRM4W{M#JoG8~YwE1iam%7g-*!34{JeZ~6D!JirCS1(=)dLjTOob_?Q80&Hvs zWWk0mQk$d9hAP6-XuS1$S-kAifFw&+nJ2e+X7PMt1hko#8vu0Y4DY$1GeU?`Ol<61 zu21^L#>Rz~nq%+5R>z)#B3)%WY(O^gX4{OVj*aTwsM3>o3)52<*b%199;Q#7d@P;~ zE~m=n6f4B-e$aR(O1Tp=$n zQsMGdWKJjuoy#BAGPr%X9L-}+Xq=}#;n5)=0M9zKJ{V_+hk$7-3buyd%?4U|Cx6{s zJ{wH43c@r?N%?11W{okX3OgdN2{@oECLTYO0-AA{1J+iaowv`e$0U_ekzUON(S3 z8ZIlp%-W<+#njv>@U<=wFz!~f-kpU;nK=;TzC#-yl@s1w$2{!~3>V3Xc|?R71&E@AfFuoYTWMh;+mHDQv#P)DtXtHHK>cm9#I~vq z@?`Xsup!|r7SkXb*BVNiaYTs;836BSuQr>XN0TD`pM>DF{=wZ~)JE4jiL1QVmG$-! z;X>a5?ozX@LqkTw{=2{WvdH~+8bHw9k%$38YDz$z-L~;l*FXDz{RF*}lM~<^Sfn1B zI4dg?pmKr85fl_k^5(95Q6C-6iyC`-g1n>YS%X8^@qZ3@eT&^hashdM!b&41jUy z=>oz4mhVL}8lF5O!JEqz#C|e_@ek1gP>i;bGAa`=2PR5{^F?vK?$8W#|879|#5V}J z)xU)8?YUwwb&HuA*4C3tM}AGuh0$NrGei--h+DT2pB;pyu*ueMx0{PshX(~~nw;Ka3CO(*$SjU1a8emwV`yGNu6srGXo)BkY^#WwT zUj$=f0cI|yrTYjg6SlH{;FeVRRnRoPW(%4iP3|$kT7vKR6J&^xvtO_K{V(?3I;zUH z>lOzdQQR1ygn}T5B8^H3NC*gubT=qScXufYQX-)e(koH5>Wp8sUH_r34?y02JkuDRyy2SrO8?1B?cxD9Xjp}ZX}c9Z=hp#bs` zOvKN-Xi5?h08yXRe#02)SQDkd`Sq@_eUJZ-99?VRSKyhkz7cOK_ROBr;IdM@yx{f- z6BPq<0&~_8`-{uaBsytjVT}HVC>t)-)6~I^4u|vbjaZUsOY~Z(VRWuA?U%-Wg8^jr zm8(xmpobxdfL(y8IW&21sQ}qB%NDmB-PR*f_>m)lGEd-te4fH9zo;Bui+z6B)*3R7 z=V13^W6=aU+F5AWD+MQu5gpHsJDMO4Ll)}-I$7NI({eGObDsm+=ufdDPSXZ+^dTj7 z<I|ZuS*T#8X#;-AW)*p3ZZ}JC>$hVv`#e1fH3_c?ka0JU=h zXnjbOHei@Zx(yw%tY!kRks@{6F&03B?dGE((gi9nMo6GvV5Loz+jYg5Qqa8|sR)C8 zQ+kDPYVS5w;{-IJh`0b!8Tm+rI9NOmDo$+XGlQ8R#De-XD~5V;5o4J~7GD1q<(OLZ z`iU_S$C6%O1k0a65HhuR8+4NG5Th8pG0)L(SLhiYX7>2&z>#94{;^Y`Gsh1F)fYFP zun{+*?mT$?zj<_;RQ0t)(|GFCTutjo6epxUKsNkH}-Bpt>C=0CG{KP|Y0# zIRK;carTdEHonAI;$iU5ON)F3T0n(fcVFLi|8Mip6L`^rsDQ(1lI+Vic7fg@ea9!3 zS%48o+EHffmp-tJ*FGD7%zXh=o{NFIguiq0igSLLQP_!y<`ZpKgKX^xg9V`b+ZT}D1!TztmUx`nk z-CG3kTR+q~O_4OpIa<@9avN?i^NzW2%Xf9Iw>5r$=P~SBpN1LT9vN{2&D*bp_H#YW zd`^qzH(3hqc>q122ik^_gqa2S9daAB#QwmY-jjG$h83thIrSO40;z~ELn%9_wsr(o zPt~VosJ5#?4i|iiw~#J<;skcJ?^A$4T0yaZ4y`wDSr9ag20Y4_-eF=*pjBzP4d{`4`7^keLagVEu=j)c@+I7V9_~Ny&Cx7~u0eyE38F>S zfQ#a&s3s6~P?&&Be(vD$f!3HNXk#q^Q)+sdHytQ?uvA%Qe;_xC5s!6k2wFNHE;b@g z7mB=)2oxfOCQOfia1b8YjGEhwMQN>YeGt(bZM5#h=;c0&t|7UII6j9$WDt2o@P{7I z2Di+C`RQ;5BLLBjGd&;XEaDwN+m{B|XzTNRHMAz^FSN;A^XcD&>iiEpgxW8GU!;& z`qnRl;9nn-ojC2aT@VLnJc?ZvPf&oU=>llBEso|@E%H@OMroQuC|L}x34OpWu|h3r z!=OpQwrm^^;*M=6TU;(c$P&%rXN5W(i{b%W<8(< zCczI%gIdSz{vmrn%9&Izmwl zHRc@DL@;z9E)v3D3g`)PNpRA}XY^yoj{~~hM?kUW6Yd_d?VSgYJSyN-fDhJ#icv*3 zO|wxT%2pChf#jDSh!{O5lrt^M6#(Z6o^NRvQs=VRk=_-dd<1*F@N~ay>k+XrJ*J9-y+aQv^10NsG?mHkd%7b5JIb4})Dn!dPS4S0# zI(6QGr(I+)oe|nktBK#80keh)38BrfI3UyJ>SAyqV_*Bo(OUi}5vK$spQ39W6fPq@ zT*`v>IUmd^l-IbC%1o)l7|z2avmYgshE^r~lT^tb$Q}t+RqqflsqxAzXtQkT4x%U( zroQ!<5lB{+Kxw!q_zS}$lYQAaYY>D2_0Amht9bczdX;Dth&r-eYg|`tY31{_OciUcr0NZ`Yoh2N`wjoXW~P2GF6m&4VV@TjmB*1(t zgHViuw!>ff(&Oaigtp{S=0oxn{jMNAH)Je3#_*KBs%@{$Q%f_inQ@;9=Nu@i+U9kX zbk|0O-y#9}!zSjUsS}{0T0$hT_Ja6VEijhIAJ#3WVEJi7@3&&zOB?q^l+1SI+Xg`E zNlUKN1yfeMa2}NIpM)bh4Nf>IjU{hLpd0^5ie6EdL|)qtupu2subS9jr{kuem28jz zjWB}nh5$VITL}!@VsW-jjEk=1TGl>0z}>SSy={_2h#%8H)UC3GHa$PfyPUL=9J>yS zL;b-QOwKYKG~d+qI}QA8NaGeA#Qd(+h-WOl;FOFE5-TA~(sBQKZM_Kz*3Q+Q>`V*b z2Nj%I@oyU6E&4r0NW8L9b}a-El4p;f3W`Ul$lx|-N_g;}A>9y<7<;&C)hBCs=M~Fs zwr82LiB=KE3gwvV6p14KX?k_jNr6Mt5x$*>8r;wYYIzzK@$?>YKf_t2{aqm`88&mo zTz18mKdx@9B<<0&>L9aIOJCb&zl@S&FbAo#8?{BY6@ovvzpF?jk>nS*Le2X;;(Tfo zX@g1G5JQ~Ks#$Z{ovnlNRJ&&NScgG%xam8SlM^WVn#AWjmc-eVLL4>WP!6tD0t9jdH@?I28!Hz>8krQ&EgFYunduGau$eP%8DrF*oeBh(^hr^rNl93MPkI$ z><@b!Pwn4YVeJC>uG<)AjOGBMmhuq$<({YW_#Ro^pw@Rf*!nQEIqZTlR@JzD2qI)c zoA#g*Oc>?erK73{w&;MErh}mX0h`bUBq5@tIovrNz+I!Enz?|epJx0s(X)ZI;R?>3 zY1ZkzV~<(0*BHBy!mmwwH-plcvC0^igFvt(Yt2drx6`5)P{EeSqTeO4(glAud8tFC zTJDgtqux`dz%|K-M$Q~&XdjcL7X)nUCB%UTK<1bSS`kN6cvX+DSp07P>M?!^Ef&{eyI9u&pLcLCVOV7w9PGm#{paEK{pY|THNe}OqtV0xtr z-zG=-7_Jv=9lJgEqtLYqpv-eShh+y9P;LJxeS8Xo3@-V0+nyNDXZXgO7#pnRgs=&$ znU3wts9Z0c&;)-lk$ZnY039zBPbc%I?NOh!KiDUeDss&r)q+3sA{-xXn3O{l|J?6C z(EwY!^UF_1Nax?;NP;qfNyeo+Ct z*ksa2dxXpQ-~0qEFD09st}*;hi@qTsll_K*g2EisnQa`RMzBb)Gl)uj+mq;r13uVW zJP|PO8ogd{24-7XGHH1!h`lU!h{xVbNBd#%l!7ShUQMs=XYJM$;=-g>01i3K1!?VTLP{t`OrwS4!vAc74lh-m~RE3nbCrb>XT4fx+Qz>dK^ZXVbuLl0M z{G-s2J7@9UBQLMa8q587hveaC>C;hA6`kiY4eDuK$E_v!+odHvqe3XoG4x|XD$)$$9FxY}NxcTBv znIhy++?FT`P=$^iohDK-&B<@TSgy=IsoPGC^W2-pF=jO60 z7toCrLbP@#^L+){n2zpq_j=bu&<-=g%BpUsbnA=B-NW40qdX{&S_Am4(kcbgQ#*c0 zQL(i;)bpgG?V!AF1XaUe!ke2ql8slOvaomsl`El&yX%(s9Tk2pi*M527`qNFd+GzI zgBVH95?WH=YJx1-0`#%2Cin(g`Yx@mQa{3);hlsReo1(O1nsV}c(p*tth({o!wTA{ zd3+tP3A`gFh|nfpGT7KGQIkHH+QdNemIT{@PLZpL`=MS&TgKxGb z_-2FcLgiB0N=-NP*H88Y4cN5`XkYJK#rkGTbB=A1iFkPMqpFk z^boa4k3-M#Tc8Fq><9DjFat^a|Ho3~i0`ypNFzU7Rp^nvHs{G%Z{LYjEomP41(jyw1wPUI#s zC;_g4`h+LuA|RMgG+(QdO%7lNzoVW@^}o2rPt#^;ea2$`%TQsPlw~zWf_#qxgn31~ zpp|@#ET`tj=Rkx)79ie2&SZ*wyy#JZEAP*xX z8Q|fy@j-V0I1EOzLOaET5(vYmd|v_0_d~nC$Q@MEf)CAX zBThyQ;6ORgWzCdK5_P0qc7bM6Rqp_XD+9ps85;x)IdrbGjsD|-JMz0#r5vz zVRRA$!0sV0mhh2YP~;zVontNYx2`C)?J3i;_)?8##eYF}Cf(oFf~3K4wms%F>OmqD zLr6D*#X;UpPgU2Wr1|N^SANM*>llRYYAd(pmj^7M(cd~*6e9(3tX-lR5FkntU)%}P z`5bjb)jY5MXtP`s0DTLARx-kxn_Zvg=W^Ql6K>fNrDI9OhDv6=aloMr zJ}optl}?-ueGb{p-!s`^TZ%`b3~VnlcaVn_mCHmJ8_^8uD^OSXSiD~u;%0EIp|&$q zHaX3@k}Y46A^^SlC14iS*rVfS;RYU1TDl)fB8zG#O*ybKd$$3ZdZQx03CL6hdBy}b zP?&`t=39z#u5eCZ1re``&?#e%BmbK0_uzSA4%S~BeovrjqLQZc^|71fy`M-vlDKj{ zM2;Hjii+Wc#t79%@(@_H4p2L&mh(4VsKi)wyHc2(F}oh%!_18zfs7n#TAoOeg>6668Sb3(et+?O zMrdt-f`7-f+z08rO%OeCCG^Ya$pzFF7$z*n;klzd8!H2!KrZ7JNLN|(GT%oatOva6 zH8?Wgpfipqse5Q>Ui>*p;o}8ZHRr{?@*oYmcLh*TZG4kH2SHFevbh>1m?0JN2rmZN zfXl|gY==)=sAsE~IK~f_+T=mpq%tgt2PuB>Z@71Aby5}kW}$8Z8?{rWXoPscie#eU zH?ui+FVKk~$H;HKMd!SA_#G`!@L22vuJE9|1l6R@x#~~8g!AAZo`{NHmSlo8YflW5 z0$(f@O7`%~mD&kH&Ok#3@E}WM3gUHoOVG(v5O=JLLDZE8ws6`INb(HA2{4@P1?ECd zbb=J;I?SAShOF=Id?v56K7P|4FKF4<}N=tioio|VJv?J(4+Dc)+sC*Pr zy_x=OnX@Tbx@duMV%jaG!QokymhF4)ye-5qQY8Ru#j#q8O~fgi-9Yx(sLnR*5KffJ zmBu;q2Glp)dL1n)cR_vs)2@2aQPWbv=0iWA$Wtz#wboPCv?-Hq;8A4BKZWwI+r+S7 z6mD!HV$62Dx7E@9Xpt)iY5cBr@JQHAwK#P8mJK}?>NIzbUP&&V=X!eZkS#Qf@3PH&+txRoz-e{iefBk3LrpWui8svkU*ztdk(nm!a={CsV<&9 zM*N@BZ~52~z19AIO7#DEC3@kb{pL1=ALE6D7LOoAUbx;41R|&KlU!gm;@}06=^5PT zN7F$#p#!D)fp!A$ZyX;ffK;E^+M)(793K++ACl|yNW6;X6L2a2ZtbE`ftCFNtMuq+g~V!Lu%?k6!HoI z1pBhiw?QB=2O&Qz2sN!iG^G|wEieznBrLf&me($;_X4(|jph*{Y+7thQs)sry?h^> z^IF*eja1l8u}!N1Tj&LFOTdP zpU=OXnnzVu2=h#=eoRWiR}qI&P3B1FBDHSd5y~$Drgh1Cr3`rc0#k4 zKeW$~YTqEAC8A~eRnLW_jC4v$XmYxuolwKZI3tm|PE z8ngvK`3BBmTn~sSwe9)9Knaj3>2?Px3<=y7)VTjfnHwNv@>68icnE`A3_VY#PZ*}D zzeq>sfiH?ur9{yO14s|A-U^5^ zCCF@%^sACL2(nS4FE}i7d2C-5&`IW@0ig7_?3MOKFq3t1_wcBx zstu(?-b8KmkclM2h=wYsw?Zb8BTPVR3Z-EzVHiAIvtJ>aKcq*`kqSQWWp46FA_|LU z>vI7}mn@4AdV|ERG;ar-n{OawFRt5Juws*|`PRgYlCnOeI2arpTngjT2Oy+5=v$XU z&eFR!>gDBz$Sf{^6_4%i1e||v{c`?AQA;7hQhPMaSW(Lbp=xD_=TB&gF9AMVJ<~GWtx14r%5iWjr|6$voxnb#$ z(GEavx+DU5Y@8wJ_OfvTGUZk}o0VNDpTpIo_Npod?8~)p7!$#|RDyNc+0Y55TmFhL zfl_;aaQEr`wLYMw8MAg$$g`S|8zYSo#>lglA?6%@8W8^4L#}*#=eJvafs;`T^Z?B) z?Imd?6Il)_VRSK?7JhuAi^oJFtVk0D0Af*v397>^3liBhl0HNt5js_7CtV-kt_B?| z#131}i+&GnF)em^pDiGl=60A>ngefRO7#wg$2xDS;Qb80i_m%1ndZTjm1;JT;NHxP}cg(>Yuna(T)0f%Kn<_Wxq@y2d{d5PQp>iqu zxyzT6>$CCVS%Nz&iaheK0rD~dhlS_`z$lP@ULQcQ2B?rQsCo;bB@R=)*<58=%fHE4Bh#PFZd9Zm@t33CFeIx>}_CZEz zPXNL%EdYRU0|6D5>fX_O|=k#yiSy#ICy3MOD*2V z86UQx)OW3M{JqGVYh6;e%yO8FFES0eBxh!-Pp@++q%Ic&y^3{Z%hf_Q57$&l3fsZN zR~NEwqk^?bR#>D7K{#(Az%qA?-ams)5st3cr;@Tl=B|Q19yN&%J<1ad#r;2Xql5Mg zME2K|KB`oT{QtV!f|+jz#dXl{EC9M-Pfl=(!{fps&S!U9vKil%2rq(3qNYGik8wxD zgt|qqj$2Ml^fnY2=Gqtu2Wv8pp8_#a%N!^n#mPr2)FGj8z&;0fngcn1b%*8T4})pBT1 zGkQknLzLVfArP5q2o_f>9){9RDe#CZy*y!?vjE@kB4i$qSiGp}FhKw)+g3??x-o%1!tj-lQi%#j@t?El6Y^s~nB1LtSm!8zS-YfepDAABztD^X>K zUj^G7rRg+_*w{K%;L5R}GMMh?*2*`>;xu%z@l;VgW9HmnN3ZBn|1iml+BvhIBxBmUT+4k0(5rg6-xUT~^f4md6(=kwlvP2dC)So*Pp2sB}H050U zFdv0uYhfTUp$x1F**U+Zn&m<%E%!8ForQ##F{YS6P4faj%@b-r-64?ZhjV0(1>y?3 zUeDkT1R}FVm=x&7nW6gB%hi=mStk!?KyHVU>L~CPbxCiNz9x_--no;?3_ETyY`)Xq z+MQut=-u+UCe0GI?^N*gbfR`(C%6Z;V0$Hl6Bv{j^Nl-=A2mD?@dfu=3I}jKEU&)+ z=K^@L5|adPUV&*DRZZ4=Nes0jE)JgayU2PwTR4H39G@pxknzH_`zFUiwwG%DbXR6L zp&9r*mSL4B&q|Mp;oiWN^O^3`4|SlGys95#QECI;%;xAwB|L>;<6kUK-tq|E-kxOl zvPo8i8fqPR@fL&+Wzjt|x`Q{yA_YdIvIW@ox1d5(L+Ih^Cr_)uB~LH_ zl|OCoj-O34KlPVnKFwv!{``6R&JCtO-(v43*)Ozx>P$QuvE}^CSB^;?NNClWY5~?va*D! z;vsn|Fdbi{N8V|*PYamY~ zq9pApbs5!kEzm2!|5)Z}R%o%Z>9kD=L<0d^j_b{2oUFhj4$ILY{oE6(wTNt`!J zpw^>)z1?XYYT_}I?O9%pr|g0R993AwObl;ws7FtBNHrs~-o%@seTNSn3L=o-G&j*} zqqBu5{o6f}2*Zam0;@`d^n;dFIRez_PLI1({4!F8F_zr}JOjQ*Azt>H7hrA9nsIlP zk^?Dkn=1l_Pd`?hi09cm)J?QEej`eKl-0;S`SjjZfzk~1TIdgb-I941I>6$&kfi$? zAU2ZjscD@#eu7!d#PP-w&5T{NN5_1KA%F1+Uqe$UvXZ&2O9b+*tgQ68iEH@CzPv%G z$Z&-LXJ(jDAUDm)&7QiZO3CEyHJ?m4t%e~`TJsE}&Q`pj@UER-!qtf3;H{EF^0VP1 zZXL{B!>Pk=4&v_W0P$$`;a*zshWE4Vj^2-eZPcks-d>Ci-%3lkp@KaFiOJga&^L@~CX9;{ZFm^hE~zCk#uL*v zbrS+<9TM%28^=h>A2;qJttN>!Y>72l5?)z_ zp3mC05%hbFnom@IDqfZ(Ykm~CTV~vT#?3vicsTl6R#$EN+b2;BOViW&_2H+YOFB(m zF$x+abT&$Eq*CFfGXs+x@)-Hl2O}jNyv4^7M+!P_Q`*vseN{wK5EDQ|jtu2%!@#DN zK#wWZkc1~>&wb6xZOrLqyOqrv6FT=enNCgFC_ZZ&7W6=*fjEqP_Hr1OxLkwJ`z#AVj|^3KILvV{#^_g#7Ly* z&*VXF{4s=l0#bNitQ3IoA^^e_?8Ju=y!)1&$kxBZQ&PiL%T3v&NP~4f?u8_er%$=i z|8zz8_@yrzS7kr=n7P=o=7Fj4<>wG`UZg+)r#p};zR4fZK&N+?7|jGxsz6xry9VI? z7`UA^$y`D%zTBWv3-4kGB6%}6VUt!S3*)WACp90}N-Cjh8W*J>4?Wfk2U3GA9duZS zE&#V)vwRFP#A1Qxwcs^e`J{T{9GA_y10+1`Cb=l^3If>W_B}L~#0T%4N&>m~OM_yn zMRFJ!(15Qtv#@Z;bAH3zqdbj*27G$IY~)cM=eAfu_OSTnA~Iwrl)>CYd7X#ut(%ni zr$I)!^mO=|MF)MItHE+^)#A5OBjud?Cup-&9>=2o$fvP5w5GP6?oe!4;_Q)T- zPS<5JlB}Flzq_O3Ga?u{fdYJ0atI%jY6+z*=wpq`f961V%NnFC^Lw^5pQLG`h#XK` z6e0uEOtM;?yp=Ao3VAFG;m!9ppL`Ash>%3bNs2t(m8Q)qE9)J$esy>eSU^e5wkA=g z5a3#)df+hkp!s+XTS+C+|MY0{w34Bp<6LJm$%3$y6GS8$fxaSHFFsTMO2~1CQ8$Vp zba)JoyGK`OK$DZLa@AMP`$SySdp6z%;L6)H@)T>nG_vnD)-E}Sqs1DLo!lTGT^P

nE88*(+|&vhbOG9DIxJ|67@ z27!=tiN(iGNrD^iWd)aPY7ra&k{^Rj?tERkJ67OQz4lb+o5xO!VeIDC7OP&(vLAbK z_YY39)vD2*7U+iIdUsLeZyLMxWanopKIJwI;`Npxb>)N{8X zAX~++kyLD4p=IJpO#nK*oHgX@FAv}hOCS)FK)y9^3iW|$W`YWJsT~+c*tOxG*DvFf z_?rvxKY(T82Fc2S{|Thihx_=|`wS{Y zf6M=!02?4Kdil1~{}zEHNwQCFCeEveHKTPbYJB0G04k?`12SrplZP?1QG>_-=wG0>dUp~5g!^WZvWeZEvWeUUH0Br?CxjfIfJeQk5-Er> z$qiIVF5@o7GKTeHVL)s|zFvfJ#PNxS$24ttcVq8jpjVldF4&8(yN5vjWzuP$xrlc0 zI2iiM#~rlr6`Ef7y4I5iZ_$Mbf&g&+dr8p9F~jfY?7@KE!o%>^9g?RS&>E+OoAG1e zk3$#!3>o?xWmzn=JOQd#k?WXtREEd-^>)6(?gM{N0G?{<|2}i?#*p??D8DLt84m#m zV=|qbK}QqO9~dhQf=hAs>~L`Kl3|ce?6q|W!jB7W6d>Zt@{<+XUWAx8FLM!K=CGyWU^7jsT z)*5&`RsBUqRo4Q^DvknBz;;mniI!~uKRD(N1LbG|5W&;=*o~a%9rTJs9%Md`c5>8+ z5BOG2-R`DkQv~B85TzXvO5`@1nLY$G)&-S3ow(IWtV^M{un+9M_-umONYqG#YNP}>|+-mYcq8R5r7w()y@T- zbRMU$mnSelhrPxx)iwjBgl8E(KDfOD1GANtm6vMB_Vk;Cg;6qvUJ zfPg@5QBf!280RJ@)AjAPSz{mi&;D70h=VKZ>s83f2Lg4g;kx3ovMQ&#t>N69oWn>; z4es+20@lSp?CIkb9y@XB6gH?6c&q9;T~MnZ!uz&4nu|I6gPjic^1d*p7(twALAV{F z==#U!ZVo~GG9>G@*_kD6Wnxm$Iu8+UqC@B!c4YR^Ks1zaos$@i;isSEYNh7SnlG_x zSK3QG2Cv%zb7e)v&ZqH{=-7pUOo75^YWwFR>|YMY)e3I=)hrdeZ+x>Uy9HdpwUI6m z6>Z$165KO9u1BuR1^0gJ>3&uc9y)q-u*r>CHCJjl%A_-`ALFT*E+f68U5l;_qjwID z`*8$Tp&%w(HO~L04deJA(Kjada6;F=U zS1mWp7%On-4F#zPRZBLkGUOxigs9NZ(qF>rIW4~I-Tqe?sn2;A1yDZfOY-ivyng_T z4Dng&U=#xpy+hS9M)tio{M27K>O*+CI=#mkiz#fAG6)p?@$F;F1@gJvv)cK3K9m#` zn5O%C)9z2q00(zU%gij>B8?vw#76;0Ff2Gme0+R9=DoqkzUM!qu6@jHBO{hEGo>XZ zE_0$@UUe342YY*`7NURE@7*C5`oWcgtfYmP@#5m*N}pLuNIY1tU0GWjFsz2DHYy)+ zpWVmK??1CZKXNo*EmEprzJ6jsMuC*yWa0${ew!4V!=BM!ZmWX5r=Q19P*ax%+pFek zoqU015sQvMSw#NE$D`ovsLh|I{W&JDZg=Za_$=? z=fh6z3%rmHO>F;w(l8_q40spuL!aW&-bIGa!g{u_NZxOJg}N*Z1ry#yK|JzT$pMGd zuz&xTT%u{;ZJi+6iEEcmWWaTB@Je7gcK=~6Qua{=c|yrj7z3)qH5xrj5$BUeWy4+ zu8?xhCbn>)8^_sKK9=VZMvzv*GZ}=QN&sfns@zmG`gaSy^9oB9-dBGE$I@8T6~7GK zMc>XfJD|5IyiaW(?UqP^n#9^$E%Z*`DZrMR$~J}eSG?nTyp~@0`u)ycJz7XEJEH_x zDLqM8f;@pzLTc(8?#}N01*bgs726cR1f-z&W_LArkDmud5K7V~cu^66s8sm(uA0w( zOmFVM-y18ArffKRfZLF!K2&#d1%)z#7srjQtje102L}e^GQ)-c-dI)B$*k3}GY|>7 zaB8@tb?Gma0LM3X=fd=~{|lorhjc)*xW^Y}W)A)uQwBt-0_VdjG!kci@2rFou>|%` zPJ@Qkgyd`=Zv-0e$4-XdM;k)|4$s^ME8iG*T}k2{>~{f1WI^u{|M;hZf^G8HMwah; zqQrlc1u*pWc>BjlV)2FcFb|*4u%$j!wR45WDtuBs#)V}}YsdGo?jSNit1iMjdXvFX zBZlMSOUR;L_?o+;`%CVrTL+E5>%+Sq8ak!&%7hgSA0qvK{@j7gpoQ^1M6nhH+XI6Q zb3Fl%!>FbBuuDvIG~NPk^M8j0f{1=^rM3omQO@mb}JR{Nd8fI(Q9}62j8_&aRNF0{rBL$gZ_ehp%L-v z&Q={1uY`c`VZPIwMTOt9)Z9OIZ5N;2*{8>;@1+X+lKhBMk|0uvUOmJS$J78y1(af|u530=Ny9P{PE4Rh5jsh4)7Y7@|Tyy^P7Wg`{1-M=I~n()hS z7)k}56Mjd1lcCBBo`8SwO6;@EFNv591D>&7n421cre`4c9YbbH7?~P_&eP6G2ob8io=g6gWd@VL#KM z5gTGhqNWGJ&ZV?g;B@XoTDZi=)89|o`yJC&_Iz`rk{h{;C2*l<9%lZvty5}`qgot_iUGmx{ z1|&SrPmy<)9sH%TEKGs|Zxr|BXpmm>nikvLM?pym5f~(8(VQ}1P^r67pOidFS(6w&5welm+`tEyZg(Ul~nfN+{;4-5~>}G8`)^op9r21=^MD4L6zdoy4 zH{dZ&IckkM2Dj#ogyv9B&DK101E0>&n>4fwr&8vP?>&jyQ#w~dh$6)UThUuRiCOOO zx?dh!nIxP;#(qPj3m5`(NZ8H^h(6^=cDe>CL#0Hia;xF-+pocD_)bp~F5+PDTGI5pgmrQ)f)$Tu=4 zRGoRz1#HE6RC-^m-LmZM@1GgjXmP4U6w86+^&OJ@m6IOl)9%L1gQqnV;iALmsD77* zo)!z=ycuYbmVH=$&SSbJ3N=@wRxvOtktoGOrG+iXq%@20sp$taEGAwwJC{_Bh=p_z zt%7>s0G%6OI<;)KR0LUE+fzQMAuX(cm2ir{2>EBJE=c-oEb$v8Rqka31SMG4ih8`t zRPJE~Gf38t8|bOz|A-%lXZS}qLQuPCPl>3T60k{KWI~OD*V{feAVXjWo)3|=>221i zb(yF3YjE;NP{?}p>W`uNdT$G@?8Yvg9}VOo*#{?YdNSu5JBl|l>wLwmkDvb8SF=`< zN7Y+V_@nL9r-aoT9yi@0?lX9(zP#Z9FA1C(Fy3ozF+!!4JtJn>$S2exEKoE&5kGL zrFYKI&H|M-6#snl>v`bqESl+VMJxJ;)ft-4IzZkJA%eyo_U5m^J{k3ZUi(vDjYp7< z8yTCH@zt4g>@ERYApEr?`2AUHp9yL1N=TOs|3buzA$jB?SbHgeNk@y+N;yMZI=1r<3QSQLVabGxV^!pwcppu>V^f=_0mrs zrAqbMZ4}xali`~=>qah-s5Cl!>j9NkNu8>8N*uFTw52*qV$ z*@muK2dKG?vU(ej;ot`$O;bZE!Oiy2fx=$oioR8UKu|9fvk1g%I_fDtXy=VxJFbww z0UNqWqI7RtvL)YAy3mdZT_=;(dxsHtih;CT%sJPOlJUyU@;{hmTUUrDdZFh~qNj=Cg8wbUzVqC)P3excdEj5SS5OmU zc+^OA?hA6Qw>-pkJnU4ohrIJ;=&X&id{L5_b)@+$Bf)gLKywd6ta06GKgwpka~u}Y z7dsSsD>FL}SNOh$xwoVPueS<_ItwnL-;oBtOp(0n3Zjgce&4MFKUx<5nQDwAm>S0; zHV~oxrgXKTHD@OTqV7&maUH0O{8_{8(oLs^aYNb$xo8!5- z#mbI}7BRYtN%JruvNirWKSEZ##N}ka^G&)by>#}K1s%FR34IzSAM8++s!q)+mLMAX#bbi*bb-XGq-YoFLgtXyv?z%j7LH_JPHWVyF`MyivFIt&d$g*;Vp3pP(gU$IJ84;z_@0m-LD#y0%ErzF~TEt8?teOob$1c{U z990)=A^*^7voY>Y(dx=0iCaU>)BnD26IM`-CB3B3t7!Rdy3*6zpAIZMIT2|nwK{BI z=o~7rW}W9YJU2fjTX(iyDu()aS3Fly*4G#LDHoISQ=XL}6Tzl{BQMm`CU^yM z#uY<%RHe)a5PnV;)uwM(sop_n{ytItm_SGx=q-lP)QCpBpmzcoesJ8Xa^@tfSmge5 z+T#LCE}=EOO4Rv}&6t4rH1kDzwA=hKb#tdxIu~68OIS|h4#@zJh&l|`t``Ym_%#GI z$5OYKR{LTH5>B%1yw#bKyNCH)6VZY7VGSB(01@jc|3+MO5-;uU~BUJ2yLQ8QIOPh;?dF|I^=G037A$i`}iFmpESF>q>gPOpJdu(F&rhY@6y` zF5wacCAW(3f_8P5b|Qsxvz!{UnwMtqWoq4$sxlZPWeg zxJ(<*R=D=h%IoTSjPuPvA~#e01GK?u4azJ>CM`qNJ7Sn9tN`Z|(tx(><2%U3=2JG)Z*R3v6IZM~R;ZB=te z6(|~|A4>)lX50(g4<`(~kV}`qBUO;Jy5vBYRh!wmdPF{4{d0Ai5DBl)Gm%Qe>#JT| z-<3SO#_9PFa$?Be^UG^&iwT{^oZIBh;{GeU`)YVX%CoF?5v3WatlZ4zh}FRTJC~9= zq_WLPf|#0@P198B)EIWoNzHsLr55UHy)|>l+4k_xXHko<8z*!HL~UEV&SD0yh$!8< zDccCa4LhDopSdcoI1h(w9S!e_%sopzk`pAx#g7`GGW6G$5$kfodeqA05MLj`bjr+$ zi~P+;&V>%1o$7r2$+eh4{ppyI(AKj~tJRz2>T;jj%%yFoqfsrXrZv$zosu2gW*J>> z=@i{|JuCCTQ`8fY^V}%9cX!F_e)tW0pL@`tQ24PLFNT_@D{+qDlqtO=hZcr!GdBc? zM@@zwNM1blR`V2m#w1K%H>4fEn6|{w zJd0`-BOf^uAW5Cr&3=7_&{vWpjbiHkN9#++Q+3Zy8?m^Mu&c}4o}rZ#qKh+l zyHi4xs;gpbb7IFyl;a)WN>TDNj-Bf1y5sj4JT>P2AXx0m3iHekOe^vHT)k49aTo8d z%4Das(#E{MW!8$7Mu2&-D6S~2B(7-U4cwc|$D*jCH`?iZCLiqTWXI3gWE!~Gc!*Ek zSc_uP^}b3g3$n?~7Ir0dy}|jpi5HUB>m`q#oF45S`8H?lc;iFOoL6e2@l@x>I6MWH zpL6O{9cE_qS1ASO-bUzu!&^NhKN;FFL#%;+y5@U|1QB!dQ&VxSXD53O743R8Z_X0) z?Iej@y>p3Hay)zBP@ZA!O*jm0`gI7+krTOoR4Gc?Sxtk3j&GG=6bPIX7Q-u+;Pe>H|t`9aSfQ-S09z98kLB8h?$$-8J4XnH$dqBkyW- zJ(PO(E05oVBuBnRqC#NBS&mn(#i|_je4KYv`?EWUB40RXO3abehD*Rku+XGR(B*=b zzkFTCv-xjuLwv66lG*OF*Vm#H6BM3yko`3o2(3{MyIU1PHw}l+w+LMXr2O^J4@3re z6yQxQ(MiW#s}kLma=fL#W6U*HhEQH9+M;Q;GS+Du&$UdduDN%fLSkXxH}`OXfgqEa zkNJlBtCzJ|jbtTa#;9%zolM?b(ys}<(pxsdeXUB6Hp`eay$;bNad)DWLvx4)ZFRHs zC^IMxhiRTzl2Mu)+4fii5To*lg6F|z&k4>?oy19aY}C@glDdpv{xY~I*JYOUsz8;g zb$DEy1$B#*$9FkD_hO&lC%$osC#dt(cEvU z%Oer*C%cMzU)yaCzAkotU(ToX;3#al2kb+iq|{Ou*m*LFD79ei9TNKRScJ6aH+u`$ zUDU#NQfd@b$RHR*t)kkV$TXqke^;;|KgGxgH8GMKPehD3#Pnk1Hmb93>l z-I-QB4P&wKzxz)whlzbj?mTr)%ukg=14&y25$PQ?+rRJTrzxXD@R@iGgZ5*y zF@B~`r@(dhSNSc7=yx&8&;WzQnRu*5uB1yp+_nSOmxCGe#p|`SucNo zB+t-9L&~Gm_|1;`->g45IUgv^4n8VNOiWC``|9ZSySVUq!Ju9M4z*98gOg{voK9WG z-yopPqVY?EO-GIe*Rps#hXRlpY={D{JYqTyCW=@9MlQ z3#VG+jPyA zJl0cKRpDqm+zX~-O;BJ#e_AnNO zv;<1tjc>mxUZyVk_)AOc0{Fz5ve51#FEoEGZ~Q~Ex_V^W8D58tv7H#mwcr zaNNOs==i=`G=&sS>&4Ki6FKgNe%AnR;G;5>z<78hdWyCkjiBmV8kWl~`UP78Blx3H z{82?cSZe?V39gVI^ORQz|5ZzCXPg#-MM?z3Xr}VV?Wh%xM$_jchhg9SYABbru8H6a zZapMrHm4h&{P2Wa%_kStCC6ZJ|8`rxbo?o|MKn||prQC%g`?kR;`amUw$S)2oG=1&%2oCMYwyd$vF^Hm5hWUwIVBV! zGDc=0bEe2V6zN7N^E^czN&UMc3 zdY-@T9{2qn_FjAKwb%Nr&&O*9J){7o(J|jfO8@tQzp%Mb2*a?1+K(-6VNkbt2oJ6$VQ#C_!v_y+`d<9v%iOD{N!Rb;S{A`rMza}3 zP@g?FB|u6_uv-9k?e$4YjHP91s~w1ClDlYP(AYw88)}F2*xF%;d{S-g2$aEQW)xPd z8Qfo1R}b-ZTUuD0S1X@74_{WH0?WD~e^my(>)^ji2&F=1XKzo>&X%;=2?+`^OdT7- z&N4F$Ef*8mEG|CN*^Z$h%h<&WdN1!vj07(#{9RbMIZNw~7AVP2F$4$2mWPer;Q2n7 zm?VrKrIp5FK?o7mfLHFwye!;eFwuSWY1H{~+{+YI<_Pfz$p>9oD~t>(x9=}Hi`%hx1*iTIMd z#81Ieuzxqvg+AxuLHpkG!_Jw?iM0BZnt56`0MTGrBV1Y2NsoQ7MVR^eiFcXUky87^ zO*dJOuYZH13zT!4BF-;=lGWLRy8yQdtL@DTU`B*wnh3)MwP_57koLyk_Uf}hdN3}8xZ^q{F_!pN%7AN=bi}cpE`lLP!W=nkP@5d@gzP0C{ZC zBL&Y^8`Hr6zf9u98yXq{hyTWpcxHG4(Uh82>g(&ECFYlJSXgU+fp&LlYr+YTF9R zbrLEWA#Eo51%P>3^sW6}yDA(I5I`E@UgC>r8T^(d^f~q~*D6=;?Bw(jVvq`(v9xwj zUqm+Z7~plR4%<_|g(&}uBW}F0)US3ah0w#iH|JpDeGW8eu!_;^5o4dJf+Fh*PROG|tCy5lceTpyqNf`$sZ{s5arUOc~o zlm-Y5{`Rc8{ZW6FAJ~8HbiyC8-UocYF3sNZ=6oP-ZCwZ@UtX+C#!kX*vrC!rId>6~wjGhqLvO~gt17g&kpk1*xWU=SbBPDD9+yjU1t zQW&g(d)PJb1=@TnA#UhG2s=R*HqjuGigM8!EfP&|}! zCE)V`v7eWte6tW|1}Zhp18=zlNbo7&-A;LpHKRW@EJ>J2F?a6RAy$W2?pu#tp;|da zp?l{m;!s)#MujSXJblCNM_%Hlu#dE01W<36mv|%!HV`7#3ZULI7m9&{qY~s3svztz zaW-q)+oS%~&WF(V9c(=S@9VRNL-~z&iIVd)mRPCb55T?1`WttKgJ(<_-X_gO{QfSM zBJ2P#WdOla!4*3ueJ5jPu%6B262f9BPx&ZQp zUvdJ=8c)GtFuNjq`Zwd6HUx|>uwuz77~DC-ci!S`cyOE!NCU+M;E33-7ZN_t~4rjMy6Xs7*`Z0edAvZPC7} zf&X&R1oMuePU}f}7`n1KQd0F}m_VB}Y(m~2AodQWd^Qh!IIOTmv0e~GC$}`xFZDd} zl!Yn&O?>V8!pPGQgmj4)mUH&uNz1%WnuZtzkmM_9S-c^c`}wq zyEn*<*e3QG`F_F6{))A|0_m6#9R36XR)_EUx7cJ1j4DC%;t#M|h>{mgyPbiXU@czY z*d7YW3j&UCPul`O(Gwj1QeQ0o{>iJ{z_Aa+it8}a5ip13l&8_rwCmvAZ_#Z5$`hJi zMR6m89l*Bv0|2Z`)@|g+#FNC~p(wIGhB`(Ropkn|0y+0IXzfcM9esK74b<3>(^|m*ai%Oh6dt+IQUQ1584k`NNgQu=FiCYiHv}h@Jy#cWTaW`9 z)^Y1?JS-%eNyO`*BHmOh03v7Jc<@l#NjGJNHR^4?6J)?)|Ihep3dT`!;(g}w`8ut( zJrv$gI0bI-PdtW5iw+)&Sbg?GRg>0}S9Ny4Xr~58AGLdnH;|3R!#F}^$j*a+s(B;- zFgt{PyAYG?j_Uz3ndmG$l=tIP&YINC|4i;bllwota*KwX3bc#sMW7i9aAAG6sOhBp z#*e8af}1qWmnrWf&c7jaTIaxp!=3jtf}53#zuFvYz8w+x9%4V{0#KN_HlYRKav-34 zqVoo|f{OrbYF?wzrPsfG8_*zjfOg9qf(qCb#}=-Jm*`*ffOGqr_V6uWovMDkA=U@r zBof~^p8)a{Yqp5AZq5_176WO88wekS6^#S|4+lsQ&a+!PH>b5nmy^EZja)WWK(u1Q z;KrOTroM;ZGDMt>KLN4Tv*8~;e7K^#dd_klQLHC!_GBVvfN7*YS`NFIW_}Y$I*bAm ztG1$BNhsgkA9S$Gbj$8&w@UOx)vPc&0-w4d!Am%nIRYh61z^UuC~W!)^eyJL0ajz> zJ^;=A=R{I~P>Hj@xA#`^67cOXN?p|nCq-U(1KKpkszghmj|-2|pHa#kKz_CWnws{n zTlouq{H8tyDb=>wy?(9(x9b0fUAtAqVDeMUlC@|5s?w`S5IAa zmpx8rV1YL*=jc z(`d0=JhR$YUen8xH@PV4$3PH_IQM=0Ox;1$Txc}325~6NKR#b@7n&~$AR&jieC7dJ z;x-E0^o^RE7jKybwfqgq7 zI7#OSI>Cb#z%kC5_B_%z3p!aU5COfrI+9Hu^6E%;FljBaL$o z0y(qyCV2Ph+wBpsTS19v$O)uu;@4irypgqG#!Q}~BM4?M|J>!{cSK|@vtf`-l=n

}xL>(w_;4sr#JQO!sK5kwv0u`cA-gssu%;5(ad+1jZr*D^jD+r-Q{ z!%*GgjUGDvu<6fUOlTf;L72DxkYphhw`^$bj># zstQyhh7fAawVy@1C0Up==$w?fTyb!lj#Ww4Ms4G3xDDdJ7=ED?2_65C!sbU=kb5>p-Lr z>p&f^HQ7;hIi|BSkz8I{D8X$>&h@O)S4AF49UKWBW$qpclJo%Ov-HzXgy!$+Fsk5M z0lMaScU0{zZvu&ii&ZKALHBg}=>%7BHHSR`%=qz8#G5pU_)YzQ_bf|0Jja{0+NyY< zRm}Te{g2{FctxL`ST7+uv$gqPi2%G{3e8L4g)Pg>Ak9Z{^CyCDY{{72*xza}#RJ58 z6(W3@3oK1%7w~F2HjEpVBGqcF}-XKZwa z6N5Jqiq+ZvcgN~k{XrA3dFC@~R<=0CL?(i&a+!(qRA+DR*m!%QHL>VByUA_FIfeA} zkEJ&a^N+IcE8H##)7Y*T+(i_ETPoGl8iglTQ@?G0*6OEOsO`643odteLtV!|00gp> z-C7EXt-E-um5b{BT{Vh2K0?~#LWg${e^73_*`s@w3AqsqWOyhDI7Ty{0ZeUTxpgVk z*+?WqxBgXM15({WV8q~xfx9;PrQZOU9EK3VBdaf^(J zL#Z`p(q|unGS{xSygzU7)dM!3N<;_Bm(cs5SZ8bwF%liNDs|3_Z0j4Asl41DdS}Pw zuWbzpyeoze-UTw#!5z8ojH|^9v$eZBYw;i#)(I$D*{<1c)D0Y9u)>f@lD-`(e_Pq^ zYO53yvYO{f+z7O+$#qXHsGMfCur77b2Yw)j!>6E|R4G})eFJuAE6WqKv<|luoHODT zJPSoK5$}&hz~HBm5$k>(@V7IQlXsE!m*r2_osEf428z8q2CljqGv#Zkn?D=Gtmu~| zPj80kD{r$NRTV7PD=~ZisurN!6@Z+7Z?^mu;XVfip;)+G6{kSszKlQO`bIZBslIaTi6{@H(yolRF(T@i zzWsHLiMuPAVj6Y|TWPbe3mx zY25r6huDCJa(*7jHcRPBjb7t90!OP3mEfrVna>yd^}ZplSjUdb2L=wmY)gI)Sjiwm za$3|px$kp-^}Kyq|I?Z(kZY@gZLy`s79`Pi{RaW>eCMt9+@lrL$-J|DYQgDYYslmx zzoq{cKpBVdqJ4DJ9fsVNTjwD{a)1^Nit&=Ef`WXn%oC6*k0;9Q(JDH71#_}l;AE$Uh zb^)-9Ot}$!^nJBa39hj5x)@Yx1OO^-z_eDd+$VA~GsyQMh3F6Oqgr%NHv}f)isx;K zta94B(iPbKcfKC=d@h!&-0RkLp;|Pw=;ZgK?ATuGdN`e*IB@(l%0e))?oCDQW^bXv z`lYCyyn$dun`QNpJSIaidX%^O1x(5yTQHuVPoPiu6GSy_%1L&F_kIeE?NG_grEh^e zpMhEWMlq?&g0%OS_yxC2lxUgLkAq;k-Y3A6TmEP((e_eQtsMsh$?SKS#aa9@(iv)1 zAP5wjp`CVC9CO{QxRtNis|46m3tV|s=oJzxU?TGii8Nirkz#MO9?evz0Z6&--YKp5qaeMamnqy4-76Q|ow#MATz2YrD zr>1pHmke`}l-bd*1mV7`$1fIgiI(asrSqsZx@wKh4Z~r;a=(133SD<#7$0W>)DwFZ zd34Q#+cY)>{1gc~QAtrzed%_qR`yOr!BYoCWfQi{>ivmW_63Lm{M1+mp<;BINb!69 zwPe}i!<8Go$#;{ZNkv#QXtulFGpu0Ztolm*eQCE9YEoleT1{D`Slp(7nXx1ADTUf4 zAU~Zo)qiLj$lIM_F%?53VE)kR=fx_BJAjC)ZYM8x`>Y-FC3W+D@Ex}=pFb#--d{-3 zMOHwj&bzO;ktsG1Q*7!;=3&R&VU{76sFB37>)Kj_p~zONy%@VHw!17ggC}zw!&*(M zvB`YXM&FOw@2rSUcC_HBN9DpA_fbst=BC+p)9H!azSF&o0_7CJDu?ZMzIUAQ6O8)& zv_0Y>^G0?ekw(l=&i9O>=<#PCw3KW;L{Zh!7wF)LvUN(xaq)eyzSQ&r1pnVB;1=ni z{$n;Y`_R;k_tSELV|DXzZ)Z!WsjwaM%gJ^k#u)0(Lm{SjR^m}%E7jg5QFh^snW>zo zBeaYRC_XjyBWokeXtMrgKM#bCoGRJ+MBFan&!NI%T4^`w&Xhl&c{lxTCiU^wpO~^M z#TR0zT?d*COoLBZ0smy50fCgr{%0lHufIYQL{T-(3St6-b>VJ}WM4x}gJB$P%bgetHuu5k(u-{mpK3 zmwahO9{|kEENQb4>aO|6(~7a}y2cwCL~Q0?VIk`w<0TjV-ZwnF^9lkejKS7mzaV={ zQoBVmVXFt?JLcQcFk_OZSFOnj7;aF@>P&TNoK@v^S|$7n%~`)*Ih>=HUkEd^zv8Dj zFnvssK-!OM8?+dE=D*vTrw+0n6<|@!-PvqGjXjHkz&N#YO#&#V=x#$IERNiF9CB!S zc%%mbo4IVpClTyFbxer;NHR{uNg_O#L(0WmE*x^iB$FLEj?kfSbp=G`Op@h^874c} z9$d%iNshH+cL_P^vZ8=1-rv)r@q$&)OoMeUS3O@Jv5DR(A|l{DMPv;+h!b;*;-zOQ z%4n;*1q`6EXs}AjywJz{h=^LIQMPtM*L&)o1MEMOLat9OklV1X>J}@zoK>?DBq(B#)3wgMy{{WcMBIrAs5_HLL7fac#+t`D7S744*u))1-cB7vJ;J%H>&T1S>mnHuuMr9wZ#D3snm-efn+-tF2mbIwuMGooIB#F22=jeAuutx@mp5 z8eDfRB;sm^^`JYQh>0_l-+SSq-4FkPFeT3KV*KwtM^fxZi~iCLfGHuKQub?E3Qht7e>wba3pnYiA~ z`~}R|WxtfpBc-a|QkMM_7rXf*pNeLl$TlRh?0sTB*SB6O?pu?)BMOa94i|#Xwccc0 zW#5h~pub&O`f4Y2Bd~l}J8*2#AhsQ)k)0u89?NGa0dmC@NVM|g*icmR1dd5WCALA* z=WpDL2{WV?p%HK}7sq(uJU=BIWB%Z-xj}sQ8@0nWX4@7o@RsyGR4`r|c~`0<7KxJD zP$~6x8Frc$$t)TkYE0{n<0Zwd-c4JNpSJI@W$Vvv*Q11QWiiz$k{P4#e^5y>;7_<6 zZyz~VoEB@9&z)KASEA_jB1Z3leu0y@WkGIwO{uw2spJ;@5HMsbHCI?1d%Q@iqpiGk zX{u_$x!v^+V;u^h=3x`>tUPSp;94M`*iEabqoJbk;U zsIh@ny5ye7O!RQ+iz;y1aS0l$w2KTtH7jFObw@QZ@dtRTFBjFbIt=7>%ykv4EH3i1 z2kW|-Y5-Lhtw0P7T<0!b1!j& z)|i=Lh|*EYOgBZ()nLkVaLWQx6sI|qTSYa4i}Z@?ZLuJWdRS;D?%N2$ASS8Ttp{Z$ zbcljtMBJKwS5Ubr*NRqR(h5X#IU#a@Sfp0ImYqQuUcTvtn==vI)m+gnr2ia@*q&ARyrL;MybQ_8Oz_C}g4 z8?X+lwE+FQK-4ufxpBv4VGRNR|7 z;Qu!cu0b^4YnTzu+D#2?o2A4ftFF~`NZIB1r4kTRXg%+sm5kZw2&#P($~USJGC4C- z9Abmc9#!*|&9tw(C&&(=X2qAz%i{y%wJ!Kqc~qb&uyg)#S6puL*s z`S7HTzu)QB?~D)E47!}{MdLNMH7~`9dDd8xL)M~w|6ejVsE!p-w zCuUsgwJg0ny%PPmLeK5oPta@^H+)~smCSra-!BlRs#RpALsCO5Qh}iNC&|GtHEASFNjFtfyh83hI53Rzp-&>ZH!qs+_h^6qjL%MOFO(n9dS=+bS$W z6}%d2;RYhIO6OXTFpw55$nb%tuGW{#J`o-%s^hC>3!a+H1Xt)3|m$y ztQ53X46UQWn+JH;o&8x)$p!|C_F&?al3xX_X2h zHwvpf7pyqlDcZ{2C?DXE)yK_#*lrA0nO@;CuUShG;tnImsd)jo`{gQ z$-!k^I{1g#qm{)6%=*{PO2x7fPoP*nJbbk)nK~XK;x^MqC9*zNKc1dB zC>e8=W$3gVL|z%p(T}=r9!5mBB5j(qpG!W!EJ&BxS2~(_eX9E=Dbb0>WCB*Qf)$T& z(-$6V#fjQZ-7Sjg?b*RD!3xEKf+b(x0`*NV^y3`PSzXi}*^DIMT6Gp~jV)H@9(kz~9Vs9&DHp6^NL9z3Q~ zp16!+i+?Xy!`9jJ?Jf~x;~Ce8$0@h!FIuF_yy>a`-Z3Hu%A{*q`h&eAyh17IFE8e% zM~tmF85kQIzvn7~1EYOE%Wj})4U$iHO?96Sm7y(?`9?O}b@J(y@rFI``HZp*%li~V zDng)!)XYWsLbf2f#j6!WU1;rB+vFFDmUy4U8);#E+#w`~dJ#V3{HW%MD3}Ixh97 zjd>viOogBIfc4e}GObO{-WaG+`?5o@^>HpyxaUyT*U`k^|=+e}y{} znxOhRP$~@o5@BGk#Nb8@c%+5R4hV(_^`JIiM=ZPCj|S)v3=j^FY9IjL0u44zf^ zAHc1DalqsPppjk^a*~oOW@Dh<(hz$TdMH{^J z9?^^UbAp$PhpIeEb_8(ixl<8rn|Z^UVM~hZBpRMsG>vVMJX?U6*jgPo2eYuoHBDL4 zKdl47i#Lj?op8-|1mU;$(Ql2I9&@3waXR*;zMGhba*?i1Yt5^L5j;E<{LY7k{O}bV zZxVFRBJ+}pFdXA(EE+r|G6esQvll(yI z;$RCz;V_E;KLIi>LM%QWP$#$io3pC{CQ)r;3tP$u1g_o_KqV=I0l(27+$VMp%!U`4 z4dd=rlSJkR=ok=!@CpM1Hl~2=xV3jST4D#&shN?SfycfJCby4+uGJ9zmh(S1InCon zY&;i`iT<7#FG{##HK*;dqj0~YAM~Jt;d2aD#xu1}6)+oBFdLFw>DcJs3*v7EW{B*4 zvku)8DAfeB*<(x;05cwYCC&=~j5ic8{h;Z)If2Y(KMKab*FKsheSa6(`wIk`5z=j& z1q8}$e{3$8Kn=mON532T@h^bMeIZEOn1SCOLcjgbOm18r-lU!Y#u6Qm(8T;;s0+%Z zG`%nitA7?ZS^>NRCoENhYpf@rEF)*cFBt-AXlvnqH`ZeS z@EC)N3M#;?>%m*^X`B-3!t^8RxSQ@$Wm%m zN{FE)DtC~R^dgIB4FX5eg&e>&m0MCF7({PKhDt$kI^EY`M(K1QCMiizH%3Es0hWIS z8*vb}UqcpJVScmSz?K8sPfQ-&2o{HyRR#5qdB!AM^qXWzJo&Nm$D_`PQCMy7wKZK3 z4DP!IJbZss#|5|vj`ux)Q`KL6e*oKTU=FmUIk*g&$O1{Spb(o=v!jO^l9-XW;a-cU zm$)xXFkygv6#Dr|__>YV$rx;V1g9tx%!vEV(`O0yFTMAj4LMRgdjv8&mZ*TB-~{lQ z{Q0CE!lU_Lg>r%Jl_{j42#!kP_acV-|Czof$%33gjww*U!B51K4B!D=8g2E_=V$Qw zW?#Gky09u#$5L3&#UuY>19+___eU5yxUkp16Q=6Ge41xBR>1Q^EfR2~ImAMgXq1Rx zELrbNO(AI&l3?&gzdPpW@ZrG-HBy4GuaXKsr=r7t{69&Ie)?l&7!J;{Q*u(5)&HB) zwnLT43WH&OYHN$M4PXMAz{~>=V(sz^csrsRUpx4}B!tjMK(1baL81DeY_@fo02RpO z>&LE&eQ( z#n|kOb*J3q9e2X zy=RM0-61Ph2|SElBZd$p8g}U+*&37r#gcO?I{kJK1h6f?-6ML&ctaiF(;Vo+%_BGq zB5d)QCl11H6NDhTL@phVVMI<>-UxJHvqjqT((;)%=eB(%>?(#0v0TXtn=r z(xC5&sNkkR;^R92fd3*Iq$|bYAINU>*~D8tyodV#L%4k~RIA_1U8{je}cWIQYyjwukpk z&Xm`f9Ka=%O2lDe-8^xO9-j_EK=|9fGXcGLD?t%@_{6w`PO>$XtrH{Iw7a^xyXQJnW%@v*mxkXeE{KYY zL7`+}xVl)_dFdl0UGrr_8RSC07`F1+tse?WmS#~gFxb@cT&LS`*R`3dAGIo`XHM8+wBQf ztpIbdoa{)BPU!yX~RbalB?>C|PJf{vzh6 zvEze$YbvhALxkcucybJ0kL1G!zZ2mSo_!$x@rpx}RJ@4m+O>1jU*A4l103c@(6L2& zzkL~3X_^zoYLTRZ!|cbef$$9cagD%19<>eWOd#3h$<;2-0K$iX$CMma*2AyvHQ=wV zKRu_M2ZPkc)~MLX<_;y7z9M581qFp-RPSD$>5mT$W04Q~))1^4Oe=C{eST;Sl+3h1 zb9TrPHP9^^835Y% z&qbm`L4wf2e~vU_Q^-e$kXc&)$8QkO>9>A7p5;4Prww$;yTS>DW_`&2G;jn8&m zaos(dBY&vWB1)ssUhs(M1*5zd>V z)X3L_N#T%2oGoo6$0LzFSIi$;`A`D_3&q4*Z)4Rn7y3JzvK?Ug8BKU zE%NnB;emmb3hjwvj`1g{&)MJGIZe9y<@LVii1SfZ5ADX9$Q>Kx8>V4SOh!C+tA$vz zzPW8~FMeyNiZY&bv~yG#Iuis7s&YXwntLo@17K-`z;+(5dQ#Nw<;;tzkBReIjcij- z)S+Kk@l-MRp20_?=k83j#m@^xPxQX-S}<|+&9`>=c0&XiPZ~0wtM`_W@mvPu5xSMa zeLB6HnA}9xPFTqlvpD*`tu9W;v2D7)a6$1ZOs)RI+rW3-rarZsc{O(2U?+)x%W|-U z1#HJL-R`8av}^-Grc(U$p_qBKDT9ccoeR^VkvBV=*;ig1tS%V^)KA;&r0q~@7uVau z)|n3}eR0ChU5_c)&EGyaY`b4&y~d7G@suDnQ*ROhw3v?X#Z+zS|<)`hF z1otm{0#{MJi|m{sp*Y;NLA1wb3Vf38i%!PwJbxf)iXiWeBJUgAn&cb9m!!h9DOSK#5T=! z-~zh4JU3Ws%cQ_0zn)HZUop7;-6K**&G`^{(0OsxPpB(1O337- zOMJzXdb3w9t4*RFza|xBsU(?XILbRzDCIXY9|%fwD?=gi*J$QS^bYP92|ptDHSO!m z(>iA1%&Mcs3l;|Tip$i?ZRVm?0=t?pP*&K1`r(Je)I`bHh+R?p@yG@|tPiZL>KeYd zkAt70j6+K99^_AgR+|z0%{%nHIqqh?`~@Gd1>v(^kCs%_)qHU9>BK|l)@j8JYgZYC zV8AzFr|s` - -``` -$ aws iam attach-role-policy --role-name masters.test-cluster-kops.cluster.k8s.local --policy-arn arn:aws:iam::0123456789:policy/rolling-upgrade-policy -``` - -### Install the rolling-upgrade-controller - -* Install the CRD using: `$ kubectl apply -f https://raw.githubusercontent.com/keikoproj/upgrade-manager/master/config/crd/bases/upgrademgr.keikoproj.io_rollingupgrades.yaml` -* Install the controller using: -`$ kubectl create -f https://raw.githubusercontent.com/keikoproj/upgrade-manager/master/deploy/rolling-upgrade-controller-deploy.yaml` -* Ensure that the rolling-upgrade-controller deployment is running. - -## Actually perform rolling-upgrade of one InstanceGroup - -### Update the nodes AutoScalingGroup - -* Update the `nodes` instance-group so that it needs a rolling-upgrade. The following command will open the specification for the nodes instance-group in an editor. Change the instance type from `c4.large` to `r4.large`. -`$ KOPS_STATE_STORE=s3://my-bucket-name kops edit ig nodes` - -* The run kops upgrade to make these changes persist and have kops modify the ASG's launch configuration -`$ KOPS_STATE_STORE=s3://my-bucket-name kops update cluster --yes` - -### Create the RollingUpgrade custom-resource (CR) that will actually do the rolling-upgrade. - -* Run the following script: - -``` bash -#!/bin/bash - -set -ex - -cat << EOF > /tmp/crd.yaml -apiVersion: upgrademgr.keikoproj.io/v1alpha1 -kind: RollingUpgrade -metadata: - generateName: rollingupgrade-sample- -spec: - asgName: nodes.test-cluster-kops.cluster.k8s.local - nodeIntervalSeconds: 300 - preDrain: - script: | - kubectl get pods --all-namespaces --field-selector spec.nodeName=${INSTANCE_NAME} - postDrain: - script: | - echo "Pods at PostDrain:" - kubectl get pods --all-namespaces --field-selector spec.nodeName=${INSTANCE_NAME} - waitSeconds: 90 - postWaitScript: | - echo "Pods at postWait:" - kubectl get pods --all-namespaces --field-selector spec.nodeName=${INSTANCE_NAME} - postTerminate: - script: | - echo "PostTerminate:" - kubectl get pods --all-namespaces -EOF - -kubectl create -f /tmp/crd.yaml -``` - -### Ensure nodes are getting updated - -* As soon as the above CR is submitted, the rolling-upgrade-controller will pick it up and start the rolling-upgrade process. -* There are multiple ways to ensure that rolling-upgrades are actually happening. - * Watch the AWS console. Existing nodes should be seen getting Terminated. New nodes coming up should be of type r4.large. - * Run `kubectl get nodes`. Some node will either have SchedulingDisabled or it could be terminated and new node should be seen coming up. - * Check the status in the actual CR. It has details of how many nodes are going to be upgraded and how many have been completed. `$ kubectl get rollingupgrade -o yaml` -* Checks the logs of the rolling-upgrade-controller for minute details of how the CR is being processed. - -## Deleting the cluster - -* Before deleting the cluster, the policy that was created explicitly will have to be deleted. - -``` -$ aws iam detach-role-policy --role-name masters.test-cluster-kops.cluster.k8s.local --policy-arn arn:aws:iam::0123456789:policy/rolling-upgrade-policy -$ aws iam delete-policy --policy-arn arn:aws:iam::0123456789:policy/rolling-upgrade-policy -``` - -* Now delete the cluster - -`$ KOPS_STATE_STORE=s3://my-bucket-name kops delete cluster test-cluster-kops.cluster.k8s.local --yes` diff --git a/examples/basic.yaml b/examples/basic.yaml deleted file mode 100644 index 93e96117..00000000 --- a/examples/basic.yaml +++ /dev/null @@ -1,9 +0,0 @@ -apiVersion: upgrademgr.keikoproj.io/v1alpha1 -kind: RollingUpgrade -metadata: - generateName: rollingupgrade-sample- -spec: - asgName: my-asg - nodeIntervalSeconds: 300 - postDrain: - waitSeconds: 90 diff --git a/examples/pre_post_drain.yaml b/examples/pre_post_drain.yaml deleted file mode 100644 index 6ecdc1d5..00000000 --- a/examples/pre_post_drain.yaml +++ /dev/null @@ -1,22 +0,0 @@ -apiVersion: upgrademgr.keikoproj.io/v1alpha1 -kind: RollingUpgrade -metadata: - generateName: rollingupgrade-sample- -spec: - asgName: my-asg-1 - nodeIntervalSeconds: 300 - preDrain: - script: | - kubectl get pods --all-namespaces --field-selector spec.nodeName=${INSTANCE_NAME} - postDrain: - script: | - echo "Pods at PostDrain:" - kubectl get pods --all-namespaces --field-selector spec.nodeName=${INSTANCE_NAME} - waitSeconds: 90 - postWaitScript: | - echo "Pods at postWait:" - kubectl get pods --all-namespaces --field-selector spec.nodeName=${INSTANCE_NAME} - postTerminate: - script: | - echo "PostTerminate:" - kubectl get pods --all-namespaces diff --git a/examples/random_update_strategy.yaml b/examples/random_update_strategy.yaml deleted file mode 100644 index 7b80af9e..00000000 --- a/examples/random_update_strategy.yaml +++ /dev/null @@ -1,26 +0,0 @@ -apiVersion: upgrademgr.keikoproj.io/v1alpha1 -kind: RollingUpgrade -metadata: - generateName: rollingupgrade-sample- -spec: - asgName: my-asg-1 - nodeIntervalSeconds: 300 - preDrain: - script: | - kubectl get pods --all-namespaces --field-selector spec.nodeName=${INSTANCE_NAME} - postDrain: - script: | - echo "Pods at PostDrain:" - kubectl get pods --all-namespaces --field-selector spec.nodeName=${INSTANCE_NAME} - waitSeconds: 90 - postWaitScript: | - echo "Pods at postWait:" - kubectl get pods --all-namespaces --field-selector spec.nodeName=${INSTANCE_NAME} - postTerminate: - script: | - echo "PostTerminate:" - kubectl get pods --all-namespaces - strategy: - type: "randomUpdate" - maxUnavailable: "100%" - drainTimeout: 120 diff --git a/examples/uniform_across_az_update_strategy.yaml b/examples/uniform_across_az_update_strategy.yaml deleted file mode 100644 index 04f9d2f0..00000000 --- a/examples/uniform_across_az_update_strategy.yaml +++ /dev/null @@ -1,26 +0,0 @@ -apiVersion: upgrademgr.keikoproj.io/v1alpha1 -kind: RollingUpgrade -metadata: - generateName: rollingupgrade-sample- -spec: - asgName: my-asg-1 - nodeIntervalSeconds: 300 - preDrain: - script: | - kubectl get pods --all-namespaces --field-selector spec.nodeName=${INSTANCE_NAME} - postDrain: - script: | - echo "Pods at PostDrain:" - kubectl get pods --all-namespaces --field-selector spec.nodeName=${INSTANCE_NAME} - waitSeconds: 90 - postWaitScript: | - echo "Pods at postWait:" - kubectl get pods --all-namespaces --field-selector spec.nodeName=${INSTANCE_NAME} - postTerminate: - script: | - echo "PostTerminate:" - kubectl get pods --all-namespaces - strategy: - type: "uniformAcrossAzUpdate" - maxUnavailable: "25%" - drainTimeout: 120 diff --git a/go.mod b/go.mod deleted file mode 100644 index 34780d7e..00000000 --- a/go.mod +++ /dev/null @@ -1,24 +0,0 @@ -module github.com/keikoproj/upgrade-manager - -go 1.15 - -require ( - github.com/aws/aws-sdk-go v1.36.11 - github.com/cucumber/godog v0.10.0 - github.com/go-logr/logr v0.1.0 - github.com/keikoproj/aws-sdk-go-cache v0.0.0-20201118182730-f6f418a4e2df - github.com/keikoproj/inverse-exp-backoff v0.0.0-20201007213207-e4a3ac0f74ab - github.com/keikoproj/kubedog v0.0.1 - github.com/onsi/ginkgo v1.14.2 - github.com/onsi/gomega v1.10.4 - github.com/sirupsen/logrus v1.6.0 - go.uber.org/zap v1.16.0 - golang.org/x/net v0.0.0-20201216054612-986b41b23924 - gopkg.in/yaml.v2 v2.4.0 - k8s.io/api v0.17.15 - k8s.io/apiextensions-apiserver v0.17.15 // indirect - k8s.io/apimachinery v0.17.15 - k8s.io/client-go v0.17.15 - k8s.io/utils v0.0.0-20191218082557-f07c713de883 // indirect - sigs.k8s.io/controller-runtime v0.4.0 -) diff --git a/go.sum b/go.sum deleted file mode 100644 index 5dd67801..00000000 --- a/go.sum +++ /dev/null @@ -1,695 +0,0 @@ -cloud.google.com/go v0.26.0 h1:e0WKqKTd5BnrG8aKH3J3h+QvEIQtSUcf2n5UZ5ZgLtQ= -cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= -cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= -cloud.google.com/go v0.38.0 h1:ROfEUZz+Gh5pa62DJWXSaonyu3StP6EA6lPEXPI6mCo= -cloud.google.com/go v0.38.0/go.mod h1:990N+gfupTy94rShfmMCWGDn0LpTmnzTp2qbd1dvSRU= -github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78/go.mod h1:LmzpDX56iTiv29bbRTIsUNlaFfuhWRQBWjQdVyAevI8= -github.com/Azure/go-autorest/autorest v0.9.0/go.mod h1:xyHB1BMZT0cuDHU7I0+g046+BFDTQ8rEZB0s4Yfa6bI= -github.com/Azure/go-autorest/autorest/adal v0.5.0/go.mod h1:8Z9fGy2MpX0PvDjB1pEgQTmVqjGhiHBW7RJJEciWzS0= -github.com/Azure/go-autorest/autorest/date v0.1.0/go.mod h1:plvfp3oPSKwf2DNjlBjWF/7vwR+cUD/ELuzDCXwHUVA= -github.com/Azure/go-autorest/autorest/mocks v0.1.0/go.mod h1:OTyCOPRA2IgIlWxVYxBee2F5Gr4kF2zd2J5cFRaIDN0= -github.com/Azure/go-autorest/autorest/mocks v0.2.0/go.mod h1:OTyCOPRA2IgIlWxVYxBee2F5Gr4kF2zd2J5cFRaIDN0= -github.com/Azure/go-autorest/logger v0.1.0/go.mod h1:oExouG+K6PryycPJfVSxi/koC6LSNgds39diKLz7Vrc= -github.com/Azure/go-autorest/tracing v0.5.0/go.mod h1:r/s2XiOKccPW3HrqB+W0TQzfbtp2fGCgRFtBroKn4Dk= -github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ= -github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= -github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo= -github.com/NYTimes/gziphandler v0.0.0-20170623195520-56545f4a5d46/go.mod h1:3wb06e3pkSAbeQ52E9H9iFoQsEEwGN64994WTCIhntQ= -github.com/PuerkitoBio/purell v1.0.0/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0= -github.com/PuerkitoBio/purell v1.1.0/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0= -github.com/PuerkitoBio/purell v1.1.1/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0= -github.com/PuerkitoBio/urlesc v0.0.0-20160726150825-5bd2802263f2/go.mod h1:uGdkoq3SwY9Y+13GIhn11/XLaGBb4BfwItxLd5jeuXE= -github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578/go.mod h1:uGdkoq3SwY9Y+13GIhn11/XLaGBb4BfwItxLd5jeuXE= -github.com/agnivade/levenshtein v1.0.1/go.mod h1:CURSv5d9Uaml+FovSIICkLbAUZ9S4RqaHDIsdSBg7lM= -github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc= -github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc= -github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0= -github.com/alecthomas/units v0.0.0-20190717042225-c3de453c63f4/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0= -github.com/andreyvit/diff v0.0.0-20170406064948-c7f18ee00883/go.mod h1:rCTlJbsFo29Kk6CurOXKm700vrz8f0KW0JNfpkRJY/8= -github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5doyWs3UAsr3K4I6qtAmlQcZDesFNEHPZAzj8= -github.com/asaskevich/govalidator v0.0.0-20180720115003-f9ffefc3facf/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY= -github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY= -github.com/aslakhellesoy/gox v1.0.100/go.mod h1:AJl542QsKKG96COVsv0N74HHzVQgDIQPceVUh1aeU2M= -github.com/aws/aws-sdk-go v1.33.8 h1:2/sOfb9oPHTRZ0lxinoaTPDcYwNa1H/SpKP4nVRBwmg= -github.com/aws/aws-sdk-go v1.33.8/go.mod h1:5zCpMtNQVjRREroY7sYe8lOMRSxkhG6MZveU8YkpAk0= -github.com/aws/aws-sdk-go v1.35.7/go.mod h1:tlPOdRjfxPBpNIwqDj61rmsnA85v9jc0Ps9+muhnW+k= -github.com/aws/aws-sdk-go v1.36.11 h1:6lVRjsmRpQwq58+YHBbBe7BZuY3l6onDBLN4twOXT7U= -github.com/aws/aws-sdk-go v1.36.11/go.mod h1:hcU610XS61/+aQV88ixoOzUoG7v3b31pl2zKMmprdro= -github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973 h1:xJ4a3vCFaGF/jqvzLMYoU8P317H5OQ+Via4RmuPwCS0= -github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q= -github.com/beorn7/perks v1.0.0 h1:HWo1m869IqiPhD389kmkxeTalrjNbbJTC8LXupb+sl0= -github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8= -github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM= -github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw= -github.com/bgentry/speakeasy v0.1.0/go.mod h1:+zsyZBPWlz7T6j88CTgSN5bM796AkVf0kBD4zp0CCIs= -github.com/blang/semver v3.5.0+incompatible/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnwebNt5EWlYSAyrTnjyyk= -github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw= -github.com/cockroachdb/datadriven v0.0.0-20190809214429-80d97fb3cbaa/go.mod h1:zn76sxSg3SzpJ0PPJaLDCu+Bu0Lg3sKTORVIj19EIF8= -github.com/coreos/bbolt v1.3.1-coreos.6/go.mod h1:iRUV2dpdMOn7Bo10OQBFzIJO9kkE559Wcmn+qkEiiKk= -github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE= -github.com/coreos/etcd v3.3.15+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE= -github.com/coreos/go-etcd v2.0.0+incompatible/go.mod h1:Jez6KQU2B/sWsbdaef3ED8NzMklzPG4d5KIOhIy30Tk= -github.com/coreos/go-oidc v2.1.0+incompatible/go.mod h1:CgnwVTmzoESiwO9qyAFEMiHoZ1nMCKZlZ9V6mm3/LKc= -github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk= -github.com/coreos/go-semver v0.3.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk= -github.com/coreos/go-systemd v0.0.0-20180511133405-39ca1b05acc7/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4= -github.com/coreos/go-systemd v0.0.0-20190321100706-95778dfbb74e/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4= -github.com/coreos/pkg v0.0.0-20160727233714-3ac0863d7acf/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA= -github.com/coreos/pkg v0.0.0-20180108230652-97fdf19511ea/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA= -github.com/cpuguy83/go-md2man v1.0.10/go.mod h1:SmD6nW6nTyfqj6ABTjUi3V3JVMnlJmwcJI5acqYI6dE= -github.com/creack/pty v1.1.7/go.mod h1:lj5s0c3V2DBrqTV7llrYr5NG6My20zk30Fl46Y7DoTY= -github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= -github.com/cucumber/gherkin-go/v11 v11.0.0 h1:cwVwN1Qn2VRSfHZNLEh5x00tPBmZcjATBWDpxsR5Xug= -github.com/cucumber/gherkin-go/v11 v11.0.0/go.mod h1:CX33k2XU2qog4e+TFjOValoq6mIUq0DmVccZs238R9w= -github.com/cucumber/godog v0.10.0 h1:W01u1+o8bRpgqJRLrclN3iAanU1jAao+TwOMoSV9g1Y= -github.com/cucumber/godog v0.10.0/go.mod h1:0Q+MOUg8Z9AhzLV+nNMbThQ2x1b17yYwGyahApTLjJA= -github.com/cucumber/messages-go/v10 v10.0.1/go.mod h1:kA5T38CBlBbYLU12TIrJ4fk4wSkVVOgyh7Enyy8WnSg= -github.com/cucumber/messages-go/v10 v10.0.3 h1:m/9SD/K/A15WP7i1aemIv7cwvUw+viS51Ui5HBw1cdE= -github.com/cucumber/messages-go/v10 v10.0.3/go.mod h1:9jMZ2Y8ZxjLY6TG2+x344nt5rXstVVDYSdS5ySfI1WY= -github.com/davecgh/go-spew v0.0.0-20151105211317-5215b55f46b2/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= -github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= -github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= -github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= -github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ= -github.com/docker/docker v0.7.3-0.20190327010347-be7ac8be2ae0/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk= -github.com/docker/go-units v0.3.3/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk= -github.com/docker/go-units v0.4.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk= -github.com/docker/spdystream v0.0.0-20160310174837-449fdfce4d96/go.mod h1:Qh8CwZgvJUkLughtfhJv5dyTYa91l1fOUCrgjqmcifM= -github.com/docopt/docopt-go v0.0.0-20180111231733-ee0de3bc6815/go.mod h1:WwZ+bS3ebgob9U8Nd0kOddGdZWjyMGR8Wziv+TBNwSE= -github.com/dustin/go-humanize v0.0.0-20171111073723-bb3d318650d4/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk= -github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk= -github.com/elazarl/goproxy v0.0.0-20170405201442-c4fc26588b6e/go.mod h1:/Zj4wYkgs4iZTTu3o/KG3Itv/qCCa8VVMlb3i9OVuzc= -github.com/emicklei/go-restful v0.0.0-20170410110728-ff4f55a20633/go.mod h1:otzb+WCGbkyDHkqmQmT5YD2WR4BBwUdeQoFo8l/7tVs= -github.com/emicklei/go-restful v2.9.5+incompatible/go.mod h1:otzb+WCGbkyDHkqmQmT5YD2WR4BBwUdeQoFo8l/7tVs= -github.com/evanphx/json-patch v4.2.0+incompatible h1:fUDGZCv/7iAN7u0puUVhvKCcsR6vRfwrJatElLBEf0I= -github.com/evanphx/json-patch v4.2.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk= -github.com/evanphx/json-patch v4.5.0+incompatible h1:ouOWdg56aJriqS0huScTkVXPC5IcNrDCXZ6OoTAWu7M= -github.com/evanphx/json-patch v4.5.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk= -github.com/evanphx/json-patch v4.9.0+incompatible h1:kLcOMZeuLAJvL2BPWLMIj5oaZQobrkAqrL+WFZwQses= -github.com/evanphx/json-patch v4.9.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk= -github.com/fatih/color v1.7.0 h1:DkWD4oS2D8LGGgTQ6IvwJJXSL5Vp2ffcQg58nFV38Ys= -github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4= -github.com/fsnotify/fsnotify v1.4.7 h1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV9I= -github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo= -github.com/fsnotify/fsnotify v1.4.9 h1:hsms1Qyu0jgnwNXIxa+/V/PDsU6CfLf6CNO8H7IWoS4= -github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ= -github.com/ghodss/yaml v0.0.0-20150909031657-73d445a93680 h1:ZktWZesgun21uEDrwW7iEV1zPCGQldM2atlJZ3TdvVM= -github.com/ghodss/yaml v0.0.0-20150909031657-73d445a93680/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04= -github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04= -github.com/globalsign/mgo v0.0.0-20180905125535-1ca0a4f7cbcb/go.mod h1:xkRDCp4j0OGD1HRkm4kmhM+pmpv3AKq5SU7GMg4oO/Q= -github.com/globalsign/mgo v0.0.0-20181015135952-eeefdecb41b8/go.mod h1:xkRDCp4j0OGD1HRkm4kmhM+pmpv3AKq5SU7GMg4oO/Q= -github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as= -github.com/go-kit/kit v0.9.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as= -github.com/go-logfmt/logfmt v0.3.0/go.mod h1:Qt1PoO58o5twSAckw1HlFXLmHsOX5/0LbT9GBnD5lWE= -github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V4qmtdjCk= -github.com/go-logr/logr v0.1.0 h1:M1Tv3VzNlEHg6uyACnRdtrploV2P7wZqH8BoQMtz0cg= -github.com/go-logr/logr v0.1.0/go.mod h1:ixOQHD9gLJUVQQ2ZOR7zLEifBX6tGkNJF4QyIY7sIas= -github.com/go-logr/zapr v0.1.0 h1:h+WVe9j6HAA01niTJPA/kKH0i7e0rLZBCwauQFcRE54= -github.com/go-logr/zapr v0.1.0/go.mod h1:tabnROwaDl0UNxkVeFRbY8bwB37GwRv0P8lg6aAiEnk= -github.com/go-openapi/analysis v0.0.0-20180825180245-b006789cd277/go.mod h1:k70tL6pCuVxPJOHXQ+wIac1FUrvNkHolPie/cLEU6hI= -github.com/go-openapi/analysis v0.17.0/go.mod h1:IowGgpVeD0vNm45So8nr+IcQ3pxVtpRoBWb8PVZO0ik= -github.com/go-openapi/analysis v0.18.0/go.mod h1:IowGgpVeD0vNm45So8nr+IcQ3pxVtpRoBWb8PVZO0ik= -github.com/go-openapi/analysis v0.19.2/go.mod h1:3P1osvZa9jKjb8ed2TPng3f0i/UY9snX6gxi44djMjk= -github.com/go-openapi/analysis v0.19.5/go.mod h1:hkEAkxagaIvIP7VTn8ygJNkd4kAYON2rCu0v0ObL0AU= -github.com/go-openapi/errors v0.17.0/go.mod h1:LcZQpmvG4wyF5j4IhA73wkLFQg+QJXOQHVjmcZxhka0= -github.com/go-openapi/errors v0.18.0/go.mod h1:LcZQpmvG4wyF5j4IhA73wkLFQg+QJXOQHVjmcZxhka0= -github.com/go-openapi/errors v0.19.2/go.mod h1:qX0BLWsyaKfvhluLejVpVNwNRdXZhEbTA4kxxpKBC94= -github.com/go-openapi/jsonpointer v0.0.0-20160704185906-46af16f9f7b1/go.mod h1:+35s3my2LFTysnkMfxsJBAMHj/DoqoB9knIWoYG/Vk0= -github.com/go-openapi/jsonpointer v0.17.0/go.mod h1:cOnomiV+CVVwFLk0A/MExoFMjwdsUdVpsRhURCKh+3M= -github.com/go-openapi/jsonpointer v0.18.0/go.mod h1:cOnomiV+CVVwFLk0A/MExoFMjwdsUdVpsRhURCKh+3M= -github.com/go-openapi/jsonpointer v0.19.2/go.mod h1:3akKfEdA7DF1sugOqz1dVQHBcuDBPKZGEoHC/NkiQRg= -github.com/go-openapi/jsonpointer v0.19.3/go.mod h1:Pl9vOtqEWErmShwVjC8pYs9cog34VGT37dQOVbmoatg= -github.com/go-openapi/jsonreference v0.0.0-20160704190145-13c6e3589ad9/go.mod h1:W3Z9FmVs9qj+KR4zFKmDPGiLdk1D9Rlm7cyMvf57TTg= -github.com/go-openapi/jsonreference v0.17.0/go.mod h1:g4xxGn04lDIRh0GJb5QlpE3HfopLOL6uZrK/VgnsK9I= -github.com/go-openapi/jsonreference v0.18.0/go.mod h1:g4xxGn04lDIRh0GJb5QlpE3HfopLOL6uZrK/VgnsK9I= -github.com/go-openapi/jsonreference v0.19.2/go.mod h1:jMjeRr2HHw6nAVajTXJ4eiUwohSTlpa0o73RUL1owJc= -github.com/go-openapi/jsonreference v0.19.3/go.mod h1:rjx6GuL8TTa9VaixXglHmQmIL98+wF9xc8zWvFonSJ8= -github.com/go-openapi/loads v0.17.0/go.mod h1:72tmFy5wsWx89uEVddd0RjRWPZm92WRLhf7AC+0+OOU= -github.com/go-openapi/loads v0.18.0/go.mod h1:72tmFy5wsWx89uEVddd0RjRWPZm92WRLhf7AC+0+OOU= -github.com/go-openapi/loads v0.19.0/go.mod h1:72tmFy5wsWx89uEVddd0RjRWPZm92WRLhf7AC+0+OOU= -github.com/go-openapi/loads v0.19.2/go.mod h1:QAskZPMX5V0C2gvfkGZzJlINuP7Hx/4+ix5jWFxsNPs= -github.com/go-openapi/loads v0.19.4/go.mod h1:zZVHonKd8DXyxyw4yfnVjPzBjIQcLt0CCsn0N0ZrQsk= -github.com/go-openapi/runtime v0.0.0-20180920151709-4f900dc2ade9/go.mod h1:6v9a6LTXWQCdL8k1AO3cvqx5OtZY/Y9wKTgaoP6YRfA= -github.com/go-openapi/runtime v0.19.0/go.mod h1:OwNfisksmmaZse4+gpV3Ne9AyMOlP1lt4sK4FXt0O64= -github.com/go-openapi/runtime v0.19.4/go.mod h1:X277bwSUBxVlCYR3r7xgZZGKVvBd/29gLDlFGtJ8NL4= -github.com/go-openapi/spec v0.0.0-20160808142527-6aced65f8501/go.mod h1:J8+jY1nAiCcj+friV/PDoE1/3eeccG9LYBs0tYvLOWc= -github.com/go-openapi/spec v0.17.0/go.mod h1:XkF/MOi14NmjsfZ8VtAKf8pIlbZzyoTvZsdfssdxcBI= -github.com/go-openapi/spec v0.18.0/go.mod h1:XkF/MOi14NmjsfZ8VtAKf8pIlbZzyoTvZsdfssdxcBI= -github.com/go-openapi/spec v0.19.2/go.mod h1:sCxk3jxKgioEJikev4fgkNmwS+3kuYdJtcsZsD5zxMY= -github.com/go-openapi/spec v0.19.3/go.mod h1:FpwSN1ksY1eteniUU7X0N/BgJ7a4WvBFVA8Lj9mJglo= -github.com/go-openapi/strfmt v0.17.0/go.mod h1:P82hnJI0CXkErkXi8IKjPbNBM6lV6+5pLP5l494TcyU= -github.com/go-openapi/strfmt v0.18.0/go.mod h1:P82hnJI0CXkErkXi8IKjPbNBM6lV6+5pLP5l494TcyU= -github.com/go-openapi/strfmt v0.19.0/go.mod h1:+uW+93UVvGGq2qGaZxdDeJqSAqBqBdl+ZPMF/cC8nDY= -github.com/go-openapi/strfmt v0.19.3/go.mod h1:0yX7dbo8mKIvc3XSKp7MNfxw4JytCfCD6+bY1AVL9LU= -github.com/go-openapi/swag v0.0.0-20160704191624-1d0bd113de87/go.mod h1:DXUve3Dpr1UfpPtxFw+EFuQ41HhCWZfha5jSVRG7C7I= -github.com/go-openapi/swag v0.17.0/go.mod h1:AByQ+nYG6gQg71GINrmuDXCPWdL640yX49/kXLo40Tg= -github.com/go-openapi/swag v0.18.0/go.mod h1:AByQ+nYG6gQg71GINrmuDXCPWdL640yX49/kXLo40Tg= -github.com/go-openapi/swag v0.19.2/go.mod h1:POnQmlKehdgb5mhVOsnJFsivZCEZ/vjK9gh66Z9tfKk= -github.com/go-openapi/swag v0.19.5/go.mod h1:POnQmlKehdgb5mhVOsnJFsivZCEZ/vjK9gh66Z9tfKk= -github.com/go-openapi/validate v0.18.0/go.mod h1:Uh4HdOzKt19xGIGm1qHf/ofbX1YQ4Y+MYsct2VUrAJ4= -github.com/go-openapi/validate v0.19.2/go.mod h1:1tRCw7m3jtI8eNWEEliiAqUIcBztB2KDnRCRMUi7GTA= -github.com/go-openapi/validate v0.19.5/go.mod h1:8DJv2CVJQ6kGNpFW6eV9N3JviE1C85nY1c2z52x1Gk4= -github.com/go-sql-driver/mysql v1.5.0/go.mod h1:DCzpHaOWr8IXmIStZouvnhqoel9Qv2LBy8hT2VhHyBg= -github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY= -github.com/gofrs/uuid v3.2.0+incompatible h1:y12jRkkFxsd7GpqdSZ+/KCs/fJbqpEXSGd4+jfEaewE= -github.com/gofrs/uuid v3.2.0+incompatible/go.mod h1:b2aQJv3Z4Fp6yNu3cdSllBxTCLRxnplIgP/c0N/04lM= -github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ= -github.com/gogo/protobuf v1.2.1/go.mod h1:hp+jE20tsWTFYpLwKvXlhS1hjn+gTNwPg2I6zVXpSg4= -github.com/gogo/protobuf v1.2.2-0.20190723190241-65acae22fc9d h1:3PaI8p3seN09VjbTYC/QWlUZdZ1qS1zGjy7LH2Wt07I= -github.com/gogo/protobuf v1.2.2-0.20190723190241-65acae22fc9d/go.mod h1:SlYgWuQ5SjCEi6WLHjHCa1yvBfUnHcTbrrZtXPKa29o= -github.com/gogo/protobuf v1.3.1 h1:DqDEcV5aeaTmdFBePNpYsp3FlcVH/2ISVVM9Qf8PSls= -github.com/gogo/protobuf v1.3.1/go.mod h1:SlYgWuQ5SjCEi6WLHjHCa1yvBfUnHcTbrrZtXPKa29o= -github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b h1:VKtxabqXZkF25pY9ekfRL6a582T4P37/31XEstQ5p58= -github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= -github.com/golang/groupcache v0.0.0-20160516000752-02826c3e7903/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= -github.com/golang/groupcache v0.0.0-20180513044358-24b0969c4cb7 h1:u4bArs140e9+AfE52mFHOXVFnOSBJBRlzTHrOPLOIhE= -github.com/golang/groupcache v0.0.0-20180513044358-24b0969c4cb7/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= -github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= -github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= -github.com/golang/protobuf v0.0.0-20161109072736-4bd1920723d7/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= -github.com/golang/protobuf v1.0.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= -github.com/golang/protobuf v1.2.0 h1:P3YflyNX/ehuJFLhxviNdFxQPkGK5cDcApsge1SqnvM= -github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= -github.com/golang/protobuf v1.3.1 h1:YF8+flBXS5eO826T4nzqPrxfhQThhXl0YzfuUPu4SBg= -github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= -github.com/golang/protobuf v1.3.2 h1:6nsPYzhq5kReh6QImI3k5qWzO4PEbvbIW2cwSfR/6xs= -github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= -github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8= -github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA= -github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs= -github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w= -github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0= -github.com/golang/protobuf v1.4.2 h1:+Z5KGCizgyZCbGh1KZqA0fcLLkwbsjIzS4aV2v7wJX0= -github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI= -github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= -github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= -github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M= -github.com/google/go-cmp v0.3.0 h1:crn/baboCvb5fXaQ0IJ1SGTsTVrWpDsCWC8EGETZijY= -github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= -github.com/google/go-cmp v0.3.1 h1:Xye71clBPdm5HgqGwUkwhbynsUJZhDbS20FvLhQ2izg= -github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= -github.com/google/go-cmp v0.4.0 h1:xsAVV57WRhGj6kEIi8ReJzQlHHqcBYCElAvkovg3B/4= -github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= -github.com/google/gofuzz v0.0.0-20161122191042-44d81051d367/go.mod h1:HP5RmnzzSNb993RKQDq4+1A4ia9nllfqcQFTQJedwGI= -github.com/google/gofuzz v1.0.0 h1:A8PeW59pxE9IoFRqBp37U+mSNaQoZ46F1f0f863XSXw= -github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= -github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs= -github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc= -github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI= -github.com/google/uuid v1.0.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= -github.com/google/uuid v1.1.1 h1:Gkbcsh/GbpXz7lPftLA3P6TYMwjCLYm83jiFQZF/3gY= -github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= -github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg= -github.com/googleapis/gnostic v0.0.0-20170729233727-0c5108395e2d/go.mod h1:sJBsCZ4ayReDTBIg8b9dl28c5xFWyhBTVRp3pOg5EKY= -github.com/googleapis/gnostic v0.3.1 h1:WeAefnSUHlBb0iJKwxFDZdbfGwkd7xRNuV+IpXMJhYk= -github.com/googleapis/gnostic v0.3.1/go.mod h1:on+2t9HRStVgn95RSsFWFz+6Q0Snyqv1awfrALZdbtU= -github.com/gophercloud/gophercloud v0.1.0/go.mod h1:vxM41WHh5uqHVBMZHzuwNOHh8XEoIEcSTewFxm1c5g8= -github.com/gorilla/websocket v0.0.0-20170926233335-4201258b820c/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ= -github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ= -github.com/gregjones/httpcache v0.0.0-20170728041850-787624de3eb7/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA= -github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA= -github.com/grpc-ecosystem/go-grpc-middleware v0.0.0-20190222133341-cfaf5686ec79/go.mod h1:FiyG127CGDf3tlThmgyCl78X/SZQqEOJBCDaAfeWzPs= -github.com/grpc-ecosystem/go-grpc-middleware v1.0.1-0.20190118093823-f849b5445de4/go.mod h1:FiyG127CGDf3tlThmgyCl78X/SZQqEOJBCDaAfeWzPs= -github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0/go.mod h1:8NvIoxWQoOIhqOTXgfV/d3M/q6VIi02HzZEHgUlZvzk= -github.com/grpc-ecosystem/grpc-gateway v1.3.0/go.mod h1:RSKVYQBd5MCa4OVpNdGskqpgL2+G+NZTnrVHpWWfpdw= -github.com/grpc-ecosystem/grpc-gateway v1.9.5/go.mod h1:vNeuVxBJEsws4ogUvrchl83t/GYV9WGTSLVdBhOQFDY= -github.com/hashicorp/go-immutable-radix v1.2.0 h1:l6UW37iCXwZkZoAbEYnptSHVE/cQ5bOTPYG5W3vf9+8= -github.com/hashicorp/go-immutable-radix v1.2.0/go.mod h1:0y9vanUI8NX6FsYoO3zeMjhV/C5i9g4Q3DwcSNZ4P60= -github.com/hashicorp/go-memdb v1.2.1 h1:wI9btDjYUOJJHTCnRlAG/TkRyD/ij7meJMrLK9X31Cc= -github.com/hashicorp/go-memdb v1.2.1/go.mod h1:OSvLJ662Jim8hMM+gWGyhktyWk2xPCnWMc7DWIqtkGA= -github.com/hashicorp/go-uuid v1.0.0 h1:RS8zrF7PhGwyNPOtxSClXXj9HA8feRnJzgnI1RJCSnM= -github.com/hashicorp/go-uuid v1.0.0/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro= -github.com/hashicorp/go-version v1.0.0/go.mod h1:fltr4n8CU8Ke44wwGCBoEymUuxUHl09ZGVZPK5anwXA= -github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= -github.com/hashicorp/golang-lru v0.5.1 h1:0hERBMJE1eitiLkihrMvRVBYAkpHzc/J3QdDN+dAcgU= -github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= -github.com/hashicorp/golang-lru v0.5.4 h1:YDjusn29QI/Das2iO9M0BHnIbxPeyuCHsjMW+lJfyTc= -github.com/hashicorp/golang-lru v0.5.4/go.mod h1:iADmTwqILo4mZ8BN3D2Q6+9jd8WM5uGBxy+E8yxSoD4= -github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ= -github.com/hpcloud/tail v1.0.0 h1:nfCOvKYfkgYP8hkirhJocXT2+zOD8yUNjXaWfTlyFKI= -github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU= -github.com/imdario/mergo v0.3.5/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA= -github.com/imdario/mergo v0.3.6/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA= -github.com/imdario/mergo v0.3.10 h1:6q5mVkdH/vYmqngx7kZQTjJ5HRsx+ImorDIEQ+beJgc= -github.com/imdario/mergo v0.3.10/go.mod h1:jmQim1M+e3UYxmgPu/WyfjB3N3VflVyUjjjwH0dnCYA= -github.com/inconshreveable/mousetrap v1.0.0 h1:Z8tu5sraLXCXIcARxBp/8cbvlwVa7Z1NHg9XEKhtSvM= -github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8= -github.com/jmespath/go-jmespath v0.3.0 h1:OS12ieG61fsCg5+qLJ+SsW9NicxNkg3b25OyT2yCeUc= -github.com/jmespath/go-jmespath v0.3.0/go.mod h1:9QtRXoHjLGCJ5IBSaohpXITPlowMeeYCZ7fLUTSywik= -github.com/jmespath/go-jmespath v0.4.0 h1:BEgLn5cpjn8UN1mAw4NjwDrS35OdebyEtFe+9YPoQUg= -github.com/jmespath/go-jmespath v0.4.0/go.mod h1:T8mJZnbsbmF+m6zOOFylbeCJqk5+pHWvzYPziyZiYoo= -github.com/jmespath/go-jmespath/internal/testify v1.5.1 h1:shLQSRRSCCPj3f2gpwzGwWFoC7ycTf1rcQZHOlsJ6N8= -github.com/jmespath/go-jmespath/internal/testify v1.5.1/go.mod h1:L3OGu8Wl2/fWfCI6z80xFu9LTZmf1ZRjMHUOPmWr69U= -github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo= -github.com/json-iterator/go v0.0.0-20180612202835-f2b4162afba3/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU= -github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU= -github.com/json-iterator/go v1.1.7 h1:KfgG9LzI+pYjr4xvmz/5H4FXjokeP+rlHLhv3iH62Fo= -github.com/json-iterator/go v1.1.7/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4= -github.com/json-iterator/go v1.1.8 h1:QiWkFLKq0T7mpzwOTu6BzNDbfTE8OLrYhVKYMLF46Ok= -github.com/json-iterator/go v1.1.8/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4= -github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU= -github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w= -github.com/karlseguin/ccache v2.0.3+incompatible h1:j68C9tWOROiOLWTS/kCGg9IcJG+ACqn5+0+t8Oh83UU= -github.com/karlseguin/ccache v2.0.3+incompatible/go.mod h1:CM9tNPzT6EdRh14+jiW8mEF9mkNZuuE51qmgGYUB93w= -github.com/karlseguin/expect v1.0.1 h1:z4wy4npwwHSWKjGWH85WNJO42VQhovxTCZDSzhjo8hY= -github.com/karlseguin/expect v1.0.1/go.mod h1:zNBxMY8P21owkeogJELCLeHIt+voOSduHYTFUbwRAV8= -github.com/keikoproj/aws-sdk-go-cache v0.0.0-20201118182730-f6f418a4e2df h1:5CIVZTNmDF4GwbyQzRGYLoG1mo2LHJSO4UwAUDNpgDw= -github.com/keikoproj/aws-sdk-go-cache v0.0.0-20201118182730-f6f418a4e2df/go.mod h1:WuCkHvglMhs9DQnwssll4dy87h352LIfN3qfyk6l6Rg= -github.com/keikoproj/inverse-exp-backoff v0.0.0-20201007213207-e4a3ac0f74ab h1:8/LbUmjJHVF8NZYHwlSWGgI731i0gFkF2posm/sAvB0= -github.com/keikoproj/inverse-exp-backoff v0.0.0-20201007213207-e4a3ac0f74ab/go.mod h1:ziu/tMrrvs8n+AI+HCZBb6wZS609fcMl5ygR2RttEE4= -github.com/keikoproj/kubedog v0.0.1 h1:SKo5g78QvlXx+JniYGvgohsB/5dm6bdj6ccyyUrnLDs= -github.com/keikoproj/kubedog v0.0.1/go.mod h1:8aRJYCL//c+RvycK3qsAfuHqyS1EP7Pa4g8M+t1wO3M= -github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q= -github.com/kisielk/errcheck v1.2.0/go.mod h1:/BMXB+zMLi60iA8Vv6Ksmxu/1UDYcXs4uQLJ+jE2L00= -github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= -github.com/konsorten/go-windows-terminal-sequences v1.0.1 h1:mweAR1A6xJ3oS2pRaGiHgQ4OO8tzTaLawm8vnODuwDk= -github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ= -github.com/konsorten/go-windows-terminal-sequences v1.0.3 h1:CE8S1cTafDpPvMhIxNJKvHsGVBgn1xWYf1NbHQhywc8= -github.com/konsorten/go-windows-terminal-sequences v1.0.3/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ= -github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc= -github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI= -github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= -github.com/kr/pretty v0.2.0/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI= -github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= -github.com/kr/pty v1.1.5/go.mod h1:9r2w37qlBe7rQ6e1fg1S/9xpWHSnaqNdHD3WcMdbPDA= -github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE= -github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= -github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= -github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= -github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ= -github.com/mailru/easyjson v0.0.0-20160728113105-d5b7844b561a/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= -github.com/mailru/easyjson v0.0.0-20180823135443-60711f1a8329/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= -github.com/mailru/easyjson v0.0.0-20190312143242-1de009706dbe/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= -github.com/mailru/easyjson v0.0.0-20190614124828-94de47d64c63/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= -github.com/mailru/easyjson v0.0.0-20190626092158-b2ccc519800e/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= -github.com/mailru/easyjson v0.7.0/go.mod h1:KAzv3t3aY1NaHWoQz1+4F1ccyAH66Jk7yos7ldAVICs= -github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU= -github.com/mattn/go-isatty v0.0.4/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNxMWT7Zi4= -github.com/mattn/go-runewidth v0.0.2/go.mod h1:LwmH8dsx7+W8Uxz3IHJYH5QSwggIsqBzpuz5H//U1FU= -github.com/matttproud/golang_protobuf_extensions v1.0.1 h1:4hp9jkHxhMHkqkrB3Ix0jegS5sx/RkqARlsWZ6pIwiU= -github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0= -github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0= -github.com/mitchellh/iochan v1.0.0/go.mod h1:JwYml1nuB7xOzsp52dPpHFffvOCDupsG0QubkSMEySY= -github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y= -github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= -github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg= -github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= -github.com/modern-go/reflect2 v0.0.0-20180320133207-05fbef0ca5da/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0= -github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742 h1:Esafd1046DLDQ0W1YjYsBW+p8U2u7vzgW2SQVmlNazg= -github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0= -github.com/modern-go/reflect2 v1.0.1 h1:9f412s+6RmYXLWZSEzVVgPGK7C2PphHj5RJrvfx9AWI= -github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0= -github.com/munnerz/goautoneg v0.0.0-20120707110453-a547fc61f48d/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ= -github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ= -github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U= -github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f/go.mod h1:ZdcZmHo+o7JKHSa8/e818NopupXU1YMK5fe1lsApnBw= -github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e h1:fD57ERR4JtEqsWbfPhv4DMiApHyliiK5xCTNVSPiaAs= -github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e/go.mod h1:zD1mROLANZcx1PVRCS0qkT7pwLkGfwJo4zjcN/Tysno= -github.com/nxadm/tail v1.4.4 h1:DQuhQpB1tVlglWS2hLQ5OV6B5r8aGxSrPc5Qo6uTN78= -github.com/nxadm/tail v1.4.4/go.mod h1:kenIhsEOeOJmVchQTgglprH7qJGnHDVpk1VPCcaMI8A= -github.com/olekukonko/tablewriter v0.0.0-20170122224234-a0225b3f23b5/go.mod h1:vsDQFd/mU46D+Z4whnwzcISnGGzXWMclvtLoiIKAKIo= -github.com/onsi/ginkgo v0.0.0-20170829012221-11459a886d9c/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= -github.com/onsi/ginkgo v1.4.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= -github.com/onsi/ginkgo v1.6.0 h1:Ix8l273rp3QzYgXSR+c8d1fTG7UPgYkOSELPhiY/YGw= -github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= -github.com/onsi/ginkgo v1.8.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= -github.com/onsi/ginkgo v1.10.1 h1:q/mM8GF/n0shIN8SaAZ0V+jnLPzen6WIVZdiwrRlMlo= -github.com/onsi/ginkgo v1.10.1/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= -github.com/onsi/ginkgo v1.12.1/go.mod h1:zj2OWP4+oCPe1qIXoGWkgMRwljMUYCdkwsT2108oapk= -github.com/onsi/ginkgo v1.14.2 h1:8mVmC9kjFFmA8H4pKMUhcblgifdkOIXPvbhN1T36q1M= -github.com/onsi/ginkgo v1.14.2/go.mod h1:iSB4RoI2tjJc9BBv4NKIKWKya62Rps+oPG/Lv9klQyY= -github.com/onsi/gomega v0.0.0-20170829124025-dcabb60a477c/go.mod h1:C1qb7wdrVGGVU+Z6iS04AVkA3Q65CEZX59MT0QO5uiA= -github.com/onsi/gomega v1.3.0/go.mod h1:C1qb7wdrVGGVU+Z6iS04AVkA3Q65CEZX59MT0QO5uiA= -github.com/onsi/gomega v1.5.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY= -github.com/onsi/gomega v1.7.0 h1:XPnZz8VVBHjVsy1vzJmRwIcSwiUO+JFfrv/xGiigmME= -github.com/onsi/gomega v1.7.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY= -github.com/onsi/gomega v1.7.1 h1:K0jcRCwNQM3vFGh1ppMtDh/+7ApJrjldlX8fA0jDTLQ= -github.com/onsi/gomega v1.7.1/go.mod h1:XdKZgCCFLUoM/7CFJVPcG8C1xQ1AJ0vpAezJrB7JYyY= -github.com/onsi/gomega v1.10.1/go.mod h1:iN09h71vgCQne3DLsj+A5owkum+a2tYe+TOCB1ybHNo= -github.com/onsi/gomega v1.10.4 h1:NiTx7EEvBzu9sFOD1zORteLSt3o8gnlvZZwSE9TnY9U= -github.com/onsi/gomega v1.10.4/go.mod h1:g/HbgYopi++010VEqkFgJHKC09uJiW9UkXvMUuKHUCQ= -github.com/pborman/uuid v1.2.0/go.mod h1:X/NO0urCmaxf9VXbdlT7C2Yzkj2IKimNn4k+gtPdI/k= -github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic= -github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU= -github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= -github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I= -github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= -github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= -github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= -github.com/pmezard/go-difflib v0.0.0-20151028094244-d8ed2627bdf0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= -github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= -github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= -github.com/pquerna/cachecontrol v0.0.0-20171018203845-0dec1b30a021/go.mod h1:prYjPmNq4d1NPVmpShWobRqXY3q7Vp+80DqgxxUrUIA= -github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw= -github.com/prometheus/client_golang v0.9.2/go.mod h1:OsXs2jCmiKlQ1lTBmv21f2mNfw4xf/QclQDMrYNZzcM= -github.com/prometheus/client_golang v1.0.0 h1:vrDKnkGzuGvhNAL56c7DBz29ZL+KxnoR0x7enabFceM= -github.com/prometheus/client_golang v1.0.0/go.mod h1:db9x61etRT2tGnBNRi70OPL5FsnadC4Ky3P0J6CfImo= -github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910 h1:idejC8f05m9MGOsuEi1ATq9shN03HrxNkD/luQvxCv8= -github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo= -github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90 h1:S/YWwWx/RA8rT8tKFRuGUZhuA90OyIBpPCXkcbwU8DE= -github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= -github.com/prometheus/client_model v0.2.0 h1:uq5h0d+GuxiXLJLNABMgp2qUWDPiLvgCzz2dUR+/W/M= -github.com/prometheus/client_model v0.2.0/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= -github.com/prometheus/common v0.0.0-20181126121408-4724e9255275/go.mod h1:daVV7qP5qjZbuso7PdcryaAu0sAZbrN9i7WWcTMWvro= -github.com/prometheus/common v0.4.1 h1:K0MGApIoQvMw27RTdJkPbr3JZ7DNbtxQNyi5STVM6Kw= -github.com/prometheus/common v0.4.1/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4= -github.com/prometheus/common v0.9.1 h1:KOMtN28tlbam3/7ZKEYKHhKoJZYYj3gMH4uc62x7X7U= -github.com/prometheus/common v0.9.1/go.mod h1:yhUN8i9wzaXS3w1O07YhxHEBxD+W35wd8bs7vj7HSQ4= -github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk= -github.com/prometheus/procfs v0.0.0-20181204211112-1dc9a6cbc91a/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk= -github.com/prometheus/procfs v0.0.2 h1:6LJUbpNm42llc4HRCuvApCSWB/WfhuNo9K98Q9sNGfs= -github.com/prometheus/procfs v0.0.2/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA= -github.com/prometheus/procfs v0.0.8 h1:+fpWZdT24pJBiqJdAwYBjPSk+5YmQzYNPYzQsdzLkt8= -github.com/prometheus/procfs v0.0.8/go.mod h1:7Qr8sr6344vo1JqZ6HhLceV9o3AJ1Ff+GxbHq6oeK9A= -github.com/remyoudompheng/bigfft v0.0.0-20170806203942-52369c62f446/go.mod h1:uYEyJGbgTkfkS4+E/PavXkNJcbFIpEtjt2B0KDQ5+9M= -github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af/go.mod h1:XWv6SoW27p1b0cqNHllgS5HIMJraePCO15w5zCzIWYg= -github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4= -github.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g= -github.com/sergi/go-diff v1.0.0/go.mod h1:0CfEIISq7TuYL3j771MWULgwwjU+GofnZX9QAmXWZgo= -github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo= -github.com/sirupsen/logrus v1.4.2 h1:SPIRibHv4MatM3XXNO2BJeFLZwZ2LvZgfQ5+UNI2im4= -github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE= -github.com/sirupsen/logrus v1.6.0 h1:UBcNElsrwanuuMsnGSlYmtmgbb23qDR5dG+6X6Oo89I= -github.com/sirupsen/logrus v1.6.0/go.mod h1:7uNnSEd1DgxDLC74fIahvMZmmYsHGZGEOFrfsX/uA88= -github.com/soheilhy/cmux v0.1.3/go.mod h1:IM3LyeVVIOuxMH7sFAkER9+bJ4dT7Ms6E4xg4kGIyLM= -github.com/soheilhy/cmux v0.1.4/go.mod h1:IM3LyeVVIOuxMH7sFAkER9+bJ4dT7Ms6E4xg4kGIyLM= -github.com/spf13/afero v1.1.2/go.mod h1:j4pytiNVoe2o6bmDsKpLACNPDBIoEAkihy7loJ1B0CQ= -github.com/spf13/afero v1.2.2/go.mod h1:9ZxEEn6pIJ8Rxe320qSDBk6AsU0r9pR7Q4OcevTdifk= -github.com/spf13/cast v1.3.0/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE= -github.com/spf13/cobra v0.0.3/go.mod h1:1l0Ry5zgKvJasoi3XT1TypsSe7PqH0Sj9dhYf7v3XqQ= -github.com/spf13/cobra v0.0.5/go.mod h1:3K3wKZymM7VvHMDS9+Akkh4K60UwM26emMESw8tLCHU= -github.com/spf13/jwalterweatherman v1.0.0/go.mod h1:cQK4TGJAtQXfYWX+Ddv3mKDzgVb68N+wFjFa4jdeBTo= -github.com/spf13/pflag v0.0.0-20170130214245-9ff6c6923cff/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4= -github.com/spf13/pflag v1.0.1/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4= -github.com/spf13/pflag v1.0.3 h1:zPAT6CGy6wXeQ7NtTnaTerfKOsV6V6F8agHXFiazDkg= -github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4= -github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA= -github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= -github.com/spf13/viper v1.3.2/go.mod h1:ZiWeW+zYFKm7srdB9IoDzzZXaJaI5eL9QjNiN/DMA2s= -github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= -github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= -github.com/stretchr/objx v0.2.0/go.mod h1:qt09Ya8vawLte6SNmTgCsAVtYtaKzEcn8ATUoHMkEqE= -github.com/stretchr/testify v0.0.0-20151208002404-e3a8ff8ce365/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= -github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= -github.com/stretchr/testify v1.3.0 h1:TivCn/peBQ7UY8ooIcPgZFpTNSz0Q2U6UrFlUfqbe0Q= -github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= -github.com/stretchr/testify v1.4.0 h1:2E4SXV/wtOkTonXsotYi4li6zVWxYlZuYNCXe9XRJyk= -github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4= -github.com/stretchr/testify v1.5.1 h1:nOGnQDM7FYENwehXlg/kFVnos3rEvtKTjRvOWSzb6H4= -github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA= -github.com/stretchr/testify v1.6.1 h1:hDPOHmpOpP40lSULcqw7IrRb/u7w6RpDC9399XyoNd0= -github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= -github.com/tidwall/pretty v1.0.0/go.mod h1:XNkn88O1ChpSDQmQeStsy+sBenx6DDtFZJxhVysOjyk= -github.com/tmc/grpc-websocket-proxy v0.0.0-20170815181823-89b8d40f7ca8/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U= -github.com/ugorji/go/codec v0.0.0-20181204163529-d75b2dcb6bc8/go.mod h1:VFNgLljTbGfSG7qAOspJ7OScBnGdDN/yBr0sguwnwf0= -github.com/urfave/cli v1.20.0/go.mod h1:70zkFmudgCuE/ngEzBv17Jvp/497gISqfk5gWijbERA= -github.com/vektah/gqlparser v1.1.2/go.mod h1:1ycwN7Ij5njmMkPPAOaRFY4rET2Enx7IkVv3vaXspKw= -github.com/wsxiaoys/terminal v0.0.0-20160513160801-0940f3fc43a0 h1:3UeQBvD0TFrlVjOeLOBz+CPAI8dnbqNSVwUwRrkp7vQ= -github.com/wsxiaoys/terminal v0.0.0-20160513160801-0940f3fc43a0/go.mod h1:IXCdmsXIht47RaVFLEdVnh1t+pgYtTAhQGj73kz+2DM= -github.com/xiang90/probing v0.0.0-20160813154853-07dd2e8dfe18/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU= -github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU= -github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q= -go.etcd.io/bbolt v1.3.3/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU= -go.etcd.io/etcd v0.0.0-20191023171146-3cf2f69b5738/go.mod h1:dnLIgRNXwCJa5e+c6mIZCrds/GIG4ncV9HhK5PX7jPg= -go.mongodb.org/mongo-driver v1.0.3/go.mod h1:u7ryQJ+DOzQmeO7zB6MHyr8jkEQvC8vH7qLUO4lqsUM= -go.mongodb.org/mongo-driver v1.1.1/go.mod h1:u7ryQJ+DOzQmeO7zB6MHyr8jkEQvC8vH7qLUO4lqsUM= -go.mongodb.org/mongo-driver v1.1.2/go.mod h1:u7ryQJ+DOzQmeO7zB6MHyr8jkEQvC8vH7qLUO4lqsUM= -go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU= -go.uber.org/atomic v0.0.0-20181018215023-8dc6146f7569/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE= -go.uber.org/atomic v1.3.2 h1:2Oa65PReHzfn29GpvgsYwloV9AVFHPDk8tYxt2c2tr4= -go.uber.org/atomic v1.3.2/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE= -go.uber.org/atomic v1.6.0 h1:Ezj3JGmsOnG1MoRWQkPBsKLe9DwWD9QeXzTRzzldNVk= -go.uber.org/atomic v1.6.0/go.mod h1:sABNBOSYdrvTF6hTgEIbc7YasKWGhgEQZyfxyTvoXHQ= -go.uber.org/multierr v0.0.0-20180122172545-ddea229ff1df/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0= -go.uber.org/multierr v1.1.0 h1:HoEmRHQPVSqub6w2z2d2EOVs2fjyFRGyofhKuyDq0QI= -go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0= -go.uber.org/multierr v1.5.0 h1:KCa4XfM8CWFCpxXRGok+Q0SS/0XBhMDbHHGABQLvD2A= -go.uber.org/multierr v1.5.0/go.mod h1:FeouvMocqHpRaaGuG9EjoKcStLC43Zu/fmqdUMPcKYU= -go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee h1:0mgffUl7nfd+FpvXMVz4IDEaUSmT1ysygQC7qYo7sG4= -go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee/go.mod h1:vJERXedbb3MVM5f9Ejo0C68/HhF8uaILCdgjnY+goOA= -go.uber.org/zap v0.0.0-20180814183419-67bc79d13d15/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q= -go.uber.org/zap v1.9.1/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q= -go.uber.org/zap v1.10.0 h1:ORx85nbTijNz8ljznvCMR1ZBIPKFn3jQrag10X2AsuM= -go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q= -go.uber.org/zap v1.16.0 h1:uFRZXykJGK9lLY4HtgSw44DnIcAM+kRBP7x5m+NpAOM= -go.uber.org/zap v1.16.0/go.mod h1:MA8QOfq0BHJwdXa996Y4dYkAqRKB8/1K1QMMZVaNZjQ= -golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= -golang.org/x/crypto v0.0.0-20181203042331-505ab145d0a9/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= -golang.org/x/crypto v0.0.0-20190211182817-74369b46fc67/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= -golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2 h1:VklqNMn3ovrHsnt90PveolxSbWFaJdECFbxSq0Mqo2M= -golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= -golang.org/x/crypto v0.0.0-20190320223903-b7391e95e576/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= -golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= -golang.org/x/crypto v0.0.0-20190611184440-5c40567a22f8 h1:1wopBVtVdWnn03fZelqdXTqk7U7zPQCb+T4rbU9ZEoU= -golang.org/x/crypto v0.0.0-20190611184440-5c40567a22f8/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= -golang.org/x/crypto v0.0.0-20190617133340-57b3e21c3d56/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= -golang.org/x/crypto v0.0.0-20190820162420-60c769a6c586/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= -golang.org/x/crypto v0.0.0-20200220183623-bac4c82f6975/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= -golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9 h1:psW17arqaxU48Z5kZ0CQnkZWQJsqcURM6tKiBApRjXI= -golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= -golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= -golang.org/x/exp v0.0.0-20190125153040-c74c464bbbf2/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= -golang.org/x/exp v0.0.0-20190312203227-4b39c73a6495/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8= -golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js= -golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= -golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU= -golang.org/x/lint v0.0.0-20190301231843-5614ed5bae6f/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= -golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= -golang.org/x/lint v0.0.0-20190930215403-16217165b5de h1:5hukYrvBGR8/eNkX5mdUezrA6JiaEZDtJb9Ei+1LlBs= -golang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= -golang.org/x/mobile v0.0.0-20190312151609-d3739f865fa6/go.mod h1:z+o9i4GpDbdi3rU15maQ/Ox0txvL9dWGYEHz965HBQE= -golang.org/x/mod v0.0.0-20190513183733-4bf6d317e70e/go.mod h1:mXi4GBBbnImb6dmsKGUJ2LatrhH/nqhxcFungHvyanc= -golang.org/x/net v0.0.0-20170114055629-f2499483f923/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= -golang.org/x/net v0.0.0-20180112015858-5ccada7d0a7b/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= -golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= -golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= -golang.org/x/net v0.0.0-20180906233101-161cd47e91fd h1:nTDtHvHSdCn1m6ITfMRqtOd/9+7a3s8RBNOZ3eYZzJA= -golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= -golang.org/x/net v0.0.0-20181005035420-146acd28ed58/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= -golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= -golang.org/x/net v0.0.0-20181201002055-351d144fa1fc/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= -golang.org/x/net v0.0.0-20181220203305-927f97764cc3/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= -golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= -golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= -golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= -golang.org/x/net v0.0.0-20190320064053-1272bf9dcd53/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= -golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= -golang.org/x/net v0.0.0-20190613194153-d28f0bde5980/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= -golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= -golang.org/x/net v0.0.0-20190812203447-cdfb69ac37fc/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= -golang.org/x/net v0.0.0-20190813141303-74dc4d7220e7/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= -golang.org/x/net v0.0.0-20190827160401-ba9fcec4b297/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= -golang.org/x/net v0.0.0-20191004110552-13f9640d40b9/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= -golang.org/x/net v0.0.0-20200202094626-16171245cfb2 h1:CCH4IOTTfewWjGOlSp+zGcjutRKlBEZQ6wTn8ozI/nI= -golang.org/x/net v0.0.0-20200202094626-16171245cfb2/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= -golang.org/x/net v0.0.0-20200520004742-59133d7f0dd7/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A= -golang.org/x/net v0.0.0-20201110031124-69a78807bb2b/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= -golang.org/x/net v0.0.0-20201202161906-c7110b5ffcbb h1:eBmm0M9fYhWpKZLjQUUKka/LtIxf46G4fxeEz5KJr9U= -golang.org/x/net v0.0.0-20201202161906-c7110b5ffcbb/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= -golang.org/x/net v0.0.0-20201216054612-986b41b23924 h1:QsnDpLLOKwHBBDa8nDws4DYNc/ryVW2vCpxCs09d4PY= -golang.org/x/net v0.0.0-20201216054612-986b41b23924/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg= -golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be h1:vEDujvNQGv4jgYKudGeI/+DAX4Jffq6hpD55MmoEvKs= -golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= -golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= -golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45 h1:SVwTIAaPC2U/AvvLNZ2a7OVsmBpC8L5BlwK1whH3hm0= -golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= -golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d h1:TzXSXBo42m9gQenoE3b9BGiEpg5IG2JkU5FkPIawgtw= -golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= -golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sys v0.0.0-20170830134202-bb24a47a89ea/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= -golang.org/x/sys v0.0.0-20180117170059-2c42eef0765b/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= -golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= -golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= -golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e h1:o3PsSEY8E4eXWkXrIP9YJALUkVZqzHJT5DOasTyn8Vs= -golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= -golang.org/x/sys v0.0.0-20181107165924-66b7b1311ac8/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= -golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= -golang.org/x/sys v0.0.0-20181205085412-a5c9d58dba9a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= -golang.org/x/sys v0.0.0-20190209173611-3b5209105503/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= -golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= -golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20190321052220-f7bb7a8bee54/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20190422165155-953cdadca894/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20190616124812-15dcb6c0061f h1:25KHgbfyiSm6vwQLbM3zZIe1v9p/3ea4Rz+nnM5K/i4= -golang.org/x/sys v0.0.0-20190616124812-15dcb6c0061f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20190826190057-c7b8b68b1456/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20190904154756-749cb33beabd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20191005200804-aed5e4c7ecf9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20191120155948-bd437916bb0e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20200519105757-fe76b779f299/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f h1:+Nyd8tzPX9R7BWHguqsrbFdRx3WQ/1ib8I44HXV5yTA= -golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20201119102817-f84b799fce68 h1:nxC68pudNYkKU6jWhgrqdreuFiOQWj1Fs7T3VrH4Pjw= -golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= -golang.org/x/text v0.0.0-20160726164857-2910a502d2bf/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= -golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg= -golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= -golang.org/x/text v0.3.1-0.20171227012246-e19ae1496984/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= -golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= -golang.org/x/text v0.3.2 h1:tW2bmiBqwgJj/UpqtC8EpXEZVYOwU0yG4iWbprSVAcs= -golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk= -golang.org/x/text v0.3.3 h1:cokOdA+Jmi5PJGXLlLllQSgYigAEfHXJAERHVMaCc2k= -golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= -golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= -golang.org/x/time v0.0.0-20181108054448-85acf8d2951c h1:fqgJT0MGcGpPgpWU7VRdRjuArfcOvC4AoJmILihzhDg= -golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= -golang.org/x/time v0.0.0-20190308202827-9d24e82272b4 h1:SvFZT6jyqRaOeXpc5h/JSfZenJ2O330aBsf7JfSUXmQ= -golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= -golang.org/x/time v0.0.0-20200630173020-3af7569d3a1e h1:EHBhcS0mlXEAVwNyO2dLfjToGsyY4j24pTs2ScHnX7s= -golang.org/x/time v0.0.0-20200630173020-3af7569d3a1e/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= -golang.org/x/tools v0.0.0-20180221164845-07fd8470d635/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= -golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= -golang.org/x/tools v0.0.0-20181011042414-1f849cf54d09/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= -golang.org/x/tools v0.0.0-20181030221726-6c7e314b6563/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= -golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= -golang.org/x/tools v0.0.0-20190125232054-d66bd3c5d5a6/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= -golang.org/x/tools v0.0.0-20190206041539-40960b6deb8e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= -golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY= -golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= -golang.org/x/tools v0.0.0-20190312151545-0bb0c0a6e846/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= -golang.org/x/tools v0.0.0-20190312170243-e65039ee4138/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= -golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= -golang.org/x/tools v0.0.0-20190614205625-5aca471b1d59/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= -golang.org/x/tools v0.0.0-20190617190820-da514acc4774/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= -golang.org/x/tools v0.0.0-20190621195816-6e04913cbbac h1:MQEvx39qSf8vyrx3XRaOe+j1UDIzKwkYOVObRgGPVqI= -golang.org/x/tools v0.0.0-20190621195816-6e04913cbbac/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= -golang.org/x/tools v0.0.0-20190920225731-5eefd052ad72/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= -golang.org/x/tools v0.0.0-20191029041327-9cc4af7d6b2c/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= -golang.org/x/tools v0.0.0-20191029190741-b9c20aec41a5 h1:hKsoRgsbwY1NafxrwTs+k64bikrLBkAgPir1TNCj3Zs= -golang.org/x/tools v0.0.0-20191029190741-b9c20aec41a5/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= -golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7 h1:9zdDQZ7Thm29KFXgAX/+yaf3eVbP7djjWp/dXAppNCc= -golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= -golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4= -golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= -gomodules.xyz/jsonpatch/v2 v2.0.1 h1:xyiBuvkD2g5n7cYzx6u2sxQvsAy4QJsZFCzGVdzOXZ0= -gomodules.xyz/jsonpatch/v2 v2.0.1/go.mod h1:IhYNNY4jnS53ZnfE4PAmpKtDpTCj1JFXc+3mwe7XcUU= -gonum.org/v1/gonum v0.0.0-20190331200053-3d26580ed485/go.mod h1:2ltnJ7xHfj0zHS40VVPYEAAMTa3ZGguvHGBSJeRWqE0= -gonum.org/v1/netlib v0.0.0-20190313105609-8cb42192e0e0/go.mod h1:wa6Ws7BG/ESfp6dHfk7C6KdzKA7wR7u/rKwOGE66zvw= -gonum.org/v1/netlib v0.0.0-20190331212654-76723241ea4e/go.mod h1:kS+toOQn6AQKjmKJ7gzohV1XkqsFehRA2FbsbkopSuQ= -google.golang.org/api v0.4.0/go.mod h1:8k5glujaEP+g9n7WNsDg8QP6cUVNI86fCNMcbazEtwE= -google.golang.org/appengine v1.1.0 h1:igQkv0AAhEIvTEpD5LIpAfav2eeVO9HBTjvKHVJPRSs= -google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= -google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= -google.golang.org/appengine v1.5.0 h1:KxkO13IPW4Lslp2bz+KHP2E3gtFlrIGNThxkZQ3g+4c= -google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= -google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc= -google.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= -google.golang.org/genproto v0.0.0-20190418145605-e7d98fc518a7/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= -google.golang.org/genproto v0.0.0-20190502173448-54afdca5d873/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= -google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c= -google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg= -google.golang.org/grpc v1.23.1/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg= -google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8= -google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0= -google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM= -google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE= -google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo= -google.golang.org/protobuf v1.23.0 h1:4MY060fB1DLGMB/7MBTLnwQUY6+F09GEiz6SsrNqyzM= -google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= -gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw= -gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= -gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY= -gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= -gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo= -gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= -gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f h1:BLraFXnmrev5lT+xlilqcH8XK9/i0At2xKjWk4p6zsU= -gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= -gopkg.in/cheggaaa/pb.v1 v1.0.25/go.mod h1:V/YB90LKu/1FcN3WVnfiiE5oMCibMjukxqG/qStrOgw= -gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI= -gopkg.in/fsnotify.v1 v1.4.7 h1:xOHLXZwVvI9hhs+cLKq5+I5onOuwQLhQwiu63xxlHs4= -gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys= -gopkg.in/inf.v0 v0.9.0/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw= -gopkg.in/inf.v0 v0.9.1 h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc= -gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw= -gopkg.in/karlseguin/expect.v1 v1.0.1 h1:9u0iUltnhFbJTHaSIH0EP+cuTU5rafIgmcsEsg2JQFw= -gopkg.in/karlseguin/expect.v1 v1.0.1/go.mod h1:uB7QIJBcclvYbwlUDkSCsGjAOMis3fP280LyhuDEf2I= -gopkg.in/natefinch/lumberjack.v2 v2.0.0/go.mod h1:l0ndWWf7gzL7RNwBG7wST/UCcT4T24xpD6X8LsfU/+k= -gopkg.in/resty.v1 v1.12.0/go.mod h1:mDo4pnntr5jdWRML875a/NmxYqAlA73dVijT2AXvQQo= -gopkg.in/square/go-jose.v2 v2.2.2/go.mod h1:M9dMgbHiYLoDGQrXy7OpJDJWiKiU//h+vD76mk0e1AI= -gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ= -gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw= -gopkg.in/yaml.v2 v2.0.0-20170812160011-eb3733d160e7/go.mod h1:JAlM8MvJe8wmxCU4Bli9HhUf9+ttbYbLASfIpnQbh74= -gopkg.in/yaml.v2 v2.0.0/go.mod h1:JAlM8MvJe8wmxCU4Bli9HhUf9+ttbYbLASfIpnQbh74= -gopkg.in/yaml.v2 v2.2.1 h1:mUhvW9EsL+naU5Q3cakzfE91YhliOondGd6ZrsDBHQE= -gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= -gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw= -gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= -gopkg.in/yaml.v2 v2.2.4 h1:/eiJrUcujPVeJ3xlSWaiNi3uSVmDGBK1pDHUHAnao1I= -gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= -gopkg.in/yaml.v2 v2.2.8 h1:obN1ZagJSUGI0Ek/LBmuj4SNLPfIny3KsKFopxRdj10= -gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= -gopkg.in/yaml.v2 v2.3.0 h1:clyUAQHOM3G0M3f5vQj7LuJrETvjVot3Z5el9nffUtU= -gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= -gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY= -gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ= -gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c h1:dUUwHk2QECo/6vqA44rthZ8ie2QXMNeKRTHCNY2nXvo= -gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= -gotest.tools v2.2.0+incompatible/go.mod h1:DsYFclhRJ6vuDpmuTbkuFWG+y2sxOXAzmJt81HFBacw= -honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= -honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= -honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= -honnef.co/go/tools v0.0.1-2019.2.3 h1:3JgtbtFHMiCmsznwGVTUWbgGov+pVqnlf1dEJTNAXeM= -honnef.co/go/tools v0.0.1-2019.2.3/go.mod h1:a3bituU0lyd329TUQxRnasdCoJDkEUEAqEt0JzvZhAg= -k8s.io/api v0.0.0-20190918155943-95b840bb6a1f/go.mod h1:uWuOHnjmNrtQomJrvEBg0c0HRNyQ+8KTEERVsK0PW48= -k8s.io/api v0.17.2 h1:NF1UFXcKN7/OOv1uxdRz3qfra8AHsPav5M93hlV9+Dc= -k8s.io/api v0.17.2/go.mod h1:BS9fjjLc4CMuqfSO8vgbHPKMt5+SF0ET6u/RVDihTo4= -k8s.io/api v0.17.15 h1:ddnV/lTRb+ihe+eo0K8npm5Ypxi0TayGQHEcJ8TRT2c= -k8s.io/api v0.17.15/go.mod h1:mCepU58Bb3HpTKL9PsivAEq7oeWHTj9eK2Drst9gPKU= -k8s.io/apiextensions-apiserver v0.0.0-20190918161926-8f644eb6e783/go.mod h1:xvae1SZB3E17UpV59AWc271W/Ph25N+bjPyR63X6tPY= -k8s.io/apiextensions-apiserver v0.17.15 h1:VXM8c4Y5xNBVXcIZTGSoxsy+YZ0QIfM58I9e7zY4PB8= -k8s.io/apiextensions-apiserver v0.17.15/go.mod h1:Qk6xT8Lbb88c9reVVWUrQJWwk24iutKH3xWd1iAXvSI= -k8s.io/apimachinery v0.0.0-20190913080033-27d36303b655/go.mod h1:nL6pwRT8NgfF8TT68DBI8uEePRt89cSvoXUVqbkWHq4= -k8s.io/apimachinery v0.17.2 h1:hwDQQFbdRlpnnsR64Asdi55GyCaIP/3WQpMmbNBeWr4= -k8s.io/apimachinery v0.17.2/go.mod h1:b9qmWdKlLuU9EBh+06BtLcSf/Mu89rWL33naRxs1uZg= -k8s.io/apimachinery v0.17.15 h1:Ne6L9kEYiRl1aCKilhzeMFSGH/w14ztxi3vBjoz4osM= -k8s.io/apimachinery v0.17.15/go.mod h1:T54ZSpncArE25c5r2PbUPsLeTpkPWY/ivafigSX6+xk= -k8s.io/apiserver v0.0.0-20190918160949-bfa5e2e684ad/go.mod h1:XPCXEwhjaFN29a8NldXA901ElnKeKLrLtREO9ZhFyhg= -k8s.io/apiserver v0.17.15/go.mod h1:D3U5E/WgntRX0vcPjGW9BInpLypADtb0YMEd257xz6w= -k8s.io/client-go v0.0.0-20190918160344-1fbdaa4c8d90/go.mod h1:J69/JveO6XESwVgG53q3Uz5OSfgsv4uxpScmmyYOOlk= -k8s.io/client-go v0.17.2 h1:ndIfkfXEGrNhLIgkr0+qhRguSD3u6DCmonepn1O6NYc= -k8s.io/client-go v0.17.2/go.mod h1:QAzRgsa0C2xl4/eVpeVAZMvikCn8Nm81yqVx3Kk9XYI= -k8s.io/client-go v0.17.15 h1:fI7P1oJimO2au8eEOAzLKN6iTMA78mLUl+EU85AUz5s= -k8s.io/client-go v0.17.15/go.mod h1:KDGIHDbyT9c20v+rmEdnGHhpQLYkYgQolBCZowcy1VM= -k8s.io/code-generator v0.0.0-20190912054826-cd179ad6a269/go.mod h1:V5BD6M4CyaN5m+VthcclXWsVcT1Hu+glwa1bi3MIsyE= -k8s.io/code-generator v0.17.15/go.mod h1:iiHz51+oTx+Z9D0vB3CH3O4HDDPWrvZyUgUYaIE9h9M= -k8s.io/component-base v0.0.0-20190918160511-547f6c5d7090/go.mod h1:933PBGtQFJky3TEwYx4aEPZ4IxqhWh3R6DCmzqIn1hA= -k8s.io/component-base v0.17.15/go.mod h1:Hi4gj6KS14OpJUtz62ofz5GquCq9qSCHvhn/NLoe8QE= -k8s.io/gengo v0.0.0-20190128074634-0689ccc1d7d6/go.mod h1:ezvh/TsK7cY6rbqRK0oQQ8IAqLxYwwyPxAX1Pzy0ii0= -k8s.io/gengo v0.0.0-20190822140433-26a664648505/go.mod h1:ezvh/TsK7cY6rbqRK0oQQ8IAqLxYwwyPxAX1Pzy0ii0= -k8s.io/klog v0.0.0-20181102134211-b9b56d5dfc92/go.mod h1:Gq+BEi5rUBO/HRz0bTSXDUcqjScdoY3a9IHpCEIOOfk= -k8s.io/klog v0.3.0 h1:0VPpR+sizsiivjIfIAQH/rl8tan6jvWkS7lU+0di3lE= -k8s.io/klog v0.3.0/go.mod h1:Gq+BEi5rUBO/HRz0bTSXDUcqjScdoY3a9IHpCEIOOfk= -k8s.io/klog v0.4.0/go.mod h1:4Bi6QPql/J/LkTDqv7R/cd3hPo4k2DG6Ptcz060Ez5I= -k8s.io/klog v1.0.0 h1:Pt+yjF5aB1xDSVbau4VsWe+dQNzA0qv1LlXdC2dF6Q8= -k8s.io/klog v1.0.0/go.mod h1:4Bi6QPql/J/LkTDqv7R/cd3hPo4k2DG6Ptcz060Ez5I= -k8s.io/kube-openapi v0.0.0-20190816220812-743ec37842bf/go.mod h1:1TqjTSzOxsLGIKfj0lK8EeCP7K1iUG65v09OM0/WG5E= -k8s.io/kube-openapi v0.0.0-20191107075043-30be4d16710a h1:UcxjrRMyNx/i/y8G7kPvLyy7rfbeuf1PYyBf973pgyU= -k8s.io/kube-openapi v0.0.0-20191107075043-30be4d16710a/go.mod h1:1TqjTSzOxsLGIKfj0lK8EeCP7K1iUG65v09OM0/WG5E= -k8s.io/kube-openapi v0.0.0-20200410145947-bcb3869e6f29 h1:NeQXVJ2XFSkRoPzRo8AId01ZER+j8oV4SZADT4iBOXQ= -k8s.io/kube-openapi v0.0.0-20200410145947-bcb3869e6f29/go.mod h1:F+5wygcW0wmRTnM3cOgIqGivxkwSWIWT5YdsDbeAOaU= -k8s.io/utils v0.0.0-20190801114015-581e00157fb1/go.mod h1:sZAwmy6armz5eXlNoLmJcl4F1QuKu7sr+mFQ0byX7Ew= -k8s.io/utils v0.0.0-20191114184206-e782cd3c129f/go.mod h1:sZAwmy6armz5eXlNoLmJcl4F1QuKu7sr+mFQ0byX7Ew= -k8s.io/utils v0.0.0-20191218082557-f07c713de883 h1:TA8t8OLS8m3/0dtTckekO0pCQ7qMnD19fsZTQEgCSKQ= -k8s.io/utils v0.0.0-20191218082557-f07c713de883/go.mod h1:sZAwmy6armz5eXlNoLmJcl4F1QuKu7sr+mFQ0byX7Ew= -modernc.org/cc v1.0.0/go.mod h1:1Sk4//wdnYJiUIxnW8ddKpaOJCF37yAdqYnkxUpaYxw= -modernc.org/golex v1.0.0/go.mod h1:b/QX9oBD/LhixY6NDh+IdGv17hgB+51fET1i2kPSmvk= -modernc.org/mathutil v1.0.0/go.mod h1:wU0vUrJsVWBZ4P6e7xtFJEhFSNsfRLJ8H458uRjg03k= -modernc.org/strutil v1.0.0/go.mod h1:lstksw84oURvj9y3tn8lGvRxyRC1S2+g5uuIzNfIOBs= -modernc.org/xc v1.0.0/go.mod h1:mRNCo0bvLjGhHO9WsyuKVU4q0ceiDDDoEeWDJHrNx8I= -sigs.k8s.io/controller-runtime v0.4.0 h1:wATM6/m+3w8lj8FXNaO6Fs/rq/vqoOjO1Q116Z9NPsg= -sigs.k8s.io/controller-runtime v0.4.0/go.mod h1:ApC79lpY3PHW9xj/w9pj+lYkLgwAAUZwfXkME1Lajns= -sigs.k8s.io/structured-merge-diff v0.0.0-20190525122527-15d366b2352e/go.mod h1:wWxsB5ozmmv/SG7nM11ayaAW51xMvak/t1r0CSlcokI= -sigs.k8s.io/structured-merge-diff v0.0.0-20190817042607-6149e4549fca/go.mod h1:IIgPezJWb76P0hotTxzDbWsMYB8APh18qZnxkomBpxA= -sigs.k8s.io/structured-merge-diff/v2 v2.0.1/go.mod h1:Wb7vfKAodbKgf6tn1Kl0VvGj7mRH6DGaRcixXEJXTsE= -sigs.k8s.io/testing_frameworks v0.1.2 h1:vK0+tvjF0BZ/RYFeZ1E6BYBwHJJXhjuZ3TdsEKH+UQM= -sigs.k8s.io/testing_frameworks v0.1.2/go.mod h1:ToQrwSC3s8Xf/lADdZp3Mktcql9CG0UAmdJG9th5i0w= -sigs.k8s.io/yaml v1.1.0 h1:4A07+ZFc2wgJwo8YNlQpr1rVlgUDlxXHhPJciaPY5gs= -sigs.k8s.io/yaml v1.1.0/go.mod h1:UJmg0vDUVViEyp3mgSv9WPwZCDxu4rQW1olrI1uml+o= diff --git a/hack/boilerplate.go.txt b/hack/boilerplate.go.txt deleted file mode 100644 index b92001fb..00000000 --- a/hack/boilerplate.go.txt +++ /dev/null @@ -1,14 +0,0 @@ -/* - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ \ No newline at end of file diff --git a/main.go b/main.go deleted file mode 100644 index f08b940e..00000000 --- a/main.go +++ /dev/null @@ -1,204 +0,0 @@ -/* - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package main - -import ( - "flag" - "fmt" - "os" - "time" - - "github.com/aws/aws-sdk-go/aws" - "github.com/aws/aws-sdk-go/aws/client" - "github.com/aws/aws-sdk-go/aws/ec2metadata" - "github.com/aws/aws-sdk-go/aws/request" - "github.com/aws/aws-sdk-go/aws/session" - "github.com/aws/aws-sdk-go/service/autoscaling" - "github.com/aws/aws-sdk-go/service/ec2" - "github.com/go-logr/logr" - "github.com/keikoproj/aws-sdk-go-cache/cache" - uberzap "go.uber.org/zap" - "go.uber.org/zap/zapcore" - "k8s.io/apimachinery/pkg/runtime" - _ "k8s.io/client-go/plugin/pkg/client/auth/gcp" - ctrl "sigs.k8s.io/controller-runtime" - "sigs.k8s.io/controller-runtime/pkg/log/zap" - - upgrademgrv1alpha1 "github.com/keikoproj/upgrade-manager/api/v1alpha1" - "github.com/keikoproj/upgrade-manager/controllers" - "github.com/keikoproj/upgrade-manager/pkg/log" - // +kubebuilder:scaffold:imports -) - -var ( - scheme = runtime.NewScheme() - setupLog = ctrl.Log.WithName("setup") -) - -var ( - CacheDefaultTTL = time.Second * 0 - DescribeAutoScalingGroupsTTL = 60 * time.Second - DescribeLaunchTemplatesTTL = 60 * time.Second - CacheMaxItems int64 = 5000 - CacheItemsToPrune uint32 = 500 -) - -func init() { - - err := upgrademgrv1alpha1.AddToScheme(scheme) - if err != nil { - panic(err) - } - // +kubebuilder:scaffold:scheme -} - -func main() { - var metricsAddr string - var enableLeaderElection bool - var namespace string - var maxParallel int - var maxAPIRetries int - var debugMode bool - var logMode string - flag.BoolVar(&debugMode, "debug", false, "enable debug logging") - flag.StringVar(&logMode, "log-format", "text", "Log mode: supported values: text, json.") - flag.StringVar(&metricsAddr, "metrics-addr", ":8080", "The address the metric endpoint binds to.") - flag.BoolVar(&enableLeaderElection, "enable-leader-election", false, - "Enable leader election for controller manager. Enabling this will ensure there is only one active controller manager.") - flag.StringVar(&namespace, "namespace", "", "The namespace in which to watch objects") - flag.IntVar(&maxParallel, "max-parallel", 10, "The max number of parallel rolling upgrades") - flag.IntVar(&maxAPIRetries, "max-api-retries", 12, "The number of maximum retries for failed/rate limited AWS API calls") - flag.Parse() - - ctrl.SetLogger(newLogger(logMode)) - - mgo := ctrl.Options{ - Scheme: scheme, - MetricsBindAddress: metricsAddr, - LeaderElection: enableLeaderElection, - } - if namespace != "" { - mgo.Namespace = namespace - setupLog.Info("Watch RollingUpgrade objects only in namespace " + namespace) - } else { - setupLog.Info("Watch RollingUpgrade objects in all namespaces") - } - mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), mgo) - if err != nil { - setupLog.Error(err, "unable to start manager") - os.Exit(1) - } - - var region string - if region, err = deriveRegion(); err != nil { - setupLog.Error(err, "unable to get region") - os.Exit(1) - } - - if debugMode { - log.SetLevel("debug") - } - - retryer := client.DefaultRetryer{ - NumMaxRetries: maxAPIRetries, - MinThrottleDelay: time.Second * 5, - MaxThrottleDelay: time.Second * 60, - MinRetryDelay: time.Second * 1, - MaxRetryDelay: time.Second * 5, - } - - config := aws.NewConfig().WithRegion(region) - config = config.WithCredentialsChainVerboseErrors(true) - config = request.WithRetryer(config, log.NewRetryLogger(retryer)) - sess, err := session.NewSession(config) - if err != nil { - log.Fatalf("failed to AWS session, %v", err) - } - - cacheCfg := cache.NewConfig(CacheDefaultTTL, CacheMaxItems, CacheItemsToPrune) - cache.AddCaching(sess, cacheCfg) - cacheCfg.SetCacheTTL("autoscaling", "DescribeAutoScalingGroups", DescribeAutoScalingGroupsTTL) - cacheCfg.SetCacheTTL("ec2", "DescribeLaunchTemplates", DescribeLaunchTemplatesTTL) - sess.Handlers.Complete.PushFront(func(r *request.Request) { - ctx := r.HTTPRequest.Context() - log.Debugf("cache hit => %v, service => %s.%s", - cache.IsCacheHit(ctx), - r.ClientInfo.ServiceName, - r.Operation.Name, - ) - }) - - logger := ctrl.Log.WithName("controllers").WithName("RollingUpgrade") - reconciler := &controllers.RollingUpgradeReconciler{ - Client: mgr.GetClient(), - Log: logger, - ClusterState: controllers.NewClusterState(), - ASGClient: autoscaling.New(sess), - EC2Client: ec2.New(sess), - CacheConfig: cacheCfg, - ScriptRunner: controllers.NewScriptRunner(logger), - } - - reconciler.SetMaxParallel(maxParallel) - - err = (reconciler).SetupWithManager(mgr) - if err != nil { - setupLog.Error(err, "unable to create controller", "controller", "RollingUpgrade") - os.Exit(1) - } - // +kubebuilder:scaffold:builder - - setupLog.Info("starting manager") - if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil { - setupLog.Error(err, "problem running manager") - os.Exit(1) - } -} - -func newLogger(logMode string) logr.Logger { - var encoder *zapcore.Encoder = nil - if logMode == "json" { - log.SetJSONFormatter() - jsonEncoder := zapcore.NewJSONEncoder(uberzap.NewProductionEncoderConfig()) - encoder = &jsonEncoder - } - - opts := []zap.Opts{zap.UseDevMode(true), zap.WriteTo(os.Stderr)} - if encoder != nil { - opts = append(opts, zap.Encoder(*encoder)) - } - logger := zap.New(opts...) - return logger -} - -func deriveRegion() (string, error) { - - if region := os.Getenv("AWS_REGION"); region != "" { - return region, nil - } - - var config aws.Config - sess := session.Must(session.NewSessionWithOptions(session.Options{ - SharedConfigState: session.SharedConfigEnable, - Config: config, - })) - c := ec2metadata.New(sess) - region, err := c.Region() - if err != nil { - return "", fmt.Errorf("cannot reach ec2metadata, if running locally export AWS_REGION: %w", err) - } - return region, nil -} diff --git a/pkg/log/log.go b/pkg/log/log.go deleted file mode 100644 index 6f42c02a..00000000 --- a/pkg/log/log.go +++ /dev/null @@ -1,205 +0,0 @@ -package log - -import ( - "github.com/sirupsen/logrus" -) - -// Logger defines a set of methods for writing application logs. -type Logger interface { - Debug(args ...interface{}) - Debugf(format string, args ...interface{}) - Debugln(args ...interface{}) - Error(args ...interface{}) - Errorf(format string, args ...interface{}) - Errorln(args ...interface{}) - Fatal(args ...interface{}) - Fatalf(format string, args ...interface{}) - Fatalln(args ...interface{}) - Info(args ...interface{}) - Infof(format string, args ...interface{}) - Infoln(args ...interface{}) - Panic(args ...interface{}) - Panicf(format string, args ...interface{}) - Panicln(args ...interface{}) - Print(args ...interface{}) - Printf(format string, args ...interface{}) - Println(args ...interface{}) - Warn(args ...interface{}) - Warnf(format string, args ...interface{}) - Warning(args ...interface{}) - Warningf(format string, args ...interface{}) - Warningln(args ...interface{}) - Warnln(args ...interface{}) -} - -var defaultLogger *logrus.Logger - -func init() { - defaultLogger = newLogrusLogger() -} - -func NewLogger() *logrus.Logger { - return newLogrusLogger() -} - -func newLogrusLogger() *logrus.Logger { - l := logrus.New() - l.Level = logrus.InfoLevel - return l -} - -func SetLevel(logLevel string) { - switch logLevel { - case "debug": - defaultLogger.Level = logrus.DebugLevel - case "warning": - defaultLogger.Level = logrus.WarnLevel - case "info": - defaultLogger.Level = logrus.InfoLevel - default: - defaultLogger.Level = logrus.InfoLevel - } -} - -// SetJSONFormatter sets JSON Formatter for default logger. -func SetJSONFormatter() { - defaultLogger.SetFormatter(&logrus.JSONFormatter{}) -} - -type Fields map[string]interface{} - -func (f Fields) With(k string, v interface{}) Fields { - f[k] = v - return f -} - -func (f Fields) WithFields(f2 Fields) Fields { - for k, v := range f2 { - f[k] = v - } - return f -} - -func WithFields(fields Fields) Logger { - return defaultLogger.WithFields(logrus.Fields(fields)) -} - -// Debug package-level convenience method. -func Debug(args ...interface{}) { - defaultLogger.Debug(args...) -} - -// Debugf package-level convenience method. -func Debugf(format string, args ...interface{}) { - defaultLogger.Debugf(format, args...) -} - -// Debugln package-level convenience method. -func Debugln(args ...interface{}) { - defaultLogger.Debugln(args...) -} - -// Error package-level convenience method. -func Error(args ...interface{}) { - defaultLogger.Error(args...) -} - -// Errorf package-level convenience method. -func Errorf(format string, args ...interface{}) { - defaultLogger.Errorf(format, args...) -} - -// Errorln package-level convenience method. -func Errorln(args ...interface{}) { - defaultLogger.Errorln(args...) -} - -// Fatal package-level convenience method. -func Fatal(args ...interface{}) { - defaultLogger.Fatal(args...) -} - -// Fatalf package-level convenience method. -func Fatalf(format string, args ...interface{}) { - defaultLogger.Fatalf(format, args...) -} - -// Fatalln package-level convenience method. -func Fatalln(args ...interface{}) { - defaultLogger.Fatalln(args...) -} - -// Info package-level convenience method. -func Info(args ...interface{}) { - defaultLogger.Info(args...) -} - -// Infof package-level convenience method. -func Infof(format string, args ...interface{}) { - defaultLogger.Infof(format, args...) -} - -// Infoln package-level convenience method. -func Infoln(args ...interface{}) { - defaultLogger.Infoln(args...) -} - -// Panic package-level convenience method. -func Panic(args ...interface{}) { - defaultLogger.Panic(args...) -} - -// Panicf package-level convenience method. -func Panicf(format string, args ...interface{}) { - defaultLogger.Panicf(format, args...) -} - -// Panicln package-level convenience method. -func Panicln(args ...interface{}) { - defaultLogger.Panicln(args...) -} - -// Print package-level convenience method. -func Print(args ...interface{}) { - defaultLogger.Print(args...) -} - -// Printf package-level convenience method. -func Printf(format string, args ...interface{}) { - defaultLogger.Printf(format, args...) -} - -// Println package-level convenience method. -func Println(args ...interface{}) { - defaultLogger.Println(args...) -} - -// Warn package-level convenience method. -func Warn(args ...interface{}) { - defaultLogger.Warn(args...) -} - -// Warnf package-level convenience method. -func Warnf(format string, args ...interface{}) { - defaultLogger.Warnf(format, args...) -} - -// Warning package-level convenience method. -func Warning(args ...interface{}) { - defaultLogger.Warning(args...) -} - -// Warningf package-level convenience method. -func Warningf(format string, args ...interface{}) { - defaultLogger.Warningf(format, args...) -} - -// Warningln package-level convenience method. -func Warningln(args ...interface{}) { - defaultLogger.Warningln(args...) -} - -// Warnln package-level convenience method. -func Warnln(args ...interface{}) { - defaultLogger.Warnln(args...) -} diff --git a/pkg/log/retry_logger.go b/pkg/log/retry_logger.go deleted file mode 100644 index ab7e4cc3..00000000 --- a/pkg/log/retry_logger.go +++ /dev/null @@ -1,44 +0,0 @@ -package log - -import ( - "fmt" - "time" - - "github.com/aws/aws-sdk-go/aws/client" - "github.com/aws/aws-sdk-go/aws/request" -) - -type RetryLogger struct { - client.DefaultRetryer -} - -var _ request.Retryer = &RetryLogger{} - -func NewRetryLogger(retryer client.DefaultRetryer) *RetryLogger { - return &RetryLogger{ - DefaultRetryer: retryer, - } -} - -func (l RetryLogger) RetryRules(r *request.Request) time.Duration { - var ( - duration = l.DefaultRetryer.RetryRules(r) - service = r.ClientInfo.ServiceName - name string - err string - ) - - if r.Operation != nil { - name = r.Operation.Name - } - method := fmt.Sprintf("%v/%v", service, name) - - if r.Error != nil { - err = fmt.Sprintf("%v", r.Error) - } else { - err = fmt.Sprintf("%d %s", r.HTTPResponse.StatusCode, r.HTTPResponse.Status) - } - Infof("retryable: %v -- %v, will retry after %v", err, method, duration) - - return duration -} diff --git a/test-bdd/bases/kustomization.yaml b/test-bdd/bases/kustomization.yaml deleted file mode 100644 index c324cfa7..00000000 --- a/test-bdd/bases/kustomization.yaml +++ /dev/null @@ -1,8 +0,0 @@ -apiVersion: kustomize.config.k8s.io/v1beta1 -kind: Kustomization -resources: -- base_crd-rbac-deployment.yaml -images: - - name: keikoproj/rolling-upgrade-controller - newTag: master - diff --git a/test-bdd/features/01_create.feature b/test-bdd/features/01_create.feature deleted file mode 100644 index fa02af85..00000000 --- a/test-bdd/features/01_create.feature +++ /dev/null @@ -1,18 +0,0 @@ -Feature: UM's RollingUpgrade Create - In order to create RollingUpgrades - As an EKS cluster operator - I need to submit the custom resource - - Background: - Given valid AWS Credentials - And a Kubernetes cluster - And an Auto Scaling Group named upgrademgr-eks-nightly-ASG - - Scenario: The ASG had a launch config update that allows nodes to join - Given the current Auto Scaling Group has the required initial settings - Then 1 node(s) with selector bdd-test=preUpgrade-label should be ready - Given I update the current Auto Scaling Group with LaunchConfigurationName set to upgrade-eks-nightly-LC-postUpgrade - And I submit the resource rolling-upgrade.yaml - Then the resource rolling-upgrade.yaml should be created - When the resource rolling-upgrade.yaml converge to selector .status.currentStatus=completed - Then 1 node(s) with selector bdd-test=postUpgrade-label should be ready diff --git a/test-bdd/main_test.go b/test-bdd/main_test.go deleted file mode 100644 index e436feba..00000000 --- a/test-bdd/main_test.go +++ /dev/null @@ -1,96 +0,0 @@ -package main - -import ( - "os" - "testing" - "time" - - "github.com/cucumber/godog" - kdog "github.com/keikoproj/kubedog" - "github.com/keikoproj/upgrade-manager/pkg/log" -) - -var t kdog.Test - -func TestMain(m *testing.M) { - opts := godog.Options{ - Format: "pretty", - Paths: []string{"features"}, - Randomize: time.Now().UTC().UnixNano(), // randomize scenario execution order - } - - // godog v0.10.0 (latest) - status := godog.TestSuite{ - Name: "godogs", - TestSuiteInitializer: InitializeTestSuite, - ScenarioInitializer: InitializeScenario, - Options: &opts, - }.Run() - - if st := m.Run(); st > status { - status = st - } - os.Exit(status) -} - -func InitializeTestSuite(ctx *godog.TestSuiteContext) { - ctx.BeforeSuite(func() { - log.Info("BDD >> trying to delete any existing test RollingUpgrade") - err := t.KubeContext.DeleteAllTestResources() - if err != nil { - log.Errorf("Failed deleting the test resources: %v", err) - } - }) - - ctx.AfterSuite(func() { - log.Infof("BDD >> scaling down the ASG %v", t.AwsContext.AsgName) - err := t.AwsContext.ScaleCurrentASG(0, 0) - if err != nil { - log.Errorf("Failed scaling down the ASG %v: %v", t.AwsContext.AsgName, err) - } - - log.Info("BDD >> deleting any existing test RollingUpgrade") - err = t.KubeContext.DeleteAllTestResources() - if err != nil { - log.Errorf("Failed deleting the test resources: %v", err) - } - }) - - t.SetTestSuite(ctx) -} - -func InitializeScenario(ctx *godog.ScenarioContext) { - ctx.AfterStep(func(s *godog.Step, err error) { - time.Sleep(time.Second * 5) - }) - - ctx.Step(`^the current Auto Scaling Group has the required initial settings$`, theRequiredInitialSettings) - - t.SetScenario(ctx) - t.Run() -} - -func theRequiredInitialSettings() error { - // Making sure the ASG has the pre-test launch config and 1 node with correct config - err := t.AwsContext.UpdateFieldOfCurrentASG("LaunchConfigurationName", "upgrade-eks-nightly-LC-preUpgrade") - if err != nil { - return err - } - err = t.AwsContext.ScaleCurrentASG(0, 0) - if err != nil { - return err - } - err = t.KubeContext.NodesWithSelectorShouldBe(0, "bdd-test=preUpgrade-label", "found") - if err != nil { - return err - } - err = t.KubeContext.NodesWithSelectorShouldBe(0, "bdd-test=postUpgrade-label", "found") - if err != nil { - return err - } - err = t.AwsContext.ScaleCurrentASG(1, 1) - if err != nil { - return err - } - return nil -} diff --git a/test-bdd/templates/rolling-upgrade.yaml b/test-bdd/templates/rolling-upgrade.yaml deleted file mode 100644 index 6ad892ec..00000000 --- a/test-bdd/templates/rolling-upgrade.yaml +++ /dev/null @@ -1,23 +0,0 @@ -apiVersion: upgrademgr.keikoproj.io/v1alpha1 -kind: RollingUpgrade -metadata: - name: test-rollup - namespace: upgrade-manager-system -spec: - asgName: upgrademgr-eks-nightly-ASG - nodeIntervalSeconds: 90 - postDrain: {} - postDrainDelaySeconds: 30 - postDrainScript: | - echo "Hello, postDrain!" - postDrainWaitScript: | - echo "Hello, postDrainWait!" - postTerminate: {} - postTerminateScript: | - echo "Hello, postTerminate!" - preDrain: {} - strategy: - mode: eager - drainTimeout: 600 - type: randomUpdate - maxUnavailable: 1 \ No newline at end of file From 28164900ddf576c644feb179bee9f4dc37db73a1 Mon Sep 17 00:00:00 2001 From: Eytan Avisror Date: Mon, 11 Jan 2021 21:57:34 -0800 Subject: [PATCH 03/74] scaffolding Signed-off-by: Eytan Avisror --- Dockerfile | 27 + Makefile | 94 +++ PROJECT | 11 + api/v1alpha1/groupversion_info.go | 36 + api/v1alpha1/rollingupgrade_types.go | 64 ++ api/v1alpha1/zz_generated.deepcopy.go | 114 ++++ config/certmanager/certificate.yaml | 25 + config/certmanager/kustomization.yaml | 5 + config/certmanager/kustomizeconfig.yaml | 16 + config/crd/kustomization.yaml | 21 + config/crd/kustomizeconfig.yaml | 19 + .../cainjection_in_rollingupgrades.yaml | 7 + .../patches/webhook_in_rollingupgrades.yaml | 14 + config/default/kustomization.yaml | 74 ++ config/default/manager_auth_proxy_patch.yaml | 26 + config/default/manager_config_patch.yaml | 20 + config/manager/controller_manager_config.yaml | 11 + config/manager/kustomization.yaml | 10 + config/manager/manager.yaml | 55 ++ config/prometheus/kustomization.yaml | 2 + config/prometheus/monitor.yaml | 16 + .../rbac/auth_proxy_client_clusterrole.yaml | 7 + config/rbac/auth_proxy_role.yaml | 13 + config/rbac/auth_proxy_role_binding.yaml | 12 + config/rbac/auth_proxy_service.yaml | 14 + config/rbac/kustomization.yaml | 12 + config/rbac/leader_election_role.yaml | 27 + config/rbac/leader_election_role_binding.yaml | 12 + config/rbac/role_binding.yaml | 12 + config/rbac/rollingupgrade_editor_role.yaml | 24 + config/rbac/rollingupgrade_viewer_role.yaml | 20 + .../upgrademgr_v1alpha1_rollingupgrade.yaml | 7 + controllers/rollingupgrade_controller.go | 63 ++ controllers/suite_test.go | 79 +++ go.mod | 12 + go.sum | 638 ++++++++++++++++++ hack/boilerplate.go.txt | 15 + main.go | 105 +++ 38 files changed, 1739 insertions(+) create mode 100644 Dockerfile create mode 100644 Makefile create mode 100644 PROJECT create mode 100644 api/v1alpha1/groupversion_info.go create mode 100644 api/v1alpha1/rollingupgrade_types.go create mode 100644 api/v1alpha1/zz_generated.deepcopy.go create mode 100644 config/certmanager/certificate.yaml create mode 100644 config/certmanager/kustomization.yaml create mode 100644 config/certmanager/kustomizeconfig.yaml create mode 100644 config/crd/kustomization.yaml create mode 100644 config/crd/kustomizeconfig.yaml create mode 100644 config/crd/patches/cainjection_in_rollingupgrades.yaml create mode 100644 config/crd/patches/webhook_in_rollingupgrades.yaml create mode 100644 config/default/kustomization.yaml create mode 100644 config/default/manager_auth_proxy_patch.yaml create mode 100644 config/default/manager_config_patch.yaml create mode 100644 config/manager/controller_manager_config.yaml create mode 100644 config/manager/kustomization.yaml create mode 100644 config/manager/manager.yaml create mode 100644 config/prometheus/kustomization.yaml create mode 100644 config/prometheus/monitor.yaml create mode 100644 config/rbac/auth_proxy_client_clusterrole.yaml create mode 100644 config/rbac/auth_proxy_role.yaml create mode 100644 config/rbac/auth_proxy_role_binding.yaml create mode 100644 config/rbac/auth_proxy_service.yaml create mode 100644 config/rbac/kustomization.yaml create mode 100644 config/rbac/leader_election_role.yaml create mode 100644 config/rbac/leader_election_role_binding.yaml create mode 100644 config/rbac/role_binding.yaml create mode 100644 config/rbac/rollingupgrade_editor_role.yaml create mode 100644 config/rbac/rollingupgrade_viewer_role.yaml create mode 100644 config/samples/upgrademgr_v1alpha1_rollingupgrade.yaml create mode 100644 controllers/rollingupgrade_controller.go create mode 100644 controllers/suite_test.go create mode 100644 go.mod create mode 100644 go.sum create mode 100644 hack/boilerplate.go.txt create mode 100644 main.go diff --git a/Dockerfile b/Dockerfile new file mode 100644 index 00000000..ce816f3b --- /dev/null +++ b/Dockerfile @@ -0,0 +1,27 @@ +# Build the manager binary +FROM golang:1.15 as builder + +WORKDIR /workspace +# Copy the Go Modules manifests +COPY go.mod go.mod +COPY go.sum go.sum +# cache deps before building and copying source so that we don't need to re-download as much +# and so that source changes don't invalidate our downloaded layer +RUN go mod download + +# Copy the go source +COPY main.go main.go +COPY api/ api/ +COPY controllers/ controllers/ + +# Build +RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GO111MODULE=on go build -a -o manager main.go + +# Use distroless as minimal base image to package the manager binary +# Refer to https://github.com/GoogleContainerTools/distroless for more details +FROM gcr.io/distroless/static:nonroot +WORKDIR / +COPY --from=builder /workspace/manager . +USER 65532:65532 + +ENTRYPOINT ["/manager"] diff --git a/Makefile b/Makefile new file mode 100644 index 00000000..c4f78a33 --- /dev/null +++ b/Makefile @@ -0,0 +1,94 @@ + +# Image URL to use all building/pushing image targets +IMG ?= controller:latest +# Produce CRDs that work back to Kubernetes 1.11 (no version conversion) +CRD_OPTIONS ?= "crd:trivialVersions=true,preserveUnknownFields=false" + +# Get the currently used golang install path (in GOPATH/bin, unless GOBIN is set) +ifeq (,$(shell go env GOBIN)) +GOBIN=$(shell go env GOPATH)/bin +else +GOBIN=$(shell go env GOBIN) +endif + +all: manager + +# Run tests +ENVTEST_ASSETS_DIR=$(shell pwd)/testbin +test: generate fmt vet manifests + mkdir -p ${ENVTEST_ASSETS_DIR} + test -f ${ENVTEST_ASSETS_DIR}/setup-envtest.sh || curl -sSLo ${ENVTEST_ASSETS_DIR}/setup-envtest.sh https://raw.githubusercontent.com/kubernetes-sigs/controller-runtime/v0.7.0/hack/setup-envtest.sh + source ${ENVTEST_ASSETS_DIR}/setup-envtest.sh; fetch_envtest_tools $(ENVTEST_ASSETS_DIR); setup_envtest_env $(ENVTEST_ASSETS_DIR); go test ./... -coverprofile cover.out + +# Build manager binary +manager: generate fmt vet + go build -o bin/manager main.go + +# Run against the configured Kubernetes cluster in ~/.kube/config +run: generate fmt vet manifests + go run ./main.go + +# Install CRDs into a cluster +install: manifests kustomize + $(KUSTOMIZE) build config/crd | kubectl apply -f - + +# Uninstall CRDs from a cluster +uninstall: manifests kustomize + $(KUSTOMIZE) build config/crd | kubectl delete -f - + +# Deploy controller in the configured Kubernetes cluster in ~/.kube/config +deploy: manifests kustomize + cd config/manager && $(KUSTOMIZE) edit set image controller=${IMG} + $(KUSTOMIZE) build config/default | kubectl apply -f - + +# UnDeploy controller from the configured Kubernetes cluster in ~/.kube/config +undeploy: + $(KUSTOMIZE) build config/default | kubectl delete -f - + +# Generate manifests e.g. CRD, RBAC etc. +manifests: controller-gen + $(CONTROLLER_GEN) $(CRD_OPTIONS) rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases + +# Run go fmt against code +fmt: + go fmt ./... + +# Run go vet against code +vet: + go vet ./... + +# Generate code +generate: controller-gen + $(CONTROLLER_GEN) object:headerFile="hack/boilerplate.go.txt" paths="./..." + +# Build the docker image +docker-build: test + docker build -t ${IMG} . + +# Push the docker image +docker-push: + docker push ${IMG} + +# Download controller-gen locally if necessary +CONTROLLER_GEN = $(shell pwd)/bin/controller-gen +controller-gen: + $(call go-get-tool,$(CONTROLLER_GEN),sigs.k8s.io/controller-tools/cmd/controller-gen@v0.4.1) + +# Download kustomize locally if necessary +KUSTOMIZE = $(shell pwd)/bin/kustomize +kustomize: + $(call go-get-tool,$(KUSTOMIZE),sigs.k8s.io/kustomize/kustomize/v3@v3.8.7) + +# go-get-tool will 'go get' any package $2 and install it to $1. +PROJECT_DIR := $(shell dirname $(abspath $(lastword $(MAKEFILE_LIST)))) +define go-get-tool +@[ -f $(1) ] || { \ +set -e ;\ +TMP_DIR=$$(mktemp -d) ;\ +cd $$TMP_DIR ;\ +go mod init tmp ;\ +echo "Downloading $(2)" ;\ +GOBIN=$(PROJECT_DIR)/bin go get $(2) ;\ +rm -rf $$TMP_DIR ;\ +} +endef diff --git a/PROJECT b/PROJECT new file mode 100644 index 00000000..2cbd982d --- /dev/null +++ b/PROJECT @@ -0,0 +1,11 @@ +domain: keikoproj.io +layout: go.kubebuilder.io/v3 +projectName: upgrade-manager +repo: github.com/keikoproj/upgrade-manager +resources: +- api: + crdVersion: v1 + group: upgrademgr + kind: RollingUpgrade + version: v1alpha1 +version: 3-alpha diff --git a/api/v1alpha1/groupversion_info.go b/api/v1alpha1/groupversion_info.go new file mode 100644 index 00000000..22f5fb7c --- /dev/null +++ b/api/v1alpha1/groupversion_info.go @@ -0,0 +1,36 @@ +/* +Copyright 2021 Intuit Inc.. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// Package v1alpha1 contains API Schema definitions for the upgrademgr v1alpha1 API group +//+kubebuilder:object:generate=true +//+groupName=upgrademgr.keikoproj.io +package v1alpha1 + +import ( + "k8s.io/apimachinery/pkg/runtime/schema" + "sigs.k8s.io/controller-runtime/pkg/scheme" +) + +var ( + // GroupVersion is group version used to register these objects + GroupVersion = schema.GroupVersion{Group: "upgrademgr.keikoproj.io", Version: "v1alpha1"} + + // SchemeBuilder is used to add go types to the GroupVersionKind scheme + SchemeBuilder = &scheme.Builder{GroupVersion: GroupVersion} + + // AddToScheme adds the types in this group-version to the given scheme. + AddToScheme = SchemeBuilder.AddToScheme +) diff --git a/api/v1alpha1/rollingupgrade_types.go b/api/v1alpha1/rollingupgrade_types.go new file mode 100644 index 00000000..caa5622a --- /dev/null +++ b/api/v1alpha1/rollingupgrade_types.go @@ -0,0 +1,64 @@ +/* +Copyright 2021 Intuit Inc.. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package v1alpha1 + +import ( + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" +) + +// EDIT THIS FILE! THIS IS SCAFFOLDING FOR YOU TO OWN! +// NOTE: json tags are required. Any new fields you add must have json tags for the fields to be serialized. + +// RollingUpgradeSpec defines the desired state of RollingUpgrade +type RollingUpgradeSpec struct { + // INSERT ADDITIONAL SPEC FIELDS - desired state of cluster + // Important: Run "make" to regenerate code after modifying this file + + // Foo is an example field of RollingUpgrade. Edit rollingupgrade_types.go to remove/update + Foo string `json:"foo,omitempty"` +} + +// RollingUpgradeStatus defines the observed state of RollingUpgrade +type RollingUpgradeStatus struct { + // INSERT ADDITIONAL STATUS FIELD - define observed state of cluster + // Important: Run "make" to regenerate code after modifying this file +} + +//+kubebuilder:object:root=true +//+kubebuilder:subresource:status + +// RollingUpgrade is the Schema for the rollingupgrades API +type RollingUpgrade struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + Spec RollingUpgradeSpec `json:"spec,omitempty"` + Status RollingUpgradeStatus `json:"status,omitempty"` +} + +//+kubebuilder:object:root=true + +// RollingUpgradeList contains a list of RollingUpgrade +type RollingUpgradeList struct { + metav1.TypeMeta `json:",inline"` + metav1.ListMeta `json:"metadata,omitempty"` + Items []RollingUpgrade `json:"items"` +} + +func init() { + SchemeBuilder.Register(&RollingUpgrade{}, &RollingUpgradeList{}) +} diff --git a/api/v1alpha1/zz_generated.deepcopy.go b/api/v1alpha1/zz_generated.deepcopy.go new file mode 100644 index 00000000..4bba20d9 --- /dev/null +++ b/api/v1alpha1/zz_generated.deepcopy.go @@ -0,0 +1,114 @@ +// +build !ignore_autogenerated + +/* +Copyright 2021 Intuit Inc.. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// Code generated by controller-gen. DO NOT EDIT. + +package v1alpha1 + +import ( + runtime "k8s.io/apimachinery/pkg/runtime" +) + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *RollingUpgrade) DeepCopyInto(out *RollingUpgrade) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ObjectMeta.DeepCopyInto(&out.ObjectMeta) + out.Spec = in.Spec + out.Status = in.Status +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RollingUpgrade. +func (in *RollingUpgrade) DeepCopy() *RollingUpgrade { + if in == nil { + return nil + } + out := new(RollingUpgrade) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *RollingUpgrade) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *RollingUpgradeList) DeepCopyInto(out *RollingUpgradeList) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ListMeta.DeepCopyInto(&out.ListMeta) + if in.Items != nil { + in, out := &in.Items, &out.Items + *out = make([]RollingUpgrade, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RollingUpgradeList. +func (in *RollingUpgradeList) DeepCopy() *RollingUpgradeList { + if in == nil { + return nil + } + out := new(RollingUpgradeList) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *RollingUpgradeList) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *RollingUpgradeSpec) DeepCopyInto(out *RollingUpgradeSpec) { + *out = *in +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RollingUpgradeSpec. +func (in *RollingUpgradeSpec) DeepCopy() *RollingUpgradeSpec { + if in == nil { + return nil + } + out := new(RollingUpgradeSpec) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *RollingUpgradeStatus) DeepCopyInto(out *RollingUpgradeStatus) { + *out = *in +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RollingUpgradeStatus. +func (in *RollingUpgradeStatus) DeepCopy() *RollingUpgradeStatus { + if in == nil { + return nil + } + out := new(RollingUpgradeStatus) + in.DeepCopyInto(out) + return out +} diff --git a/config/certmanager/certificate.yaml b/config/certmanager/certificate.yaml new file mode 100644 index 00000000..52d86618 --- /dev/null +++ b/config/certmanager/certificate.yaml @@ -0,0 +1,25 @@ +# The following manifests contain a self-signed issuer CR and a certificate CR. +# More document can be found at https://docs.cert-manager.io +# WARNING: Targets CertManager v1.0. Check https://cert-manager.io/docs/installation/upgrading/ for breaking changes. +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: selfsigned-issuer + namespace: system +spec: + selfSigned: {} +--- +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: serving-cert # this name should match the one appeared in kustomizeconfig.yaml + namespace: system +spec: + # $(SERVICE_NAME) and $(SERVICE_NAMESPACE) will be substituted by kustomize + dnsNames: + - $(SERVICE_NAME).$(SERVICE_NAMESPACE).svc + - $(SERVICE_NAME).$(SERVICE_NAMESPACE).svc.cluster.local + issuerRef: + kind: Issuer + name: selfsigned-issuer + secretName: webhook-server-cert # this secret will not be prefixed, since it's not managed by kustomize diff --git a/config/certmanager/kustomization.yaml b/config/certmanager/kustomization.yaml new file mode 100644 index 00000000..bebea5a5 --- /dev/null +++ b/config/certmanager/kustomization.yaml @@ -0,0 +1,5 @@ +resources: +- certificate.yaml + +configurations: +- kustomizeconfig.yaml diff --git a/config/certmanager/kustomizeconfig.yaml b/config/certmanager/kustomizeconfig.yaml new file mode 100644 index 00000000..90d7c313 --- /dev/null +++ b/config/certmanager/kustomizeconfig.yaml @@ -0,0 +1,16 @@ +# This configuration is for teaching kustomize how to update name ref and var substitution +nameReference: +- kind: Issuer + group: cert-manager.io + fieldSpecs: + - kind: Certificate + group: cert-manager.io + path: spec/issuerRef/name + +varReference: +- kind: Certificate + group: cert-manager.io + path: spec/commonName +- kind: Certificate + group: cert-manager.io + path: spec/dnsNames diff --git a/config/crd/kustomization.yaml b/config/crd/kustomization.yaml new file mode 100644 index 00000000..7b0bdd48 --- /dev/null +++ b/config/crd/kustomization.yaml @@ -0,0 +1,21 @@ +# This kustomization.yaml is not intended to be run by itself, +# since it depends on service name and namespace that are out of this kustomize package. +# It should be run by config/default +resources: +- bases/upgrademgr.keikoproj.io_rollingupgrades.yaml +#+kubebuilder:scaffold:crdkustomizeresource + +patchesStrategicMerge: +# [WEBHOOK] To enable webhook, uncomment all the sections with [WEBHOOK] prefix. +# patches here are for enabling the conversion webhook for each CRD +#- patches/webhook_in_rollingupgrades.yaml +#+kubebuilder:scaffold:crdkustomizewebhookpatch + +# [CERTMANAGER] To enable webhook, uncomment all the sections with [CERTMANAGER] prefix. +# patches here are for enabling the CA injection for each CRD +#- patches/cainjection_in_rollingupgrades.yaml +#+kubebuilder:scaffold:crdkustomizecainjectionpatch + +# the following config is for teaching kustomize how to do kustomization for CRDs. +configurations: +- kustomizeconfig.yaml diff --git a/config/crd/kustomizeconfig.yaml b/config/crd/kustomizeconfig.yaml new file mode 100644 index 00000000..ec5c150a --- /dev/null +++ b/config/crd/kustomizeconfig.yaml @@ -0,0 +1,19 @@ +# This file is for teaching kustomize how to substitute name and namespace reference in CRD +nameReference: +- kind: Service + version: v1 + fieldSpecs: + - kind: CustomResourceDefinition + version: v1 + group: apiextensions.k8s.io + path: spec/conversion/webhook/clientConfig/service/name + +namespace: +- kind: CustomResourceDefinition + version: v1 + group: apiextensions.k8s.io + path: spec/conversion/webhook/clientConfig/service/namespace + create: false + +varReference: +- path: metadata/annotations diff --git a/config/crd/patches/cainjection_in_rollingupgrades.yaml b/config/crd/patches/cainjection_in_rollingupgrades.yaml new file mode 100644 index 00000000..0aea76d9 --- /dev/null +++ b/config/crd/patches/cainjection_in_rollingupgrades.yaml @@ -0,0 +1,7 @@ +# The following patch adds a directive for certmanager to inject CA into the CRD +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + cert-manager.io/inject-ca-from: $(CERTIFICATE_NAMESPACE)/$(CERTIFICATE_NAME) + name: rollingupgrades.upgrademgr.keikoproj.io diff --git a/config/crd/patches/webhook_in_rollingupgrades.yaml b/config/crd/patches/webhook_in_rollingupgrades.yaml new file mode 100644 index 00000000..93193e9f --- /dev/null +++ b/config/crd/patches/webhook_in_rollingupgrades.yaml @@ -0,0 +1,14 @@ +# The following patch enables a conversion webhook for the CRD +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: rollingupgrades.upgrademgr.keikoproj.io +spec: + conversion: + strategy: Webhook + webhook: + clientConfig: + service: + namespace: system + name: webhook-service + path: /convert diff --git a/config/default/kustomization.yaml b/config/default/kustomization.yaml new file mode 100644 index 00000000..79e04547 --- /dev/null +++ b/config/default/kustomization.yaml @@ -0,0 +1,74 @@ +# Adds namespace to all resources. +namespace: test-system + +# Value of this field is prepended to the +# names of all resources, e.g. a deployment named +# "wordpress" becomes "alices-wordpress". +# Note that it should also match with the prefix (text before '-') of the namespace +# field above. +namePrefix: test- + +# Labels to add to all resources and selectors. +#commonLabels: +# someName: someValue + +bases: +- ../crd +- ../rbac +- ../manager +# [WEBHOOK] To enable webhook, uncomment all the sections with [WEBHOOK] prefix including the one in +# crd/kustomization.yaml +#- ../webhook +# [CERTMANAGER] To enable cert-manager, uncomment all sections with 'CERTMANAGER'. 'WEBHOOK' components are required. +#- ../certmanager +# [PROMETHEUS] To enable prometheus monitor, uncomment all sections with 'PROMETHEUS'. +#- ../prometheus + +patchesStrategicMerge: +# Protect the /metrics endpoint by putting it behind auth. +# If you want your controller-manager to expose the /metrics +# endpoint w/o any authn/z, please comment the following line. +- manager_auth_proxy_patch.yaml + +# Mount the controller config file for loading manager configurations +# through a ComponentConfig type +#- manager_config_patch.yaml + +# [WEBHOOK] To enable webhook, uncomment all the sections with [WEBHOOK] prefix including the one in +# crd/kustomization.yaml +#- manager_webhook_patch.yaml + +# [CERTMANAGER] To enable cert-manager, uncomment all sections with 'CERTMANAGER'. +# Uncomment 'CERTMANAGER' sections in crd/kustomization.yaml to enable the CA injection in the admission webhooks. +# 'CERTMANAGER' needs to be enabled to use ca injection +#- webhookcainjection_patch.yaml + +# the following config is for teaching kustomize how to do var substitution +vars: +# [CERTMANAGER] To enable cert-manager, uncomment all sections with 'CERTMANAGER' prefix. +#- name: CERTIFICATE_NAMESPACE # namespace of the certificate CR +# objref: +# kind: Certificate +# group: cert-manager.io +# version: v1 +# name: serving-cert # this name should match the one in certificate.yaml +# fieldref: +# fieldpath: metadata.namespace +#- name: CERTIFICATE_NAME +# objref: +# kind: Certificate +# group: cert-manager.io +# version: v1 +# name: serving-cert # this name should match the one in certificate.yaml +#- name: SERVICE_NAMESPACE # namespace of the service +# objref: +# kind: Service +# version: v1 +# name: webhook-service +# fieldref: +# fieldpath: metadata.namespace +#- name: SERVICE_NAME +# objref: +# kind: Service +# version: v1 +# name: webhook-service diff --git a/config/default/manager_auth_proxy_patch.yaml b/config/default/manager_auth_proxy_patch.yaml new file mode 100644 index 00000000..49b1f1ab --- /dev/null +++ b/config/default/manager_auth_proxy_patch.yaml @@ -0,0 +1,26 @@ +# This patch inject a sidecar container which is a HTTP proxy for the +# controller manager, it performs RBAC authorization against the Kubernetes API using SubjectAccessReviews. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: controller-manager + namespace: system +spec: + template: + spec: + containers: + - name: kube-rbac-proxy + image: gcr.io/kubebuilder/kube-rbac-proxy:v0.5.0 + args: + - "--secure-listen-address=0.0.0.0:8443" + - "--upstream=http://127.0.0.1:8080/" + - "--logtostderr=true" + - "--v=10" + ports: + - containerPort: 8443 + name: https + - name: manager + args: + - "--health-probe-bind-address=:8081" + - "--metrics-bind-address=127.0.0.1:8080" + - "--leader-elect" diff --git a/config/default/manager_config_patch.yaml b/config/default/manager_config_patch.yaml new file mode 100644 index 00000000..6c400155 --- /dev/null +++ b/config/default/manager_config_patch.yaml @@ -0,0 +1,20 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: controller-manager + namespace: system +spec: + template: + spec: + containers: + - name: manager + args: + - "--config=controller_manager_config.yaml" + volumeMounts: + - name: manager-config + mountPath: /controller_manager_config.yaml + subPath: controller_manager_config.yaml + volumes: + - name: manager-config + configMap: + name: manager-config diff --git a/config/manager/controller_manager_config.yaml b/config/manager/controller_manager_config.yaml new file mode 100644 index 00000000..79b2b804 --- /dev/null +++ b/config/manager/controller_manager_config.yaml @@ -0,0 +1,11 @@ +apiVersion: controller-runtime.sigs.k8s.io/v1alpha1 +kind: ControllerManagerConfig +health: + healthProbeBindAddress: :8081 +metrics: + bindAddress: 127.0.0.1:8080 +webhook: + port: 9443 +leaderElection: + leaderElect: true + resourceName: d6edb06e.keikoproj.io diff --git a/config/manager/kustomization.yaml b/config/manager/kustomization.yaml new file mode 100644 index 00000000..2bcd3eea --- /dev/null +++ b/config/manager/kustomization.yaml @@ -0,0 +1,10 @@ +resources: +- manager.yaml + +generatorOptions: + disableNameSuffixHash: true + +configMapGenerator: +- name: manager-config + files: + - controller_manager_config.yaml diff --git a/config/manager/manager.yaml b/config/manager/manager.yaml new file mode 100644 index 00000000..70e3f6ac --- /dev/null +++ b/config/manager/manager.yaml @@ -0,0 +1,55 @@ +apiVersion: v1 +kind: Namespace +metadata: + labels: + control-plane: controller-manager + name: system +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: controller-manager + namespace: system + labels: + control-plane: controller-manager +spec: + selector: + matchLabels: + control-plane: controller-manager + replicas: 1 + template: + metadata: + labels: + control-plane: controller-manager + spec: + securityContext: + runAsUser: 65532 + containers: + - command: + - /manager + args: + - --leader-elect + image: controller:latest + name: manager + securityContext: + allowPrivilegeEscalation: false + livenessProbe: + httpGet: + path: /healthz + port: 8081 + initialDelaySeconds: 15 + periodSeconds: 20 + readinessProbe: + httpGet: + path: /readyz + port: 8081 + initialDelaySeconds: 5 + periodSeconds: 10 + resources: + limits: + cpu: 100m + memory: 30Mi + requests: + cpu: 100m + memory: 20Mi + terminationGracePeriodSeconds: 10 diff --git a/config/prometheus/kustomization.yaml b/config/prometheus/kustomization.yaml new file mode 100644 index 00000000..ed137168 --- /dev/null +++ b/config/prometheus/kustomization.yaml @@ -0,0 +1,2 @@ +resources: +- monitor.yaml diff --git a/config/prometheus/monitor.yaml b/config/prometheus/monitor.yaml new file mode 100644 index 00000000..9b8047b7 --- /dev/null +++ b/config/prometheus/monitor.yaml @@ -0,0 +1,16 @@ + +# Prometheus Monitor Service (Metrics) +apiVersion: monitoring.coreos.com/v1 +kind: ServiceMonitor +metadata: + labels: + control-plane: controller-manager + name: controller-manager-metrics-monitor + namespace: system +spec: + endpoints: + - path: /metrics + port: https + selector: + matchLabels: + control-plane: controller-manager diff --git a/config/rbac/auth_proxy_client_clusterrole.yaml b/config/rbac/auth_proxy_client_clusterrole.yaml new file mode 100644 index 00000000..bd4af137 --- /dev/null +++ b/config/rbac/auth_proxy_client_clusterrole.yaml @@ -0,0 +1,7 @@ +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: metrics-reader +rules: +- nonResourceURLs: ["/metrics"] + verbs: ["get"] diff --git a/config/rbac/auth_proxy_role.yaml b/config/rbac/auth_proxy_role.yaml new file mode 100644 index 00000000..618f5e41 --- /dev/null +++ b/config/rbac/auth_proxy_role.yaml @@ -0,0 +1,13 @@ +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: proxy-role +rules: +- apiGroups: ["authentication.k8s.io"] + resources: + - tokenreviews + verbs: ["create"] +- apiGroups: ["authorization.k8s.io"] + resources: + - subjectaccessreviews + verbs: ["create"] diff --git a/config/rbac/auth_proxy_role_binding.yaml b/config/rbac/auth_proxy_role_binding.yaml new file mode 100644 index 00000000..48ed1e4b --- /dev/null +++ b/config/rbac/auth_proxy_role_binding.yaml @@ -0,0 +1,12 @@ +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: proxy-rolebinding +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: proxy-role +subjects: +- kind: ServiceAccount + name: default + namespace: system diff --git a/config/rbac/auth_proxy_service.yaml b/config/rbac/auth_proxy_service.yaml new file mode 100644 index 00000000..6cf656be --- /dev/null +++ b/config/rbac/auth_proxy_service.yaml @@ -0,0 +1,14 @@ +apiVersion: v1 +kind: Service +metadata: + labels: + control-plane: controller-manager + name: controller-manager-metrics-service + namespace: system +spec: + ports: + - name: https + port: 8443 + targetPort: https + selector: + control-plane: controller-manager diff --git a/config/rbac/kustomization.yaml b/config/rbac/kustomization.yaml new file mode 100644 index 00000000..66c28338 --- /dev/null +++ b/config/rbac/kustomization.yaml @@ -0,0 +1,12 @@ +resources: +- role.yaml +- role_binding.yaml +- leader_election_role.yaml +- leader_election_role_binding.yaml +# Comment the following 4 lines if you want to disable +# the auth proxy (https://github.com/brancz/kube-rbac-proxy) +# which protects your /metrics endpoint. +- auth_proxy_service.yaml +- auth_proxy_role.yaml +- auth_proxy_role_binding.yaml +- auth_proxy_client_clusterrole.yaml diff --git a/config/rbac/leader_election_role.yaml b/config/rbac/leader_election_role.yaml new file mode 100644 index 00000000..6334cc51 --- /dev/null +++ b/config/rbac/leader_election_role.yaml @@ -0,0 +1,27 @@ +# permissions to do leader election. +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: leader-election-role +rules: +- apiGroups: + - "" + - coordination.k8s.io + resources: + - configmaps + - leases + verbs: + - get + - list + - watch + - create + - update + - patch + - delete +- apiGroups: + - "" + resources: + - events + verbs: + - create + - patch diff --git a/config/rbac/leader_election_role_binding.yaml b/config/rbac/leader_election_role_binding.yaml new file mode 100644 index 00000000..eed16906 --- /dev/null +++ b/config/rbac/leader_election_role_binding.yaml @@ -0,0 +1,12 @@ +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: leader-election-rolebinding +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: leader-election-role +subjects: +- kind: ServiceAccount + name: default + namespace: system diff --git a/config/rbac/role_binding.yaml b/config/rbac/role_binding.yaml new file mode 100644 index 00000000..8f265870 --- /dev/null +++ b/config/rbac/role_binding.yaml @@ -0,0 +1,12 @@ +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: manager-rolebinding +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: manager-role +subjects: +- kind: ServiceAccount + name: default + namespace: system diff --git a/config/rbac/rollingupgrade_editor_role.yaml b/config/rbac/rollingupgrade_editor_role.yaml new file mode 100644 index 00000000..00ffd93e --- /dev/null +++ b/config/rbac/rollingupgrade_editor_role.yaml @@ -0,0 +1,24 @@ +# permissions for end users to edit rollingupgrades. +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: rollingupgrade-editor-role +rules: +- apiGroups: + - upgrademgr.keikoproj.io + resources: + - rollingupgrades + verbs: + - create + - delete + - get + - list + - patch + - update + - watch +- apiGroups: + - upgrademgr.keikoproj.io + resources: + - rollingupgrades/status + verbs: + - get diff --git a/config/rbac/rollingupgrade_viewer_role.yaml b/config/rbac/rollingupgrade_viewer_role.yaml new file mode 100644 index 00000000..2eb5bcf4 --- /dev/null +++ b/config/rbac/rollingupgrade_viewer_role.yaml @@ -0,0 +1,20 @@ +# permissions for end users to view rollingupgrades. +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: rollingupgrade-viewer-role +rules: +- apiGroups: + - upgrademgr.keikoproj.io + resources: + - rollingupgrades + verbs: + - get + - list + - watch +- apiGroups: + - upgrademgr.keikoproj.io + resources: + - rollingupgrades/status + verbs: + - get diff --git a/config/samples/upgrademgr_v1alpha1_rollingupgrade.yaml b/config/samples/upgrademgr_v1alpha1_rollingupgrade.yaml new file mode 100644 index 00000000..bac61e87 --- /dev/null +++ b/config/samples/upgrademgr_v1alpha1_rollingupgrade.yaml @@ -0,0 +1,7 @@ +apiVersion: upgrademgr.keikoproj.io/v1alpha1 +kind: RollingUpgrade +metadata: + name: rollingupgrade-sample +spec: + # Add fields here + foo: bar diff --git a/controllers/rollingupgrade_controller.go b/controllers/rollingupgrade_controller.go new file mode 100644 index 00000000..351872ec --- /dev/null +++ b/controllers/rollingupgrade_controller.go @@ -0,0 +1,63 @@ +/* +Copyright 2021 Intuit Inc.. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package controllers + +import ( + "context" + + "github.com/go-logr/logr" + "k8s.io/apimachinery/pkg/runtime" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/client" + + upgrademgrv1alpha1 "github.com/keikoproj/upgrade-manager/api/v1alpha1" +) + +// RollingUpgradeReconciler reconciles a RollingUpgrade object +type RollingUpgradeReconciler struct { + client.Client + Log logr.Logger + Scheme *runtime.Scheme +} + +//+kubebuilder:rbac:groups=upgrademgr.keikoproj.io,resources=rollingupgrades,verbs=get;list;watch;create;update;patch;delete +//+kubebuilder:rbac:groups=upgrademgr.keikoproj.io,resources=rollingupgrades/status,verbs=get;update;patch +//+kubebuilder:rbac:groups=upgrademgr.keikoproj.io,resources=rollingupgrades/finalizers,verbs=update + +// Reconcile is part of the main kubernetes reconciliation loop which aims to +// move the current state of the cluster closer to the desired state. +// TODO(user): Modify the Reconcile function to compare the state specified by +// the RollingUpgrade object against the actual cluster state, and then +// perform operations to make the cluster state reflect the state specified by +// the user. +// +// For more details, check Reconcile and its Result here: +// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.7.0/pkg/reconcile +func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { + _ = r.Log.WithValues("rollingupgrade", req.NamespacedName) + + // your logic here + + return ctrl.Result{}, nil +} + +// SetupWithManager sets up the controller with the Manager. +func (r *RollingUpgradeReconciler) SetupWithManager(mgr ctrl.Manager) error { + return ctrl.NewControllerManagedBy(mgr). + For(&upgrademgrv1alpha1.RollingUpgrade{}). + Complete(r) +} diff --git a/controllers/suite_test.go b/controllers/suite_test.go new file mode 100644 index 00000000..75a6ef2f --- /dev/null +++ b/controllers/suite_test.go @@ -0,0 +1,79 @@ +/* +Copyright 2021 Intuit Inc.. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package controllers + +import ( + "path/filepath" + "testing" + + . "github.com/onsi/ginkgo" + . "github.com/onsi/gomega" + "k8s.io/client-go/kubernetes/scheme" + "k8s.io/client-go/rest" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/envtest" + "sigs.k8s.io/controller-runtime/pkg/envtest/printer" + logf "sigs.k8s.io/controller-runtime/pkg/log" + "sigs.k8s.io/controller-runtime/pkg/log/zap" + + upgrademgrv1alpha1 "github.com/keikoproj/upgrade-manager/api/v1alpha1" + //+kubebuilder:scaffold:imports +) + +// These tests use Ginkgo (BDD-style Go testing framework). Refer to +// http://onsi.github.io/ginkgo/ to learn more about Ginkgo. + +var cfg *rest.Config +var k8sClient client.Client +var testEnv *envtest.Environment + +func TestAPIs(t *testing.T) { + RegisterFailHandler(Fail) + + RunSpecsWithDefaultAndCustomReporters(t, + "Controller Suite", + []Reporter{printer.NewlineReporter{}}) +} + +var _ = BeforeSuite(func() { + logf.SetLogger(zap.New(zap.WriteTo(GinkgoWriter), zap.UseDevMode(true))) + + By("bootstrapping test environment") + testEnv = &envtest.Environment{ + CRDDirectoryPaths: []string{filepath.Join("..", "config", "crd", "bases")}, + } + + cfg, err := testEnv.Start() + Expect(err).NotTo(HaveOccurred()) + Expect(cfg).NotTo(BeNil()) + + err = upgrademgrv1alpha1.AddToScheme(scheme.Scheme) + Expect(err).NotTo(HaveOccurred()) + + //+kubebuilder:scaffold:scheme + + k8sClient, err = client.New(cfg, client.Options{Scheme: scheme.Scheme}) + Expect(err).NotTo(HaveOccurred()) + Expect(k8sClient).NotTo(BeNil()) + +}, 60) + +var _ = AfterSuite(func() { + By("tearing down the test environment") + err := testEnv.Stop() + Expect(err).NotTo(HaveOccurred()) +}) diff --git a/go.mod b/go.mod new file mode 100644 index 00000000..a943e11c --- /dev/null +++ b/go.mod @@ -0,0 +1,12 @@ +module github.com/keikoproj/upgrade-manager + +go 1.15 + +require ( + github.com/go-logr/logr v0.3.0 + github.com/onsi/ginkgo v1.14.1 + github.com/onsi/gomega v1.10.2 + k8s.io/apimachinery v0.19.2 + k8s.io/client-go v0.19.2 + sigs.k8s.io/controller-runtime v0.7.0 +) diff --git a/go.sum b/go.sum new file mode 100644 index 00000000..ed5f385e --- /dev/null +++ b/go.sum @@ -0,0 +1,638 @@ +cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= +cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= +cloud.google.com/go v0.38.0/go.mod h1:990N+gfupTy94rShfmMCWGDn0LpTmnzTp2qbd1dvSRU= +cloud.google.com/go v0.44.1/go.mod h1:iSa0KzasP4Uvy3f1mN/7PiObzGgflwredwwASm/v6AU= +cloud.google.com/go v0.44.2/go.mod h1:60680Gw3Yr4ikxnPRS/oxxkBccT6SA1yMk63TGekxKY= +cloud.google.com/go v0.45.1/go.mod h1:RpBamKRgapWJb87xiFSdk4g1CME7QZg3uwTez+TSTjc= +cloud.google.com/go v0.46.3/go.mod h1:a6bKKbmY7er1mI7TEI4lsAkts/mkhTSZK8w33B4RAg0= +cloud.google.com/go v0.51.0 h1:PvKAVQWCtlGUSlZkGW3QLelKaWq7KYv/MW1EboG8bfM= +cloud.google.com/go v0.51.0/go.mod h1:hWtGJ6gnXH+KgDv+V0zFGDvpi07n3z8ZNj3T1RW0Gcw= +cloud.google.com/go/bigquery v1.0.1/go.mod h1:i/xbL2UlR5RvWAURpBYZTtm/cXjCha9lbfbpx4poX+o= +cloud.google.com/go/datastore v1.0.0/go.mod h1:LXYbyblFSglQ5pkeyhO+Qmw7ukd3C+pD7TKLgZqpHYE= +cloud.google.com/go/pubsub v1.0.1/go.mod h1:R0Gpsv3s54REJCy4fxDixWD93lHJMoZTyQ2kNxGRt3I= +cloud.google.com/go/storage v1.0.0/go.mod h1:IhtSnM/ZTZV8YYJWCY8RULGVqBDmpoyjwiyrjsg+URw= +dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU= +github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78/go.mod h1:LmzpDX56iTiv29bbRTIsUNlaFfuhWRQBWjQdVyAevI8= +github.com/Azure/go-autorest/autorest v0.9.0/go.mod h1:xyHB1BMZT0cuDHU7I0+g046+BFDTQ8rEZB0s4Yfa6bI= +github.com/Azure/go-autorest/autorest v0.9.6 h1:5YWtOnckcudzIw8lPPBcWOnmIFWMtHci1ZWAZulMSx0= +github.com/Azure/go-autorest/autorest v0.9.6/go.mod h1:/FALq9T/kS7b5J5qsQ+RSTUdAmGFqi0vUdVNNx8q630= +github.com/Azure/go-autorest/autorest/adal v0.5.0/go.mod h1:8Z9fGy2MpX0PvDjB1pEgQTmVqjGhiHBW7RJJEciWzS0= +github.com/Azure/go-autorest/autorest/adal v0.8.2 h1:O1X4oexUxnZCaEUGsvMnr8ZGj8HI37tNezwY4npRqA0= +github.com/Azure/go-autorest/autorest/adal v0.8.2/go.mod h1:ZjhuQClTqx435SRJ2iMlOxPYt3d2C/T/7TiQCVZSn3Q= +github.com/Azure/go-autorest/autorest/date v0.1.0/go.mod h1:plvfp3oPSKwf2DNjlBjWF/7vwR+cUD/ELuzDCXwHUVA= +github.com/Azure/go-autorest/autorest/date v0.2.0 h1:yW+Zlqf26583pE43KhfnhFcdmSWlm5Ew6bxipnr/tbM= +github.com/Azure/go-autorest/autorest/date v0.2.0/go.mod h1:vcORJHLJEh643/Ioh9+vPmf1Ij9AEBM5FuBIXLmIy0g= +github.com/Azure/go-autorest/autorest/mocks v0.1.0/go.mod h1:OTyCOPRA2IgIlWxVYxBee2F5Gr4kF2zd2J5cFRaIDN0= +github.com/Azure/go-autorest/autorest/mocks v0.2.0/go.mod h1:OTyCOPRA2IgIlWxVYxBee2F5Gr4kF2zd2J5cFRaIDN0= +github.com/Azure/go-autorest/autorest/mocks v0.3.0 h1:qJumjCaCudz+OcqE9/XtEPfvtOjOmKaui4EOpFI6zZc= +github.com/Azure/go-autorest/autorest/mocks v0.3.0/go.mod h1:a8FDP3DYzQ4RYfVAxAN3SVSiiO77gL2j2ronKKP0syM= +github.com/Azure/go-autorest/logger v0.1.0 h1:ruG4BSDXONFRrZZJ2GUXDiUyVpayPmb1GnWeHDdaNKY= +github.com/Azure/go-autorest/logger v0.1.0/go.mod h1:oExouG+K6PryycPJfVSxi/koC6LSNgds39diKLz7Vrc= +github.com/Azure/go-autorest/tracing v0.5.0 h1:TRn4WjSnkcSy5AEG3pnbtFSwNtwzjr4VYyQflFE619k= +github.com/Azure/go-autorest/tracing v0.5.0/go.mod h1:r/s2XiOKccPW3HrqB+W0TQzfbtp2fGCgRFtBroKn4Dk= +github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ= +github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= +github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo= +github.com/NYTimes/gziphandler v0.0.0-20170623195520-56545f4a5d46/go.mod h1:3wb06e3pkSAbeQ52E9H9iFoQsEEwGN64994WTCIhntQ= +github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU= +github.com/PuerkitoBio/purell v1.0.0/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0= +github.com/PuerkitoBio/purell v1.1.0/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0= +github.com/PuerkitoBio/purell v1.1.1/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0= +github.com/PuerkitoBio/urlesc v0.0.0-20160726150825-5bd2802263f2/go.mod h1:uGdkoq3SwY9Y+13GIhn11/XLaGBb4BfwItxLd5jeuXE= +github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578/go.mod h1:uGdkoq3SwY9Y+13GIhn11/XLaGBb4BfwItxLd5jeuXE= +github.com/agnivade/levenshtein v1.0.1/go.mod h1:CURSv5d9Uaml+FovSIICkLbAUZ9S4RqaHDIsdSBg7lM= +github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc= +github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc= +github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0= +github.com/alecthomas/units v0.0.0-20190717042225-c3de453c63f4/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0= +github.com/andreyvit/diff v0.0.0-20170406064948-c7f18ee00883/go.mod h1:rCTlJbsFo29Kk6CurOXKm700vrz8f0KW0JNfpkRJY/8= +github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5doyWs3UAsr3K4I6qtAmlQcZDesFNEHPZAzj8= +github.com/asaskevich/govalidator v0.0.0-20180720115003-f9ffefc3facf/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY= +github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY= +github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q= +github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8= +github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM= +github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw= +github.com/bgentry/speakeasy v0.1.0/go.mod h1:+zsyZBPWlz7T6j88CTgSN5bM796AkVf0kBD4zp0CCIs= +github.com/blang/semver v3.5.0+incompatible/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnwebNt5EWlYSAyrTnjyyk= +github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU= +github.com/cespare/xxhash v1.1.0 h1:a6HrQnmkObjyL+Gs60czilIUGqrzKutQD6XZog3p+ko= +github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc= +github.com/cespare/xxhash/v2 v2.1.1 h1:6MnRN8NT7+YBpUIWxHtefFZOKTAPgGjpQSxqLNn0+qY= +github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= +github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI= +github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI= +github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU= +github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw= +github.com/cockroachdb/datadriven v0.0.0-20190809214429-80d97fb3cbaa/go.mod h1:zn76sxSg3SzpJ0PPJaLDCu+Bu0Lg3sKTORVIj19EIF8= +github.com/coreos/bbolt v1.3.2/go.mod h1:iRUV2dpdMOn7Bo10OQBFzIJO9kkE559Wcmn+qkEiiKk= +github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE= +github.com/coreos/go-oidc v2.1.0+incompatible/go.mod h1:CgnwVTmzoESiwO9qyAFEMiHoZ1nMCKZlZ9V6mm3/LKc= +github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk= +github.com/coreos/go-semver v0.3.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk= +github.com/coreos/go-systemd v0.0.0-20180511133405-39ca1b05acc7/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4= +github.com/coreos/go-systemd v0.0.0-20190321100706-95778dfbb74e/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4= +github.com/coreos/pkg v0.0.0-20160727233714-3ac0863d7acf/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA= +github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA= +github.com/cpuguy83/go-md2man/v2 v2.0.0/go.mod h1:maD7wRr/U5Z6m/iR4s+kqSMx2CaBsrgA7czyZG/E6dU= +github.com/creack/pty v1.1.7/go.mod h1:lj5s0c3V2DBrqTV7llrYr5NG6My20zk30Fl46Y7DoTY= +github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= +github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/dgrijalva/jwt-go v3.2.0+incompatible h1:7qlOGliEKZXTDg6OTjfoBKDXWrumCAMpl/TFQ4/5kLM= +github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ= +github.com/dgryski/go-sip13 v0.0.0-20181026042036-e10d5fee7954/go.mod h1:vAd38F8PWV+bWy6jNmig1y/TA+kYO4g3RSRF0IAv0no= +github.com/docker/go-units v0.3.3/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk= +github.com/docker/go-units v0.4.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk= +github.com/docker/spdystream v0.0.0-20160310174837-449fdfce4d96/go.mod h1:Qh8CwZgvJUkLughtfhJv5dyTYa91l1fOUCrgjqmcifM= +github.com/docopt/docopt-go v0.0.0-20180111231733-ee0de3bc6815/go.mod h1:WwZ+bS3ebgob9U8Nd0kOddGdZWjyMGR8Wziv+TBNwSE= +github.com/dustin/go-humanize v0.0.0-20171111073723-bb3d318650d4/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk= +github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk= +github.com/elazarl/goproxy v0.0.0-20180725130230-947c36da3153/go.mod h1:/Zj4wYkgs4iZTTu3o/KG3Itv/qCCa8VVMlb3i9OVuzc= +github.com/emicklei/go-restful v0.0.0-20170410110728-ff4f55a20633/go.mod h1:otzb+WCGbkyDHkqmQmT5YD2WR4BBwUdeQoFo8l/7tVs= +github.com/emicklei/go-restful v2.9.5+incompatible/go.mod h1:otzb+WCGbkyDHkqmQmT5YD2WR4BBwUdeQoFo8l/7tVs= +github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4= +github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c= +github.com/evanphx/json-patch v4.5.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk= +github.com/evanphx/json-patch v4.9.0+incompatible h1:kLcOMZeuLAJvL2BPWLMIj5oaZQobrkAqrL+WFZwQses= +github.com/evanphx/json-patch v4.9.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk= +github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4= +github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo= +github.com/fsnotify/fsnotify v1.4.9 h1:hsms1Qyu0jgnwNXIxa+/V/PDsU6CfLf6CNO8H7IWoS4= +github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ= +github.com/ghodss/yaml v0.0.0-20150909031657-73d445a93680/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04= +github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04= +github.com/globalsign/mgo v0.0.0-20180905125535-1ca0a4f7cbcb/go.mod h1:xkRDCp4j0OGD1HRkm4kmhM+pmpv3AKq5SU7GMg4oO/Q= +github.com/globalsign/mgo v0.0.0-20181015135952-eeefdecb41b8/go.mod h1:xkRDCp4j0OGD1HRkm4kmhM+pmpv3AKq5SU7GMg4oO/Q= +github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8= +github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as= +github.com/go-kit/kit v0.9.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as= +github.com/go-logfmt/logfmt v0.3.0/go.mod h1:Qt1PoO58o5twSAckw1HlFXLmHsOX5/0LbT9GBnD5lWE= +github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V4qmtdjCk= +github.com/go-logr/logr v0.1.0/go.mod h1:ixOQHD9gLJUVQQ2ZOR7zLEifBX6tGkNJF4QyIY7sIas= +github.com/go-logr/logr v0.2.0/go.mod h1:z6/tIYblkpsD+a4lm/fGIIU9mZ+XfAiaFtq7xTgseGU= +github.com/go-logr/logr v0.3.0 h1:q4c+kbcR0d5rSurhBR8dIgieOaYpXtsdTYfx22Cu6rs= +github.com/go-logr/logr v0.3.0/go.mod h1:z6/tIYblkpsD+a4lm/fGIIU9mZ+XfAiaFtq7xTgseGU= +github.com/go-logr/zapr v0.2.0 h1:v6Ji8yBW77pva6NkJKQdHLAJKrIJKRHz0RXwPqCHSR4= +github.com/go-logr/zapr v0.2.0/go.mod h1:qhKdvif7YF5GI9NWEpyxTSSBdGmzkNguibrdCNVPunU= +github.com/go-openapi/analysis v0.0.0-20180825180245-b006789cd277/go.mod h1:k70tL6pCuVxPJOHXQ+wIac1FUrvNkHolPie/cLEU6hI= +github.com/go-openapi/analysis v0.17.0/go.mod h1:IowGgpVeD0vNm45So8nr+IcQ3pxVtpRoBWb8PVZO0ik= +github.com/go-openapi/analysis v0.18.0/go.mod h1:IowGgpVeD0vNm45So8nr+IcQ3pxVtpRoBWb8PVZO0ik= +github.com/go-openapi/analysis v0.19.2/go.mod h1:3P1osvZa9jKjb8ed2TPng3f0i/UY9snX6gxi44djMjk= +github.com/go-openapi/analysis v0.19.5/go.mod h1:hkEAkxagaIvIP7VTn8ygJNkd4kAYON2rCu0v0ObL0AU= +github.com/go-openapi/errors v0.17.0/go.mod h1:LcZQpmvG4wyF5j4IhA73wkLFQg+QJXOQHVjmcZxhka0= +github.com/go-openapi/errors v0.18.0/go.mod h1:LcZQpmvG4wyF5j4IhA73wkLFQg+QJXOQHVjmcZxhka0= +github.com/go-openapi/errors v0.19.2/go.mod h1:qX0BLWsyaKfvhluLejVpVNwNRdXZhEbTA4kxxpKBC94= +github.com/go-openapi/jsonpointer v0.0.0-20160704185906-46af16f9f7b1/go.mod h1:+35s3my2LFTysnkMfxsJBAMHj/DoqoB9knIWoYG/Vk0= +github.com/go-openapi/jsonpointer v0.17.0/go.mod h1:cOnomiV+CVVwFLk0A/MExoFMjwdsUdVpsRhURCKh+3M= +github.com/go-openapi/jsonpointer v0.18.0/go.mod h1:cOnomiV+CVVwFLk0A/MExoFMjwdsUdVpsRhURCKh+3M= +github.com/go-openapi/jsonpointer v0.19.2/go.mod h1:3akKfEdA7DF1sugOqz1dVQHBcuDBPKZGEoHC/NkiQRg= +github.com/go-openapi/jsonpointer v0.19.3/go.mod h1:Pl9vOtqEWErmShwVjC8pYs9cog34VGT37dQOVbmoatg= +github.com/go-openapi/jsonreference v0.0.0-20160704190145-13c6e3589ad9/go.mod h1:W3Z9FmVs9qj+KR4zFKmDPGiLdk1D9Rlm7cyMvf57TTg= +github.com/go-openapi/jsonreference v0.17.0/go.mod h1:g4xxGn04lDIRh0GJb5QlpE3HfopLOL6uZrK/VgnsK9I= +github.com/go-openapi/jsonreference v0.18.0/go.mod h1:g4xxGn04lDIRh0GJb5QlpE3HfopLOL6uZrK/VgnsK9I= +github.com/go-openapi/jsonreference v0.19.2/go.mod h1:jMjeRr2HHw6nAVajTXJ4eiUwohSTlpa0o73RUL1owJc= +github.com/go-openapi/jsonreference v0.19.3/go.mod h1:rjx6GuL8TTa9VaixXglHmQmIL98+wF9xc8zWvFonSJ8= +github.com/go-openapi/loads v0.17.0/go.mod h1:72tmFy5wsWx89uEVddd0RjRWPZm92WRLhf7AC+0+OOU= +github.com/go-openapi/loads v0.18.0/go.mod h1:72tmFy5wsWx89uEVddd0RjRWPZm92WRLhf7AC+0+OOU= +github.com/go-openapi/loads v0.19.0/go.mod h1:72tmFy5wsWx89uEVddd0RjRWPZm92WRLhf7AC+0+OOU= +github.com/go-openapi/loads v0.19.2/go.mod h1:QAskZPMX5V0C2gvfkGZzJlINuP7Hx/4+ix5jWFxsNPs= +github.com/go-openapi/loads v0.19.4/go.mod h1:zZVHonKd8DXyxyw4yfnVjPzBjIQcLt0CCsn0N0ZrQsk= +github.com/go-openapi/runtime v0.0.0-20180920151709-4f900dc2ade9/go.mod h1:6v9a6LTXWQCdL8k1AO3cvqx5OtZY/Y9wKTgaoP6YRfA= +github.com/go-openapi/runtime v0.19.0/go.mod h1:OwNfisksmmaZse4+gpV3Ne9AyMOlP1lt4sK4FXt0O64= +github.com/go-openapi/runtime v0.19.4/go.mod h1:X277bwSUBxVlCYR3r7xgZZGKVvBd/29gLDlFGtJ8NL4= +github.com/go-openapi/spec v0.0.0-20160808142527-6aced65f8501/go.mod h1:J8+jY1nAiCcj+friV/PDoE1/3eeccG9LYBs0tYvLOWc= +github.com/go-openapi/spec v0.17.0/go.mod h1:XkF/MOi14NmjsfZ8VtAKf8pIlbZzyoTvZsdfssdxcBI= +github.com/go-openapi/spec v0.18.0/go.mod h1:XkF/MOi14NmjsfZ8VtAKf8pIlbZzyoTvZsdfssdxcBI= +github.com/go-openapi/spec v0.19.2/go.mod h1:sCxk3jxKgioEJikev4fgkNmwS+3kuYdJtcsZsD5zxMY= +github.com/go-openapi/spec v0.19.3/go.mod h1:FpwSN1ksY1eteniUU7X0N/BgJ7a4WvBFVA8Lj9mJglo= +github.com/go-openapi/strfmt v0.17.0/go.mod h1:P82hnJI0CXkErkXi8IKjPbNBM6lV6+5pLP5l494TcyU= +github.com/go-openapi/strfmt v0.18.0/go.mod h1:P82hnJI0CXkErkXi8IKjPbNBM6lV6+5pLP5l494TcyU= +github.com/go-openapi/strfmt v0.19.0/go.mod h1:+uW+93UVvGGq2qGaZxdDeJqSAqBqBdl+ZPMF/cC8nDY= +github.com/go-openapi/strfmt v0.19.3/go.mod h1:0yX7dbo8mKIvc3XSKp7MNfxw4JytCfCD6+bY1AVL9LU= +github.com/go-openapi/swag v0.0.0-20160704191624-1d0bd113de87/go.mod h1:DXUve3Dpr1UfpPtxFw+EFuQ41HhCWZfha5jSVRG7C7I= +github.com/go-openapi/swag v0.17.0/go.mod h1:AByQ+nYG6gQg71GINrmuDXCPWdL640yX49/kXLo40Tg= +github.com/go-openapi/swag v0.18.0/go.mod h1:AByQ+nYG6gQg71GINrmuDXCPWdL640yX49/kXLo40Tg= +github.com/go-openapi/swag v0.19.2/go.mod h1:POnQmlKehdgb5mhVOsnJFsivZCEZ/vjK9gh66Z9tfKk= +github.com/go-openapi/swag v0.19.5/go.mod h1:POnQmlKehdgb5mhVOsnJFsivZCEZ/vjK9gh66Z9tfKk= +github.com/go-openapi/validate v0.18.0/go.mod h1:Uh4HdOzKt19xGIGm1qHf/ofbX1YQ4Y+MYsct2VUrAJ4= +github.com/go-openapi/validate v0.19.2/go.mod h1:1tRCw7m3jtI8eNWEEliiAqUIcBztB2KDnRCRMUi7GTA= +github.com/go-openapi/validate v0.19.5/go.mod h1:8DJv2CVJQ6kGNpFW6eV9N3JviE1C85nY1c2z52x1Gk4= +github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY= +github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ= +github.com/gogo/protobuf v1.2.1/go.mod h1:hp+jE20tsWTFYpLwKvXlhS1hjn+gTNwPg2I6zVXpSg4= +github.com/gogo/protobuf v1.3.1 h1:DqDEcV5aeaTmdFBePNpYsp3FlcVH/2ISVVM9Qf8PSls= +github.com/gogo/protobuf v1.3.1/go.mod h1:SlYgWuQ5SjCEi6WLHjHCa1yvBfUnHcTbrrZtXPKa29o= +github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= +github.com/golang/groupcache v0.0.0-20160516000752-02826c3e7903/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= +github.com/golang/groupcache v0.0.0-20190129154638-5b532d6fd5ef/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= +github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= +github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7 h1:5ZkaAPbicIKTF2I64qf5Fh8Aa83Q/dnOafMYV0OMwjA= +github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= +github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= +github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= +github.com/golang/mock v1.3.1/go.mod h1:sBzyDLLjw3U8JLTeZvSv8jJB+tU5PVekmnlKIyFUx0Y= +github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= +github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= +github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= +github.com/golang/protobuf v1.3.3/go.mod h1:vzj43D7+SQXF/4pzW/hwtAqwc6iTitCiVSaWz5lYuqw= +github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8= +github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA= +github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs= +github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w= +github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0= +github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QDs8UjoX8= +github.com/golang/protobuf v1.4.2 h1:+Z5KGCizgyZCbGh1KZqA0fcLLkwbsjIzS4aV2v7wJX0= +github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI= +github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= +github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= +github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M= +github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= +github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= +github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= +github.com/google/go-cmp v0.5.2 h1:X2ev0eStA3AbceY54o37/0PQ/UWqKEiiO2dKL5OPaFM= +github.com/google/go-cmp v0.5.2/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= +github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= +github.com/google/gofuzz v1.1.0 h1:Hsa8mG0dQ46ij8Sl2AYJDUv1oA9/d6Vk+3LG99Oe02g= +github.com/google/gofuzz v1.1.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= +github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs= +github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc= +github.com/google/pprof v0.0.0-20190515194954-54271f7e092f/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc= +github.com/google/pprof v0.0.0-20191218002539-d4f498aebedc/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM= +github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI= +github.com/google/uuid v1.0.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/google/uuid v1.1.1 h1:Gkbcsh/GbpXz7lPftLA3P6TYMwjCLYm83jiFQZF/3gY= +github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg= +github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk= +github.com/googleapis/gnostic v0.4.1/go.mod h1:LRhVm6pbyptWbWbuZ38d1eyptfvIytN3ir6b65WBswg= +github.com/googleapis/gnostic v0.5.1 h1:A8Yhf6EtqTv9RMsU6MQTyrtV1TjWlR6xU9BsZIwuTCM= +github.com/googleapis/gnostic v0.5.1/go.mod h1:6U4PtQXGIEt/Z3h5MAT7FNofLnw9vXk2cUuW7uA/OeU= +github.com/gorilla/websocket v0.0.0-20170926233335-4201258b820c/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ= +github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ= +github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA= +github.com/grpc-ecosystem/go-grpc-middleware v1.0.0/go.mod h1:FiyG127CGDf3tlThmgyCl78X/SZQqEOJBCDaAfeWzPs= +github.com/grpc-ecosystem/go-grpc-middleware v1.0.1-0.20190118093823-f849b5445de4/go.mod h1:FiyG127CGDf3tlThmgyCl78X/SZQqEOJBCDaAfeWzPs= +github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0/go.mod h1:8NvIoxWQoOIhqOTXgfV/d3M/q6VIi02HzZEHgUlZvzk= +github.com/grpc-ecosystem/grpc-gateway v1.9.0/go.mod h1:vNeuVxBJEsws4ogUvrchl83t/GYV9WGTSLVdBhOQFDY= +github.com/grpc-ecosystem/grpc-gateway v1.9.5/go.mod h1:vNeuVxBJEsws4ogUvrchl83t/GYV9WGTSLVdBhOQFDY= +github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= +github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= +github.com/hashicorp/golang-lru v0.5.4 h1:YDjusn29QI/Das2iO9M0BHnIbxPeyuCHsjMW+lJfyTc= +github.com/hashicorp/golang-lru v0.5.4/go.mod h1:iADmTwqILo4mZ8BN3D2Q6+9jd8WM5uGBxy+E8yxSoD4= +github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ= +github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU= +github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc= +github.com/imdario/mergo v0.3.5/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA= +github.com/imdario/mergo v0.3.10 h1:6q5mVkdH/vYmqngx7kZQTjJ5HRsx+ImorDIEQ+beJgc= +github.com/imdario/mergo v0.3.10/go.mod h1:jmQim1M+e3UYxmgPu/WyfjB3N3VflVyUjjjwH0dnCYA= +github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8= +github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo= +github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU= +github.com/json-iterator/go v1.1.7/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4= +github.com/json-iterator/go v1.1.10 h1:Kz6Cvnvv2wGdaG/V8yMvfkmNiXq9Ya2KUv4rouJJr68= +github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4= +github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU= +github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk= +github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w= +github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q= +github.com/kisielk/errcheck v1.2.0/go.mod h1:/BMXB+zMLi60iA8Vv6Ksmxu/1UDYcXs4uQLJ+jE2L00= +github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= +github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ= +github.com/konsorten/go-windows-terminal-sequences v1.0.3/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ= +github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc= +github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= +github.com/kr/pretty v0.2.0 h1:s5hAObm+yFO5uHYt5dYjxi2rXrsnmRpJx4OYvIWUaQs= +github.com/kr/pretty v0.2.0/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI= +github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= +github.com/kr/pty v1.1.5/go.mod h1:9r2w37qlBe7rQ6e1fg1S/9xpWHSnaqNdHD3WcMdbPDA= +github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE= +github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= +github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ= +github.com/mailru/easyjson v0.0.0-20160728113105-d5b7844b561a/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= +github.com/mailru/easyjson v0.0.0-20180823135443-60711f1a8329/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= +github.com/mailru/easyjson v0.0.0-20190312143242-1de009706dbe/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= +github.com/mailru/easyjson v0.0.0-20190614124828-94de47d64c63/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= +github.com/mailru/easyjson v0.0.0-20190626092158-b2ccc519800e/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= +github.com/mailru/easyjson v0.7.0/go.mod h1:KAzv3t3aY1NaHWoQz1+4F1ccyAH66Jk7yos7ldAVICs= +github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU= +github.com/mattn/go-isatty v0.0.4/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNxMWT7Zi4= +github.com/mattn/go-runewidth v0.0.2/go.mod h1:LwmH8dsx7+W8Uxz3IHJYH5QSwggIsqBzpuz5H//U1FU= +github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0= +github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369 h1:I0XW9+e1XWDxdcEniV4rQAIOPUGDq67JSCiRCgGCZLI= +github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369/go.mod h1:BSXmuO+STAnVfrANrmjBb36TMTDstsz7MSK+HVaYKv4= +github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0= +github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y= +github.com/moby/term v0.0.0-20200312100748-672ec06f55cd/go.mod h1:DdlQx2hp0Ss5/fLikoLlEeIYiATotOjgB//nb973jeo= +github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= +github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg= +github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= +github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0= +github.com/modern-go/reflect2 v1.0.1 h1:9f412s+6RmYXLWZSEzVVgPGK7C2PphHj5RJrvfx9AWI= +github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0= +github.com/munnerz/goautoneg v0.0.0-20120707110453-a547fc61f48d/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ= +github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ= +github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U= +github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f/go.mod h1:ZdcZmHo+o7JKHSa8/e818NopupXU1YMK5fe1lsApnBw= +github.com/nxadm/tail v1.4.4 h1:DQuhQpB1tVlglWS2hLQ5OV6B5r8aGxSrPc5Qo6uTN78= +github.com/nxadm/tail v1.4.4/go.mod h1:kenIhsEOeOJmVchQTgglprH7qJGnHDVpk1VPCcaMI8A= +github.com/oklog/ulid v1.3.1/go.mod h1:CirwcVhetQ6Lv90oh/F+FBtV6XMibvdAFo93nm5qn4U= +github.com/olekukonko/tablewriter v0.0.0-20170122224234-a0225b3f23b5/go.mod h1:vsDQFd/mU46D+Z4whnwzcISnGGzXWMclvtLoiIKAKIo= +github.com/onsi/ginkgo v0.0.0-20170829012221-11459a886d9c/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= +github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= +github.com/onsi/ginkgo v1.11.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= +github.com/onsi/ginkgo v1.12.1/go.mod h1:zj2OWP4+oCPe1qIXoGWkgMRwljMUYCdkwsT2108oapk= +github.com/onsi/ginkgo v1.14.1 h1:jMU0WaQrP0a/YAEq8eJmJKjBoMs+pClEr1vDMlM/Do4= +github.com/onsi/ginkgo v1.14.1/go.mod h1:iSB4RoI2tjJc9BBv4NKIKWKya62Rps+oPG/Lv9klQyY= +github.com/onsi/gomega v0.0.0-20170829124025-dcabb60a477c/go.mod h1:C1qb7wdrVGGVU+Z6iS04AVkA3Q65CEZX59MT0QO5uiA= +github.com/onsi/gomega v1.7.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY= +github.com/onsi/gomega v1.7.1/go.mod h1:XdKZgCCFLUoM/7CFJVPcG8C1xQ1AJ0vpAezJrB7JYyY= +github.com/onsi/gomega v1.10.1/go.mod h1:iN09h71vgCQne3DLsj+A5owkum+a2tYe+TOCB1ybHNo= +github.com/onsi/gomega v1.10.2 h1:aY/nuoWlKJud2J6U0E3NWsjlg+0GtwXxgEqthRdzlcs= +github.com/onsi/gomega v1.10.2/go.mod h1:iN09h71vgCQne3DLsj+A5owkum+a2tYe+TOCB1ybHNo= +github.com/pborman/uuid v1.2.0/go.mod h1:X/NO0urCmaxf9VXbdlT7C2Yzkj2IKimNn4k+gtPdI/k= +github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic= +github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU= +github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= +github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= +github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= +github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= +github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= +github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/pquerna/cachecontrol v0.0.0-20171018203845-0dec1b30a021/go.mod h1:prYjPmNq4d1NPVmpShWobRqXY3q7Vp+80DqgxxUrUIA= +github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw= +github.com/prometheus/client_golang v0.9.3/go.mod h1:/TN21ttK/J9q6uSwhBd54HahCDft0ttaMvbicHlPoso= +github.com/prometheus/client_golang v1.0.0/go.mod h1:db9x61etRT2tGnBNRi70OPL5FsnadC4Ky3P0J6CfImo= +github.com/prometheus/client_golang v1.7.1 h1:NTGy1Ja9pByO+xAeH/qiWnLrKtr3hJPNjaVUwnjpdpA= +github.com/prometheus/client_golang v1.7.1/go.mod h1:PY5Wy2awLA44sXw4AOSfFBetzPP4j5+D6mVACh+pe2M= +github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo= +github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= +github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= +github.com/prometheus/client_model v0.2.0 h1:uq5h0d+GuxiXLJLNABMgp2qUWDPiLvgCzz2dUR+/W/M= +github.com/prometheus/client_model v0.2.0/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= +github.com/prometheus/common v0.0.0-20181113130724-41aa239b4cce/go.mod h1:daVV7qP5qjZbuso7PdcryaAu0sAZbrN9i7WWcTMWvro= +github.com/prometheus/common v0.4.0/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4= +github.com/prometheus/common v0.4.1/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4= +github.com/prometheus/common v0.10.0 h1:RyRA7RzGXQZiW+tGMr7sxa85G1z0yOpM1qq5c8lNawc= +github.com/prometheus/common v0.10.0/go.mod h1:Tlit/dnDKsSWFlCLTWaA1cyBgKHSMdTB80sz/V91rCo= +github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk= +github.com/prometheus/procfs v0.0.0-20190507164030-5867b95ac084/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA= +github.com/prometheus/procfs v0.0.2/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA= +github.com/prometheus/procfs v0.1.3 h1:F0+tqvhOksq22sc6iCHF5WGlWjdwj92p0udFh1VFBS8= +github.com/prometheus/procfs v0.1.3/go.mod h1:lV6e/gmhEcM9IjHGsFOCxxuZ+z1YqCvr4OA4YeYWdaU= +github.com/prometheus/tsdb v0.7.1/go.mod h1:qhTCs0VvXwvX/y3TZrWD7rabWM+ijKTux40TwIPHuXU= +github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af/go.mod h1:XWv6SoW27p1b0cqNHllgS5HIMJraePCO15w5zCzIWYg= +github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4= +github.com/russross/blackfriday/v2 v2.0.1/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM= +github.com/sergi/go-diff v1.0.0/go.mod h1:0CfEIISq7TuYL3j771MWULgwwjU+GofnZX9QAmXWZgo= +github.com/shurcooL/sanitized_anchor_name v1.0.0/go.mod h1:1NzhyTcUVG4SuEtjjoZeVRXNmyL/1OwPU0+IJeTBvfc= +github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo= +github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE= +github.com/sirupsen/logrus v1.6.0/go.mod h1:7uNnSEd1DgxDLC74fIahvMZmmYsHGZGEOFrfsX/uA88= +github.com/soheilhy/cmux v0.1.4/go.mod h1:IM3LyeVVIOuxMH7sFAkER9+bJ4dT7Ms6E4xg4kGIyLM= +github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA= +github.com/spf13/afero v1.1.2/go.mod h1:j4pytiNVoe2o6bmDsKpLACNPDBIoEAkihy7loJ1B0CQ= +github.com/spf13/afero v1.2.2/go.mod h1:9ZxEEn6pIJ8Rxe320qSDBk6AsU0r9pR7Q4OcevTdifk= +github.com/spf13/cast v1.3.0/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE= +github.com/spf13/cobra v0.0.3/go.mod h1:1l0Ry5zgKvJasoi3XT1TypsSe7PqH0Sj9dhYf7v3XqQ= +github.com/spf13/cobra v1.0.0/go.mod h1:/6GTrnGXV9HjY+aR4k0oJ5tcvakLuG6EuKReYlHNrgE= +github.com/spf13/jwalterweatherman v1.0.0/go.mod h1:cQK4TGJAtQXfYWX+Ddv3mKDzgVb68N+wFjFa4jdeBTo= +github.com/spf13/pflag v0.0.0-20170130214245-9ff6c6923cff/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4= +github.com/spf13/pflag v1.0.1/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4= +github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4= +github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA= +github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= +github.com/spf13/viper v1.4.0/go.mod h1:PTJ7Z/lr49W6bUbkmS1V3by4uWynFiR9p7+dSq/yZzE= +github.com/stoewer/go-strcase v1.2.0/go.mod h1:IBiWB2sKIp3wVVQ3Y035++gc+knqhUQag1KpM8ahLw8= +github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= +github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= +github.com/stretchr/objx v0.2.0/go.mod h1:qt09Ya8vawLte6SNmTgCsAVtYtaKzEcn8ATUoHMkEqE= +github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= +github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= +github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4= +github.com/stretchr/testify v1.5.1 h1:nOGnQDM7FYENwehXlg/kFVnos3rEvtKTjRvOWSzb6H4= +github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA= +github.com/tidwall/pretty v1.0.0/go.mod h1:XNkn88O1ChpSDQmQeStsy+sBenx6DDtFZJxhVysOjyk= +github.com/tmc/grpc-websocket-proxy v0.0.0-20170815181823-89b8d40f7ca8/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U= +github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U= +github.com/ugorji/go v1.1.4/go.mod h1:uQMGLiO92mf5W77hV/PUCpI3pbzQx3CRekS0kk+RGrc= +github.com/urfave/cli v1.20.0/go.mod h1:70zkFmudgCuE/ngEzBv17Jvp/497gISqfk5gWijbERA= +github.com/vektah/gqlparser v1.1.2/go.mod h1:1ycwN7Ij5njmMkPPAOaRFY4rET2Enx7IkVv3vaXspKw= +github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU= +github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q= +github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= +go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU= +go.etcd.io/bbolt v1.3.3/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU= +go.etcd.io/bbolt v1.3.5/go.mod h1:G5EMThwa9y8QZGBClrRx5EY+Yw9kAhnjy3bSjsnlVTQ= +go.etcd.io/etcd v0.5.0-alpha.5.0.20200819165624-17cef6e3e9d5/go.mod h1:skWido08r9w6Lq/w70DO5XYIKMu4QFu1+4VsqLQuJy8= +go.mongodb.org/mongo-driver v1.0.3/go.mod h1:u7ryQJ+DOzQmeO7zB6MHyr8jkEQvC8vH7qLUO4lqsUM= +go.mongodb.org/mongo-driver v1.1.1/go.mod h1:u7ryQJ+DOzQmeO7zB6MHyr8jkEQvC8vH7qLUO4lqsUM= +go.mongodb.org/mongo-driver v1.1.2/go.mod h1:u7ryQJ+DOzQmeO7zB6MHyr8jkEQvC8vH7qLUO4lqsUM= +go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU= +go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8= +go.opencensus.io v0.22.2/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw= +go.uber.org/atomic v1.3.2/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE= +go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE= +go.uber.org/atomic v1.6.0 h1:Ezj3JGmsOnG1MoRWQkPBsKLe9DwWD9QeXzTRzzldNVk= +go.uber.org/atomic v1.6.0/go.mod h1:sABNBOSYdrvTF6hTgEIbc7YasKWGhgEQZyfxyTvoXHQ= +go.uber.org/goleak v1.1.10 h1:z+mqJhf6ss6BSfSM671tgKyZBFPTTJM+HLxnhPC3wu0= +go.uber.org/goleak v1.1.10/go.mod h1:8a7PlsEVH3e/a/GLqe5IIrQx6GzcnRmZEufDUTk4A7A= +go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0= +go.uber.org/multierr v1.5.0 h1:KCa4XfM8CWFCpxXRGok+Q0SS/0XBhMDbHHGABQLvD2A= +go.uber.org/multierr v1.5.0/go.mod h1:FeouvMocqHpRaaGuG9EjoKcStLC43Zu/fmqdUMPcKYU= +go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee h1:0mgffUl7nfd+FpvXMVz4IDEaUSmT1ysygQC7qYo7sG4= +go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee/go.mod h1:vJERXedbb3MVM5f9Ejo0C68/HhF8uaILCdgjnY+goOA= +go.uber.org/zap v1.8.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q= +go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q= +go.uber.org/zap v1.15.0 h1:ZZCA22JRF2gQE5FoNmhmrf7jeJJ2uhqDUNRYKm8dvmM= +go.uber.org/zap v1.15.0/go.mod h1:Mb2vm2krFEG5DV0W9qcHBYFtp/Wku1cvYaqPsS/WYfc= +golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= +golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= +golang.org/x/crypto v0.0.0-20190320223903-b7391e95e576/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= +golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= +golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= +golang.org/x/crypto v0.0.0-20190611184440-5c40567a22f8/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= +golang.org/x/crypto v0.0.0-20190617133340-57b3e21c3d56/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= +golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= +golang.org/x/crypto v0.0.0-20191206172530-e9b2fee46413/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= +golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9 h1:psW17arqaxU48Z5kZ0CQnkZWQJsqcURM6tKiBApRjXI= +golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= +golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= +golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= +golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8= +golang.org/x/exp v0.0.0-20190829153037-c13cbed26979/go.mod h1:86+5VVa7VpoJ4kLfm080zCjGlMRFzhUhsZKEZO7MGek= +golang.org/x/exp v0.0.0-20191227195350-da58074b4299/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4= +golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js= +golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0= +golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= +golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU= +golang.org/x/lint v0.0.0-20190301231843-5614ed5bae6f/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= +golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= +golang.org/x/lint v0.0.0-20190409202823-959b441ac422/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= +golang.org/x/lint v0.0.0-20190909230951-414d861bb4ac/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= +golang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= +golang.org/x/lint v0.0.0-20191125180803-fdd1cda4f05f h1:J5lckAjkw6qYlOZNj90mLYNTEKDvWeuc1yieZ8qUzUE= +golang.org/x/lint v0.0.0-20191125180803-fdd1cda4f05f/go.mod h1:5qLYkcX4OjUUV8bRuDixDT3tpyyb+LUpUlRWLxfhWrs= +golang.org/x/mobile v0.0.0-20190312151609-d3739f865fa6/go.mod h1:z+o9i4GpDbdi3rU15maQ/Ox0txvL9dWGYEHz965HBQE= +golang.org/x/mobile v0.0.0-20190719004257-d2bd2a29d028/go.mod h1:E/iHnbuqvinMTCcRqshq8CkpyQDoeVncDDYHnLhea+o= +golang.org/x/mod v0.0.0-20190513183733-4bf6d317e70e/go.mod h1:mXi4GBBbnImb6dmsKGUJ2LatrhH/nqhxcFungHvyanc= +golang.org/x/mod v0.1.0/go.mod h1:0QHyrYULN0/3qlju5TqG8bIK38QM8yzMo5ekMj3DlcY= +golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg= +golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= +golang.org/x/mod v0.3.0 h1:RM4zey1++hCTbCVQfnWeKs9/IEsaBLA8vTkd0WVtmH4= +golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= +golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20181005035420-146acd28ed58/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20181220203305-927f97764cc3/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= +golang.org/x/net v0.0.0-20190320064053-1272bf9dcd53/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= +golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= +golang.org/x/net v0.0.0-20190501004415-9ce7a6920f09/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= +golang.org/x/net v0.0.0-20190503192946-f4e77d36d62c/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= +golang.org/x/net v0.0.0-20190522155817-f3200d17e092/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks= +golang.org/x/net v0.0.0-20190603091049-60506f45cf65/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks= +golang.org/x/net v0.0.0-20190613194153-d28f0bde5980/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20190813141303-74dc4d7220e7/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20190827160401-ba9fcec4b297/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20191209160850-c0dbc17a3553/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20200324143707-d3edc9973b7e/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A= +golang.org/x/net v0.0.0-20200520004742-59133d7f0dd7/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A= +golang.org/x/net v0.0.0-20200707034311-ab3426394381 h1:VXak5I6aEWmAXeQjA+QSZzlgNrpq9mjcfDemuexIKsU= +golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA= +golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= +golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= +golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= +golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6 h1:pE8b58s1HRDMi8RDc79m0HISf9D4TzseP40cEA6IGfs= +golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= +golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20181107165924-66b7b1311ac8/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20190321052220-f7bb7a8bee54/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20190422165155-953cdadca894/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20190507160741-ecd444e8653b/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20190606165138-5da285871e9c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20190616124812-15dcb6c0061f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20190624142023-c5567b49c5d0/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20190826190057-c7b8b68b1456/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20190904154756-749cb33beabd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20191005200804-aed5e4c7ecf9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20191120155948-bd437916bb0e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20191228213918-04cbcbbfeed8/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200106162015-b016eb3dc98e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200302150141-5c8b2ff67527/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200519105757-fe76b779f299/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200615200032-f1bc736245b1/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200622214017-ed371f2e16b4 h1:5/PjkGUjvEU5Gl6BxmvKRPpqo2uNMv4rcHBMwzk/st8= +golang.org/x/sys v0.0.0-20200622214017-ed371f2e16b4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= +golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= +golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk= +golang.org/x/text v0.3.3 h1:cokOdA+Jmi5PJGXLlLllQSgYigAEfHXJAERHVMaCc2k= +golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= +golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= +golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= +golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= +golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= +golang.org/x/time v0.0.0-20200630173020-3af7569d3a1e h1:EHBhcS0mlXEAVwNyO2dLfjToGsyY4j24pTs2ScHnX7s= +golang.org/x/time v0.0.0-20200630173020-3af7569d3a1e/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= +golang.org/x/tools v0.0.0-20180221164845-07fd8470d635/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20181011042414-1f849cf54d09/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20181030221726-6c7e314b6563/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20190125232054-d66bd3c5d5a6/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY= +golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= +golang.org/x/tools v0.0.0-20190312151545-0bb0c0a6e846/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= +golang.org/x/tools v0.0.0-20190312170243-e65039ee4138/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= +golang.org/x/tools v0.0.0-20190425150028-36563e24a262/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= +golang.org/x/tools v0.0.0-20190506145303-2d16b83fe98c/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= +golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= +golang.org/x/tools v0.0.0-20190606124116-d0a3d012864b/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= +golang.org/x/tools v0.0.0-20190614205625-5aca471b1d59/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= +golang.org/x/tools v0.0.0-20190617190820-da514acc4774/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= +golang.org/x/tools v0.0.0-20190621195816-6e04913cbbac/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= +golang.org/x/tools v0.0.0-20190624222133-a101b041ded4/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= +golang.org/x/tools v0.0.0-20190628153133-6cdbf07be9d0/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= +golang.org/x/tools v0.0.0-20190816200558-6889da9d5479/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= +golang.org/x/tools v0.0.0-20190911174233-4f2ddba30aff/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= +golang.org/x/tools v0.0.0-20191012152004-8de300cfc20a/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= +golang.org/x/tools v0.0.0-20191029041327-9cc4af7d6b2c/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= +golang.org/x/tools v0.0.0-20191029190741-b9c20aec41a5/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= +golang.org/x/tools v0.0.0-20191108193012-7d206e10da11/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= +golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= +golang.org/x/tools v0.0.0-20191125144606-a911d9008d1f/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= +golang.org/x/tools v0.0.0-20191227053925-7b8e75db28f4/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= +golang.org/x/tools v0.0.0-20200616133436-c1934b75d054 h1:HHeAlu5H9b71C+Fx0K+1dGgVFN1DM1/wz4aoGOA5qS8= +golang.org/x/tools v0.0.0-20200616133436-c1934b75d054/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= +golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4= +golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +gomodules.xyz/jsonpatch/v2 v2.1.0 h1:Phva6wqu+xR//Njw6iorylFFgn/z547tw5Ne3HZPQ+k= +gomodules.xyz/jsonpatch/v2 v2.1.0/go.mod h1:IhYNNY4jnS53ZnfE4PAmpKtDpTCj1JFXc+3mwe7XcUU= +google.golang.org/api v0.4.0/go.mod h1:8k5glujaEP+g9n7WNsDg8QP6cUVNI86fCNMcbazEtwE= +google.golang.org/api v0.7.0/go.mod h1:WtwebWUNSVBH/HAw79HIFXZNqEvBhG+Ra+ax0hx3E3M= +google.golang.org/api v0.8.0/go.mod h1:o4eAsZoiT+ibD93RtjEohWalFOjRDx6CVaqeizhEnKg= +google.golang.org/api v0.9.0/go.mod h1:o4eAsZoiT+ibD93RtjEohWalFOjRDx6CVaqeizhEnKg= +google.golang.org/api v0.15.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI= +google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= +google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= +google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= +google.golang.org/appengine v1.6.1/go.mod h1:i06prIuMbXzDqacNJfV5OdTW448YApPu5ww/cMBSeb0= +google.golang.org/appengine v1.6.5/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc= +google.golang.org/appengine v1.6.6 h1:lMO5rYAqUxkmaj76jAkRUvt5JZgFymx/+Q5Mzfivuhc= +google.golang.org/appengine v1.6.6/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc= +google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc= +google.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= +google.golang.org/genproto v0.0.0-20190418145605-e7d98fc518a7/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= +google.golang.org/genproto v0.0.0-20190425155659-357c62f0e4bb/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= +google.golang.org/genproto v0.0.0-20190502173448-54afdca5d873/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= +google.golang.org/genproto v0.0.0-20190801165951-fa694d86fc64/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc= +google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc= +google.golang.org/genproto v0.0.0-20190911173649-1774047e7e51/go.mod h1:IbNlFCBrqXvoKpeg0TB2l7cyZUmoaFKYIwrEpbDKLA8= +google.golang.org/genproto v0.0.0-20191230161307-f3c370f40bfb/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc= +google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo= +google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c= +google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38= +google.golang.org/grpc v1.21.0/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM= +google.golang.org/grpc v1.21.1/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM= +google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg= +google.golang.org/grpc v1.26.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk= +google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk= +google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8= +google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0= +google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM= +google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE= +google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo= +google.golang.org/protobuf v1.22.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= +google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= +google.golang.org/protobuf v1.23.1-0.20200526195155-81db48ad09cc/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= +google.golang.org/protobuf v1.24.0 h1:UhZDfRO8JRQru4/+LlLE0BRKGF8L+PICnvYZmx/fEGA= +google.golang.org/protobuf v1.24.0/go.mod h1:r/3tXBNzIEhYS9I1OUVjXDlt8tc493IdKGjtUeSXeh4= +gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo= +gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/cheggaaa/pb.v1 v1.0.25/go.mod h1:V/YB90LKu/1FcN3WVnfiiE5oMCibMjukxqG/qStrOgw= +gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI= +gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys= +gopkg.in/inf.v0 v0.9.1 h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc= +gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw= +gopkg.in/natefinch/lumberjack.v2 v2.0.0/go.mod h1:l0ndWWf7gzL7RNwBG7wST/UCcT4T24xpD6X8LsfU/+k= +gopkg.in/resty.v1 v1.12.0/go.mod h1:mDo4pnntr5jdWRML875a/NmxYqAlA73dVijT2AXvQQo= +gopkg.in/square/go-jose.v2 v2.2.2/go.mod h1:M9dMgbHiYLoDGQrXy7OpJDJWiKiU//h+vD76mk0e1AI= +gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ= +gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw= +gopkg.in/yaml.v2 v2.0.0-20170812160011-eb3733d160e7/go.mod h1:JAlM8MvJe8wmxCU4Bli9HhUf9+ttbYbLASfIpnQbh74= +gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= +gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= +gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= +gopkg.in/yaml.v2 v2.2.5/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= +gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= +gopkg.in/yaml.v2 v2.3.0 h1:clyUAQHOM3G0M3f5vQj7LuJrETvjVot3Z5el9nffUtU= +gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= +gopkg.in/yaml.v3 v3.0.0-20200615113413-eeeca48fe776 h1:tQIYjPdBoyREyB9XMu+nnTclpTYkz2zFM+lzLJFO4gQ= +gopkg.in/yaml.v3 v3.0.0-20200615113413-eeeca48fe776/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= +gotest.tools v2.2.0+incompatible/go.mod h1:DsYFclhRJ6vuDpmuTbkuFWG+y2sxOXAzmJt81HFBacw= +gotest.tools/v3 v3.0.2/go.mod h1:3SzNCllyD9/Y+b5r9JIKQ474KzkZyqLqEfYqMsX94Bk= +honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= +honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= +honnef.co/go/tools v0.0.0-20190418001031-e561f6794a2a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= +honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= +honnef.co/go/tools v0.0.1-2019.2.3 h1:3JgtbtFHMiCmsznwGVTUWbgGov+pVqnlf1dEJTNAXeM= +honnef.co/go/tools v0.0.1-2019.2.3/go.mod h1:a3bituU0lyd329TUQxRnasdCoJDkEUEAqEt0JzvZhAg= +k8s.io/api v0.19.2 h1:q+/krnHWKsL7OBZg/rxnycsl9569Pud76UJ77MvKXms= +k8s.io/api v0.19.2/go.mod h1:IQpK0zFQ1xc5iNIQPqzgoOwuFugaYHK4iCknlAQP9nI= +k8s.io/apiextensions-apiserver v0.19.2 h1:oG84UwiDsVDu7dlsGQs5GySmQHCzMhknfhFExJMz9tA= +k8s.io/apiextensions-apiserver v0.19.2/go.mod h1:EYNjpqIAvNZe+svXVx9j4uBaVhTB4C94HkY3w058qcg= +k8s.io/apimachinery v0.19.2 h1:5Gy9vQpAGTKHPVOh5c4plE274X8D/6cuEiTO2zve7tc= +k8s.io/apimachinery v0.19.2/go.mod h1:DnPGDnARWFvYa3pMHgSxtbZb7gpzzAZ1pTfaUNDVlmA= +k8s.io/apiserver v0.19.2/go.mod h1:FreAq0bJ2vtZFj9Ago/X0oNGC51GfubKK/ViOKfVAOA= +k8s.io/client-go v0.19.2 h1:gMJuU3xJZs86L1oQ99R4EViAADUPMHHtS9jFshasHSc= +k8s.io/client-go v0.19.2/go.mod h1:S5wPhCqyDNAlzM9CnEdgTGV4OqhsW3jGO1UM1epwfJA= +k8s.io/code-generator v0.19.2/go.mod h1:moqLn7w0t9cMs4+5CQyxnfA/HV8MF6aAVENF+WZZhgk= +k8s.io/component-base v0.19.2 h1:jW5Y9RcZTb79liEhW3XDVTW7MuvEGP0tQZnfSX6/+gs= +k8s.io/component-base v0.19.2/go.mod h1:g5LrsiTiabMLZ40AR6Hl45f088DevyGY+cCE2agEIVo= +k8s.io/gengo v0.0.0-20200413195148-3a45101e95ac/go.mod h1:ezvh/TsK7cY6rbqRK0oQQ8IAqLxYwwyPxAX1Pzy0ii0= +k8s.io/gengo v0.0.0-20200428234225-8167cfdcfc14/go.mod h1:ezvh/TsK7cY6rbqRK0oQQ8IAqLxYwwyPxAX1Pzy0ii0= +k8s.io/klog/v2 v2.0.0/go.mod h1:PBfzABfn139FHAV07az/IF9Wp1bkk3vpT2XSJ76fSDE= +k8s.io/klog/v2 v2.2.0 h1:XRvcwJozkgZ1UQJmfMGpvRthQHOvihEhYtDfAaxMz/A= +k8s.io/klog/v2 v2.2.0/go.mod h1:Od+F08eJP+W3HUb4pSrPpgp9DGU4GzlpG/TmITuYh/Y= +k8s.io/kube-openapi v0.0.0-20200805222855-6aeccd4b50c6 h1:+WnxoVtG8TMiudHBSEtrVL1egv36TkkJm+bA8AxicmQ= +k8s.io/kube-openapi v0.0.0-20200805222855-6aeccd4b50c6/go.mod h1:UuqjUnNftUyPE5H64/qeyjQoUZhGpeFDVdxjTeEVN2o= +k8s.io/utils v0.0.0-20200729134348-d5654de09c73/go.mod h1:jPW/WVKK9YHAvNhRxK0md/EJ228hCsBRufyofKtW8HA= +k8s.io/utils v0.0.0-20200912215256-4140de9c8800 h1:9ZNvfPvVIEsp/T1ez4GQuzCcCTEQWhovSofhqR73A6g= +k8s.io/utils v0.0.0-20200912215256-4140de9c8800/go.mod h1:jPW/WVKK9YHAvNhRxK0md/EJ228hCsBRufyofKtW8HA= +rsc.io/binaryregexp v0.2.0/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8= +sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.0.9/go.mod h1:dzAXnQbTRyDlZPJX2SUPEqvnB+j7AJjtlox7PEwigU0= +sigs.k8s.io/controller-runtime v0.7.0 h1:bU20IBBEPccWz5+zXpLnpVsgBYxqclaHu1pVDl/gEt8= +sigs.k8s.io/controller-runtime v0.7.0/go.mod h1:pJ3YBrJiAqMAZKi6UVGuE98ZrroV1p+pIhoHsMm9wdU= +sigs.k8s.io/structured-merge-diff/v4 v4.0.1 h1:YXTMot5Qz/X1iBRJhAt+vI+HVttY0WkSqqhKxQ0xVbA= +sigs.k8s.io/structured-merge-diff/v4 v4.0.1/go.mod h1:bJZC9H9iH24zzfZ/41RGcq60oK1F7G282QMXDPYydCw= +sigs.k8s.io/yaml v1.1.0/go.mod h1:UJmg0vDUVViEyp3mgSv9WPwZCDxu4rQW1olrI1uml+o= +sigs.k8s.io/yaml v1.2.0 h1:kr/MCeFWJWTwyaHoR9c8EjH9OumOmoF9YGiZd7lFm/Q= +sigs.k8s.io/yaml v1.2.0/go.mod h1:yfXDCHCao9+ENCvLSE62v9VSji2MKu5jeNfTrofGhJc= diff --git a/hack/boilerplate.go.txt b/hack/boilerplate.go.txt new file mode 100644 index 00000000..503c748b --- /dev/null +++ b/hack/boilerplate.go.txt @@ -0,0 +1,15 @@ +/* +Copyright 2021 Intuit Inc.. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ \ No newline at end of file diff --git a/main.go b/main.go new file mode 100644 index 00000000..e35ce3cb --- /dev/null +++ b/main.go @@ -0,0 +1,105 @@ +/* +Copyright 2021 Intuit Inc.. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package main + +import ( + "flag" + "os" + + // Import all Kubernetes client auth plugins (e.g. Azure, GCP, OIDC, etc.) + // to ensure that exec-entrypoint and run can make use of them. + _ "k8s.io/client-go/plugin/pkg/client/auth" + + "k8s.io/apimachinery/pkg/runtime" + utilruntime "k8s.io/apimachinery/pkg/util/runtime" + clientgoscheme "k8s.io/client-go/kubernetes/scheme" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/healthz" + "sigs.k8s.io/controller-runtime/pkg/log/zap" + + upgrademgrv1alpha1 "github.com/keikoproj/upgrade-manager/api/v1alpha1" + "github.com/keikoproj/upgrade-manager/controllers" + //+kubebuilder:scaffold:imports +) + +var ( + scheme = runtime.NewScheme() + setupLog = ctrl.Log.WithName("setup") +) + +func init() { + utilruntime.Must(clientgoscheme.AddToScheme(scheme)) + + utilruntime.Must(upgrademgrv1alpha1.AddToScheme(scheme)) + //+kubebuilder:scaffold:scheme +} + +func main() { + var metricsAddr string + var enableLeaderElection bool + var probeAddr string + flag.StringVar(&metricsAddr, "metrics-bind-address", ":8080", "The address the metric endpoint binds to.") + flag.StringVar(&probeAddr, "health-probe-bind-address", ":8081", "The address the probe endpoint binds to.") + flag.BoolVar(&enableLeaderElection, "leader-elect", false, + "Enable leader election for controller manager. "+ + "Enabling this will ensure there is only one active controller manager.") + opts := zap.Options{ + Development: true, + } + opts.BindFlags(flag.CommandLine) + flag.Parse() + + ctrl.SetLogger(zap.New(zap.UseFlagOptions(&opts))) + + mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{ + Scheme: scheme, + MetricsBindAddress: metricsAddr, + Port: 9443, + HealthProbeBindAddress: probeAddr, + LeaderElection: enableLeaderElection, + LeaderElectionID: "d6edb06e.keikoproj.io", + }) + if err != nil { + setupLog.Error(err, "unable to start manager") + os.Exit(1) + } + + if err = (&controllers.RollingUpgradeReconciler{ + Client: mgr.GetClient(), + Log: ctrl.Log.WithName("controllers").WithName("RollingUpgrade"), + Scheme: mgr.GetScheme(), + }).SetupWithManager(mgr); err != nil { + setupLog.Error(err, "unable to create controller", "controller", "RollingUpgrade") + os.Exit(1) + } + //+kubebuilder:scaffold:builder + + if err := mgr.AddHealthzCheck("health", healthz.Ping); err != nil { + setupLog.Error(err, "unable to set up health check") + os.Exit(1) + } + if err := mgr.AddReadyzCheck("check", healthz.Ping); err != nil { + setupLog.Error(err, "unable to set up ready check") + os.Exit(1) + } + + setupLog.Info("starting manager") + if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil { + setupLog.Error(err, "problem running manager") + os.Exit(1) + } +} From 87afbd687855b02c1f3ea96523ea4c65fa23817f Mon Sep 17 00:00:00 2001 From: Eytan Avisror Date: Mon, 11 Jan 2021 22:17:16 -0800 Subject: [PATCH 04/74] add API Signed-off-by: Eytan Avisror --- api/v1alpha1/groupversion_info.go | 2 +- api/v1alpha1/rollingupgrade_types.go | 107 ++++++++++-- api/v1alpha1/zz_generated.deepcopy.go | 120 +++++++++++++- ...grademgr.keikoproj.io_rollingupgrades.yaml | 155 ++++++++++++++++++ config/rbac/role.yaml | 69 ++++++++ controllers/rollingupgrade_controller.go | 26 ++- controllers/suite_test.go | 2 +- go.mod | 1 + go.sum | 1 + hack/boilerplate.go.txt | 2 +- main.go | 2 +- 11 files changed, 453 insertions(+), 34 deletions(-) create mode 100644 config/crd/bases/upgrademgr.keikoproj.io_rollingupgrades.yaml create mode 100644 config/rbac/role.yaml diff --git a/api/v1alpha1/groupversion_info.go b/api/v1alpha1/groupversion_info.go index 22f5fb7c..dd7c9b64 100644 --- a/api/v1alpha1/groupversion_info.go +++ b/api/v1alpha1/groupversion_info.go @@ -1,5 +1,5 @@ /* -Copyright 2021 Intuit Inc.. +Copyright 2021 Intuit Inc. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. diff --git a/api/v1alpha1/rollingupgrade_types.go b/api/v1alpha1/rollingupgrade_types.go index caa5622a..edf13ec3 100644 --- a/api/v1alpha1/rollingupgrade_types.go +++ b/api/v1alpha1/rollingupgrade_types.go @@ -1,5 +1,5 @@ /* -Copyright 2021 Intuit Inc.. +Copyright 2021 Intuit Inc. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -17,29 +17,44 @@ limitations under the License. package v1alpha1 import ( + "fmt" + + corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/util/intstr" ) -// EDIT THIS FILE! THIS IS SCAFFOLDING FOR YOU TO OWN! -// NOTE: json tags are required. Any new fields you add must have json tags for the fields to be serialized. - // RollingUpgradeSpec defines the desired state of RollingUpgrade type RollingUpgradeSpec struct { - // INSERT ADDITIONAL SPEC FIELDS - desired state of cluster - // Important: Run "make" to regenerate code after modifying this file - - // Foo is an example field of RollingUpgrade. Edit rollingupgrade_types.go to remove/update - Foo string `json:"foo,omitempty"` + PostDrainDelaySeconds int `json:"postDrainDelaySeconds,omitempty"` + NodeIntervalSeconds int `json:"nodeIntervalSeconds,omitempty"` + AsgName string `json:"asgName,omitempty"` + PreDrain PreDrainSpec `json:"preDrain,omitempty"` + PostDrain PostDrainSpec `json:"postDrain,omitempty"` + PostTerminate PostTerminateSpec `json:"postTerminate,omitempty"` + Strategy UpdateStrategy `json:"strategy,omitempty"` + IgnoreDrainFailures bool `json:"ignoreDrainFailures,omitempty"` + ForceRefresh bool `json:"forceRefresh,omitempty"` + ReadinessGates []NodeReadinessGate `json:"readinessGates,omitempty"` } // RollingUpgradeStatus defines the observed state of RollingUpgrade type RollingUpgradeStatus struct { - // INSERT ADDITIONAL STATUS FIELD - define observed state of cluster - // Important: Run "make" to regenerate code after modifying this file + CurrentStatus string `json:"currentStatus,omitempty"` + StartTime string `json:"startTime,omitempty"` + EndTime string `json:"endTime,omitempty"` + TotalProcessingTime string `json:"totalProcessingTime,omitempty"` + NodesProcessed int `json:"nodesProcessed,omitempty"` + TotalNodes int `json:"totalNodes,omitempty"` + Conditions []RollingUpgradeCondition `json:"conditions,omitempty"` } -//+kubebuilder:object:root=true -//+kubebuilder:subresource:status +// +kubebuilder:object:root=true +// +kubebuilder:subresource:status +// +kubebuilder:resource:path=rollingupgrades,scope=Namespaced,shortName=ru +// +kubebuilder:printcolumn:name="Status",type="string",JSONPath=".status.currentStatus",description="current status of the rollingupgarde" +// +kubebuilder:printcolumn:name="TotalNodes",type="string",JSONPath=".status.totalNodes",description="total nodes involved in the rollingupgarde" +// +kubebuilder:printcolumn:name="NodesProcessed",type="string",JSONPath=".status.nodesProcessed",description="current number of nodes processed in the rollingupgarde" // RollingUpgrade is the Schema for the rollingupgrades API type RollingUpgrade struct { @@ -62,3 +77,69 @@ type RollingUpgradeList struct { func init() { SchemeBuilder.Register(&RollingUpgrade{}, &RollingUpgradeList{}) } + +// PreDrainSpec contains the fields for actions taken before draining the node. +type PreDrainSpec struct { + Script string `json:"script,omitempty"` +} + +// PostDrainSpec contains the fields for actions taken after draining the node. +type PostDrainSpec struct { + Script string `json:"script,omitempty"` + WaitSeconds int64 `json:"waitSeconds,omitempty"` + PostWaitScript string `json:"postWaitScript,omitempty"` +} + +// PostTerminateSpec contains the fields for actions taken after terminating the node. +type PostTerminateSpec struct { + Script string `json:"script,omitempty"` +} + +type NodeReadinessGate struct { + MatchLabels map[string]string `json:"matchLabels,omitempty" protobuf:"bytes,1,rep,name=matchLabels"` +} + +const ( + // Status + StatusRunning = "running" + StatusComplete = "completed" + StatusError = "error" + + // Conditions + UpgradeComplete UpgradeConditionType = "Complete" +) + +// RollingUpgradeCondition describes the state of the RollingUpgrade +type RollingUpgradeCondition struct { + Type UpgradeConditionType `json:"type,omitempty"` + Status corev1.ConditionStatus `json:"status,omitempty"` +} + +type UpdateStrategyType string +type UpdateStrategyMode string +type UpgradeConditionType string + +const ( + RandomUpdateStrategy UpdateStrategyType = "randomUpdate" + UniformAcrossAzUpdateStrategy UpdateStrategyType = "uniformAcrossAzUpdate" + + UpdateStrategyModeLazy UpdateStrategyMode = "lazy" + UpdateStrategyModeEager UpdateStrategyMode = "eager" +) + +// UpdateStrategy holds the information needed to perform update based on different update strategies +type UpdateStrategy struct { + Type UpdateStrategyType `json:"type,omitempty"` + Mode UpdateStrategyMode `json:"mode,omitempty"` + MaxUnavailable intstr.IntOrString `json:"maxUnavailable,omitempty"` + DrainTimeout int `json:"drainTimeout"` +} + +func (c UpdateStrategyMode) String() string { + return string(c) +} + +// NamespacedName returns namespaced name of the object. +func (r RollingUpgrade) NamespacedName() string { + return fmt.Sprintf("%s/%s", r.Namespace, r.Name) +} diff --git a/api/v1alpha1/zz_generated.deepcopy.go b/api/v1alpha1/zz_generated.deepcopy.go index 4bba20d9..f9016b52 100644 --- a/api/v1alpha1/zz_generated.deepcopy.go +++ b/api/v1alpha1/zz_generated.deepcopy.go @@ -1,7 +1,7 @@ // +build !ignore_autogenerated /* -Copyright 2021 Intuit Inc.. +Copyright 2021 Intuit Inc. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -24,13 +24,80 @@ import ( runtime "k8s.io/apimachinery/pkg/runtime" ) +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *NodeReadinessGate) DeepCopyInto(out *NodeReadinessGate) { + *out = *in + if in.MatchLabels != nil { + in, out := &in.MatchLabels, &out.MatchLabels + *out = make(map[string]string, len(*in)) + for key, val := range *in { + (*out)[key] = val + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new NodeReadinessGate. +func (in *NodeReadinessGate) DeepCopy() *NodeReadinessGate { + if in == nil { + return nil + } + out := new(NodeReadinessGate) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *PostDrainSpec) DeepCopyInto(out *PostDrainSpec) { + *out = *in +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new PostDrainSpec. +func (in *PostDrainSpec) DeepCopy() *PostDrainSpec { + if in == nil { + return nil + } + out := new(PostDrainSpec) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *PostTerminateSpec) DeepCopyInto(out *PostTerminateSpec) { + *out = *in +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new PostTerminateSpec. +func (in *PostTerminateSpec) DeepCopy() *PostTerminateSpec { + if in == nil { + return nil + } + out := new(PostTerminateSpec) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *PreDrainSpec) DeepCopyInto(out *PreDrainSpec) { + *out = *in +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new PreDrainSpec. +func (in *PreDrainSpec) DeepCopy() *PreDrainSpec { + if in == nil { + return nil + } + out := new(PreDrainSpec) + in.DeepCopyInto(out) + return out +} + // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *RollingUpgrade) DeepCopyInto(out *RollingUpgrade) { *out = *in out.TypeMeta = in.TypeMeta in.ObjectMeta.DeepCopyInto(&out.ObjectMeta) - out.Spec = in.Spec - out.Status = in.Status + in.Spec.DeepCopyInto(&out.Spec) + in.Status.DeepCopyInto(&out.Status) } // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RollingUpgrade. @@ -51,6 +118,21 @@ func (in *RollingUpgrade) DeepCopyObject() runtime.Object { return nil } +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *RollingUpgradeCondition) DeepCopyInto(out *RollingUpgradeCondition) { + *out = *in +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RollingUpgradeCondition. +func (in *RollingUpgradeCondition) DeepCopy() *RollingUpgradeCondition { + if in == nil { + return nil + } + out := new(RollingUpgradeCondition) + in.DeepCopyInto(out) + return out +} + // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *RollingUpgradeList) DeepCopyInto(out *RollingUpgradeList) { *out = *in @@ -86,6 +168,17 @@ func (in *RollingUpgradeList) DeepCopyObject() runtime.Object { // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *RollingUpgradeSpec) DeepCopyInto(out *RollingUpgradeSpec) { *out = *in + out.PreDrain = in.PreDrain + out.PostDrain = in.PostDrain + out.PostTerminate = in.PostTerminate + out.Strategy = in.Strategy + if in.ReadinessGates != nil { + in, out := &in.ReadinessGates, &out.ReadinessGates + *out = make([]NodeReadinessGate, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } } // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RollingUpgradeSpec. @@ -101,6 +194,11 @@ func (in *RollingUpgradeSpec) DeepCopy() *RollingUpgradeSpec { // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *RollingUpgradeStatus) DeepCopyInto(out *RollingUpgradeStatus) { *out = *in + if in.Conditions != nil { + in, out := &in.Conditions, &out.Conditions + *out = make([]RollingUpgradeCondition, len(*in)) + copy(*out, *in) + } } // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RollingUpgradeStatus. @@ -112,3 +210,19 @@ func (in *RollingUpgradeStatus) DeepCopy() *RollingUpgradeStatus { in.DeepCopyInto(out) return out } + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *UpdateStrategy) DeepCopyInto(out *UpdateStrategy) { + *out = *in + out.MaxUnavailable = in.MaxUnavailable +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new UpdateStrategy. +func (in *UpdateStrategy) DeepCopy() *UpdateStrategy { + if in == nil { + return nil + } + out := new(UpdateStrategy) + in.DeepCopyInto(out) + return out +} diff --git a/config/crd/bases/upgrademgr.keikoproj.io_rollingupgrades.yaml b/config/crd/bases/upgrademgr.keikoproj.io_rollingupgrades.yaml new file mode 100644 index 00000000..7f0bfc18 --- /dev/null +++ b/config/crd/bases/upgrademgr.keikoproj.io_rollingupgrades.yaml @@ -0,0 +1,155 @@ + +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.4.1 + creationTimestamp: null + name: rollingupgrades.upgrademgr.keikoproj.io +spec: + group: upgrademgr.keikoproj.io + names: + kind: RollingUpgrade + listKind: RollingUpgradeList + plural: rollingupgrades + shortNames: + - ru + singular: rollingupgrade + scope: Namespaced + versions: + - additionalPrinterColumns: + - description: current status of the rollingupgarde + jsonPath: .status.currentStatus + name: Status + type: string + - description: total nodes involved in the rollingupgarde + jsonPath: .status.totalNodes + name: TotalNodes + type: string + - description: current number of nodes processed in the rollingupgarde + jsonPath: .status.nodesProcessed + name: NodesProcessed + type: string + name: v1alpha1 + schema: + openAPIV3Schema: + description: RollingUpgrade is the Schema for the rollingupgrades API + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation + of an object. Servers should convert recognized schemas to the latest + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this + object represents. Servers may infer this from the endpoint the client + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' + type: string + metadata: + type: object + spec: + description: RollingUpgradeSpec defines the desired state of RollingUpgrade + properties: + asgName: + type: string + forceRefresh: + type: boolean + ignoreDrainFailures: + type: boolean + nodeIntervalSeconds: + type: integer + postDrain: + description: PostDrainSpec contains the fields for actions taken after + draining the node. + properties: + postWaitScript: + type: string + script: + type: string + waitSeconds: + format: int64 + type: integer + type: object + postDrainDelaySeconds: + type: integer + postTerminate: + description: PostTerminateSpec contains the fields for actions taken + after terminating the node. + properties: + script: + type: string + type: object + preDrain: + description: PreDrainSpec contains the fields for actions taken before + draining the node. + properties: + script: + type: string + type: object + readinessGates: + items: + properties: + matchLabels: + additionalProperties: + type: string + type: object + type: object + type: array + strategy: + description: UpdateStrategy holds the information needed to perform + update based on different update strategies + properties: + drainTimeout: + type: integer + maxUnavailable: + anyOf: + - type: integer + - type: string + x-kubernetes-int-or-string: true + mode: + type: string + type: + type: string + required: + - drainTimeout + type: object + type: object + status: + description: RollingUpgradeStatus defines the observed state of RollingUpgrade + properties: + conditions: + items: + description: RollingUpgradeCondition describes the state of the + RollingUpgrade + properties: + status: + type: string + type: + type: string + type: object + type: array + currentStatus: + type: string + endTime: + type: string + nodesProcessed: + type: integer + startTime: + type: string + totalNodes: + type: integer + totalProcessingTime: + type: string + type: object + type: object + served: true + storage: true + subresources: + status: {} +status: + acceptedNames: + kind: "" + plural: "" + conditions: [] + storedVersions: [] diff --git a/config/rbac/role.yaml b/config/rbac/role.yaml new file mode 100644 index 00000000..5c4ea57f --- /dev/null +++ b/config/rbac/role.yaml @@ -0,0 +1,69 @@ + +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + creationTimestamp: null + name: manager-role +rules: +- apiGroups: + - apps + - extensions + resources: + - daemonsets + - replicasets + - statefulsets + verbs: + - get +- apiGroups: + - batch + resources: + - jobs + verbs: + - get +- apiGroups: + - "" + resources: + - events + verbs: + - create +- apiGroups: + - "" + resources: + - nodes + verbs: + - get + - list + - patch +- apiGroups: + - "" + resources: + - pods + verbs: + - list +- apiGroups: + - "" + resources: + - pods/eviction + verbs: + - create +- apiGroups: + - upgrademgr.keikoproj.io + resources: + - rollingupgrades + verbs: + - create + - delete + - get + - list + - patch + - update + - watch +- apiGroups: + - upgrademgr.keikoproj.io + resources: + - rollingupgrades/status + verbs: + - get + - patch + - update diff --git a/controllers/rollingupgrade_controller.go b/controllers/rollingupgrade_controller.go index 351872ec..f2dbe883 100644 --- a/controllers/rollingupgrade_controller.go +++ b/controllers/rollingupgrade_controller.go @@ -1,5 +1,5 @@ /* -Copyright 2021 Intuit Inc.. +Copyright 2021 Intuit Inc. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -34,19 +34,17 @@ type RollingUpgradeReconciler struct { Scheme *runtime.Scheme } -//+kubebuilder:rbac:groups=upgrademgr.keikoproj.io,resources=rollingupgrades,verbs=get;list;watch;create;update;patch;delete -//+kubebuilder:rbac:groups=upgrademgr.keikoproj.io,resources=rollingupgrades/status,verbs=get;update;patch -//+kubebuilder:rbac:groups=upgrademgr.keikoproj.io,resources=rollingupgrades/finalizers,verbs=update - -// Reconcile is part of the main kubernetes reconciliation loop which aims to -// move the current state of the cluster closer to the desired state. -// TODO(user): Modify the Reconcile function to compare the state specified by -// the RollingUpgrade object against the actual cluster state, and then -// perform operations to make the cluster state reflect the state specified by -// the user. -// -// For more details, check Reconcile and its Result here: -// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.7.0/pkg/reconcile +// +kubebuilder:rbac:groups=upgrademgr.keikoproj.io,resources=rollingupgrades,verbs=get;list;watch;create;update;patch;delete +// +kubebuilder:rbac:groups=upgrademgr.keikoproj.io,resources=rollingupgrades/status,verbs=get;update;patch +// +kubebuilder:rbac:groups=core,resources=nodes,verbs=get;list;patch +// +kubebuilder:rbac:groups=core,resources=pods,verbs=list +// +kubebuilder:rbac:groups=core,resources=events,verbs=create +// +kubebuilder:rbac:groups=core,resources=pods/eviction,verbs=create +// +kubebuilder:rbac:groups=extensions;apps,resources=daemonsets;replicasets;statefulsets,verbs=get +// +kubebuilder:rbac:groups=batch,resources=jobs,verbs=get + +// Reconcile reads that state of the cluster for a RollingUpgrade object and makes changes based on the state read +// and the details in the RollingUpgrade.Spec func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { _ = r.Log.WithValues("rollingupgrade", req.NamespacedName) diff --git a/controllers/suite_test.go b/controllers/suite_test.go index 75a6ef2f..61786df2 100644 --- a/controllers/suite_test.go +++ b/controllers/suite_test.go @@ -1,5 +1,5 @@ /* -Copyright 2021 Intuit Inc.. +Copyright 2021 Intuit Inc. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. diff --git a/go.mod b/go.mod index a943e11c..46a0b2a8 100644 --- a/go.mod +++ b/go.mod @@ -6,6 +6,7 @@ require ( github.com/go-logr/logr v0.3.0 github.com/onsi/ginkgo v1.14.1 github.com/onsi/gomega v1.10.2 + k8s.io/api v0.19.2 k8s.io/apimachinery v0.19.2 k8s.io/client-go v0.19.2 sigs.k8s.io/controller-runtime v0.7.0 diff --git a/go.sum b/go.sum index ed5f385e..1a4d3c60 100644 --- a/go.sum +++ b/go.sum @@ -611,6 +611,7 @@ k8s.io/apiextensions-apiserver v0.19.2 h1:oG84UwiDsVDu7dlsGQs5GySmQHCzMhknfhFExJ k8s.io/apiextensions-apiserver v0.19.2/go.mod h1:EYNjpqIAvNZe+svXVx9j4uBaVhTB4C94HkY3w058qcg= k8s.io/apimachinery v0.19.2 h1:5Gy9vQpAGTKHPVOh5c4plE274X8D/6cuEiTO2zve7tc= k8s.io/apimachinery v0.19.2/go.mod h1:DnPGDnARWFvYa3pMHgSxtbZb7gpzzAZ1pTfaUNDVlmA= +k8s.io/apimachinery v0.20.1 h1:LAhz8pKbgR8tUwn7boK+b2HZdt7MiTu2mkYtFMUjTRQ= k8s.io/apiserver v0.19.2/go.mod h1:FreAq0bJ2vtZFj9Ago/X0oNGC51GfubKK/ViOKfVAOA= k8s.io/client-go v0.19.2 h1:gMJuU3xJZs86L1oQ99R4EViAADUPMHHtS9jFshasHSc= k8s.io/client-go v0.19.2/go.mod h1:S5wPhCqyDNAlzM9CnEdgTGV4OqhsW3jGO1UM1epwfJA= diff --git a/hack/boilerplate.go.txt b/hack/boilerplate.go.txt index 503c748b..74601199 100644 --- a/hack/boilerplate.go.txt +++ b/hack/boilerplate.go.txt @@ -1,5 +1,5 @@ /* -Copyright 2021 Intuit Inc.. +Copyright 2021 Intuit Inc. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. diff --git a/main.go b/main.go index e35ce3cb..f4ed6121 100644 --- a/main.go +++ b/main.go @@ -1,5 +1,5 @@ /* -Copyright 2021 Intuit Inc.. +Copyright 2021 Intuit Inc. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. From 25644a6f3cb95a4bac012eb72ad829a449e39116 Mon Sep 17 00:00:00 2001 From: Eytan Avisror Date: Tue, 12 Jan 2021 00:38:16 -0800 Subject: [PATCH 05/74] initial code Signed-off-by: Eytan Avisror --- api/v1alpha1/rollingupgrade_types.go | 4 + controllers/cloud.go | 19 ++ controllers/common/log/log.go | 205 +++++++++++++++++++++ controllers/common/log/retry.go | 44 +++++ controllers/providers/aws/autoscaling.go | 19 ++ controllers/providers/aws/ec2.go | 19 ++ controllers/providers/aws/utils.go | 52 ++++++ controllers/providers/kubernetes/events.go | 19 ++ controllers/providers/kubernetes/nodes.go | 19 ++ controllers/providers/kubernetes/utils.go | 83 +++++++++ controllers/rollingupgrade_controller.go | 77 +++++++- controllers/state.go | 19 ++ controllers/suite_test.go | 79 -------- controllers/upgrade.go | 19 ++ go.mod | 6 +- go.sum | 28 ++- main.go | 157 ++++++++++++++-- 17 files changed, 763 insertions(+), 105 deletions(-) create mode 100644 controllers/cloud.go create mode 100644 controllers/common/log/log.go create mode 100644 controllers/common/log/retry.go create mode 100644 controllers/providers/aws/autoscaling.go create mode 100644 controllers/providers/aws/ec2.go create mode 100644 controllers/providers/aws/utils.go create mode 100644 controllers/providers/kubernetes/events.go create mode 100644 controllers/providers/kubernetes/nodes.go create mode 100644 controllers/providers/kubernetes/utils.go create mode 100644 controllers/state.go delete mode 100644 controllers/suite_test.go create mode 100644 controllers/upgrade.go diff --git a/api/v1alpha1/rollingupgrade_types.go b/api/v1alpha1/rollingupgrade_types.go index edf13ec3..f472a48b 100644 --- a/api/v1alpha1/rollingupgrade_types.go +++ b/api/v1alpha1/rollingupgrade_types.go @@ -143,3 +143,7 @@ func (c UpdateStrategyMode) String() string { func (r RollingUpgrade) NamespacedName() string { return fmt.Sprintf("%s/%s", r.Namespace, r.Name) } + +func (r RollingUpgrade) ScalingGroupName() string { + return r.Spec.AsgName +} diff --git a/controllers/cloud.go b/controllers/cloud.go new file mode 100644 index 00000000..b6a8d7a0 --- /dev/null +++ b/controllers/cloud.go @@ -0,0 +1,19 @@ +/* +Copyright 2021 Intuit Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package controllers + +// TODO: Resource discovery for AWS & Kubernetes diff --git a/controllers/common/log/log.go b/controllers/common/log/log.go new file mode 100644 index 00000000..6f42c02a --- /dev/null +++ b/controllers/common/log/log.go @@ -0,0 +1,205 @@ +package log + +import ( + "github.com/sirupsen/logrus" +) + +// Logger defines a set of methods for writing application logs. +type Logger interface { + Debug(args ...interface{}) + Debugf(format string, args ...interface{}) + Debugln(args ...interface{}) + Error(args ...interface{}) + Errorf(format string, args ...interface{}) + Errorln(args ...interface{}) + Fatal(args ...interface{}) + Fatalf(format string, args ...interface{}) + Fatalln(args ...interface{}) + Info(args ...interface{}) + Infof(format string, args ...interface{}) + Infoln(args ...interface{}) + Panic(args ...interface{}) + Panicf(format string, args ...interface{}) + Panicln(args ...interface{}) + Print(args ...interface{}) + Printf(format string, args ...interface{}) + Println(args ...interface{}) + Warn(args ...interface{}) + Warnf(format string, args ...interface{}) + Warning(args ...interface{}) + Warningf(format string, args ...interface{}) + Warningln(args ...interface{}) + Warnln(args ...interface{}) +} + +var defaultLogger *logrus.Logger + +func init() { + defaultLogger = newLogrusLogger() +} + +func NewLogger() *logrus.Logger { + return newLogrusLogger() +} + +func newLogrusLogger() *logrus.Logger { + l := logrus.New() + l.Level = logrus.InfoLevel + return l +} + +func SetLevel(logLevel string) { + switch logLevel { + case "debug": + defaultLogger.Level = logrus.DebugLevel + case "warning": + defaultLogger.Level = logrus.WarnLevel + case "info": + defaultLogger.Level = logrus.InfoLevel + default: + defaultLogger.Level = logrus.InfoLevel + } +} + +// SetJSONFormatter sets JSON Formatter for default logger. +func SetJSONFormatter() { + defaultLogger.SetFormatter(&logrus.JSONFormatter{}) +} + +type Fields map[string]interface{} + +func (f Fields) With(k string, v interface{}) Fields { + f[k] = v + return f +} + +func (f Fields) WithFields(f2 Fields) Fields { + for k, v := range f2 { + f[k] = v + } + return f +} + +func WithFields(fields Fields) Logger { + return defaultLogger.WithFields(logrus.Fields(fields)) +} + +// Debug package-level convenience method. +func Debug(args ...interface{}) { + defaultLogger.Debug(args...) +} + +// Debugf package-level convenience method. +func Debugf(format string, args ...interface{}) { + defaultLogger.Debugf(format, args...) +} + +// Debugln package-level convenience method. +func Debugln(args ...interface{}) { + defaultLogger.Debugln(args...) +} + +// Error package-level convenience method. +func Error(args ...interface{}) { + defaultLogger.Error(args...) +} + +// Errorf package-level convenience method. +func Errorf(format string, args ...interface{}) { + defaultLogger.Errorf(format, args...) +} + +// Errorln package-level convenience method. +func Errorln(args ...interface{}) { + defaultLogger.Errorln(args...) +} + +// Fatal package-level convenience method. +func Fatal(args ...interface{}) { + defaultLogger.Fatal(args...) +} + +// Fatalf package-level convenience method. +func Fatalf(format string, args ...interface{}) { + defaultLogger.Fatalf(format, args...) +} + +// Fatalln package-level convenience method. +func Fatalln(args ...interface{}) { + defaultLogger.Fatalln(args...) +} + +// Info package-level convenience method. +func Info(args ...interface{}) { + defaultLogger.Info(args...) +} + +// Infof package-level convenience method. +func Infof(format string, args ...interface{}) { + defaultLogger.Infof(format, args...) +} + +// Infoln package-level convenience method. +func Infoln(args ...interface{}) { + defaultLogger.Infoln(args...) +} + +// Panic package-level convenience method. +func Panic(args ...interface{}) { + defaultLogger.Panic(args...) +} + +// Panicf package-level convenience method. +func Panicf(format string, args ...interface{}) { + defaultLogger.Panicf(format, args...) +} + +// Panicln package-level convenience method. +func Panicln(args ...interface{}) { + defaultLogger.Panicln(args...) +} + +// Print package-level convenience method. +func Print(args ...interface{}) { + defaultLogger.Print(args...) +} + +// Printf package-level convenience method. +func Printf(format string, args ...interface{}) { + defaultLogger.Printf(format, args...) +} + +// Println package-level convenience method. +func Println(args ...interface{}) { + defaultLogger.Println(args...) +} + +// Warn package-level convenience method. +func Warn(args ...interface{}) { + defaultLogger.Warn(args...) +} + +// Warnf package-level convenience method. +func Warnf(format string, args ...interface{}) { + defaultLogger.Warnf(format, args...) +} + +// Warning package-level convenience method. +func Warning(args ...interface{}) { + defaultLogger.Warning(args...) +} + +// Warningf package-level convenience method. +func Warningf(format string, args ...interface{}) { + defaultLogger.Warningf(format, args...) +} + +// Warningln package-level convenience method. +func Warningln(args ...interface{}) { + defaultLogger.Warningln(args...) +} + +// Warnln package-level convenience method. +func Warnln(args ...interface{}) { + defaultLogger.Warnln(args...) +} diff --git a/controllers/common/log/retry.go b/controllers/common/log/retry.go new file mode 100644 index 00000000..ab7e4cc3 --- /dev/null +++ b/controllers/common/log/retry.go @@ -0,0 +1,44 @@ +package log + +import ( + "fmt" + "time" + + "github.com/aws/aws-sdk-go/aws/client" + "github.com/aws/aws-sdk-go/aws/request" +) + +type RetryLogger struct { + client.DefaultRetryer +} + +var _ request.Retryer = &RetryLogger{} + +func NewRetryLogger(retryer client.DefaultRetryer) *RetryLogger { + return &RetryLogger{ + DefaultRetryer: retryer, + } +} + +func (l RetryLogger) RetryRules(r *request.Request) time.Duration { + var ( + duration = l.DefaultRetryer.RetryRules(r) + service = r.ClientInfo.ServiceName + name string + err string + ) + + if r.Operation != nil { + name = r.Operation.Name + } + method := fmt.Sprintf("%v/%v", service, name) + + if r.Error != nil { + err = fmt.Sprintf("%v", r.Error) + } else { + err = fmt.Sprintf("%d %s", r.HTTPResponse.StatusCode, r.HTTPResponse.Status) + } + Infof("retryable: %v -- %v, will retry after %v", err, method, duration) + + return duration +} diff --git a/controllers/providers/aws/autoscaling.go b/controllers/providers/aws/autoscaling.go new file mode 100644 index 00000000..6d2e4e15 --- /dev/null +++ b/controllers/providers/aws/autoscaling.go @@ -0,0 +1,19 @@ +/* +Copyright 2021 Intuit Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package aws + +// TODO: Autoscaling API Calls diff --git a/controllers/providers/aws/ec2.go b/controllers/providers/aws/ec2.go new file mode 100644 index 00000000..d8cdd1c6 --- /dev/null +++ b/controllers/providers/aws/ec2.go @@ -0,0 +1,19 @@ +/* +Copyright 2021 Intuit Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package aws + +// TODO: EC2 API Calls diff --git a/controllers/providers/aws/utils.go b/controllers/providers/aws/utils.go new file mode 100644 index 00000000..13122422 --- /dev/null +++ b/controllers/providers/aws/utils.go @@ -0,0 +1,52 @@ +/* +Copyright 2021 Intuit Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package aws + +import ( + "fmt" + "os" + + "github.com/aws/aws-sdk-go/aws" + "github.com/aws/aws-sdk-go/aws/ec2metadata" + "github.com/aws/aws-sdk-go/aws/session" + "github.com/aws/aws-sdk-go/service/autoscaling/autoscalingiface" + "github.com/aws/aws-sdk-go/service/ec2/ec2iface" +) + +type AmazonClientSet struct { + AsgClient autoscalingiface.AutoScalingAPI + Ec2Client ec2iface.EC2API +} + +func DeriveRegion() (string, error) { + + if region := os.Getenv("AWS_REGION"); region != "" { + return region, nil + } + + var config aws.Config + sess := session.Must(session.NewSessionWithOptions(session.Options{ + SharedConfigState: session.SharedConfigEnable, + Config: config, + })) + c := ec2metadata.New(sess) + region, err := c.Region() + if err != nil { + return "", fmt.Errorf("cannot reach ec2metadata, if running locally export AWS_REGION: %w", err) + } + return region, nil +} diff --git a/controllers/providers/kubernetes/events.go b/controllers/providers/kubernetes/events.go new file mode 100644 index 00000000..4351aec8 --- /dev/null +++ b/controllers/providers/kubernetes/events.go @@ -0,0 +1,19 @@ +/* +Copyright 2021 Intuit Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package kubernetes + +// TODO: Kubernetes Events API calls diff --git a/controllers/providers/kubernetes/nodes.go b/controllers/providers/kubernetes/nodes.go new file mode 100644 index 00000000..7a0e60b4 --- /dev/null +++ b/controllers/providers/kubernetes/nodes.go @@ -0,0 +1,19 @@ +/* +Copyright 2021 Intuit Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package kubernetes + +// TODO: Kubernetes Nodes API calls diff --git a/controllers/providers/kubernetes/utils.go b/controllers/providers/kubernetes/utils.go new file mode 100644 index 00000000..a4a251ac --- /dev/null +++ b/controllers/providers/kubernetes/utils.go @@ -0,0 +1,83 @@ +/* +Copyright 2021 Intuit Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package kubernetes + +import ( + "fmt" + "os" + "os/user" + + "k8s.io/client-go/kubernetes" + "k8s.io/client-go/rest" + "k8s.io/client-go/tools/clientcmd" +) + +// Placeholder Kubernetes helper functions + +type KubernetesClientSet struct { + Kubernetes kubernetes.Interface +} + +func GetKubernetesClient() (kubernetes.Interface, error) { + var config *rest.Config + config, err := GetKubernetesConfig() + if err != nil { + return nil, err + } + client, err := kubernetes.NewForConfig(config) + if err != nil { + return nil, err + } + return client, nil +} + +func GetKubernetesConfig() (*rest.Config, error) { + var config *rest.Config + config, err := rest.InClusterConfig() + if err != nil { + config, err = GetKubernetesLocalConfig() + if err != nil { + return nil, err + } + return config, nil + } + return config, nil +} + +func GetKubernetesLocalConfig() (*rest.Config, error) { + var kubePath string + if os.Getenv("KUBECONFIG") != "" { + kubePath = os.Getenv("KUBECONFIG") + } else { + usr, err := user.Current() + if err != nil { + return nil, err + } + kubePath = usr.HomeDir + "/.kube/config" + } + + if kubePath == "" { + err := fmt.Errorf("failed to get kubeconfig path") + return nil, err + } + + config, err := clientcmd.BuildConfigFromFlags("", kubePath) + if err != nil { + return nil, err + } + return config, nil +} diff --git a/controllers/rollingupgrade_controller.go b/controllers/rollingupgrade_controller.go index f2dbe883..a530eaa0 100644 --- a/controllers/rollingupgrade_controller.go +++ b/controllers/rollingupgrade_controller.go @@ -18,20 +18,34 @@ package controllers import ( "context" + "strings" + "sync" "github.com/go-logr/logr" + "github.com/keikoproj/aws-sdk-go-cache/cache" + upgrademgrv1alpha1 "github.com/keikoproj/upgrade-manager/api/v1alpha1" + awsprovider "github.com/keikoproj/upgrade-manager/controllers/providers/aws" + kubeprovider "github.com/keikoproj/upgrade-manager/controllers/providers/kubernetes" + kerrors "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/runtime" ctrl "sigs.k8s.io/controller-runtime" "sigs.k8s.io/controller-runtime/pkg/client" - - upgrademgrv1alpha1 "github.com/keikoproj/upgrade-manager/api/v1alpha1" + "sigs.k8s.io/controller-runtime/pkg/reconcile" ) // RollingUpgradeReconciler reconciles a RollingUpgrade object type RollingUpgradeReconciler struct { client.Client - Log logr.Logger - Scheme *runtime.Scheme + logr.Logger + Scheme *runtime.Scheme + AdmissionMap sync.Map + CacheConfig *cache.Config + Auth *RollingUpgradeAuthenticator +} + +type RollingUpgradeAuthenticator struct { + awsprovider.AmazonClientSet + kubeprovider.KubernetesClientSet } // +kubebuilder:rbac:groups=upgrademgr.keikoproj.io,resources=rollingupgrades,verbs=get;list;watch;create;update;patch;delete @@ -46,9 +60,60 @@ type RollingUpgradeReconciler struct { // Reconcile reads that state of the cluster for a RollingUpgrade object and makes changes based on the state read // and the details in the RollingUpgrade.Spec func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { - _ = r.Log.WithValues("rollingupgrade", req.NamespacedName) + rollingUpgrade := &upgrademgrv1alpha1.RollingUpgrade{} + err := r.Get(ctx, req.NamespacedName, rollingUpgrade) + if err != nil { + if kerrors.IsNotFound(err) { + r.AdmissionMap.Delete(req.NamespacedName) + r.Info("deleted object from admission map", "name", req.NamespacedName) + return ctrl.Result{}, nil + } + return ctrl.Result{}, err + } + + // If the resource is being deleted, remove it from the admissionMap + if !rollingUpgrade.DeletionTimestamp.IsZero() { + r.AdmissionMap.Delete(req.NamespacedName) + r.Info("deleted object from admission map", "name", req.NamespacedName) + return reconcile.Result{}, nil + } + + // TODO: VALIDATION + SET DEFAULTS + + // Migrate r.setDefaultsForRollingUpdateStrategy & r.validateRollingUpgradeObj into v1alpha1 RollingUpgrade.Validate() + // Include setting of defaults in validation + + // if ok, err := rollingUpgrade.Validate(); !ok { + // return reconcile.Result{}, err + // } + + var ( + scalingGroupName = rollingUpgrade.ScalingGroupName() + inProgress bool + ) + + r.AdmissionMap.Range(func(k, v interface{}) bool { + val := v.(string) + if strings.EqualFold(val, scalingGroupName) { + r.Info("object already being processed", "name", k, "scalingGroup", scalingGroupName) + inProgress = true + return false + } + return true + }) + + if inProgress { + return ctrl.Result{}, nil + } + + r.Info("admitted new rollingupgrade", "name", req.NamespacedName, "scalingGroup", scalingGroupName) + r.AdmissionMap.Store(req.NamespacedName, scalingGroupName) + + // TODO: Cloud Discovery - discover AWS / K8s resources + + // TODO: State - determine state / requeue - // your logic here + // TODO: Process - rotate nodes return ctrl.Result{}, nil } diff --git a/controllers/state.go b/controllers/state.go new file mode 100644 index 00000000..5fbf1c8c --- /dev/null +++ b/controllers/state.go @@ -0,0 +1,19 @@ +/* +Copyright 2021 Intuit Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package controllers + +// TODO: State discovery logic diff --git a/controllers/suite_test.go b/controllers/suite_test.go deleted file mode 100644 index 61786df2..00000000 --- a/controllers/suite_test.go +++ /dev/null @@ -1,79 +0,0 @@ -/* -Copyright 2021 Intuit Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package controllers - -import ( - "path/filepath" - "testing" - - . "github.com/onsi/ginkgo" - . "github.com/onsi/gomega" - "k8s.io/client-go/kubernetes/scheme" - "k8s.io/client-go/rest" - "sigs.k8s.io/controller-runtime/pkg/client" - "sigs.k8s.io/controller-runtime/pkg/envtest" - "sigs.k8s.io/controller-runtime/pkg/envtest/printer" - logf "sigs.k8s.io/controller-runtime/pkg/log" - "sigs.k8s.io/controller-runtime/pkg/log/zap" - - upgrademgrv1alpha1 "github.com/keikoproj/upgrade-manager/api/v1alpha1" - //+kubebuilder:scaffold:imports -) - -// These tests use Ginkgo (BDD-style Go testing framework). Refer to -// http://onsi.github.io/ginkgo/ to learn more about Ginkgo. - -var cfg *rest.Config -var k8sClient client.Client -var testEnv *envtest.Environment - -func TestAPIs(t *testing.T) { - RegisterFailHandler(Fail) - - RunSpecsWithDefaultAndCustomReporters(t, - "Controller Suite", - []Reporter{printer.NewlineReporter{}}) -} - -var _ = BeforeSuite(func() { - logf.SetLogger(zap.New(zap.WriteTo(GinkgoWriter), zap.UseDevMode(true))) - - By("bootstrapping test environment") - testEnv = &envtest.Environment{ - CRDDirectoryPaths: []string{filepath.Join("..", "config", "crd", "bases")}, - } - - cfg, err := testEnv.Start() - Expect(err).NotTo(HaveOccurred()) - Expect(cfg).NotTo(BeNil()) - - err = upgrademgrv1alpha1.AddToScheme(scheme.Scheme) - Expect(err).NotTo(HaveOccurred()) - - //+kubebuilder:scaffold:scheme - - k8sClient, err = client.New(cfg, client.Options{Scheme: scheme.Scheme}) - Expect(err).NotTo(HaveOccurred()) - Expect(k8sClient).NotTo(BeNil()) - -}, 60) - -var _ = AfterSuite(func() { - By("tearing down the test environment") - err := testEnv.Stop() - Expect(err).NotTo(HaveOccurred()) -}) diff --git a/controllers/upgrade.go b/controllers/upgrade.go new file mode 100644 index 00000000..b26d42fd --- /dev/null +++ b/controllers/upgrade.go @@ -0,0 +1,19 @@ +/* +Copyright 2021 Intuit Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package controllers + +// TODO: main node rotation logic diff --git a/go.mod b/go.mod index 46a0b2a8..99b95918 100644 --- a/go.mod +++ b/go.mod @@ -3,9 +3,11 @@ module github.com/keikoproj/upgrade-manager go 1.15 require ( + github.com/aws/aws-sdk-go v1.36.24 github.com/go-logr/logr v0.3.0 - github.com/onsi/ginkgo v1.14.1 - github.com/onsi/gomega v1.10.2 + github.com/keikoproj/aws-sdk-go-cache v0.0.0-20201118182730-f6f418a4e2df + github.com/sirupsen/logrus v1.6.0 + go.uber.org/zap v1.15.0 k8s.io/api v0.19.2 k8s.io/apimachinery v0.19.2 k8s.io/client-go v0.19.2 diff --git a/go.sum b/go.sum index 1a4d3c60..9b81a34f 100644 --- a/go.sum +++ b/go.sum @@ -49,6 +49,9 @@ github.com/andreyvit/diff v0.0.0-20170406064948-c7f18ee00883/go.mod h1:rCTlJbsFo github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5doyWs3UAsr3K4I6qtAmlQcZDesFNEHPZAzj8= github.com/asaskevich/govalidator v0.0.0-20180720115003-f9ffefc3facf/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY= github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY= +github.com/aws/aws-sdk-go v1.35.7/go.mod h1:tlPOdRjfxPBpNIwqDj61rmsnA85v9jc0Ps9+muhnW+k= +github.com/aws/aws-sdk-go v1.36.24 h1:uVuio0zA5ideP3DGZDpIoExQJd0WcoNUVlNZaKwBnf8= +github.com/aws/aws-sdk-go v1.36.24/go.mod h1:hcU610XS61/+aQV88ixoOzUoG7v3b31pl2zKMmprdro= github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q= github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8= github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM= @@ -163,6 +166,7 @@ github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7a github.com/gogo/protobuf v1.2.1/go.mod h1:hp+jE20tsWTFYpLwKvXlhS1hjn+gTNwPg2I6zVXpSg4= github.com/gogo/protobuf v1.3.1 h1:DqDEcV5aeaTmdFBePNpYsp3FlcVH/2ISVVM9Qf8PSls= github.com/gogo/protobuf v1.3.1/go.mod h1:SlYgWuQ5SjCEi6WLHjHCa1yvBfUnHcTbrrZtXPKa29o= +github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b h1:VKtxabqXZkF25pY9ekfRL6a582T4P37/31XEstQ5p58= github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= github.com/golang/groupcache v0.0.0-20160516000752-02826c3e7903/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= github.com/golang/groupcache v0.0.0-20190129154638-5b532d6fd5ef/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= @@ -227,6 +231,10 @@ github.com/imdario/mergo v0.3.5/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJ github.com/imdario/mergo v0.3.10 h1:6q5mVkdH/vYmqngx7kZQTjJ5HRsx+ImorDIEQ+beJgc= github.com/imdario/mergo v0.3.10/go.mod h1:jmQim1M+e3UYxmgPu/WyfjB3N3VflVyUjjjwH0dnCYA= github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8= +github.com/jmespath/go-jmespath v0.4.0 h1:BEgLn5cpjn8UN1mAw4NjwDrS35OdebyEtFe+9YPoQUg= +github.com/jmespath/go-jmespath v0.4.0/go.mod h1:T8mJZnbsbmF+m6zOOFylbeCJqk5+pHWvzYPziyZiYoo= +github.com/jmespath/go-jmespath/internal/testify v1.5.1 h1:shLQSRRSCCPj3f2gpwzGwWFoC7ycTf1rcQZHOlsJ6N8= +github.com/jmespath/go-jmespath/internal/testify v1.5.1/go.mod h1:L3OGu8Wl2/fWfCI6z80xFu9LTZmf1ZRjMHUOPmWr69U= github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo= github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU= github.com/json-iterator/go v1.1.7/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4= @@ -235,10 +243,17 @@ github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/ github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU= github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk= github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w= +github.com/karlseguin/ccache v2.0.3+incompatible h1:j68C9tWOROiOLWTS/kCGg9IcJG+ACqn5+0+t8Oh83UU= +github.com/karlseguin/ccache v2.0.3+incompatible/go.mod h1:CM9tNPzT6EdRh14+jiW8mEF9mkNZuuE51qmgGYUB93w= +github.com/karlseguin/expect v1.0.1 h1:z4wy4npwwHSWKjGWH85WNJO42VQhovxTCZDSzhjo8hY= +github.com/karlseguin/expect v1.0.1/go.mod h1:zNBxMY8P21owkeogJELCLeHIt+voOSduHYTFUbwRAV8= +github.com/keikoproj/aws-sdk-go-cache v0.0.0-20201118182730-f6f418a4e2df h1:5CIVZTNmDF4GwbyQzRGYLoG1mo2LHJSO4UwAUDNpgDw= +github.com/keikoproj/aws-sdk-go-cache v0.0.0-20201118182730-f6f418a4e2df/go.mod h1:WuCkHvglMhs9DQnwssll4dy87h352LIfN3qfyk6l6Rg= github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q= github.com/kisielk/errcheck v1.2.0/go.mod h1:/BMXB+zMLi60iA8Vv6Ksmxu/1UDYcXs4uQLJ+jE2L00= github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ= +github.com/konsorten/go-windows-terminal-sequences v1.0.3 h1:CE8S1cTafDpPvMhIxNJKvHsGVBgn1xWYf1NbHQhywc8= github.com/konsorten/go-windows-terminal-sequences v1.0.3/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ= github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc= github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= @@ -313,11 +328,13 @@ github.com/prometheus/client_model v0.2.0/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6T github.com/prometheus/common v0.0.0-20181113130724-41aa239b4cce/go.mod h1:daVV7qP5qjZbuso7PdcryaAu0sAZbrN9i7WWcTMWvro= github.com/prometheus/common v0.4.0/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4= github.com/prometheus/common v0.4.1/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4= +github.com/prometheus/common v0.9.1/go.mod h1:yhUN8i9wzaXS3w1O07YhxHEBxD+W35wd8bs7vj7HSQ4= github.com/prometheus/common v0.10.0 h1:RyRA7RzGXQZiW+tGMr7sxa85G1z0yOpM1qq5c8lNawc= github.com/prometheus/common v0.10.0/go.mod h1:Tlit/dnDKsSWFlCLTWaA1cyBgKHSMdTB80sz/V91rCo= github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk= github.com/prometheus/procfs v0.0.0-20190507164030-5867b95ac084/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA= github.com/prometheus/procfs v0.0.2/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA= +github.com/prometheus/procfs v0.0.8/go.mod h1:7Qr8sr6344vo1JqZ6HhLceV9o3AJ1Ff+GxbHq6oeK9A= github.com/prometheus/procfs v0.1.3 h1:F0+tqvhOksq22sc6iCHF5WGlWjdwj92p0udFh1VFBS8= github.com/prometheus/procfs v0.1.3/go.mod h1:lV6e/gmhEcM9IjHGsFOCxxuZ+z1YqCvr4OA4YeYWdaU= github.com/prometheus/tsdb v0.7.1/go.mod h1:qhTCs0VvXwvX/y3TZrWD7rabWM+ijKTux40TwIPHuXU= @@ -328,6 +345,7 @@ github.com/sergi/go-diff v1.0.0/go.mod h1:0CfEIISq7TuYL3j771MWULgwwjU+GofnZX9QAm github.com/shurcooL/sanitized_anchor_name v1.0.0/go.mod h1:1NzhyTcUVG4SuEtjjoZeVRXNmyL/1OwPU0+IJeTBvfc= github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo= github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE= +github.com/sirupsen/logrus v1.6.0 h1:UBcNElsrwanuuMsnGSlYmtmgbb23qDR5dG+6X6Oo89I= github.com/sirupsen/logrus v1.6.0/go.mod h1:7uNnSEd1DgxDLC74fIahvMZmmYsHGZGEOFrfsX/uA88= github.com/soheilhy/cmux v0.1.4/go.mod h1:IM3LyeVVIOuxMH7sFAkER9+bJ4dT7Ms6E4xg4kGIyLM= github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA= @@ -358,6 +376,8 @@ github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1 github.com/ugorji/go v1.1.4/go.mod h1:uQMGLiO92mf5W77hV/PUCpI3pbzQx3CRekS0kk+RGrc= github.com/urfave/cli v1.20.0/go.mod h1:70zkFmudgCuE/ngEzBv17Jvp/497gISqfk5gWijbERA= github.com/vektah/gqlparser v1.1.2/go.mod h1:1ycwN7Ij5njmMkPPAOaRFY4rET2Enx7IkVv3vaXspKw= +github.com/wsxiaoys/terminal v0.0.0-20160513160801-0940f3fc43a0 h1:3UeQBvD0TFrlVjOeLOBz+CPAI8dnbqNSVwUwRrkp7vQ= +github.com/wsxiaoys/terminal v0.0.0-20160513160801-0940f3fc43a0/go.mod h1:IXCdmsXIht47RaVFLEdVnh1t+pgYtTAhQGj73kz+2DM= github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU= github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q= github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= @@ -441,11 +461,14 @@ golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLL golang.org/x/net v0.0.0-20190813141303-74dc4d7220e7/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20190827160401-ba9fcec4b297/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20191209160850-c0dbc17a3553/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20200202094626-16171245cfb2/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20200324143707-d3edc9973b7e/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A= golang.org/x/net v0.0.0-20200520004742-59133d7f0dd7/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A= golang.org/x/net v0.0.0-20200707034311-ab3426394381 h1:VXak5I6aEWmAXeQjA+QSZzlgNrpq9mjcfDemuexIKsU= golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA= +golang.org/x/net v0.0.0-20201110031124-69a78807bb2b h1:uwuIcX0g4Yl1NC5XAz37xsr2lTtcqevgzYNVt49waME= +golang.org/x/net v0.0.0-20201110031124-69a78807bb2b/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= @@ -486,6 +509,8 @@ golang.org/x/sys v0.0.0-20200519105757-fe76b779f299/go.mod h1:h1NjWce9XRLGQEsW7w golang.org/x/sys v0.0.0-20200615200032-f1bc736245b1/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200622214017-ed371f2e16b4 h1:5/PjkGUjvEU5Gl6BxmvKRPpqo2uNMv4rcHBMwzk/st8= golang.org/x/sys v0.0.0-20200622214017-ed371f2e16b4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f h1:+Nyd8tzPX9R7BWHguqsrbFdRx3WQ/1ib8I44HXV5yTA= +golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk= @@ -582,6 +607,8 @@ gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI= gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys= gopkg.in/inf.v0 v0.9.1 h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc= gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw= +gopkg.in/karlseguin/expect.v1 v1.0.1 h1:9u0iUltnhFbJTHaSIH0EP+cuTU5rafIgmcsEsg2JQFw= +gopkg.in/karlseguin/expect.v1 v1.0.1/go.mod h1:uB7QIJBcclvYbwlUDkSCsGjAOMis3fP280LyhuDEf2I= gopkg.in/natefinch/lumberjack.v2 v2.0.0/go.mod h1:l0ndWWf7gzL7RNwBG7wST/UCcT4T24xpD6X8LsfU/+k= gopkg.in/resty.v1 v1.12.0/go.mod h1:mDo4pnntr5jdWRML875a/NmxYqAlA73dVijT2AXvQQo= gopkg.in/square/go-jose.v2 v2.2.2/go.mod h1:M9dMgbHiYLoDGQrXy7OpJDJWiKiU//h+vD76mk0e1AI= @@ -611,7 +638,6 @@ k8s.io/apiextensions-apiserver v0.19.2 h1:oG84UwiDsVDu7dlsGQs5GySmQHCzMhknfhFExJ k8s.io/apiextensions-apiserver v0.19.2/go.mod h1:EYNjpqIAvNZe+svXVx9j4uBaVhTB4C94HkY3w058qcg= k8s.io/apimachinery v0.19.2 h1:5Gy9vQpAGTKHPVOh5c4plE274X8D/6cuEiTO2zve7tc= k8s.io/apimachinery v0.19.2/go.mod h1:DnPGDnARWFvYa3pMHgSxtbZb7gpzzAZ1pTfaUNDVlmA= -k8s.io/apimachinery v0.20.1 h1:LAhz8pKbgR8tUwn7boK+b2HZdt7MiTu2mkYtFMUjTRQ= k8s.io/apiserver v0.19.2/go.mod h1:FreAq0bJ2vtZFj9Ago/X0oNGC51GfubKK/ViOKfVAOA= k8s.io/client-go v0.19.2 h1:gMJuU3xJZs86L1oQ99R4EViAADUPMHHtS9jFshasHSc= k8s.io/client-go v0.19.2/go.mod h1:S5wPhCqyDNAlzM9CnEdgTGV4OqhsW3jGO1UM1epwfJA= diff --git a/main.go b/main.go index f4ed6121..bde12695 100644 --- a/main.go +++ b/main.go @@ -19,26 +19,47 @@ package main import ( "flag" "os" + "time" // Import all Kubernetes client auth plugins (e.g. Azure, GCP, OIDC, etc.) // to ensure that exec-entrypoint and run can make use of them. _ "k8s.io/client-go/plugin/pkg/client/auth" + "github.com/aws/aws-sdk-go/aws" + "github.com/aws/aws-sdk-go/aws/client" + "github.com/aws/aws-sdk-go/aws/request" + "github.com/aws/aws-sdk-go/aws/session" + "github.com/aws/aws-sdk-go/service/autoscaling" + "github.com/aws/aws-sdk-go/service/ec2" + "github.com/go-logr/logr" + "github.com/keikoproj/aws-sdk-go-cache/cache" + upgrademgrv1alpha1 "github.com/keikoproj/upgrade-manager/api/v1alpha1" + "github.com/keikoproj/upgrade-manager/controllers" + "github.com/keikoproj/upgrade-manager/controllers/common/log" + awsprovider "github.com/keikoproj/upgrade-manager/controllers/providers/aws" + kubeprovider "github.com/keikoproj/upgrade-manager/controllers/providers/kubernetes" + uberzap "go.uber.org/zap" + "go.uber.org/zap/zapcore" "k8s.io/apimachinery/pkg/runtime" utilruntime "k8s.io/apimachinery/pkg/util/runtime" clientgoscheme "k8s.io/client-go/kubernetes/scheme" ctrl "sigs.k8s.io/controller-runtime" "sigs.k8s.io/controller-runtime/pkg/healthz" "sigs.k8s.io/controller-runtime/pkg/log/zap" - - upgrademgrv1alpha1 "github.com/keikoproj/upgrade-manager/api/v1alpha1" - "github.com/keikoproj/upgrade-manager/controllers" //+kubebuilder:scaffold:imports ) var ( scheme = runtime.NewScheme() - setupLog = ctrl.Log.WithName("setup") + setupLog = ctrl.Log.WithName("main") +) + +var ( + CacheDefaultTTL = time.Second * 0 + DescribeAutoScalingGroupsTTL = 60 * time.Second + DescribeLaunchTemplatesTTL = 60 * time.Second + CacheMaxItems int64 = 5000 + CacheItemsToPrune uint32 = 500 ) func init() { @@ -49,43 +70,128 @@ func init() { } func main() { - var metricsAddr string - var enableLeaderElection bool - var probeAddr string - flag.StringVar(&metricsAddr, "metrics-bind-address", ":8080", "The address the metric endpoint binds to.") + + var ( + metricsAddr string + probeAddr string + enableLeaderElection bool + namespace string + maxParallel int + maxAPIRetries int + debugMode bool + logMode string + ) + + flag.BoolVar(&debugMode, "debug", false, "enable debug logging") + flag.StringVar(&logMode, "log-format", "text", "Log mode: supported values: text, json.") + flag.StringVar(&metricsAddr, "metrics-addr", ":8080", "The address the metric endpoint binds to.") flag.StringVar(&probeAddr, "health-probe-bind-address", ":8081", "The address the probe endpoint binds to.") - flag.BoolVar(&enableLeaderElection, "leader-elect", false, - "Enable leader election for controller manager. "+ - "Enabling this will ensure there is only one active controller manager.") + flag.BoolVar(&enableLeaderElection, "enable-leader-election", false, + "Enable leader election for controller manager. Enabling this will ensure there is only one active controller manager.") + flag.StringVar(&namespace, "namespace", "", "The namespace in which to watch objects") + flag.IntVar(&maxParallel, "max-parallel", 10, "The max number of parallel rolling upgrades") + flag.IntVar(&maxAPIRetries, "max-api-retries", 12, "The number of maximum retries for failed/rate limited AWS API calls") + opts := zap.Options{ Development: true, } opts.BindFlags(flag.CommandLine) flag.Parse() - ctrl.SetLogger(zap.New(zap.UseFlagOptions(&opts))) + ctrl.SetLogger(newLogger(logMode)) - mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{ + ctrlOpts := ctrl.Options{ Scheme: scheme, MetricsBindAddress: metricsAddr, Port: 9443, HealthProbeBindAddress: probeAddr, LeaderElection: enableLeaderElection, LeaderElectionID: "d6edb06e.keikoproj.io", - }) + } + + if namespace != "" { + ctrlOpts.Namespace = namespace + setupLog.Info("starting watch in namespaced mode", "namespace", namespace) + } else { + setupLog.Info("starting watch in all namespaces") + } + + mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrlOpts) if err != nil { setupLog.Error(err, "unable to start manager") os.Exit(1) } + var region string + if region, err = awsprovider.DeriveRegion(); err != nil { + setupLog.Error(err, "unable to get region") + os.Exit(1) + } + + if debugMode { + log.SetLevel("debug") + } + + retryer := client.DefaultRetryer{ + NumMaxRetries: maxAPIRetries, + MinThrottleDelay: time.Second * 5, + MaxThrottleDelay: time.Second * 60, + MinRetryDelay: time.Second * 1, + MaxRetryDelay: time.Second * 5, + } + + config := aws.NewConfig().WithRegion(region) + config = config.WithCredentialsChainVerboseErrors(true) + config = request.WithRetryer(config, log.NewRetryLogger(retryer)) + sess, err := session.NewSession(config) + if err != nil { + setupLog.Error(err, "failed to create an AWS session") + os.Exit(1) + } + + cacheCfg := cache.NewConfig(CacheDefaultTTL, CacheMaxItems, CacheItemsToPrune) + cache.AddCaching(sess, cacheCfg) + cacheCfg.SetCacheTTL("autoscaling", "DescribeAutoScalingGroups", DescribeAutoScalingGroupsTTL) + cacheCfg.SetCacheTTL("ec2", "DescribeLaunchTemplates", DescribeLaunchTemplatesTTL) + sess.Handlers.Complete.PushFront(func(r *request.Request) { + ctx := r.HTTPRequest.Context() + log.Debugf("cache hit => %v, service => %s.%s", + cache.IsCacheHit(ctx), + r.ClientInfo.ServiceName, + r.Operation.Name, + ) + }) + + kube, err := kubeprovider.GetKubernetesClient() + if err != nil { + setupLog.Error(err, "unable to create kubernetes client") + os.Exit(1) + } + + awsClient := awsprovider.AmazonClientSet{ + Ec2Client: ec2.New(sess), + AsgClient: autoscaling.New(sess), + } + + kubeClient := kubeprovider.KubernetesClientSet{ + Kubernetes: kube, + } + + logger := ctrl.Log.WithName("controllers").WithName("RollingUpgrade") if err = (&controllers.RollingUpgradeReconciler{ - Client: mgr.GetClient(), - Log: ctrl.Log.WithName("controllers").WithName("RollingUpgrade"), - Scheme: mgr.GetScheme(), + Client: mgr.GetClient(), + Logger: logger, + Scheme: mgr.GetScheme(), + CacheConfig: cacheCfg, + Auth: &controllers.RollingUpgradeAuthenticator{ + AmazonClientSet: awsClient, + KubernetesClientSet: kubeClient, + }, }).SetupWithManager(mgr); err != nil { setupLog.Error(err, "unable to create controller", "controller", "RollingUpgrade") os.Exit(1) } + //+kubebuilder:scaffold:builder if err := mgr.AddHealthzCheck("health", healthz.Ping); err != nil { @@ -102,4 +208,21 @@ func main() { setupLog.Error(err, "problem running manager") os.Exit(1) } + +} + +func newLogger(logMode string) logr.Logger { + var encoder *zapcore.Encoder = nil + if logMode == "json" { + log.SetJSONFormatter() + jsonEncoder := zapcore.NewJSONEncoder(uberzap.NewProductionEncoderConfig()) + encoder = &jsonEncoder + } + + opts := []zap.Opts{zap.UseDevMode(true), zap.WriteTo(os.Stderr)} + if encoder != nil { + opts = append(opts, zap.Encoder(*encoder)) + } + logger := zap.New(opts...) + return logger } From 8f33f1e94f7ae1636eb1402070ca7fce888b7f31 Mon Sep 17 00:00:00 2001 From: Eytan Avisror Date: Tue, 12 Jan 2021 15:09:27 -0800 Subject: [PATCH 06/74] add more scaffolding Signed-off-by: Eytan Avisror --- api/v1alpha1/rollingupgrade_types.go | 17 ++++++- controllers/cloud.go | 43 +++++++++++++++++ controllers/common/log/log.go | 16 +++++++ controllers/common/log/retry.go | 16 +++++++ controllers/common/utils.go | 29 +++++++++++ controllers/rollingupgrade_controller.go | 61 +++++++++++++++++++++--- controllers/state.go | 13 ++++- controllers/upgrade.go | 6 +++ main.go | 9 +++- 9 files changed, 198 insertions(+), 12 deletions(-) create mode 100644 controllers/common/utils.go diff --git a/api/v1alpha1/rollingupgrade_types.go b/api/v1alpha1/rollingupgrade_types.go index f472a48b..947e9049 100644 --- a/api/v1alpha1/rollingupgrade_types.go +++ b/api/v1alpha1/rollingupgrade_types.go @@ -101,6 +101,7 @@ type NodeReadinessGate struct { const ( // Status + StatusInit = "init" StatusRunning = "running" StatusComplete = "completed" StatusError = "error" @@ -109,6 +110,10 @@ const ( UpgradeComplete UpgradeConditionType = "Complete" ) +var ( + FiniteStates = []string{StatusComplete, StatusError} +) + // RollingUpgradeCondition describes the state of the RollingUpgrade type RollingUpgradeCondition struct { Type UpgradeConditionType `json:"type,omitempty"` @@ -140,10 +145,18 @@ func (c UpdateStrategyMode) String() string { } // NamespacedName returns namespaced name of the object. -func (r RollingUpgrade) NamespacedName() string { +func (r *RollingUpgrade) NamespacedName() string { return fmt.Sprintf("%s/%s", r.Namespace, r.Name) } -func (r RollingUpgrade) ScalingGroupName() string { +func (r *RollingUpgrade) ScalingGroupName() string { return r.Spec.AsgName } + +func (r *RollingUpgrade) CurrentStatus() string { + return r.Status.CurrentStatus +} + +func (r *RollingUpgrade) SetCurrentStatus(status string) { + r.Status.CurrentStatus = status +} diff --git a/controllers/cloud.go b/controllers/cloud.go index b6a8d7a0..dc314c76 100644 --- a/controllers/cloud.go +++ b/controllers/cloud.go @@ -16,4 +16,47 @@ limitations under the License. package controllers +import ( + "github.com/aws/aws-sdk-go/service/autoscaling" + "github.com/aws/aws-sdk-go/service/ec2" + "github.com/go-logr/logr" + corev1 "k8s.io/api/core/v1" +) + // TODO: Resource discovery for AWS & Kubernetes + +type DiscoveredState struct { + *RollingUpgradeAuthenticator + logr.Logger + ClusterNodes []corev1.NodeList + LaunchTemplates []*ec2.LaunchTemplate + ScalingGroups []*autoscaling.Group + InProgressInstances []*ec2.Instance +} + +func NewDiscoveredState(auth *RollingUpgradeAuthenticator, logger logr.Logger) *DiscoveredState { + return &DiscoveredState{ + RollingUpgradeAuthenticator: auth, + Logger: logger, + } +} + +func (d *DiscoveredState) Discover() error { + + // DescribeLaunchTemplatesPages + + // DescribeAutoScalingGroupsPages + + // DescribeInstancesPages with filter upgrademgr.keikoproj.io/state=in-progress + + // List Nodes + + return nil +} + +func (d *DiscoveredState) IsConfigurationDrift() bool { + + // Check if launch template / launch config mismatch from scaling group + + return false +} diff --git a/controllers/common/log/log.go b/controllers/common/log/log.go index 6f42c02a..4d6f588b 100644 --- a/controllers/common/log/log.go +++ b/controllers/common/log/log.go @@ -1,3 +1,19 @@ +/* +Copyright 2021 Intuit Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + package log import ( diff --git a/controllers/common/log/retry.go b/controllers/common/log/retry.go index ab7e4cc3..9cdaaae9 100644 --- a/controllers/common/log/retry.go +++ b/controllers/common/log/retry.go @@ -1,3 +1,19 @@ +/* +Copyright 2021 Intuit Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + package log import ( diff --git a/controllers/common/utils.go b/controllers/common/utils.go new file mode 100644 index 00000000..5bb3ca44 --- /dev/null +++ b/controllers/common/utils.go @@ -0,0 +1,29 @@ +/* +Copyright 2021 Intuit Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package common + +import "strings" + +// ContainsEqualFold returns true if a given slice 'slice' contains string 's' under unicode case-folding +func ContainsEqualFold(slice []string, s string) bool { + for _, item := range slice { + if strings.EqualFold(item, s) { + return true + } + } + return false +} diff --git a/controllers/rollingupgrade_controller.go b/controllers/rollingupgrade_controller.go index a530eaa0..f10a9631 100644 --- a/controllers/rollingupgrade_controller.go +++ b/controllers/rollingupgrade_controller.go @@ -20,16 +20,20 @@ import ( "context" "strings" "sync" + "time" "github.com/go-logr/logr" "github.com/keikoproj/aws-sdk-go-cache/cache" + "github.com/keikoproj/upgrade-manager/api/v1alpha1" upgrademgrv1alpha1 "github.com/keikoproj/upgrade-manager/api/v1alpha1" + "github.com/keikoproj/upgrade-manager/controllers/common" awsprovider "github.com/keikoproj/upgrade-manager/controllers/providers/aws" kubeprovider "github.com/keikoproj/upgrade-manager/controllers/providers/kubernetes" kerrors "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/runtime" ctrl "sigs.k8s.io/controller-runtime" "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/controller" "sigs.k8s.io/controller-runtime/pkg/reconcile" ) @@ -41,6 +45,8 @@ type RollingUpgradeReconciler struct { AdmissionMap sync.Map CacheConfig *cache.Config Auth *RollingUpgradeAuthenticator + Cloud *DiscoveredState + maxParallel int } type RollingUpgradeAuthenticator struct { @@ -74,7 +80,15 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque // If the resource is being deleted, remove it from the admissionMap if !rollingUpgrade.DeletionTimestamp.IsZero() { r.AdmissionMap.Delete(req.NamespacedName) - r.Info("deleted object from admission map", "name", req.NamespacedName) + r.Info("rolling upgrade deleted", "name", req.NamespacedName) + return reconcile.Result{}, nil + } + + // Stop processing upgrades which are in finite state + currentStatus := rollingUpgrade.CurrentStatus() + if common.ContainsEqualFold(v1alpha1.FiniteStates, currentStatus) { + r.AdmissionMap.Delete(req.NamespacedName) + r.Info("rolling upgrade ended", "name", req.NamespacedName, "status", currentStatus) return reconcile.Result{}, nil } @@ -92,10 +106,15 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque inProgress bool ) + // Defer a status update on the resource + defer r.UpdateStatus(rollingUpgrade) + + // handle condition where multiple resources submitted targeting the same scaling group by requeing r.AdmissionMap.Range(func(k, v interface{}) bool { val := v.(string) - if strings.EqualFold(val, scalingGroupName) { - r.Info("object already being processed", "name", k, "scalingGroup", scalingGroupName) + resource := k.(string) + if strings.EqualFold(val, scalingGroupName) && !strings.EqualFold(resource, rollingUpgrade.NamespacedName()) { + r.Info("object already being processed by existing resource", "resource", resource, "scalingGroup", scalingGroupName) inProgress = true return false } @@ -103,17 +122,31 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque }) if inProgress { - return ctrl.Result{}, nil + // requeue any resources which are already being processed by a different resource, until the resource is completed/deleted + return ctrl.Result{RequeueAfter: time.Second * 30}, nil } r.Info("admitted new rollingupgrade", "name", req.NamespacedName, "scalingGroup", scalingGroupName) r.AdmissionMap.Store(req.NamespacedName, scalingGroupName) + rollingUpgrade.SetCurrentStatus(v1alpha1.StatusInit) - // TODO: Cloud Discovery - discover AWS / K8s resources + discoveredState := NewDiscoveredState(r.Auth, r.Logger) + if err := discoveredState.Discover(); err != nil { + rollingUpgrade.SetCurrentStatus(v1alpha1.StatusError) + return ctrl.Result{}, err + } - // TODO: State - determine state / requeue + // determine and set state + r.DiscoverState(rollingUpgrade) - // TODO: Process - rotate nodes + // process node rotation + if !common.ContainsEqualFold(v1alpha1.FiniteStates, rollingUpgrade.CurrentStatus()) { + if err := r.RotateNodes(rollingUpgrade); err != nil { + rollingUpgrade.SetCurrentStatus(v1alpha1.StatusError) + return ctrl.Result{}, err + } + return reconcile.Result{RequeueAfter: time.Second * 10}, nil + } return ctrl.Result{}, nil } @@ -122,5 +155,19 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque func (r *RollingUpgradeReconciler) SetupWithManager(mgr ctrl.Manager) error { return ctrl.NewControllerManagedBy(mgr). For(&upgrademgrv1alpha1.RollingUpgrade{}). + WithOptions(controller.Options{MaxConcurrentReconciles: r.maxParallel}). Complete(r) } + +func (r *RollingUpgradeReconciler) SetMaxParallel(n int) { + if n >= 1 { + r.Info("setting max parallel reconcile", "value", n) + r.maxParallel = n + } +} + +func (r *RollingUpgradeReconciler) UpdateStatus(rollingUpgrade *v1alpha1.RollingUpgrade) { + if err := r.Status().Update(context.Background(), rollingUpgrade); err != nil { + r.Error(err, "failed to update status", "name", rollingUpgrade.NamespacedName()) + } +} diff --git a/controllers/state.go b/controllers/state.go index 5fbf1c8c..9163aca5 100644 --- a/controllers/state.go +++ b/controllers/state.go @@ -16,4 +16,15 @@ limitations under the License. package controllers -// TODO: State discovery logic +import "github.com/keikoproj/upgrade-manager/api/v1alpha1" + +func (r *RollingUpgradeReconciler) DiscoverState(rollingUpgrade *v1alpha1.RollingUpgrade) { + + if rollingUpgrade.CurrentStatus() != v1alpha1.StatusInit { + return + } + + // Detect if launch template / config drifted via discovered state + + // set currentStatus to running or completed +} diff --git a/controllers/upgrade.go b/controllers/upgrade.go index b26d42fd..becd8109 100644 --- a/controllers/upgrade.go +++ b/controllers/upgrade.go @@ -16,4 +16,10 @@ limitations under the License. package controllers +import "github.com/keikoproj/upgrade-manager/api/v1alpha1" + // TODO: main node rotation logic +func (r *RollingUpgradeReconciler) RotateNodes(rollingUpgrade *v1alpha1.RollingUpgrade) error { + + return nil +} diff --git a/main.go b/main.go index bde12695..1490a770 100644 --- a/main.go +++ b/main.go @@ -178,7 +178,8 @@ func main() { } logger := ctrl.Log.WithName("controllers").WithName("RollingUpgrade") - if err = (&controllers.RollingUpgradeReconciler{ + + reconciler := &controllers.RollingUpgradeReconciler{ Client: mgr.GetClient(), Logger: logger, Scheme: mgr.GetScheme(), @@ -187,7 +188,11 @@ func main() { AmazonClientSet: awsClient, KubernetesClientSet: kubeClient, }, - }).SetupWithManager(mgr); err != nil { + } + + reconciler.SetMaxParallel(maxParallel) + + if err = (reconciler).SetupWithManager(mgr); err != nil { setupLog.Error(err, "unable to create controller", "controller", "RollingUpgrade") os.Exit(1) } From 41bd5713bf23a214a5e08e3fd051c3b743062ce9 Mon Sep 17 00:00:00 2001 From: Eytan Avisror Date: Tue, 12 Jan 2021 17:53:22 -0800 Subject: [PATCH 07/74] Add kubernetes API calls Signed-off-by: Eytan Avisror --- api/v1alpha1/rollingupgrade_types.go | 5 ++ controllers/providers/kubernetes/events.go | 81 +++++++++++++++++++++- controllers/providers/kubernetes/nodes.go | 18 ++++- controllers/rollingupgrade_controller.go | 12 ++-- main.go | 1 + 5 files changed, 107 insertions(+), 10 deletions(-) diff --git a/api/v1alpha1/rollingupgrade_types.go b/api/v1alpha1/rollingupgrade_types.go index 947e9049..0bcd91ff 100644 --- a/api/v1alpha1/rollingupgrade_types.go +++ b/api/v1alpha1/rollingupgrade_types.go @@ -160,3 +160,8 @@ func (r *RollingUpgrade) CurrentStatus() string { func (r *RollingUpgrade) SetCurrentStatus(status string) { r.Status.CurrentStatus = status } + +// Migrate r.setDefaultsForRollingUpdateStrategy & r.validateRollingUpgradeObj into v1alpha1 RollingUpgrade.Validate() +func (r *RollingUpgrade) Validate() (bool, error) { + return true, nil +} diff --git a/controllers/providers/kubernetes/events.go b/controllers/providers/kubernetes/events.go index 4351aec8..2de0f95d 100644 --- a/controllers/providers/kubernetes/events.go +++ b/controllers/providers/kubernetes/events.go @@ -16,4 +16,83 @@ limitations under the License. package kubernetes -// TODO: Kubernetes Events API calls +import ( + "context" + "encoding/json" + "fmt" + "math/rand" + "time" + + "github.com/go-logr/logr" + "github.com/keikoproj/upgrade-manager/api/v1alpha1" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/client-go/kubernetes" +) + +type EventWriter struct { + kubernetes.Interface + logr.Logger +} + +func NewEventWriter(k kubernetes.Interface, logger logr.Logger) *EventWriter { + return &EventWriter{ + Interface: k, + Logger: logger, + } +} + +// EventReason defines the reason of an event +type EventReason string + +// EventLevel defines the level of an event +type EventLevel string + +const ( + // EventLevelNormal is the level of a normal event + EventLevelNormal = "Normal" + // EventLevelWarning is the level of a warning event + EventLevelWarning = "Warning" + // EventReasonRUStarted Rolling Upgrade Started + EventReasonRUStarted EventReason = "RollingUpgradeStarted" + // EventReasonRUInstanceStarted Rolling Upgrade for Instance has started + EventReasonRUInstanceStarted EventReason = "RollingUpgradeInstanceStarted" + // EventReasonRUInstanceFinished Rolling Upgrade for Instance has finished + EventReasonRUInstanceFinished EventReason = "RollingUpgradeInstanceFinished" + // EventReasonRUFinished Rolling Upgrade Finished + EventReasonRUFinished EventReason = "RollingUpgradeFinished" +) + +func (w *EventWriter) CreateEvent(rollingUpgrade *v1alpha1.RollingUpgrade, reason EventReason, level string, msgFields map[string]string) { + b, _ := json.Marshal(msgFields) + msgPayload := string(b) + t := metav1.Time{Time: time.Now()} + event := &corev1.Event{ + ObjectMeta: metav1.ObjectMeta{ + Name: fmt.Sprintf("%v-%v.%v", rollingUpgrade.GetName(), time.Now().Unix(), rand.Int()), + }, + Source: corev1.EventSource{ + Component: "upgrade-manager", + }, + InvolvedObject: corev1.ObjectReference{ + Kind: "RollingUpgrade", + Name: rollingUpgrade.GetName(), + Namespace: rollingUpgrade.GetNamespace(), + ResourceVersion: rollingUpgrade.GetResourceVersion(), + APIVersion: v1alpha1.GroupVersion.Version, + UID: rollingUpgrade.GetUID(), + }, + Reason: string(reason), + Message: msgPayload, + Type: level, + Count: 1, + FirstTimestamp: t, + LastTimestamp: t, + } + + w.V(1).Info("publishing event", "event", event) + _, err := w.CoreV1().Events(rollingUpgrade.GetNamespace()).Create(context.Background(), event, metav1.CreateOptions{}) + if err != nil { + w.Error(err, "failed to publish event") + } +} diff --git a/controllers/providers/kubernetes/nodes.go b/controllers/providers/kubernetes/nodes.go index 7a0e60b4..3e75b922 100644 --- a/controllers/providers/kubernetes/nodes.go +++ b/controllers/providers/kubernetes/nodes.go @@ -16,4 +16,20 @@ limitations under the License. package kubernetes -// TODO: Kubernetes Nodes API calls +import ( + "context" + + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/client-go/kubernetes" +) + +// ListClusterNodes gets a list of all nodes in the cluster +func ListClusterNodes(k kubernetes.Interface) (*corev1.NodeList, error) { + var nodes *corev1.NodeList + nodes, err := k.CoreV1().Nodes().List(context.Background(), metav1.ListOptions{}) + if err != nil { + return nodes, err + } + return nodes, nil +} diff --git a/controllers/rollingupgrade_controller.go b/controllers/rollingupgrade_controller.go index f10a9631..d05b1696 100644 --- a/controllers/rollingupgrade_controller.go +++ b/controllers/rollingupgrade_controller.go @@ -46,6 +46,7 @@ type RollingUpgradeReconciler struct { CacheConfig *cache.Config Auth *RollingUpgradeAuthenticator Cloud *DiscoveredState + EventWriter *kubeprovider.EventWriter maxParallel int } @@ -92,14 +93,9 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque return reconcile.Result{}, nil } - // TODO: VALIDATION + SET DEFAULTS - - // Migrate r.setDefaultsForRollingUpdateStrategy & r.validateRollingUpgradeObj into v1alpha1 RollingUpgrade.Validate() - // Include setting of defaults in validation - - // if ok, err := rollingUpgrade.Validate(); !ok { - // return reconcile.Result{}, err - // } + if ok, err := rollingUpgrade.Validate(); !ok { + return reconcile.Result{}, err + } var ( scalingGroupName = rollingUpgrade.ScalingGroupName() diff --git a/main.go b/main.go index 1490a770..f33ba854 100644 --- a/main.go +++ b/main.go @@ -188,6 +188,7 @@ func main() { AmazonClientSet: awsClient, KubernetesClientSet: kubeClient, }, + EventWriter: kubeprovider.NewEventWriter(kubeClient.Kubernetes, logger), } reconciler.SetMaxParallel(maxParallel) From 335fb4f914eecdc0cc73f68c77c5842319322353 Mon Sep 17 00:00:00 2001 From: Eytan Avisror Date: Tue, 12 Jan 2021 18:12:06 -0800 Subject: [PATCH 08/74] aws API calls Signed-off-by: Eytan Avisror --- controllers/providers/aws/autoscaling.go | 16 +++++++++++++++- controllers/providers/aws/ec2.go | 16 +++++++++++++++- controllers/providers/kubernetes/events.go | 11 +++++------ controllers/providers/kubernetes/nodes.go | 5 ++--- controllers/rollingupgrade_controller.go | 4 ++-- main.go | 6 +++--- 6 files changed, 42 insertions(+), 16 deletions(-) diff --git a/controllers/providers/aws/autoscaling.go b/controllers/providers/aws/autoscaling.go index 6d2e4e15..2a49c6bc 100644 --- a/controllers/providers/aws/autoscaling.go +++ b/controllers/providers/aws/autoscaling.go @@ -16,4 +16,18 @@ limitations under the License. package aws -// TODO: Autoscaling API Calls +import ( + "github.com/aws/aws-sdk-go/service/autoscaling" +) + +func (a *AmazonClientSet) DescribeScalingGroups() ([]*autoscaling.Group, error) { + scalingGroups := []*autoscaling.Group{} + err := a.AsgClient.DescribeAutoScalingGroupsPages(&autoscaling.DescribeAutoScalingGroupsInput{}, func(page *autoscaling.DescribeAutoScalingGroupsOutput, lastPage bool) bool { + scalingGroups = append(scalingGroups, page.AutoScalingGroups...) + return page.NextToken != nil + }) + if err != nil { + return scalingGroups, err + } + return scalingGroups, nil +} diff --git a/controllers/providers/aws/ec2.go b/controllers/providers/aws/ec2.go index d8cdd1c6..f3f7258e 100644 --- a/controllers/providers/aws/ec2.go +++ b/controllers/providers/aws/ec2.go @@ -16,4 +16,18 @@ limitations under the License. package aws -// TODO: EC2 API Calls +import ( + "github.com/aws/aws-sdk-go/service/ec2" +) + +func (a *AmazonClientSet) DescribeLaunchTemplates() ([]*ec2.LaunchTemplate, error) { + launchTemplates := []*ec2.LaunchTemplate{} + err := a.Ec2Client.DescribeLaunchTemplatesPages(&ec2.DescribeLaunchTemplatesInput{}, func(page *ec2.DescribeLaunchTemplatesOutput, lastPage bool) bool { + launchTemplates = append(launchTemplates, page.LaunchTemplates...) + return page.NextToken != nil + }) + if err != nil { + return launchTemplates, err + } + return launchTemplates, nil +} diff --git a/controllers/providers/kubernetes/events.go b/controllers/providers/kubernetes/events.go index 2de0f95d..6c9ac2bd 100644 --- a/controllers/providers/kubernetes/events.go +++ b/controllers/providers/kubernetes/events.go @@ -27,18 +27,17 @@ import ( "github.com/keikoproj/upgrade-manager/api/v1alpha1" corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - "k8s.io/client-go/kubernetes" ) type EventWriter struct { - kubernetes.Interface + *KubernetesClientSet logr.Logger } -func NewEventWriter(k kubernetes.Interface, logger logr.Logger) *EventWriter { +func NewEventWriter(k *KubernetesClientSet, logger logr.Logger) *EventWriter { return &EventWriter{ - Interface: k, - Logger: logger, + KubernetesClientSet: k, + Logger: logger, } } @@ -91,7 +90,7 @@ func (w *EventWriter) CreateEvent(rollingUpgrade *v1alpha1.RollingUpgrade, reaso } w.V(1).Info("publishing event", "event", event) - _, err := w.CoreV1().Events(rollingUpgrade.GetNamespace()).Create(context.Background(), event, metav1.CreateOptions{}) + _, err := w.Kubernetes.CoreV1().Events(rollingUpgrade.GetNamespace()).Create(context.Background(), event, metav1.CreateOptions{}) if err != nil { w.Error(err, "failed to publish event") } diff --git a/controllers/providers/kubernetes/nodes.go b/controllers/providers/kubernetes/nodes.go index 3e75b922..02501dde 100644 --- a/controllers/providers/kubernetes/nodes.go +++ b/controllers/providers/kubernetes/nodes.go @@ -21,13 +21,12 @@ import ( corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - "k8s.io/client-go/kubernetes" ) // ListClusterNodes gets a list of all nodes in the cluster -func ListClusterNodes(k kubernetes.Interface) (*corev1.NodeList, error) { +func (k *KubernetesClientSet) ListClusterNodes() (*corev1.NodeList, error) { var nodes *corev1.NodeList - nodes, err := k.CoreV1().Nodes().List(context.Background(), metav1.ListOptions{}) + nodes, err := k.Kubernetes.CoreV1().Nodes().List(context.Background(), metav1.ListOptions{}) if err != nil { return nodes, err } diff --git a/controllers/rollingupgrade_controller.go b/controllers/rollingupgrade_controller.go index d05b1696..917a249d 100644 --- a/controllers/rollingupgrade_controller.go +++ b/controllers/rollingupgrade_controller.go @@ -51,8 +51,8 @@ type RollingUpgradeReconciler struct { } type RollingUpgradeAuthenticator struct { - awsprovider.AmazonClientSet - kubeprovider.KubernetesClientSet + *awsprovider.AmazonClientSet + *kubeprovider.KubernetesClientSet } // +kubebuilder:rbac:groups=upgrademgr.keikoproj.io,resources=rollingupgrades,verbs=get;list;watch;create;update;patch;delete diff --git a/main.go b/main.go index f33ba854..3ed1fea3 100644 --- a/main.go +++ b/main.go @@ -168,12 +168,12 @@ func main() { os.Exit(1) } - awsClient := awsprovider.AmazonClientSet{ + awsClient := &awsprovider.AmazonClientSet{ Ec2Client: ec2.New(sess), AsgClient: autoscaling.New(sess), } - kubeClient := kubeprovider.KubernetesClientSet{ + kubeClient := &kubeprovider.KubernetesClientSet{ Kubernetes: kube, } @@ -188,7 +188,7 @@ func main() { AmazonClientSet: awsClient, KubernetesClientSet: kubeClient, }, - EventWriter: kubeprovider.NewEventWriter(kubeClient.Kubernetes, logger), + EventWriter: kubeprovider.NewEventWriter(kubeClient, logger), } reconciler.SetMaxParallel(maxParallel) From 6b8dad50834cf884647f8e3186fc5a0d73925f64 Mon Sep 17 00:00:00 2001 From: Eytan Avisror Date: Tue, 12 Jan 2021 20:39:31 -0800 Subject: [PATCH 09/74] AWS API calls & Drift detection Signed-off-by: Eytan Avisror --- api/v1alpha1/rollingupgrade_types.go | 4 + controllers/cloud.go | 43 +++++++---- controllers/providers/aws/ec2.go | 26 +++++++ controllers/providers/aws/utils.go | 34 +++++++++ controllers/providers/kubernetes/utils.go | 14 ++++ controllers/rollingupgrade_controller.go | 14 +--- controllers/state.go | 30 -------- controllers/upgrade.go | 89 ++++++++++++++++++++++- go.mod | 1 + 9 files changed, 200 insertions(+), 55 deletions(-) delete mode 100644 controllers/state.go diff --git a/api/v1alpha1/rollingupgrade_types.go b/api/v1alpha1/rollingupgrade_types.go index 0bcd91ff..a7aec6d7 100644 --- a/api/v1alpha1/rollingupgrade_types.go +++ b/api/v1alpha1/rollingupgrade_types.go @@ -161,6 +161,10 @@ func (r *RollingUpgrade) SetCurrentStatus(status string) { r.Status.CurrentStatus = status } +func (r *RollingUpgrade) IsForceRefresh() bool { + return r.Spec.ForceRefresh +} + // Migrate r.setDefaultsForRollingUpdateStrategy & r.validateRollingUpgradeObj into v1alpha1 RollingUpgrade.Validate() func (r *RollingUpgrade) Validate() (bool, error) { return true, nil diff --git a/controllers/cloud.go b/controllers/cloud.go index dc314c76..1bf779e4 100644 --- a/controllers/cloud.go +++ b/controllers/cloud.go @@ -20,18 +20,24 @@ import ( "github.com/aws/aws-sdk-go/service/autoscaling" "github.com/aws/aws-sdk-go/service/ec2" "github.com/go-logr/logr" + corev1 "k8s.io/api/core/v1" + + "github.com/pkg/errors" ) -// TODO: Resource discovery for AWS & Kubernetes +var ( + instanceStateTagKey = "upgrademgr.keikoproj.io/state" + inProgressTagValue = "in-progress" +) type DiscoveredState struct { *RollingUpgradeAuthenticator logr.Logger - ClusterNodes []corev1.NodeList + ClusterNodes *corev1.NodeList LaunchTemplates []*ec2.LaunchTemplate ScalingGroups []*autoscaling.Group - InProgressInstances []*ec2.Instance + InProgressInstances []string } func NewDiscoveredState(auth *RollingUpgradeAuthenticator, logger logr.Logger) *DiscoveredState { @@ -43,20 +49,29 @@ func NewDiscoveredState(auth *RollingUpgradeAuthenticator, logger logr.Logger) * func (d *DiscoveredState) Discover() error { - // DescribeLaunchTemplatesPages + launchTemplates, err := d.AmazonClientSet.DescribeLaunchTemplates() + if err != nil { + return errors.Wrap(err, "failed to discover launch templates") + } + d.LaunchTemplates = launchTemplates - // DescribeAutoScalingGroupsPages + scalingGroups, err := d.AmazonClientSet.DescribeScalingGroups() + if err != nil { + return errors.Wrap(err, "failed to discover scaling groups") + } + d.ScalingGroups = scalingGroups - // DescribeInstancesPages with filter upgrademgr.keikoproj.io/state=in-progress + inProgressInstances, err := d.AmazonClientSet.DescribeTaggedInstanceIDs(instanceStateTagKey, inProgressTagValue) + if err != nil { + return errors.Wrap(err, "failed to discover ec2 instances") + } + d.InProgressInstances = inProgressInstances - // List Nodes + nodes, err := d.KubernetesClientSet.ListClusterNodes() + if err != nil { + return errors.Wrap(err, "failed to discover cluster nodes") + } + d.ClusterNodes = nodes return nil } - -func (d *DiscoveredState) IsConfigurationDrift() bool { - - // Check if launch template / launch config mismatch from scaling group - - return false -} diff --git a/controllers/providers/aws/ec2.go b/controllers/providers/aws/ec2.go index f3f7258e..f7c7644a 100644 --- a/controllers/providers/aws/ec2.go +++ b/controllers/providers/aws/ec2.go @@ -17,6 +17,9 @@ limitations under the License. package aws import ( + "fmt" + + "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/service/ec2" ) @@ -31,3 +34,26 @@ func (a *AmazonClientSet) DescribeLaunchTemplates() ([]*ec2.LaunchTemplate, erro } return launchTemplates, nil } + +func (a *AmazonClientSet) DescribeTaggedInstanceIDs(tagKey, tagValue string) ([]string, error) { + instances := []string{} + key := fmt.Sprintf("tag:%v", tagKey) + input := &ec2.DescribeInstancesInput{ + Filters: []*ec2.Filter{ + { + Name: aws.String(key), + Values: aws.StringSlice([]string{tagValue}), + }, + }, + } + + err := a.Ec2Client.DescribeInstancesPages(input, func(page *ec2.DescribeInstancesOutput, lastPage bool) bool { + for _, res := range page.Reservations { + for _, instance := range res.Instances { + instances = append(instances, aws.StringValue(instance.InstanceId)) + } + } + return page.NextToken != nil + }) + return instances, err +} diff --git a/controllers/providers/aws/utils.go b/controllers/providers/aws/utils.go index 13122422..efb01ab4 100644 --- a/controllers/providers/aws/utils.go +++ b/controllers/providers/aws/utils.go @@ -19,11 +19,15 @@ package aws import ( "fmt" "os" + "strconv" + "strings" "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/aws/ec2metadata" "github.com/aws/aws-sdk-go/aws/session" + "github.com/aws/aws-sdk-go/service/autoscaling" "github.com/aws/aws-sdk-go/service/autoscaling/autoscalingiface" + "github.com/aws/aws-sdk-go/service/ec2" "github.com/aws/aws-sdk-go/service/ec2/ec2iface" ) @@ -50,3 +54,33 @@ func DeriveRegion() (string, error) { } return region, nil } + +func SelectScalingGroup(name string, groups []*autoscaling.Group) *autoscaling.Group { + for _, group := range groups { + groupName := aws.StringValue(group.AutoScalingGroupName) + if strings.EqualFold(groupName, name) { + return group + } + } + return &autoscaling.Group{} +} + +// func ListScalingInstanceIDs(group *autoscaling.Group) []string { +// instanceIDs := make([]string, 0) +// for _, instance := range group.Instances { +// instanceID := aws.StringValue(instance.InstanceId) +// instanceIDs = append(instanceIDs, instanceID) +// } +// return instanceIDs +// } + +func GetTemplateLatestVersion(templates []*ec2.LaunchTemplate, templateName string) string { + for _, template := range templates { + name := aws.StringValue(template.LaunchTemplateName) + if strings.EqualFold(name, templateName) { + versionInt := aws.Int64Value(template.LatestVersionNumber) + return strconv.FormatInt(versionInt, 10) + } + } + return "0" +} diff --git a/controllers/providers/kubernetes/utils.go b/controllers/providers/kubernetes/utils.go index a4a251ac..7f37508f 100644 --- a/controllers/providers/kubernetes/utils.go +++ b/controllers/providers/kubernetes/utils.go @@ -20,6 +20,9 @@ import ( "fmt" "os" "os/user" + "strings" + + corev1 "k8s.io/api/core/v1" "k8s.io/client-go/kubernetes" "k8s.io/client-go/rest" @@ -81,3 +84,14 @@ func GetKubernetesLocalConfig() (*rest.Config, error) { } return config, nil } + +func SelectNodeByInstanceID(instanceID string, nodes *corev1.NodeList) corev1.Node { + for _, node := range nodes.Items { + tokens := strings.Split(node.Spec.ProviderID, "/") + nodeID := tokens[len(tokens)-1] + if strings.EqualFold(instanceID, nodeID) { + return node + } + } + return corev1.Node{} +} diff --git a/controllers/rollingupgrade_controller.go b/controllers/rollingupgrade_controller.go index 917a249d..46f954df 100644 --- a/controllers/rollingupgrade_controller.go +++ b/controllers/rollingupgrade_controller.go @@ -132,19 +132,13 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque return ctrl.Result{}, err } - // determine and set state - r.DiscoverState(rollingUpgrade) - // process node rotation - if !common.ContainsEqualFold(v1alpha1.FiniteStates, rollingUpgrade.CurrentStatus()) { - if err := r.RotateNodes(rollingUpgrade); err != nil { - rollingUpgrade.SetCurrentStatus(v1alpha1.StatusError) - return ctrl.Result{}, err - } - return reconcile.Result{RequeueAfter: time.Second * 10}, nil + if err := r.RotateNodes(rollingUpgrade); err != nil { + rollingUpgrade.SetCurrentStatus(v1alpha1.StatusError) + return ctrl.Result{}, err } - return ctrl.Result{}, nil + return reconcile.Result{RequeueAfter: time.Second * 10}, nil } // SetupWithManager sets up the controller with the Manager. diff --git a/controllers/state.go b/controllers/state.go deleted file mode 100644 index 9163aca5..00000000 --- a/controllers/state.go +++ /dev/null @@ -1,30 +0,0 @@ -/* -Copyright 2021 Intuit Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package controllers - -import "github.com/keikoproj/upgrade-manager/api/v1alpha1" - -func (r *RollingUpgradeReconciler) DiscoverState(rollingUpgrade *v1alpha1.RollingUpgrade) { - - if rollingUpgrade.CurrentStatus() != v1alpha1.StatusInit { - return - } - - // Detect if launch template / config drifted via discovered state - - // set currentStatus to running or completed -} diff --git a/controllers/upgrade.go b/controllers/upgrade.go index becd8109..720db6d3 100644 --- a/controllers/upgrade.go +++ b/controllers/upgrade.go @@ -16,10 +16,97 @@ limitations under the License. package controllers -import "github.com/keikoproj/upgrade-manager/api/v1alpha1" +import ( + "strings" + + "github.com/aws/aws-sdk-go/aws" + "github.com/aws/aws-sdk-go/service/autoscaling" + "github.com/keikoproj/upgrade-manager/api/v1alpha1" + awsprovider "github.com/keikoproj/upgrade-manager/controllers/providers/aws" + kubeprovider "github.com/keikoproj/upgrade-manager/controllers/providers/kubernetes" +) // TODO: main node rotation logic func (r *RollingUpgradeReconciler) RotateNodes(rollingUpgrade *v1alpha1.RollingUpgrade) error { return nil } + +func (r *RollingUpgradeReconciler) IsInstanceDrifted(rollingUpgrade *v1alpha1.RollingUpgrade, instance *autoscaling.Instance) bool { + + var ( + scalingGroupName = rollingUpgrade.ScalingGroupName() + scalingGroup = awsprovider.SelectScalingGroup(scalingGroupName, r.Cloud.ScalingGroups) + instanceID = aws.StringValue(instance.InstanceId) + ) + + // check if there is atleast one node that meets the force-referesh criteria + if rollingUpgrade.IsForceRefresh() { + var ( + node = kubeprovider.SelectNodeByInstanceID(instanceID, r.Cloud.ClusterNodes) + nodeCreationTime = node.CreationTimestamp.Time + upgradeCreationTime = rollingUpgrade.CreationTimestamp.Time + ) + if nodeCreationTime.Before(upgradeCreationTime) { + r.Info("rolling upgrade configured for forced refresh", "name", rollingUpgrade.NamespacedName(), "instance", instanceID) + return true + } + } + + if scalingGroup.LaunchConfigurationName != nil { + if instance.LaunchConfigurationName == nil { + r.Info("launch configuration name differs", "name", rollingUpgrade.NamespacedName(), "instance", instanceID) + return true + } + launchConfigName := aws.StringValue(scalingGroup.LaunchConfigurationName) + instanceConfigName := aws.StringValue(instance.LaunchConfigurationName) + if !strings.EqualFold(launchConfigName, instanceConfigName) { + r.Info("launch configuration name differs", "name", rollingUpgrade.NamespacedName(), "instance", instanceID) + return true + } + } else if scalingGroup.LaunchTemplate != nil { + if instance.LaunchTemplate == nil { + r.Info("launch template name differs", "name", rollingUpgrade.NamespacedName(), "instance", instanceID) + return true + } + + var ( + launchTemplateName = aws.StringValue(scalingGroup.LaunchTemplate.LaunchTemplateName) + instanceTemplateName = aws.StringValue(instance.LaunchTemplate.LaunchTemplateName) + instanceTemplateVersion = aws.StringValue(instance.LaunchTemplate.Version) + templateVersion = awsprovider.GetTemplateLatestVersion(r.Cloud.LaunchTemplates, launchTemplateName) + ) + + if !strings.EqualFold(launchTemplateName, instanceTemplateName) { + r.Info("launch template name differs", "name", rollingUpgrade.NamespacedName(), "instance", instanceID) + return true + } else if !strings.EqualFold(instanceTemplateVersion, templateVersion) { + r.Info("launch template version differs", "name", rollingUpgrade.NamespacedName(), "instance", instanceID) + return true + } + + } else if scalingGroup.MixedInstancesPolicy != nil { + if instance.LaunchTemplate == nil { + r.Info("launch template name differs", "name", rollingUpgrade.NamespacedName(), "instance", instanceID) + return true + } + + var ( + launchTemplateName = aws.StringValue(scalingGroup.MixedInstancesPolicy.LaunchTemplate.LaunchTemplateSpecification.LaunchTemplateName) + instanceTemplateName = aws.StringValue(instance.LaunchTemplate.LaunchTemplateName) + instanceTemplateVersion = aws.StringValue(instance.LaunchTemplate.Version) + templateVersion = awsprovider.GetTemplateLatestVersion(r.Cloud.LaunchTemplates, launchTemplateName) + ) + + if !strings.EqualFold(launchTemplateName, instanceTemplateName) { + r.Info("launch template name differs", "name", rollingUpgrade.NamespacedName(), "instance", instanceID) + return true + } else if !strings.EqualFold(instanceTemplateVersion, templateVersion) { + r.Info("launch template version differs", "name", rollingUpgrade.NamespacedName(), "instance", instanceID) + return true + } + } + + r.Info("node refresh not required", "name", rollingUpgrade.NamespacedName(), "instance", instanceID) + return false +} diff --git a/go.mod b/go.mod index 99b95918..6299d030 100644 --- a/go.mod +++ b/go.mod @@ -6,6 +6,7 @@ require ( github.com/aws/aws-sdk-go v1.36.24 github.com/go-logr/logr v0.3.0 github.com/keikoproj/aws-sdk-go-cache v0.0.0-20201118182730-f6f418a4e2df + github.com/pkg/errors v0.9.1 github.com/sirupsen/logrus v1.6.0 go.uber.org/zap v1.15.0 k8s.io/api v0.19.2 From 5cb9efb087c1cf315c1b6e05b9b07a1a7d1f8c1a Mon Sep 17 00:00:00 2001 From: Eytan Avisror Date: Wed, 13 Jan 2021 16:54:03 -0800 Subject: [PATCH 10/74] initial rotation logic Signed-off-by: Eytan Avisror --- api/v1alpha1/rollingupgrade_types.go | 64 +++++++++++-- controllers/common/utils.go | 4 +- controllers/providers/aws/utils.go | 31 ++++++ controllers/upgrade.go | 137 +++++++++++++++++++++++++++ 4 files changed, 228 insertions(+), 8 deletions(-) diff --git a/api/v1alpha1/rollingupgrade_types.go b/api/v1alpha1/rollingupgrade_types.go index a7aec6d7..c3aa3085 100644 --- a/api/v1alpha1/rollingupgrade_types.go +++ b/api/v1alpha1/rollingupgrade_types.go @@ -40,13 +40,15 @@ type RollingUpgradeSpec struct { // RollingUpgradeStatus defines the observed state of RollingUpgrade type RollingUpgradeStatus struct { - CurrentStatus string `json:"currentStatus,omitempty"` - StartTime string `json:"startTime,omitempty"` - EndTime string `json:"endTime,omitempty"` - TotalProcessingTime string `json:"totalProcessingTime,omitempty"` - NodesProcessed int `json:"nodesProcessed,omitempty"` - TotalNodes int `json:"totalNodes,omitempty"` - Conditions []RollingUpgradeCondition `json:"conditions,omitempty"` + CurrentStatus string `json:"currentStatus,omitempty"` + StartTime string `json:"startTime,omitempty"` + EndTime string `json:"endTime,omitempty"` + TotalProcessingTime string `json:"totalProcessingTime,omitempty"` + NodesProcessed int `json:"nodesProcessed,omitempty"` + TotalNodes int `json:"totalNodes,omitempty"` + Conditions []RollingUpgradeCondition `json:"conditions,omitempty"` + LastNodeTerminationTime metav1.Time `json:"lastTerminationTime,omitempty"` + LastNodeDrainTime metav1.Time `json:"lastDrainTime,omitempty"` } // +kubebuilder:object:root=true @@ -157,10 +159,58 @@ func (r *RollingUpgrade) CurrentStatus() string { return r.Status.CurrentStatus } +func (r *RollingUpgrade) UpdateStrategyType() UpdateStrategyType { + return r.Spec.Strategy.Type +} + +func (r *RollingUpgrade) MaxUnavailable() intstr.IntOrString { + return r.Spec.Strategy.MaxUnavailable +} + +func (r *RollingUpgrade) LastNodeTerminationTime() metav1.Time { + return r.Status.LastNodeTerminationTime +} + +func (r *RollingUpgrade) LastNodeDrainTime() metav1.Time { + return r.Status.LastNodeDrainTime +} + +func (r *RollingUpgrade) NodeIntervalSeconds() int { + return r.Spec.NodeIntervalSeconds +} + +func (r *RollingUpgrade) PostDrainDelaySeconds() int { + return r.Spec.PostDrainDelaySeconds +} + func (r *RollingUpgrade) SetCurrentStatus(status string) { r.Status.CurrentStatus = status } +func (r *RollingUpgrade) SetStartTime(t string) { + r.Status.StartTime = t +} + +func (r *RollingUpgrade) StartTime() string { + return r.Status.StartTime +} + +func (r *RollingUpgrade) SetEndTime(t string) { + r.Status.EndTime = t +} + +func (r *RollingUpgrade) EndTime() string { + return r.Status.EndTime +} + +func (r *RollingUpgrade) SetTotalNodes(n int) { + r.Status.TotalNodes = n +} + +func (r *RollingUpgrade) SetNodesProcessed(n int) { + r.Status.NodesProcessed = n +} + func (r *RollingUpgrade) IsForceRefresh() bool { return r.Spec.ForceRefresh } diff --git a/controllers/common/utils.go b/controllers/common/utils.go index 5bb3ca44..4b67ce3f 100644 --- a/controllers/common/utils.go +++ b/controllers/common/utils.go @@ -16,7 +16,9 @@ limitations under the License. package common -import "strings" +import ( + "strings" +) // ContainsEqualFold returns true if a given slice 'slice' contains string 's' under unicode case-folding func ContainsEqualFold(slice []string, s string) bool { diff --git a/controllers/providers/aws/utils.go b/controllers/providers/aws/utils.go index efb01ab4..e5e01bd2 100644 --- a/controllers/providers/aws/utils.go +++ b/controllers/providers/aws/utils.go @@ -19,6 +19,7 @@ package aws import ( "fmt" "os" + "sort" "strconv" "strings" @@ -65,6 +66,36 @@ func SelectScalingGroup(name string, groups []*autoscaling.Group) *autoscaling.G return &autoscaling.Group{} } +func SelectScalingGroupInstance(instanceID string, group *autoscaling.Group) *autoscaling.Instance { + for _, instance := range group.Instances { + selectedID := aws.StringValue(instance.InstanceId) + if strings.EqualFold(instanceID, selectedID) { + return instance + } + } + return &autoscaling.Instance{} +} + +func GetScalingAZs(instances []*autoscaling.Instance) []string { + AZs := make([]string, 0) + for _, instance := range instances { + AZ := aws.StringValue(instance.AvailabilityZone) + AZs = append(AZs, AZ) + } + sort.Strings(AZs) + return AZs +} + +func SelectInstancesByAZ(instances []*autoscaling.Group) *autoscaling.Instance { + for _, instance := range group.Instances { + selectedID := aws.StringValue(instance.InstanceId) + if strings.EqualFold(instanceID, selectedID) { + return instance + } + } + return &autoscaling.Instance{} +} + // func ListScalingInstanceIDs(group *autoscaling.Group) []string { // instanceIDs := make([]string, 0) // for _, instance := range group.Instances { diff --git a/controllers/upgrade.go b/controllers/upgrade.go index 720db6d3..31bb193a 100644 --- a/controllers/upgrade.go +++ b/controllers/upgrade.go @@ -18,20 +18,157 @@ package controllers import ( "strings" + "time" "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/service/autoscaling" "github.com/keikoproj/upgrade-manager/api/v1alpha1" awsprovider "github.com/keikoproj/upgrade-manager/controllers/providers/aws" kubeprovider "github.com/keikoproj/upgrade-manager/controllers/providers/kubernetes" + "k8s.io/apimachinery/pkg/util/intstr" ) // TODO: main node rotation logic func (r *RollingUpgradeReconciler) RotateNodes(rollingUpgrade *v1alpha1.RollingUpgrade) error { + var ( + lastTerminationTime = rollingUpgrade.LastNodeTerminationTime() + nodeInterval = rollingUpgrade.NodeIntervalSeconds() + lastDrainTime = rollingUpgrade.LastNodeDrainTime() + drainInterval = rollingUpgrade.PostDrainDelaySeconds() + ) + rollingUpgrade.SetCurrentStatus(v1alpha1.StatusRunning) + + // set status start time + if rollingUpgrade.StartTime() == "" { + rollingUpgrade.SetStartTime(time.Now().Format(time.RFC3339)) + } + + if !lastTerminationTime.IsZero() || !lastDrainTime.IsZero() { + + // Check if we are still waiting on a termination delay + if time.Since(lastTerminationTime.Time).Seconds() < float64(nodeInterval) { + r.Info("reconcile requeue due to termination interval wait", "name", rollingUpgrade.NamespacedName()) + return nil + } + + // Check if we are still waiting on a drain delay + if time.Since(lastDrainTime.Time).Seconds() < float64(drainInterval) { + r.Info("reconcile requeue due to drain interval wait", "name", rollingUpgrade.NamespacedName()) + return nil + } + } + + var ( + scalingGroup = awsprovider.SelectScalingGroup(rollingUpgrade.ScalingGroupName(), r.Cloud.ScalingGroups) + ) + + rollingUpgrade.SetTotalNodes(len(scalingGroup.Instances)) + rotationTargets := r.SelectTargets(rollingUpgrade, scalingGroup) + r.ReplaceNodeBatch(rollingUpgrade, rotationTargets) return nil } +func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.RollingUpgrade, batch []*autoscaling.Instance) (bool, error) { + var ( + mode = rollingUpgrade.StrategyMode() + ) + + switch mode { + case v1alpha1.UpdateStrategyModeEager: + for _, target := range batch { + // Add in-progress tag + + // Standby + + // Wait for desired nodes + + // Issue drain/scripts concurrently - set lastDrainTime + + // Is drained? + + // Terminate - set lastTerminateTime + } + case v1alpha1.UpdateStrategyModeLazy: + for _, target := range batch { + // Add in-progress tag + + // Issue drain/scripts concurrently - set lastDrainTime + + // Is drained? + + // Terminate - set lastTerminateTime + } + } + +} + +func (r *RollingUpgradeReconciler) SelectTargets(rollingUpgrade *v1alpha1.RollingUpgrade, scalingGroup *autoscaling.Group) []*autoscaling.Instance { + var ( + batchSize = rollingUpgrade.MaxUnavailable() + totalNodes = len(scalingGroup.Instances) + targets = make([]*autoscaling.Instance, 0) + ) + + var unavailableInt int + if batchSize.Type == intstr.String { + unavailableInt, _ = intstr.GetValueFromIntOrPercent(&batchSize, totalNodes, true) + } else { + unavailableInt = batchSize.IntValue() + } + + // first process all in progress instances + for _, instance := range r.Cloud.InProgressInstances { + selectedInstance := awsprovider.SelectScalingGroupInstance(instance, scalingGroup) + targets = append(targets, selectedInstance) + } + + if len(targets) > 0 { + if unavailableInt > len(targets) { + unavailableInt = len(targets) + } + return targets[:unavailableInt] + } + + // select via strategy if there are no in-progress instances + if rollingUpgrade.UpdateStrategyType() == v1alpha1.RandomUpdateStrategy { + for _, instance := range scalingGroup.Instances { + if r.IsInstanceDrifted(rollingUpgrade, instance) { + targets = append(targets, instance) + } + } + if unavailableInt > len(targets) { + unavailableInt = len(targets) + } + return targets[:unavailableInt] + + } else if rollingUpgrade.UpdateStrategyType() == v1alpha1.UniformAcrossAzUpdateStrategy { + for _, instance := range scalingGroup.Instances { + if r.IsInstanceDrifted(rollingUpgrade, instance) { + targets = append(targets, instance) + } + } + + var AZtargets = make([]*autoscaling.Instance, 0) + AZs := awsprovider.GetScalingAZs(targets) + if len(AZs) == 0 { + return AZtargets + } + for _, target := range targets { + AZ := aws.StringValue(target.AvailabilityZone) + if strings.EqualFold(AZ, AZs[0]) { + AZtargets = append(AZtargets, target) + } + } + if unavailableInt > len(AZtargets) { + unavailableInt = len(AZtargets) + } + return AZtargets[:unavailableInt] + } + + return targets +} + func (r *RollingUpgradeReconciler) IsInstanceDrifted(rollingUpgrade *v1alpha1.RollingUpgrade, instance *autoscaling.Instance) bool { var ( From 59e9b0dd962f712b4db685d8b14fb7106987010c Mon Sep 17 00:00:00 2001 From: Shreyas Badiger <7680410+shreyas-badiger@users.noreply.github.com> Date: Wed, 13 Jan 2021 16:55:34 -0800 Subject: [PATCH 11/74] Implemented RollingUpgrade object validation. (#176) * Validation step to check Nodes and ASG launch configs Signed-off-by: shreyas-badiger * Validating launch definition after a rolling upgrade Signed-off-by: shreyas-badiger --- api/v1alpha1/rollingupgrade_types.go | 48 ++++++++++++++++++++++++++-- 1 file changed, 46 insertions(+), 2 deletions(-) diff --git a/api/v1alpha1/rollingupgrade_types.go b/api/v1alpha1/rollingupgrade_types.go index c3aa3085..10b9110b 100644 --- a/api/v1alpha1/rollingupgrade_types.go +++ b/api/v1alpha1/rollingupgrade_types.go @@ -17,7 +17,10 @@ limitations under the License. package v1alpha1 import ( + "common" "fmt" + "strconv" + "strings" corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" @@ -113,7 +116,9 @@ const ( ) var ( - FiniteStates = []string{StatusComplete, StatusError} + FiniteStates = []string{StatusComplete, StatusError} + AllowedStrategyType = []string{string(RandomUpdateStrategy), string(UniformAcrossAzUpdateStrategy)} + AllowedStrategyMode = []string{string(UpdateStrategyModeLazy), string(UpdateStrategyModeEager)} ) // RollingUpgradeCondition describes the state of the RollingUpgrade @@ -215,7 +220,46 @@ func (r *RollingUpgrade) IsForceRefresh() bool { return r.Spec.ForceRefresh } -// Migrate r.setDefaultsForRollingUpdateStrategy & r.validateRollingUpgradeObj into v1alpha1 RollingUpgrade.Validate() func (r *RollingUpgrade) Validate() (bool, error) { + strategy := r.Spec.Strategy + + // validating the Type value + if strategy.Type == "" { + r.Spec.Strategy.Type = RandomUpdateStrategy + } else if !common.ContainsEqualFold(AllowedStrategyType, strategy.Type) { + err := fmt.Errorf("%s: Invalid value for startegy Type - %d", r.Name, strategy.MaxUnavailable.IntVal) + return false, err + } + + // validating the Mode value + if strategy.Mode == "" { + r.Spec.Strategy.Mode = UpdateStrategyModeLazy + } else if !common.ContainsEqualFold(AllowedStrategyMode, strategy.Mode) { + err := fmt.Errorf("%s: Invalid value for startegy Mode - %d", r.Name, strategy.MaxUnavailable.IntVal) + return false, err + } + + // validating the maxUnavailable value + if strategy.MaxUnavailable.Type == intstr.Int && strategy.MaxUnavailable.IntVal == 0 { + r.Spec.Strategy.MaxUnavailable.IntVal = 1 + } else if strategy.MaxUnavailable.Type == intstr.Int && strategy.MaxUnavailable.IntVal < 0 { + err := fmt.Errorf("%s: Invalid value for startegy maxUnavailable - %d", r.Name, strategy.MaxUnavailable.IntVal) + return false, err + } else if strategy.MaxUnavailable.Type == intstr.String { + intValue, _ := strconv.Atoi(strings.Trim(strategy.MaxUnavailable.StrVal, "%")) + if intValue <= 0 || intValue > 100 { + err := fmt.Errorf("%s: Invalid value for startegy maxUnavailable - %s", r.Name, strategy.MaxUnavailable.StrVal) + return false, err + } + } + + // validating the DrainTimeout value + if strategy.DrainTimeout == 0 { + r.Spec.Strategy.DrainTimeout = -1 + } else if strategy.DrainTimeout < -1 { + err := fmt.Errorf("%s: Invalid value for startegy DrainTimeout - %d", r.Name, strategy.MaxUnavailable.IntVal) + return false, err + } + return true, nil } From 7cb15b0d0ea3b371c5f3940bf2860a2a4870ceb0 Mon Sep 17 00:00:00 2001 From: Shreyas Badiger <7680410+shreyas-badiger@users.noreply.github.com> Date: Wed, 27 Jan 2021 17:12:06 -0800 Subject: [PATCH 12/74] Fix all the "make vet" errors in Controller V2 branch. (#177) * Validation step to check Nodes and ASG launch configs Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Validating launch definition after a rolling upgrade Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Resolve error log message and return statement Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Adding Functional Test (#113) * Adding BDD, workflow and badge * Changing CI workflow job name * Adding make manifests * Clarifying cron time zone comment Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * release 0.13 (#115) * release 0.13 * Update CHANGELOG.md Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * bump version (#116) Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Repo selection for CI and BDD workflows & CI step for releases (#117) * CI-BDD not on forks & Step for releases (#2) * Testing CI-BDD not on forks & Step for releases * Adding step for image with tag git-tag Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Terminate unjoined nodes Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Resolving PR comments Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Set version and update CHANGELOG for version 0.14. (#121) Co-authored-by: Shri Javadekar Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Bump version to 0.15-dev. Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Fix typo in README.md. (#125) Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Ignore the terminated instance during upgrade Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Added WARNING prefix in the logging Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Apply suggestions from code review Co-authored-by: Kevin Downey Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Capitalize sprintf to Sprintf Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Upgrade to Go 1.15 (#128) Signed-off-by: Oleg Atamanenko Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Fix few typos and simplify error returns, remove redundant types (#131) Signed-off-by: Oleg Atamanenko Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Readiness gates implementation for eager mode (#130) Signed-off-by: Oleg Atamanenko Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Adding Functional Test (#113) * Adding BDD, workflow and badge * Changing CI workflow job name * Adding make manifests * Clarifying cron time zone comment Signed-off-by: sbadiger * Validation step to check Nodes and ASG launch configs (#112) * Validation step to check Nodes and ASG launch configs * Validating launch definition after a rolling upgrade * Resolve error log message and return statement Co-authored-by: Eytan Avisror Signed-off-by: sbadiger * release 0.13 (#115) * release 0.13 * Update CHANGELOG.md Signed-off-by: sbadiger * bump version (#116) Signed-off-by: sbadiger * Repo selection for CI and BDD workflows & CI step for releases (#117) * CI-BDD not on forks & Step for releases (#2) * Testing CI-BDD not on forks & Step for releases * Adding step for image with tag git-tag Signed-off-by: sbadiger * Terminate unjoined nodes (#120) * Validation step to check Nodes and ASG launch configs * Validating launch definition after a rolling upgrade * Resolve error log message and return statement * Terminate unjoined nodes * Resolving PR comments Co-authored-by: Eytan Avisror Signed-off-by: sbadiger * Set version and update CHANGELOG for version 0.14. (#121) Co-authored-by: Shri Javadekar Signed-off-by: sbadiger * Bump version to 0.15-dev. Signed-off-by: sbadiger * Fix bug when switching to launch templates (#136) * Update rollingupgrade_controller.go * Update rollingupgrade_controller.go Signed-off-by: Eytan Avisror * spacing fixes Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * Extract script runner to a separate type; fix work with env. variables (#132) Signed-off-by: Oleg Atamanenko Signed-off-by: sbadiger * Set version and update CHANGELOG for version v0.15 (#137) Signed-off-by: Shri Javadekar Signed-off-by: sbadiger * Bump version to v0.16-dev. Signed-off-by: Shri Javadekar Signed-off-by: sbadiger * Propagate parent env variables to allow to talk with API Server (#144) Signed-off-by: Oleg Atamanenko Signed-off-by: sbadiger * Bump Golang CI action to fix failed CI run (#146) Signed-off-by: Oleg Atamanenko Signed-off-by: sbadiger * Simplify (#145) Signed-off-by: Oleg Atamanenko Signed-off-by: sbadiger * Add Expiration to cache and do not refresh ASG if cache is not expired (#143) Signed-off-by: Oleg Atamanenko Co-authored-by: Shri Javadekar Signed-off-by: sbadiger * Fix documentation for uniform across AZ Update strategy and fix typos (#147) Signed-off-by: Oleg Atamanenko Signed-off-by: sbadiger * Move cluster state from package level to a cluster state impl (#148) Signed-off-by: Oleg Atamanenko Signed-off-by: sbadiger * Simplify work with intstr type. (#149) Signed-off-by: Oleg Atamanenko Signed-off-by: sbadiger * If instance is in standby mode already, just return (#138) Signed-off-by: Oleg Atamanenko Co-authored-by: Shri Javadekar Signed-off-by: sbadiger * Handle terminated instances gracefully. (#150) Signed-off-by: Oleg Atamanenko Signed-off-by: sbadiger * Template version comparison fix (#155) * get template version Signed-off-by: Eytan Avisror * fix tests Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * release 0.16 (#157) Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * bump version to 0.17-dev (#158) Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * Don't uncordon node on failure to run postDrain script when IgnoreDrainFailures set (#151) * Don't uncordon node on failure to run postDrain script when IgnoreDrainFailures set Signed-off-by: Adam Malcontenti-Wilson * Test node uncordon when postDrain / postDrainWait script fails Signed-off-by: Adam Malcontenti-Wilson Signed-off-by: sbadiger * Abort on strategy failure instead of continuing (#152) * Abort on strategy failure instead of continuing Signed-off-by: Adam Malcontenti-Wilson * Remove unformatted error message placeholder Signed-off-by: Adam Malcontenti-Wilson * Explictly specify strategy for tests Signed-off-by: Adam Malcontenti-Wilson Signed-off-by: sbadiger * use NamespacedName (#160) Signed-off-by: Eytan Avisror Co-authored-by: Shri Javadekar Signed-off-by: sbadiger * Set version and update CHANGELOG for version v0.17 (#161) Signed-off-by: Shri Javadekar Signed-off-by: sbadiger * Bump version to v0.18-dev (#162) Signed-off-by: Shri Javadekar Signed-off-by: sbadiger * Move constants to types so that they can be reused (#167) Signed-off-by: Oleg Atamanenko Signed-off-by: sbadiger * Remove separate module for pkg/log (#168) Signed-off-by: Oleg Atamanenko Co-authored-by: Shri Javadekar Signed-off-by: sbadiger * Bump dependencies. (#169) Signed-off-by: Oleg Atamanenko Signed-off-by: sbadiger * use standard fmt.Errorf to format error message; unify error format (#171) Signed-off-by: Oleg Atamanenko Signed-off-by: sbadiger * Fix namespaced name order (#170) Signed-off-by: Oleg Atamanenko Signed-off-by: sbadiger * Add instance id to the logs (#173) Signed-off-by: Oleg Atamanenko Co-authored-by: Shri Javadekar Signed-off-by: sbadiger * Bump golang and busybox (#172) Signed-off-by: Oleg Atamanenko Co-authored-by: Shri Javadekar Signed-off-by: sbadiger * Expose template list and other execution errors to logs (#166) * Log and return wrapped launchtemplate error Signed-off-by: Adam Malcontenti-Wilson * Expose execution error in logs Signed-off-by: Adam Malcontenti-Wilson Signed-off-by: sbadiger * output can contain other messages from API Server, so be more relaxed (#174) Signed-off-by: Oleg Atamanenko Signed-off-by: sbadiger * Delete README.md Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * delete all Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * scaffolding Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * add API Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * initial code Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * add more scaffolding Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * Add kubernetes API calls Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * aws API calls Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * AWS API calls & Drift detection Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * validate() function Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * modified validate() Signed-off-by: sbadiger * modified validate() Signed-off-by: sbadiger * initial rotation logic Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * basic script_runner without any modifications Signed-off-by: sbadiger * Fix all the vet related errors Signed-off-by: sbadiger Co-authored-by: Alfredo Garo <44888596+garomonegro@users.noreply.github.com> Co-authored-by: Eytan Avisror Co-authored-by: Shri Javadekar Co-authored-by: Shri Javadekar Co-authored-by: Shri Javadekar Co-authored-by: Craig Robson Co-authored-by: Kevin Downey Co-authored-by: Oleg Atamanenko Co-authored-by: Shreyas Badiger <7680410+hard-fault@users.noreply.github.com> Co-authored-by: Adam Malcontenti-Wilson Co-authored-by: Adam Malcontenti-Wilson Co-authored-by: Eytan Avisror --- api/v1alpha1/rollingupgrade_types.go | 8 ++++++-- controllers/providers/aws/utils.go | 19 ++++++++++--------- controllers/upgrade.go | 12 +++++++++++- 3 files changed, 27 insertions(+), 12 deletions(-) diff --git a/api/v1alpha1/rollingupgrade_types.go b/api/v1alpha1/rollingupgrade_types.go index 10b9110b..5804dc57 100644 --- a/api/v1alpha1/rollingupgrade_types.go +++ b/api/v1alpha1/rollingupgrade_types.go @@ -17,11 +17,12 @@ limitations under the License. package v1alpha1 import ( - "common" + "fmt" "strconv" "strings" + "github.com/keikoproj/upgrade-manager/controllers/common" corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/intstr" @@ -220,13 +221,16 @@ func (r *RollingUpgrade) IsForceRefresh() bool { return r.Spec.ForceRefresh } +func (r *RollingUpgrade) StrategyMode() UpdateStrategyMode { + return r.Spec.Strategy.Mode +} func (r *RollingUpgrade) Validate() (bool, error) { strategy := r.Spec.Strategy // validating the Type value if strategy.Type == "" { r.Spec.Strategy.Type = RandomUpdateStrategy - } else if !common.ContainsEqualFold(AllowedStrategyType, strategy.Type) { + } else if !common.ContainsEqualFold(AllowedStrategyType, string(strategy.Type)) { err := fmt.Errorf("%s: Invalid value for startegy Type - %d", r.Name, strategy.MaxUnavailable.IntVal) return false, err } diff --git a/controllers/providers/aws/utils.go b/controllers/providers/aws/utils.go index e5e01bd2..f14bc819 100644 --- a/controllers/providers/aws/utils.go +++ b/controllers/providers/aws/utils.go @@ -86,15 +86,16 @@ func GetScalingAZs(instances []*autoscaling.Instance) []string { return AZs } -func SelectInstancesByAZ(instances []*autoscaling.Group) *autoscaling.Instance { - for _, instance := range group.Instances { - selectedID := aws.StringValue(instance.InstanceId) - if strings.EqualFold(instanceID, selectedID) { - return instance - } - } - return &autoscaling.Instance{} -} +// func SelectInstancesByAZ(instances []*autoscaling.Group) *autoscaling.Instance { +// for _, instance := range group.Instances { +// selectedID := aws.StringValue(instance.InstanceId) +// if strings.EqualFold(instanceID, selectedID) { +// return instance +// } +// } +// return &autoscaling.Instance{} +// } + // func ListScalingInstanceIDs(group *autoscaling.Group) []string { // instanceIDs := make([]string, 0) diff --git a/controllers/upgrade.go b/controllers/upgrade.go index 31bb193a..5dd7ac33 100644 --- a/controllers/upgrade.go +++ b/controllers/upgrade.go @@ -77,12 +77,21 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol switch mode { case v1alpha1.UpdateStrategyModeEager: for _, target := range batch { + _ = target // Add in-progress tag // Standby // Wait for desired nodes + // predrain script + + // Issue drain/scripts concurrently - set lastDrainTime + + // post drain script + + // Wait for desired nodes + // Issue drain/scripts concurrently - set lastDrainTime // Is drained? @@ -91,6 +100,7 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol } case v1alpha1.UpdateStrategyModeLazy: for _, target := range batch { + _ = target // Add in-progress tag // Issue drain/scripts concurrently - set lastDrainTime @@ -100,7 +110,7 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol // Terminate - set lastTerminateTime } } - + return true, nil } func (r *RollingUpgradeReconciler) SelectTargets(rollingUpgrade *v1alpha1.RollingUpgrade, scalingGroup *autoscaling.Group) []*autoscaling.Instance { From 2c1d8e79634a9fd85075cbf2f05b64eb40033e99 Mon Sep 17 00:00:00 2001 From: Shreyas Badiger <7680410+shreyas-badiger@users.noreply.github.com> Date: Wed, 3 Feb 2021 15:44:48 -0800 Subject: [PATCH 13/74] Controller v2: Implementation of Instance termination (#178) * fix make vet errors. Signed-off-by: sbadiger * Terminate instances and run v2 for first time. Signed-off-by: sbadiger * Addressing review comments Signed-off-by: sbadiger * addressing more review comments Signed-off-by: sbadiger * Log error message Signed-off-by: sbadiger * error handling for instance tagging Signed-off-by: sbadiger --- api/v1alpha1/rollingupgrade_types.go | 3 +-- controllers/providers/aws/autoscaling.go | 22 +++++++++++++++ controllers/providers/aws/ec2.go | 14 ++++++++++ controllers/providers/aws/utils.go | 1 - controllers/rollingupgrade_controller.go | 6 ++--- controllers/upgrade.go | 34 +++++++++++++++++++++++- 6 files changed, 73 insertions(+), 7 deletions(-) diff --git a/api/v1alpha1/rollingupgrade_types.go b/api/v1alpha1/rollingupgrade_types.go index 5804dc57..f7bb97a6 100644 --- a/api/v1alpha1/rollingupgrade_types.go +++ b/api/v1alpha1/rollingupgrade_types.go @@ -17,7 +17,6 @@ limitations under the License. package v1alpha1 import ( - "fmt" "strconv" "strings" @@ -238,7 +237,7 @@ func (r *RollingUpgrade) Validate() (bool, error) { // validating the Mode value if strategy.Mode == "" { r.Spec.Strategy.Mode = UpdateStrategyModeLazy - } else if !common.ContainsEqualFold(AllowedStrategyMode, strategy.Mode) { + } else if !common.ContainsEqualFold(AllowedStrategyMode, string(strategy.Mode)) { err := fmt.Errorf("%s: Invalid value for startegy Mode - %d", r.Name, strategy.MaxUnavailable.IntVal) return false, err } diff --git a/controllers/providers/aws/autoscaling.go b/controllers/providers/aws/autoscaling.go index 2a49c6bc..d578f1c8 100644 --- a/controllers/providers/aws/autoscaling.go +++ b/controllers/providers/aws/autoscaling.go @@ -17,9 +17,18 @@ limitations under the License. package aws import ( + "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/service/autoscaling" ) +var ( + TerminatingInstanceStates = []string{ + autoscaling.LifecycleStateTerminating, + autoscaling.LifecycleStateTerminatingWait, + autoscaling.LifecycleStateTerminatingProceed, + } +) + func (a *AmazonClientSet) DescribeScalingGroups() ([]*autoscaling.Group, error) { scalingGroups := []*autoscaling.Group{} err := a.AsgClient.DescribeAutoScalingGroupsPages(&autoscaling.DescribeAutoScalingGroupsInput{}, func(page *autoscaling.DescribeAutoScalingGroupsOutput, lastPage bool) bool { @@ -31,3 +40,16 @@ func (a *AmazonClientSet) DescribeScalingGroups() ([]*autoscaling.Group, error) } return scalingGroups, nil } + +func (a *AmazonClientSet) TerminateInstance(instance *autoscaling.Instance) error { + instanceID := aws.StringValue(instance.InstanceId) + input := &autoscaling.TerminateInstanceInAutoScalingGroupInput{ + InstanceId: aws.String(instanceID), + ShouldDecrementDesiredCapacity: aws.Bool(false), + } + + if _, err := a.AsgClient.TerminateInstanceInAutoScalingGroup(input); err != nil { + return err + } + return nil +} diff --git a/controllers/providers/aws/ec2.go b/controllers/providers/aws/ec2.go index f7c7644a..c0fde6a8 100644 --- a/controllers/providers/aws/ec2.go +++ b/controllers/providers/aws/ec2.go @@ -57,3 +57,17 @@ func (a *AmazonClientSet) DescribeTaggedInstanceIDs(tagKey, tagValue string) ([] }) return instances, err } + +func (a *AmazonClientSet) TagEC2instance(instanceID, tagKey, tagValue string) error { + input := &ec2.CreateTagsInput{ + Resources: aws.StringSlice([]string{instanceID}), + Tags: []*ec2.Tag{ + { + Key: aws.String(tagKey), + Value: aws.String(tagValue), + }, + }, + } + _, err := a.Ec2Client.CreateTags(input) + return err +} diff --git a/controllers/providers/aws/utils.go b/controllers/providers/aws/utils.go index f14bc819..95d22402 100644 --- a/controllers/providers/aws/utils.go +++ b/controllers/providers/aws/utils.go @@ -96,7 +96,6 @@ func GetScalingAZs(instances []*autoscaling.Instance) []string { // return &autoscaling.Instance{} // } - // func ListScalingInstanceIDs(group *autoscaling.Group) []string { // instanceIDs := make([]string, 0) // for _, instance := range group.Instances { diff --git a/controllers/rollingupgrade_controller.go b/controllers/rollingupgrade_controller.go index 46f954df..2ae22aa2 100644 --- a/controllers/rollingupgrade_controller.go +++ b/controllers/rollingupgrade_controller.go @@ -123,11 +123,11 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque } r.Info("admitted new rollingupgrade", "name", req.NamespacedName, "scalingGroup", scalingGroupName) - r.AdmissionMap.Store(req.NamespacedName, scalingGroupName) + r.AdmissionMap.Store(rollingUpgrade.NamespacedName(), scalingGroupName) rollingUpgrade.SetCurrentStatus(v1alpha1.StatusInit) - discoveredState := NewDiscoveredState(r.Auth, r.Logger) - if err := discoveredState.Discover(); err != nil { + r.Cloud = NewDiscoveredState(r.Auth, r.Logger) + if err := r.Cloud.Discover(); err != nil { rollingUpgrade.SetCurrentStatus(v1alpha1.StatusError) return ctrl.Result{}, err } diff --git a/controllers/upgrade.go b/controllers/upgrade.go index 5dd7ac33..f23b8d46 100644 --- a/controllers/upgrade.go +++ b/controllers/upgrade.go @@ -23,6 +23,7 @@ import ( "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/service/autoscaling" "github.com/keikoproj/upgrade-manager/api/v1alpha1" + "github.com/keikoproj/upgrade-manager/controllers/common" awsprovider "github.com/keikoproj/upgrade-manager/controllers/providers/aws" kubeprovider "github.com/keikoproj/upgrade-manager/controllers/providers/kubernetes" "k8s.io/apimachinery/pkg/util/intstr" @@ -63,8 +64,17 @@ func (r *RollingUpgradeReconciler) RotateNodes(rollingUpgrade *v1alpha1.RollingU ) rollingUpgrade.SetTotalNodes(len(scalingGroup.Instances)) + + // check if all instances are rotated. + if r.IsScalingGroupDrifted(rollingUpgrade) { + rollingUpgrade.SetCurrentStatus(v1alpha1.StatusComplete) + return nil + } + rotationTargets := r.SelectTargets(rollingUpgrade, scalingGroup) - r.ReplaceNodeBatch(rollingUpgrade, rotationTargets) + if ok, err := r.ReplaceNodeBatch(rollingUpgrade, rotationTargets); !ok { + return err + } return nil } @@ -79,6 +89,9 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol for _, target := range batch { _ = target // Add in-progress tag + if err := r.Auth.TagEC2instance(aws.StringValue(target.InstanceId), instanceStateTagKey, inProgressTagValue); err != nil { + r.Error(err, "failed to set instance tag", "name", rollingUpgrade.NamespacedName(), "instance", aws.StringValue(target.InstanceId)) + } // Standby @@ -97,6 +110,10 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol // Is drained? // Terminate - set lastTerminateTime + if err := r.Auth.TerminateInstance(target); err != nil { + r.Info("failed to terminate instance", "name", rollingUpgrade.NamespacedName(), "instance", aws.StringValue(target.InstanceId), "message", err) + return true, nil + } } case v1alpha1.UpdateStrategyModeLazy: for _, target := range batch { @@ -187,6 +204,10 @@ func (r *RollingUpgradeReconciler) IsInstanceDrifted(rollingUpgrade *v1alpha1.Ro instanceID = aws.StringValue(instance.InstanceId) ) + // if an instance is in terminating state, ignore. + if common.ContainsEqualFold(awsprovider.TerminatingInstanceStates, aws.StringValue(instance.LifecycleState)) { + return false + } // check if there is atleast one node that meets the force-referesh criteria if rollingUpgrade.IsForceRefresh() { var ( @@ -257,3 +278,14 @@ func (r *RollingUpgradeReconciler) IsInstanceDrifted(rollingUpgrade *v1alpha1.Ro r.Info("node refresh not required", "name", rollingUpgrade.NamespacedName(), "instance", instanceID) return false } + +func (r *RollingUpgradeReconciler) IsScalingGroupDrifted(rollingUpgrade *v1alpha1.RollingUpgrade) bool { + r.Info("checking if rolling upgrade is completed", "name", rollingUpgrade.NamespacedName()) + scalingGroup := awsprovider.SelectScalingGroup(rollingUpgrade.ScalingGroupName(), r.Cloud.ScalingGroups) + for _, instance := range scalingGroup.Instances { + if r.IsInstanceDrifted(rollingUpgrade, instance) { + return false + } + } + return true +} From dd6a332e22319503353939dc16cff9aa58423ce1 Mon Sep 17 00:00:00 2001 From: Eytan Avisror Date: Tue, 9 Feb 2021 20:18:43 -0800 Subject: [PATCH 14/74] Migrate Script Runner (#179) * Basic script runner Signed-off-by: Eytan Avisror * Update upgrade.go Signed-off-by: Eytan Avisror --- api/v1alpha1/rollingupgrade_types.go | 32 +++++ api/v1alpha1/zz_generated.deepcopy.go | 2 + ...grademgr.keikoproj.io_rollingupgrades.yaml | 6 + controllers/providers/aws/ec2.go | 4 + controllers/providers/aws/utils.go | 10 ++ controllers/rollingupgrade_controller.go | 6 +- controllers/script_runner.go | 133 ++++++++++++++++++ controllers/script_runner_test.go | 48 +++++++ controllers/upgrade.go | 41 +++++- go.mod | 1 + main.go | 3 + 11 files changed, 279 insertions(+), 7 deletions(-) create mode 100644 controllers/script_runner.go create mode 100644 controllers/script_runner_test.go diff --git a/api/v1alpha1/rollingupgrade_types.go b/api/v1alpha1/rollingupgrade_types.go index f7bb97a6..88d408b2 100644 --- a/api/v1alpha1/rollingupgrade_types.go +++ b/api/v1alpha1/rollingupgrade_types.go @@ -54,6 +54,17 @@ type RollingUpgradeStatus struct { LastNodeDrainTime metav1.Time `json:"lastDrainTime,omitempty"` } +func (s *RollingUpgradeStatus) SetCondition(cond RollingUpgradeCondition) { + // if condition exists, overwrite, otherwise append + for ix, c := range s.Conditions { + if c.Type == cond.Type { + s.Conditions[ix] = cond + return + } + } + s.Conditions = append(s.Conditions, cond) +} + // +kubebuilder:object:root=true // +kubebuilder:subresource:status // +kubebuilder:resource:path=rollingupgrades,scope=Namespaced,shortName=ru @@ -160,6 +171,22 @@ func (r *RollingUpgrade) ScalingGroupName() string { return r.Spec.AsgName } +func (r *RollingUpgrade) PostTerminateScript() string { + return r.Spec.PostTerminate.Script +} + +func (r *RollingUpgrade) PostWaitScript() string { + return r.Spec.PostDrain.PostWaitScript +} + +func (r *RollingUpgrade) PreDrainScript() string { + return r.Spec.PreDrain.Script +} + +func (r *RollingUpgrade) PostDrainScript() string { + return r.Spec.PostDrain.Script +} + func (r *RollingUpgrade) CurrentStatus() string { return r.Status.CurrentStatus } @@ -216,6 +243,10 @@ func (r *RollingUpgrade) SetNodesProcessed(n int) { r.Status.NodesProcessed = n } +func (r *RollingUpgrade) GetStatus() RollingUpgradeStatus { + return r.Status +} + func (r *RollingUpgrade) IsForceRefresh() bool { return r.Spec.ForceRefresh } @@ -223,6 +254,7 @@ func (r *RollingUpgrade) IsForceRefresh() bool { func (r *RollingUpgrade) StrategyMode() UpdateStrategyMode { return r.Spec.Strategy.Mode } + func (r *RollingUpgrade) Validate() (bool, error) { strategy := r.Spec.Strategy diff --git a/api/v1alpha1/zz_generated.deepcopy.go b/api/v1alpha1/zz_generated.deepcopy.go index f9016b52..5d613e18 100644 --- a/api/v1alpha1/zz_generated.deepcopy.go +++ b/api/v1alpha1/zz_generated.deepcopy.go @@ -199,6 +199,8 @@ func (in *RollingUpgradeStatus) DeepCopyInto(out *RollingUpgradeStatus) { *out = make([]RollingUpgradeCondition, len(*in)) copy(*out, *in) } + in.LastNodeTerminationTime.DeepCopyInto(&out.LastNodeTerminationTime) + in.LastNodeDrainTime.DeepCopyInto(&out.LastNodeDrainTime) } // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RollingUpgradeStatus. diff --git a/config/crd/bases/upgrademgr.keikoproj.io_rollingupgrades.yaml b/config/crd/bases/upgrademgr.keikoproj.io_rollingupgrades.yaml index 7f0bfc18..f8eb03d1 100644 --- a/config/crd/bases/upgrademgr.keikoproj.io_rollingupgrades.yaml +++ b/config/crd/bases/upgrademgr.keikoproj.io_rollingupgrades.yaml @@ -133,6 +133,12 @@ spec: type: string endTime: type: string + lastDrainTime: + format: date-time + type: string + lastTerminationTime: + format: date-time + type: string nodesProcessed: type: integer startTime: diff --git a/controllers/providers/aws/ec2.go b/controllers/providers/aws/ec2.go index c0fde6a8..cf548620 100644 --- a/controllers/providers/aws/ec2.go +++ b/controllers/providers/aws/ec2.go @@ -44,6 +44,10 @@ func (a *AmazonClientSet) DescribeTaggedInstanceIDs(tagKey, tagValue string) ([] Name: aws.String(key), Values: aws.StringSlice([]string{tagValue}), }, + { + Name: aws.String("instance-state-name"), + Values: aws.StringSlice([]string{"pending", "running"}), + }, }, } diff --git a/controllers/providers/aws/utils.go b/controllers/providers/aws/utils.go index 95d22402..3db1cbc5 100644 --- a/controllers/providers/aws/utils.go +++ b/controllers/providers/aws/utils.go @@ -86,6 +86,16 @@ func GetScalingAZs(instances []*autoscaling.Instance) []string { return AZs } +func GetInstanceIDs(instances []*autoscaling.Instance) []string { + IDs := make([]string, 0) + for _, instance := range instances { + ID := aws.StringValue(instance.InstanceId) + IDs = append(IDs, ID) + } + sort.Strings(IDs) + return IDs +} + // func SelectInstancesByAZ(instances []*autoscaling.Group) *autoscaling.Instance { // for _, instance := range group.Instances { // selectedID := aws.StringValue(instance.InstanceId) diff --git a/controllers/rollingupgrade_controller.go b/controllers/rollingupgrade_controller.go index 2ae22aa2..fb61c94b 100644 --- a/controllers/rollingupgrade_controller.go +++ b/controllers/rollingupgrade_controller.go @@ -25,7 +25,6 @@ import ( "github.com/go-logr/logr" "github.com/keikoproj/aws-sdk-go-cache/cache" "github.com/keikoproj/upgrade-manager/api/v1alpha1" - upgrademgrv1alpha1 "github.com/keikoproj/upgrade-manager/api/v1alpha1" "github.com/keikoproj/upgrade-manager/controllers/common" awsprovider "github.com/keikoproj/upgrade-manager/controllers/providers/aws" kubeprovider "github.com/keikoproj/upgrade-manager/controllers/providers/kubernetes" @@ -48,6 +47,7 @@ type RollingUpgradeReconciler struct { Cloud *DiscoveredState EventWriter *kubeprovider.EventWriter maxParallel int + ScriptRunner ScriptRunner } type RollingUpgradeAuthenticator struct { @@ -67,7 +67,7 @@ type RollingUpgradeAuthenticator struct { // Reconcile reads that state of the cluster for a RollingUpgrade object and makes changes based on the state read // and the details in the RollingUpgrade.Spec func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { - rollingUpgrade := &upgrademgrv1alpha1.RollingUpgrade{} + rollingUpgrade := &v1alpha1.RollingUpgrade{} err := r.Get(ctx, req.NamespacedName, rollingUpgrade) if err != nil { if kerrors.IsNotFound(err) { @@ -144,7 +144,7 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque // SetupWithManager sets up the controller with the Manager. func (r *RollingUpgradeReconciler) SetupWithManager(mgr ctrl.Manager) error { return ctrl.NewControllerManagedBy(mgr). - For(&upgrademgrv1alpha1.RollingUpgrade{}). + For(&v1alpha1.RollingUpgrade{}). WithOptions(controller.Options{MaxConcurrentReconciles: r.maxParallel}). Complete(r) } diff --git a/controllers/script_runner.go b/controllers/script_runner.go new file mode 100644 index 00000000..fe51a571 --- /dev/null +++ b/controllers/script_runner.go @@ -0,0 +1,133 @@ +/* +Copyright 2019 Intuit, Inc.. +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package controllers + +import ( + "fmt" + "os" + "os/exec" + + "github.com/go-logr/logr" + "github.com/keikoproj/upgrade-manager/api/v1alpha1" +) + +const ( + ShellBinary = "/bin/sh" +) + +type ScriptRunner struct { + logr.Logger +} + +type ScriptTarget struct { + InstanceID string + NodeName string + UpgradeObject *v1alpha1.RollingUpgrade +} + +func NewScriptRunner(logger logr.Logger) ScriptRunner { + return ScriptRunner{ + Logger: logger, + } +} + +func (r *ScriptRunner) getEnv(target ScriptTarget) []string { + var ( + asgNameEnv = "ASG_NAME" + instanceIdEnv = "INSTANCE_ID" + instanceNameEnv = "INSTANCE_NAME" + ) + return []string{ + fmt.Sprintf("%s=%s", asgNameEnv, target.UpgradeObject.ScalingGroupName()), + fmt.Sprintf("%s=%s", instanceIdEnv, target.InstanceID), + fmt.Sprintf("%s=%s", instanceNameEnv, target.NodeName), + } +} + +func (r *ScriptRunner) runScript(script string, target ScriptTarget) (string, error) { + r.Info("running script", "script", script, "name", target.UpgradeObject.NamespacedName()) + command := exec.Command(ShellBinary, "-c", script) + command.Env = append(os.Environ(), r.getEnv(target)...) + + out, err := command.CombinedOutput() + if err != nil { + return string(out), err + } + return string(out), nil +} + +func (r *ScriptRunner) PostTerminate(target ScriptTarget) error { + script := target.UpgradeObject.PostTerminateScript() + if script == "" { + return nil + } + + out, err := r.runScript(script, target) + if err != nil { + r.Info("script execution failed", "output", out, "stage", "PostTerminate", "script", script, "name", target.UpgradeObject.NamespacedName(), "target", target.NodeName) + return err + } + + r.Info("script execution succeeded", "output", out, "stage", "PostTerminate", "script", script, "name", target.UpgradeObject.NamespacedName(), "target", target.NodeName) + + return nil +} + +func (r *ScriptRunner) PreDrain(target ScriptTarget) error { + script := target.UpgradeObject.PreDrainScript() + if script == "" { + return nil + } + + out, err := r.runScript(script, target) + if err != nil { + r.Info("script execution failed", "output", out, "stage", "PreDrain", "script", script, "name", target.UpgradeObject.NamespacedName(), "target", target.NodeName) + return err + } + + r.Info("script execution succeeded", "output", out, "stage", "PreDrain", "script", script, "name", target.UpgradeObject.NamespacedName(), "target", target.NodeName) + return nil +} + +func (r *ScriptRunner) PostDrain(target ScriptTarget) error { + script := target.UpgradeObject.PostDrainScript() + if script == "" { + return nil + } + + out, err := r.runScript(script, target) + if err != nil { + r.Info("script execution failed", "output", out, "stage", "PostDrain", "script", script, "name", target.UpgradeObject.NamespacedName(), "target", target.NodeName) + return err + } + + r.Info("script execution succeeded", "output", out, "stage", "PostDrain", "script", script, "name", target.UpgradeObject.NamespacedName(), "target", target.NodeName) + return nil +} + +func (r *ScriptRunner) PostWait(target ScriptTarget) error { + script := target.UpgradeObject.PostWaitScript() + if script == "" { + return nil + } + + out, err := r.runScript(script, target) + if err != nil { + r.Info("script execution failed", "output", out, "stage", "PostWait", "script", script, "name", target.UpgradeObject.NamespacedName(), "target", target.NodeName) + return err + } + + r.Info("script execution succeeded", "output", out, "stage", "PostWait", "script", script, "name", target.UpgradeObject.NamespacedName(), "target", target.NodeName) + return nil +} diff --git a/controllers/script_runner_test.go b/controllers/script_runner_test.go new file mode 100644 index 00000000..e6e8236b --- /dev/null +++ b/controllers/script_runner_test.go @@ -0,0 +1,48 @@ +package controllers + +import ( + "testing" + + "github.com/keikoproj/upgrade-manager/api/v1alpha1" + "github.com/onsi/gomega" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + runtimelog "sigs.k8s.io/controller-runtime/pkg/log" +) + +func TestScriptSuccess(t *testing.T) { + g := gomega.NewGomegaWithT(t) + r := &ScriptRunner{Logger: runtimelog.NullLogger{}} + target := ScriptTarget{ + InstanceID: "instance", + NodeName: "node", + UpgradeObject: &v1alpha1.RollingUpgrade{ + ObjectMeta: metav1.ObjectMeta{ + Name: "foo", + Namespace: "default", + }, + }, + } + + out, err := r.runScript("echo hello", target) + g.Expect(err).To(gomega.BeNil()) + g.Expect(out).To(gomega.Equal("hello\n")) +} + +func TestScriptFailure(t *testing.T) { + g := gomega.NewGomegaWithT(t) + r := &ScriptRunner{Logger: runtimelog.NullLogger{}} + target := ScriptTarget{ + InstanceID: "instance", + NodeName: "node", + UpgradeObject: &v1alpha1.RollingUpgrade{ + ObjectMeta: metav1.ObjectMeta{ + Name: "foo", + Namespace: "default", + }, + }, + } + + out, err := r.runScript("echo this will fail; exit 1", target) + g.Expect(err).To(gomega.Not(gomega.BeNil())) + g.Expect(out).To(gomega.Not(gomega.Equal(""))) +} diff --git a/controllers/upgrade.go b/controllers/upgrade.go index f23b8d46..beaa3b8c 100644 --- a/controllers/upgrade.go +++ b/controllers/upgrade.go @@ -84,13 +84,30 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol mode = rollingUpgrade.StrategyMode() ) + r.Info("rotating batch", "instances", awsprovider.GetInstanceIDs(batch), "name", rollingUpgrade.NamespacedName()) + switch mode { case v1alpha1.UpdateStrategyModeEager: + + // TODO: THE BELOW LOOP IS A TEMPORARY PLACEHOLDER FOR TESTING PURPOSES + // WE SHOULD SWITCH TO PARALLEL PROCESSING OF TARGETS IN A BATCH VIA GOROUTINES + for _, target := range batch { - _ = target + + var ( + instanceID = aws.StringValue(target.InstanceId) + node = kubeprovider.SelectNodeByInstanceID(instanceID, r.Cloud.ClusterNodes) + nodeName = node.GetName() + scriptTarget = ScriptTarget{ + InstanceID: instanceID, + NodeName: nodeName, + UpgradeObject: rollingUpgrade, + } + ) + // Add in-progress tag - if err := r.Auth.TagEC2instance(aws.StringValue(target.InstanceId), instanceStateTagKey, inProgressTagValue); err != nil { - r.Error(err, "failed to set instance tag", "name", rollingUpgrade.NamespacedName(), "instance", aws.StringValue(target.InstanceId)) + if err := r.Auth.TagEC2instance(instanceID, instanceStateTagKey, inProgressTagValue); err != nil { + r.Error(err, "failed to set instance tag", "name", rollingUpgrade.NamespacedName(), "instance", instanceID) } // Standby @@ -98,10 +115,16 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol // Wait for desired nodes // predrain script + if err := r.ScriptRunner.PreDrain(scriptTarget); err != nil { + return false, err + } // Issue drain/scripts concurrently - set lastDrainTime // post drain script + if err := r.ScriptRunner.PostDrain(scriptTarget); err != nil { + return false, err + } // Wait for desired nodes @@ -109,11 +132,21 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol // Is drained? + // Post Wait Script + if err := r.ScriptRunner.PostWait(scriptTarget); err != nil { + return false, err + } + // Terminate - set lastTerminateTime if err := r.Auth.TerminateInstance(target); err != nil { - r.Info("failed to terminate instance", "name", rollingUpgrade.NamespacedName(), "instance", aws.StringValue(target.InstanceId), "message", err) + r.Info("failed to terminate instance", "name", rollingUpgrade.NamespacedName(), "instance", instanceID, "message", err) return true, nil } + + // Post Wait Script + if err := r.ScriptRunner.PostTerminate(scriptTarget); err != nil { + return false, err + } } case v1alpha1.UpdateStrategyModeLazy: for _, target := range batch { diff --git a/go.mod b/go.mod index 6299d030..816421ef 100644 --- a/go.mod +++ b/go.mod @@ -6,6 +6,7 @@ require ( github.com/aws/aws-sdk-go v1.36.24 github.com/go-logr/logr v0.3.0 github.com/keikoproj/aws-sdk-go-cache v0.0.0-20201118182730-f6f418a4e2df + github.com/onsi/gomega v1.10.2 github.com/pkg/errors v0.9.1 github.com/sirupsen/logrus v1.6.0 go.uber.org/zap v1.15.0 diff --git a/main.go b/main.go index 3ed1fea3..3a5ef2b0 100644 --- a/main.go +++ b/main.go @@ -189,6 +189,9 @@ func main() { KubernetesClientSet: kubeClient, }, EventWriter: kubeprovider.NewEventWriter(kubeClient, logger), + ScriptRunner: controllers.ScriptRunner{ + Logger: logger, + }, } reconciler.SetMaxParallel(maxParallel) From 57df5a5ed914688a810e52703cc19e39036a0ba5 Mon Sep 17 00:00:00 2001 From: Shreyas Badiger <7680410+shreyas-badiger@users.noreply.github.com> Date: Fri, 12 Feb 2021 09:28:07 -0800 Subject: [PATCH 15/74] Implemented node drain. (#181) --- api/v1alpha1/rollingupgrade_types.go | 12 +++++++ controllers/providers/kubernetes/nodes.go | 43 +++++++++++++++++++++++ controllers/upgrade.go | 20 ++++++++--- 3 files changed, 71 insertions(+), 4 deletions(-) diff --git a/api/v1alpha1/rollingupgrade_types.go b/api/v1alpha1/rollingupgrade_types.go index 88d408b2..f0fc2d00 100644 --- a/api/v1alpha1/rollingupgrade_types.go +++ b/api/v1alpha1/rollingupgrade_types.go @@ -171,6 +171,10 @@ func (r *RollingUpgrade) ScalingGroupName() string { return r.Spec.AsgName } +func (r *RollingUpgrade) DrainTimeout() int { + return r.Spec.Strategy.DrainTimeout +} + func (r *RollingUpgrade) PostTerminateScript() string { return r.Spec.PostTerminate.Script } @@ -203,10 +207,18 @@ func (r *RollingUpgrade) LastNodeTerminationTime() metav1.Time { return r.Status.LastNodeTerminationTime } +func (r *RollingUpgrade) SetLastNodeTerminationTime(t metav1.Time) { + r.Status.LastNodeTerminationTime = t +} + func (r *RollingUpgrade) LastNodeDrainTime() metav1.Time { return r.Status.LastNodeDrainTime } +func (r *RollingUpgrade) SetLastNodeDrainTime(t metav1.Time) { + r.Status.LastNodeDrainTime = t +} + func (r *RollingUpgrade) NodeIntervalSeconds() int { return r.Spec.NodeIntervalSeconds } diff --git a/controllers/providers/kubernetes/nodes.go b/controllers/providers/kubernetes/nodes.go index 02501dde..4ae2c0c7 100644 --- a/controllers/providers/kubernetes/nodes.go +++ b/controllers/providers/kubernetes/nodes.go @@ -18,9 +18,15 @@ package kubernetes import ( "context" + "fmt" + "os" + "time" corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/client-go/kubernetes" + drain "k8s.io/kubectl/pkg/drain" ) // ListClusterNodes gets a list of all nodes in the cluster @@ -32,3 +38,40 @@ func (k *KubernetesClientSet) ListClusterNodes() (*corev1.NodeList, error) { } return nodes, nil } + +// DrainNode cordons and drains a node. +func (k *KubernetesClientSet) DrainNode(node *corev1.Node, PostDrainDelaySeconds time.Duration, DrainTimeout int, client kubernetes.Interface) error { + if client == nil { + return fmt.Errorf("K8sClient not set") + } + + if node == nil { + return fmt.Errorf("node not set") + } + + helper := &drain.Helper{ + Client: client, + Force: true, + GracePeriodSeconds: -1, + IgnoreAllDaemonSets: true, + Out: os.Stdout, + ErrOut: os.Stdout, + DeleteEmptyDirData: true, + Timeout: time.Duration(DrainTimeout) * time.Second, + } + + if err := drain.RunCordonOrUncordon(helper, node, true); err != nil { + if apierrors.IsNotFound(err) { + return err + } + return fmt.Errorf("error cordoning node: %v", err) + } + + if err := drain.RunNodeDrain(helper, node.Name); err != nil { + if apierrors.IsNotFound(err) { + return err + } + return fmt.Errorf("error draining node: %v", err) + } + return nil +} diff --git a/controllers/upgrade.go b/controllers/upgrade.go index beaa3b8c..6e682692 100644 --- a/controllers/upgrade.go +++ b/controllers/upgrade.go @@ -17,6 +17,7 @@ limitations under the License. package controllers import ( + "reflect" "strings" "time" @@ -26,6 +27,8 @@ import ( "github.com/keikoproj/upgrade-manager/controllers/common" awsprovider "github.com/keikoproj/upgrade-manager/controllers/providers/aws" kubeprovider "github.com/keikoproj/upgrade-manager/controllers/providers/kubernetes" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/intstr" ) @@ -66,7 +69,7 @@ func (r *RollingUpgradeReconciler) RotateNodes(rollingUpgrade *v1alpha1.RollingU rollingUpgrade.SetTotalNodes(len(scalingGroup.Instances)) // check if all instances are rotated. - if r.IsScalingGroupDrifted(rollingUpgrade) { + if !r.IsScalingGroupDrifted(rollingUpgrade) { rollingUpgrade.SetCurrentStatus(v1alpha1.StatusComplete) return nil } @@ -119,7 +122,15 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol return false, err } - // Issue drain/scripts concurrently - set lastDrainTime + // Issue drain concurrently - set lastDrainTime + if node := kubeprovider.SelectNodeByInstanceID(instanceID, r.Cloud.ClusterNodes); !reflect.DeepEqual(node, corev1.Node{}) { + r.Info("draining the node", "name", rollingUpgrade.NamespacedName(), "instance", instanceID, "node name", node.Name) + if err := r.Auth.DrainNode(&node, time.Duration(rollingUpgrade.PostDrainDelaySeconds()), rollingUpgrade.DrainTimeout(), r.Auth.Kubernetes); err != nil { + r.Error(err, "failed to drain node", "name", rollingUpgrade.NamespacedName(), "instance", instanceID, "node name", node.Name) + return false, err + } + } + rollingUpgrade.SetLastNodeDrainTime(metav1.Time{Time: time.Now()}) // post drain script if err := r.ScriptRunner.PostDrain(scriptTarget); err != nil { @@ -142,6 +153,7 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol r.Info("failed to terminate instance", "name", rollingUpgrade.NamespacedName(), "instance", instanceID, "message", err) return true, nil } + rollingUpgrade.SetLastNodeTerminationTime(metav1.Time{Time: time.Now()}) // Post Wait Script if err := r.ScriptRunner.PostTerminate(scriptTarget); err != nil { @@ -317,8 +329,8 @@ func (r *RollingUpgradeReconciler) IsScalingGroupDrifted(rollingUpgrade *v1alpha scalingGroup := awsprovider.SelectScalingGroup(rollingUpgrade.ScalingGroupName(), r.Cloud.ScalingGroups) for _, instance := range scalingGroup.Instances { if r.IsInstanceDrifted(rollingUpgrade, instance) { - return false + return true } } - return true + return false } From 11d3ae695152a35bc3faba2f587ed322d48dc604 Mon Sep 17 00:00:00 2001 From: Shreyas Badiger <7680410+shreyas-badiger@users.noreply.github.com> Date: Wed, 17 Feb 2021 16:25:00 -0800 Subject: [PATCH 16/74] Eager mode implementation (#183) * Eager mode implementation Signed-off-by: sbadiger --- controllers/providers/aws/autoscaling.go | 15 +++++++ controllers/providers/aws/utils.go | 11 +++++ controllers/providers/kubernetes/utils.go | 33 ++++++++++++++- controllers/upgrade.go | 50 +++++++++++++++++++---- 4 files changed, 99 insertions(+), 10 deletions(-) diff --git a/controllers/providers/aws/autoscaling.go b/controllers/providers/aws/autoscaling.go index d578f1c8..6bfef0a6 100644 --- a/controllers/providers/aws/autoscaling.go +++ b/controllers/providers/aws/autoscaling.go @@ -53,3 +53,18 @@ func (a *AmazonClientSet) TerminateInstance(instance *autoscaling.Instance) erro } return nil } + +func (a *AmazonClientSet) SetInstanceStandBy(instance *autoscaling.Instance, scalingGroupName string) error { + instanceID := aws.StringValue(instance.InstanceId) + input := &autoscaling.EnterStandbyInput{ + AutoScalingGroupName: aws.String(scalingGroupName), + InstanceIds: aws.StringSlice([]string{instanceID}), + ShouldDecrementDesiredCapacity: aws.Bool(false), + } + + if _, err := a.AsgClient.EnterStandby(input); err != nil { + return err + } + + return nil +} diff --git a/controllers/providers/aws/utils.go b/controllers/providers/aws/utils.go index 3db1cbc5..28e5c340 100644 --- a/controllers/providers/aws/utils.go +++ b/controllers/providers/aws/utils.go @@ -125,3 +125,14 @@ func GetTemplateLatestVersion(templates []*ec2.LaunchTemplate, templateName stri } return "0" } + +func GetInServiceInstances(scalingGroup *autoscaling.Group) []string { + instances := scalingGroup.Instances + inServiceInstances := []string{} + for _, instance := range instances { + if aws.StringValue(instance.LifecycleState) == autoscaling.LifecycleStateInService { + inServiceInstances = append(inServiceInstances, aws.StringValue(instance.InstanceId)) + } + } + return inServiceInstances +} diff --git a/controllers/providers/kubernetes/utils.go b/controllers/providers/kubernetes/utils.go index 7f37508f..79f148b8 100644 --- a/controllers/providers/kubernetes/utils.go +++ b/controllers/providers/kubernetes/utils.go @@ -24,6 +24,7 @@ import ( corev1 "k8s.io/api/core/v1" + "github.com/keikoproj/upgrade-manager/api/v1alpha1" "k8s.io/client-go/kubernetes" "k8s.io/client-go/rest" "k8s.io/client-go/tools/clientcmd" @@ -87,11 +88,39 @@ func GetKubernetesLocalConfig() (*rest.Config, error) { func SelectNodeByInstanceID(instanceID string, nodes *corev1.NodeList) corev1.Node { for _, node := range nodes.Items { - tokens := strings.Split(node.Spec.ProviderID, "/") - nodeID := tokens[len(tokens)-1] + nodeID := GetNodeInstanceID(node) if strings.EqualFold(instanceID, nodeID) { return node } } return corev1.Node{} } + +func GetNodeInstanceID(node corev1.Node) string { + tokens := strings.Split(node.Spec.ProviderID, "/") + nodeInstanceID := tokens[len(tokens)-1] + return nodeInstanceID +} + +func IsNodeReady(node corev1.Node) bool { + for _, condition := range node.Status.Conditions { + if condition.Type == corev1.NodeReady && condition.Status == corev1.ConditionTrue { + return true + } + } + return false +} + +func IsNodePassesReadinessGates(node corev1.Node, requiredReadinessGates []v1alpha1.NodeReadinessGate) bool { + if len(requiredReadinessGates) == 0 { + return true + } + for _, gate := range requiredReadinessGates { + for key, value := range gate.MatchLabels { + if node.Labels[key] != value { + return false + } + } + } + return true +} diff --git a/controllers/upgrade.go b/controllers/upgrade.go index 6e682692..76a5d78f 100644 --- a/controllers/upgrade.go +++ b/controllers/upgrade.go @@ -111,13 +111,25 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol // Add in-progress tag if err := r.Auth.TagEC2instance(instanceID, instanceStateTagKey, inProgressTagValue); err != nil { r.Error(err, "failed to set instance tag", "name", rollingUpgrade.NamespacedName(), "instance", instanceID) + return false, err } // Standby + if aws.StringValue(target.LifecycleState) == autoscaling.LifecycleStateInService { + r.Info("setting instance to stand-by", "name", rollingUpgrade.NamespacedName(), "instance", instanceID) + if err := r.Auth.SetInstanceStandBy(target, rollingUpgrade.Spec.AsgName); err != nil { + r.Error(err, "couldn't set the instance to stand-by", "name", rollingUpgrade.NamespacedName(), "instance", instanceID) + return false, err + } + } // Wait for desired nodes + if !r.DesiredNodesReady(rollingUpgrade) { + r.Info("new node is yet to join the cluster") + return true, nil + } - // predrain script + // Predrain script if err := r.ScriptRunner.PreDrain(scriptTarget); err != nil { return false, err } @@ -137,25 +149,20 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol return false, err } - // Wait for desired nodes - - // Issue drain/scripts concurrently - set lastDrainTime - - // Is drained? - // Post Wait Script if err := r.ScriptRunner.PostWait(scriptTarget); err != nil { return false, err } // Terminate - set lastTerminateTime + r.Info("terminating instance", "name", rollingUpgrade.NamespacedName(), "instance", instanceID) if err := r.Auth.TerminateInstance(target); err != nil { r.Info("failed to terminate instance", "name", rollingUpgrade.NamespacedName(), "instance", instanceID, "message", err) return true, nil } rollingUpgrade.SetLastNodeTerminationTime(metav1.Time{Time: time.Now()}) - // Post Wait Script + // Post Terminate Script if err := r.ScriptRunner.PostTerminate(scriptTarget); err != nil { return false, err } @@ -334,3 +341,30 @@ func (r *RollingUpgradeReconciler) IsScalingGroupDrifted(rollingUpgrade *v1alpha } return false } + +func (r *RollingUpgradeReconciler) DesiredNodesReady(rollingUpgrade *v1alpha1.RollingUpgrade) bool { + var ( + scalingGroup = awsprovider.SelectScalingGroup(rollingUpgrade.ScalingGroupName(), r.Cloud.ScalingGroups) + desiredInstances = aws.Int64Value(scalingGroup.DesiredCapacity) + readyNodes = 0 + ) + + // wait for desired instances + inServiceInstances := awsprovider.GetInServiceInstances(scalingGroup) + if len(inServiceInstances) != int(desiredInstances) { + return false + } + + // wait for desired nodes + for _, node := range r.Cloud.ClusterNodes.Items { + instanceID := kubeprovider.GetNodeInstanceID(node) + if common.ContainsEqualFold(inServiceInstances, instanceID) && kubeprovider.IsNodeReady(node) && kubeprovider.IsNodePassesReadinessGates(node, rollingUpgrade.Spec.ReadinessGates) { + readyNodes++ + } + } + if readyNodes != int(desiredInstances) { + return false + } + + return true +} From 14e950e1bed5e28ce51e0416f953f68617bb979a Mon Sep 17 00:00:00 2001 From: Sheldon Shao Date: Tue, 2 Mar 2021 13:46:05 -0800 Subject: [PATCH 17/74] Metrics features (#189) Signed-off-by: xshao --- api/v1alpha1/rollingupgrade_types.go | 106 ++++++++++++++++++ api/v1alpha1/rollingupgrade_types_test.go | 47 ++++++++ api/v1alpha1/zz_generated.deepcopy.go | 60 ++++++++++ ...grademgr.keikoproj.io_rollingupgrades.yaml | 33 ++++++ controllers/common/metrics.go | 74 ++++++++++++ controllers/common/metrics_test.go | 29 +++++ controllers/providers/kubernetes/utils.go | 10 +- controllers/upgrade.go | 28 +++++ go.mod | 8 +- main.go | 5 + 10 files changed, 393 insertions(+), 7 deletions(-) create mode 100644 api/v1alpha1/rollingupgrade_types_test.go create mode 100644 controllers/common/metrics.go create mode 100644 controllers/common/metrics_test.go diff --git a/api/v1alpha1/rollingupgrade_types.go b/api/v1alpha1/rollingupgrade_types.go index f0fc2d00..bc3447a8 100644 --- a/api/v1alpha1/rollingupgrade_types.go +++ b/api/v1alpha1/rollingupgrade_types.go @@ -20,6 +20,7 @@ import ( "fmt" "strconv" "strings" + "time" "github.com/keikoproj/upgrade-manager/controllers/common" corev1 "k8s.io/api/core/v1" @@ -52,6 +53,85 @@ type RollingUpgradeStatus struct { Conditions []RollingUpgradeCondition `json:"conditions,omitempty"` LastNodeTerminationTime metav1.Time `json:"lastTerminationTime,omitempty"` LastNodeDrainTime metav1.Time `json:"lastDrainTime,omitempty"` + + Statistics []*RollingUpgradeStatistics `json:"statistics,omitempty"` + InProcessingNodes map[string]*NodeInProcessing `json:"inProcessingNodes,omitempty"` +} + +// RollingUpgrade Statistics, includes summary(sum/count) from each step +type RollingUpgradeStatistics struct { + StepName RollingUpgradeStep `json:"stepName,omitempty"` + DurationSum metav1.Duration `json:"durationSum,omitempty"` + DurationCount int32 `json:"durationCount,omitempty"` +} + +// Node In-processing +type NodeInProcessing struct { + NodeName string `json:"nodeName,omitempty"` + StepName RollingUpgradeStep `json:"stepName,omitempty"` + UpgradeStartTime metav1.Time `json:"upgradeStartTime,omitempty"` + StepStartTime metav1.Time `json:"stepStartTime,omitempty"` + StepEndTime metav1.Time `json:"stepEndTime,omitempty"` +} + +// Add one step duration +func (s *RollingUpgradeStatus) addStepDuration(asgName string, stepName RollingUpgradeStep, duration time.Duration) { + // if step exists, add count and sum, otherwise append + for _, s := range s.Statistics { + if s.StepName == stepName { + s.DurationSum = metav1.Duration{ + Duration: s.DurationSum.Duration + duration, + } + s.DurationCount += 1 + return + } + } + s.Statistics = append(s.Statistics, &RollingUpgradeStatistics{ + StepName: stepName, + DurationSum: metav1.Duration{ + Duration: duration, + }, + DurationCount: 1, + }) + + //Add to system level statistics + common.AddRollingUpgradeStepDuration(asgName, string(stepName), duration) +} + +// Node turns onto step +func (s *RollingUpgradeStatus) NodeStep(asgName string, nodeName string, stepName RollingUpgradeStep) { + if s.InProcessingNodes == nil { + s.InProcessingNodes = make(map[string]*NodeInProcessing) + } + var inProcessingNode *NodeInProcessing + if n, ok := s.InProcessingNodes[nodeName]; !ok { + inProcessingNode = &NodeInProcessing{ + NodeName: nodeName, + StepName: stepName, + UpgradeStartTime: metav1.Now(), + StepStartTime: metav1.Now(), + } + s.InProcessingNodes[nodeName] = inProcessingNode + } else { + inProcessingNode = n + n.StepEndTime = metav1.Now() + var duration = n.StepEndTime.Sub(n.StepStartTime.Time) + if stepName == NodeRotationCompleted { + //Add overall and remove the node from in-processing map + var total = n.StepEndTime.Sub(n.UpgradeStartTime.Time) + s.addStepDuration(asgName, inProcessingNode.StepName, duration) + s.addStepDuration(asgName, NodeRotationTotal, total) + delete(s.InProcessingNodes, nodeName) + } else if inProcessingNode.StepName != stepName { //Still same step + var oldOrder = NodeRotationStepOrders[inProcessingNode.StepName] + var newOrder = NodeRotationStepOrders[stepName] + if newOrder > oldOrder { //Make sure the steps running in order + s.addStepDuration(asgName, inProcessingNode.StepName, duration) + n.StepStartTime = metav1.Now() + inProcessingNode.StepName = stepName + } + } + } } func (s *RollingUpgradeStatus) SetCondition(cond RollingUpgradeCondition) { @@ -115,6 +195,8 @@ type NodeReadinessGate struct { MatchLabels map[string]string `json:"matchLabels,omitempty" protobuf:"bytes,1,rep,name=matchLabels"` } +type RollingUpgradeStep string + const ( // Status StatusInit = "init" @@ -124,8 +206,32 @@ const ( // Conditions UpgradeComplete UpgradeConditionType = "Complete" + + NodeRotationTotal RollingUpgradeStep = "total" + + NodeRotationKickoff RollingUpgradeStep = "kickoff" + NodeRotationDesiredNodeReady RollingUpgradeStep = "desired_node_ready" + NodeRotationPredrainScript RollingUpgradeStep = "predrain_script" + NodeRotationDrain RollingUpgradeStep = "drain" + NodeRotationPostdrainScript RollingUpgradeStep = "postdrain_script" + NodeRotationPostWait RollingUpgradeStep = "post_wait" + NodeRotationTerminate RollingUpgradeStep = "terminate" + NodeRotationPostTerminate RollingUpgradeStep = "post_terminate" + NodeRotationCompleted RollingUpgradeStep = "completed" ) +var NodeRotationStepOrders = map[RollingUpgradeStep]int{ + NodeRotationKickoff: 10, + NodeRotationDesiredNodeReady: 20, + NodeRotationPredrainScript: 30, + NodeRotationDrain: 40, + NodeRotationPostdrainScript: 50, + NodeRotationPostWait: 60, + NodeRotationTerminate: 70, + NodeRotationPostTerminate: 80, + NodeRotationCompleted: 1000, +} + var ( FiniteStates = []string{StatusComplete, StatusError} AllowedStrategyType = []string{string(RandomUpdateStrategy), string(UniformAcrossAzUpdateStrategy)} diff --git a/api/v1alpha1/rollingupgrade_types_test.go b/api/v1alpha1/rollingupgrade_types_test.go new file mode 100644 index 00000000..5e752341 --- /dev/null +++ b/api/v1alpha1/rollingupgrade_types_test.go @@ -0,0 +1,47 @@ +package v1alpha1 + +import ( + "github.com/onsi/gomega" + "testing" +) + +// Test +func TestNodeTurnsOntoStep(t *testing.T) { + g := gomega.NewGomegaWithT(t) + + r := &RollingUpgradeStatus{} + + r.NodeStep("test-asg", "node-1", NodeRotationKickoff) + + g.Expect(r.InProcessingNodes).NotTo(gomega.BeNil()) + g.Expect(r.Statistics).To(gomega.BeNil()) + + r.NodeStep("test-asg", "node-1", NodeRotationDesiredNodeReady) + + g.Expect(r.Statistics).NotTo(gomega.BeNil()) + g.Expect(len(r.Statistics)).To(gomega.Equal(1)) + g.Expect(r.Statistics[0].StepName).To(gomega.Equal(NodeRotationKickoff)) + + //Retry desired_node_ready + r.NodeStep("test-asg", "node-1", NodeRotationDesiredNodeReady) + g.Expect(len(r.Statistics)).To(gomega.Equal(1)) + g.Expect(r.Statistics[0].StepName).To(gomega.Equal(NodeRotationKickoff)) + + //Retry desired_node_ready again + r.NodeStep("test-asg", "node-1", NodeRotationDesiredNodeReady) + g.Expect(len(r.Statistics)).To(gomega.Equal(1)) + g.Expect(r.Statistics[0].StepName).To(gomega.Equal(NodeRotationKickoff)) + + //Completed + r.NodeStep("test-asg", "node-1", NodeRotationCompleted) + g.Expect(len(r.Statistics)).To(gomega.Equal(3)) + g.Expect(r.Statistics[1].StepName).To(gomega.Equal(NodeRotationDesiredNodeReady)) + g.Expect(r.Statistics[2].StepName).To(gomega.Equal(NodeRotationTotal)) + + //Second node + r.NodeStep("test-asg", "node-2", NodeRotationKickoff) + g.Expect(len(r.Statistics)).To(gomega.Equal(3)) + + r.NodeStep("test-asg", "node-2", NodeRotationDesiredNodeReady) + g.Expect(len(r.Statistics)).To(gomega.Equal(3)) +} diff --git a/api/v1alpha1/zz_generated.deepcopy.go b/api/v1alpha1/zz_generated.deepcopy.go index 5d613e18..1c76fdc4 100644 --- a/api/v1alpha1/zz_generated.deepcopy.go +++ b/api/v1alpha1/zz_generated.deepcopy.go @@ -24,6 +24,24 @@ import ( runtime "k8s.io/apimachinery/pkg/runtime" ) +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *NodeInProcessing) DeepCopyInto(out *NodeInProcessing) { + *out = *in + in.UpgradeStartTime.DeepCopyInto(&out.UpgradeStartTime) + in.StepStartTime.DeepCopyInto(&out.StepStartTime) + in.StepEndTime.DeepCopyInto(&out.StepEndTime) +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new NodeInProcessing. +func (in *NodeInProcessing) DeepCopy() *NodeInProcessing { + if in == nil { + return nil + } + out := new(NodeInProcessing) + in.DeepCopyInto(out) + return out +} + // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *NodeReadinessGate) DeepCopyInto(out *NodeReadinessGate) { *out = *in @@ -191,6 +209,22 @@ func (in *RollingUpgradeSpec) DeepCopy() *RollingUpgradeSpec { return out } +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *RollingUpgradeStatistics) DeepCopyInto(out *RollingUpgradeStatistics) { + *out = *in + out.DurationSum = in.DurationSum +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RollingUpgradeStatistics. +func (in *RollingUpgradeStatistics) DeepCopy() *RollingUpgradeStatistics { + if in == nil { + return nil + } + out := new(RollingUpgradeStatistics) + in.DeepCopyInto(out) + return out +} + // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *RollingUpgradeStatus) DeepCopyInto(out *RollingUpgradeStatus) { *out = *in @@ -201,6 +235,32 @@ func (in *RollingUpgradeStatus) DeepCopyInto(out *RollingUpgradeStatus) { } in.LastNodeTerminationTime.DeepCopyInto(&out.LastNodeTerminationTime) in.LastNodeDrainTime.DeepCopyInto(&out.LastNodeDrainTime) + if in.Statistics != nil { + in, out := &in.Statistics, &out.Statistics + *out = make([]*RollingUpgradeStatistics, len(*in)) + for i := range *in { + if (*in)[i] != nil { + in, out := &(*in)[i], &(*out)[i] + *out = new(RollingUpgradeStatistics) + **out = **in + } + } + } + if in.InProcessingNodes != nil { + in, out := &in.InProcessingNodes, &out.InProcessingNodes + *out = make(map[string]*NodeInProcessing, len(*in)) + for key, val := range *in { + var outVal *NodeInProcessing + if val == nil { + (*out)[key] = nil + } else { + in, out := &val, &outVal + *out = new(NodeInProcessing) + (*in).DeepCopyInto(*out) + } + (*out)[key] = outVal + } + } } // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RollingUpgradeStatus. diff --git a/config/crd/bases/upgrademgr.keikoproj.io_rollingupgrades.yaml b/config/crd/bases/upgrademgr.keikoproj.io_rollingupgrades.yaml index f8eb03d1..615ffff6 100644 --- a/config/crd/bases/upgrademgr.keikoproj.io_rollingupgrades.yaml +++ b/config/crd/bases/upgrademgr.keikoproj.io_rollingupgrades.yaml @@ -133,6 +133,25 @@ spec: type: string endTime: type: string + inProcessingNodes: + additionalProperties: + description: Node In-processing + properties: + nodeName: + type: string + stepEndTime: + format: date-time + type: string + stepName: + type: string + stepStartTime: + format: date-time + type: string + upgradeStartTime: + format: date-time + type: string + type: object + type: object lastDrainTime: format: date-time type: string @@ -143,6 +162,20 @@ spec: type: integer startTime: type: string + statistics: + items: + description: RollingUpgrade Statistics, includes summary(sum/count) + from each step + properties: + durationCount: + format: int32 + type: integer + durationSum: + type: string + stepName: + type: string + type: object + type: array totalNodes: type: integer totalProcessingTime: diff --git a/controllers/common/metrics.go b/controllers/common/metrics.go new file mode 100644 index 00000000..649429dc --- /dev/null +++ b/controllers/common/metrics.go @@ -0,0 +1,74 @@ +package common + +import ( + "github.com/keikoproj/upgrade-manager/controllers/common/log" + "github.com/prometheus/client_golang/prometheus" + "reflect" + "sigs.k8s.io/controller-runtime/pkg/metrics" + "strings" + "time" +) + +//All cluster level node upgrade statistics + +var nodeRotationTotal = prometheus.NewHistogram( + prometheus.HistogramOpts{ + Namespace: "node", + Name: "rotation_total_seconds", + Help: "Node rotation total", + Buckets: []float64{ + 10.0, + 30.0, + 60.0, + 90.0, + 120.0, + 180.0, + 300.0, + 600.0, + 900.0, + }, + }) + +var stepSummaries = make(map[string]map[string]prometheus.Summary) + +func InitMetrics() { + metrics.Registry.MustRegister(nodeRotationTotal) +} + +// Add rolling update step duration when the step is completed +func AddRollingUpgradeStepDuration(asgName string, stepName string, duration time.Duration) { + if strings.EqualFold(stepName, "total") { //Histogram + nodeRotationTotal.Observe(duration.Seconds()) + } else { //Summary + var steps map[string]prometheus.Summary + if m, ok := stepSummaries[asgName]; !ok { + steps = make(map[string]prometheus.Summary) + stepSummaries[asgName] = steps + } else { + steps = m + } + + var summary prometheus.Summary + if s, ok := steps[stepName]; !ok { + summary = prometheus.NewSummary( + prometheus.SummaryOpts{ + Namespace: "node", + Name: stepName + "_seconds", + Help: "Summary for node " + stepName, + ConstLabels: prometheus.Labels{"asg": asgName}, + }) + err := metrics.Registry.Register(summary) + if err != nil { + if reflect.TypeOf(err).String() == "prometheus.AlreadyRegisteredError" { + log.Warnf("summary was registered again, ASG: %s, step: %s", asgName, stepName) + } else { + log.Errorf("register summary error, ASG: %s, step: %s, %v", asgName, stepName, err) + } + } + steps[stepName] = summary + } else { + summary = s + } + summary.Observe(duration.Seconds()) + } +} diff --git a/controllers/common/metrics_test.go b/controllers/common/metrics_test.go new file mode 100644 index 00000000..296ae336 --- /dev/null +++ b/controllers/common/metrics_test.go @@ -0,0 +1,29 @@ +package common + +import ( + "github.com/onsi/gomega" + "testing" +) + +func TestAddRollingUpgradeStepDuration(t *testing.T) { + g := gomega.NewGomegaWithT(t) + + g.Expect(stepSummaries["test-asg"]).To(gomega.BeNil()) + AddRollingUpgradeStepDuration("test-asg", "kickoff", 1) + + g.Expect(stepSummaries["test-asg"]).NotTo(gomega.BeNil()) + g.Expect(stepSummaries["test-asg"]["kickoff"]).NotTo(gomega.BeNil()) + + //Test duplicate + AddRollingUpgradeStepDuration("test-asg", "kickoff", 1) + g.Expect(stepSummaries["test-asg"]["kickoff"]).NotTo(gomega.BeNil()) + + //Test duplicate + delete(stepSummaries["test-asg"], "kickoff") + AddRollingUpgradeStepDuration("test-asg", "kickoff", 1) + g.Expect(stepSummaries["test-asg"]["kickoff"]).NotTo(gomega.BeNil()) + + //Test total + AddRollingUpgradeStepDuration("test-asg", "total", 1) + g.Expect(stepSummaries["test-asg"]["kickoff"]).NotTo(gomega.BeNil()) +} diff --git a/controllers/providers/kubernetes/utils.go b/controllers/providers/kubernetes/utils.go index 79f148b8..6ea58764 100644 --- a/controllers/providers/kubernetes/utils.go +++ b/controllers/providers/kubernetes/utils.go @@ -87,10 +87,12 @@ func GetKubernetesLocalConfig() (*rest.Config, error) { } func SelectNodeByInstanceID(instanceID string, nodes *corev1.NodeList) corev1.Node { - for _, node := range nodes.Items { - nodeID := GetNodeInstanceID(node) - if strings.EqualFold(instanceID, nodeID) { - return node + if nodes != nil { + for _, node := range nodes.Items { + nodeID := GetNodeInstanceID(node) + if strings.EqualFold(instanceID, nodeID) { + return node + } } } return corev1.Node{} diff --git a/controllers/upgrade.go b/controllers/upgrade.go index 76a5d78f..f008876c 100644 --- a/controllers/upgrade.go +++ b/controllers/upgrade.go @@ -108,6 +108,9 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol } ) + //Add statistics + rollingUpgrade.Status.NodeStep(rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationKickoff) + // Add in-progress tag if err := r.Auth.TagEC2instance(instanceID, instanceStateTagKey, inProgressTagValue); err != nil { r.Error(err, "failed to set instance tag", "name", rollingUpgrade.NamespacedName(), "instance", instanceID) @@ -123,12 +126,18 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol } } + // Turns onto desired nodes + rollingUpgrade.Status.NodeStep(rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationDesiredNodeReady) + // Wait for desired nodes if !r.DesiredNodesReady(rollingUpgrade) { r.Info("new node is yet to join the cluster") return true, nil } + // Turns onto PreDrain script + rollingUpgrade.Status.NodeStep(rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationPredrainScript) + // Predrain script if err := r.ScriptRunner.PreDrain(scriptTarget); err != nil { return false, err @@ -137,6 +146,10 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol // Issue drain concurrently - set lastDrainTime if node := kubeprovider.SelectNodeByInstanceID(instanceID, r.Cloud.ClusterNodes); !reflect.DeepEqual(node, corev1.Node{}) { r.Info("draining the node", "name", rollingUpgrade.NamespacedName(), "instance", instanceID, "node name", node.Name) + + // Turns onto NodeRotationDrain + rollingUpgrade.Status.NodeStep(rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationDrain) + if err := r.Auth.DrainNode(&node, time.Duration(rollingUpgrade.PostDrainDelaySeconds()), rollingUpgrade.DrainTimeout(), r.Auth.Kubernetes); err != nil { r.Error(err, "failed to drain node", "name", rollingUpgrade.NamespacedName(), "instance", instanceID, "node name", node.Name) return false, err @@ -144,16 +157,25 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol } rollingUpgrade.SetLastNodeDrainTime(metav1.Time{Time: time.Now()}) + // Turns onto NodeRotationPostdrainScript + rollingUpgrade.Status.NodeStep(rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationPostdrainScript) + // post drain script if err := r.ScriptRunner.PostDrain(scriptTarget); err != nil { return false, err } + // Turns onto NodeRotationPostWait + rollingUpgrade.Status.NodeStep(rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationPostWait) + // Post Wait Script if err := r.ScriptRunner.PostWait(scriptTarget); err != nil { return false, err } + // Turns onto NodeRotationTerminate + rollingUpgrade.Status.NodeStep(rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationTerminate) + // Terminate - set lastTerminateTime r.Info("terminating instance", "name", rollingUpgrade.NamespacedName(), "instance", instanceID) if err := r.Auth.TerminateInstance(target); err != nil { @@ -162,10 +184,16 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol } rollingUpgrade.SetLastNodeTerminationTime(metav1.Time{Time: time.Now()}) + // Turns onto NodeRotationTerminate + rollingUpgrade.Status.NodeStep(rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationPostTerminate) + // Post Terminate Script if err := r.ScriptRunner.PostTerminate(scriptTarget); err != nil { return false, err } + + // Turns onto NodeRotationCompleted + rollingUpgrade.Status.NodeStep(rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationCompleted) } case v1alpha1.UpdateStrategyModeLazy: for _, target := range batch { diff --git a/go.mod b/go.mod index 816421ef..b444654f 100644 --- a/go.mod +++ b/go.mod @@ -8,10 +8,12 @@ require ( github.com/keikoproj/aws-sdk-go-cache v0.0.0-20201118182730-f6f418a4e2df github.com/onsi/gomega v1.10.2 github.com/pkg/errors v0.9.1 + github.com/prometheus/client_golang v1.7.1 github.com/sirupsen/logrus v1.6.0 go.uber.org/zap v1.15.0 - k8s.io/api v0.19.2 - k8s.io/apimachinery v0.19.2 - k8s.io/client-go v0.19.2 + k8s.io/api v0.20.4 + k8s.io/apimachinery v0.20.4 + k8s.io/client-go v0.20.4 + k8s.io/kubectl v0.20.4 sigs.k8s.io/controller-runtime v0.7.0 ) diff --git a/main.go b/main.go index 3a5ef2b0..c6fd6fbc 100644 --- a/main.go +++ b/main.go @@ -18,6 +18,7 @@ package main import ( "flag" + "github.com/keikoproj/upgrade-manager/controllers/common" "os" "time" @@ -67,6 +68,8 @@ func init() { utilruntime.Must(upgrademgrv1alpha1.AddToScheme(scheme)) //+kubebuilder:scaffold:scheme + + common.InitMetrics() } func main() { @@ -212,6 +215,8 @@ func main() { os.Exit(1) } + setupLog.Info("registering prometheus") + setupLog.Info("starting manager") if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil { setupLog.Error(err, "problem running manager") From 728dae9bf748c5129d3420f5ec4991adcfff1dae Mon Sep 17 00:00:00 2001 From: Shreyas Badiger <7680410+shreyas-badiger@users.noreply.github.com> Date: Wed, 10 Mar 2021 17:01:58 -0800 Subject: [PATCH 18/74] Process the batch rotation in parallel (#192) * Process the batch rotation in parallel Signed-off-by: sbadiger * addressing review comments Signed-off-by: sbadiger --- controllers/cloud.go | 2 +- controllers/providers/kubernetes/nodes.go | 4 +- controllers/providers/kubernetes/utils.go | 17 +- controllers/rollingupgrade_controller.go | 26 ++- controllers/upgrade.go | 214 ++++++++++++++-------- 5 files changed, 172 insertions(+), 91 deletions(-) diff --git a/controllers/cloud.go b/controllers/cloud.go index 1bf779e4..2fea2bd7 100644 --- a/controllers/cloud.go +++ b/controllers/cloud.go @@ -68,7 +68,7 @@ func (d *DiscoveredState) Discover() error { d.InProgressInstances = inProgressInstances nodes, err := d.KubernetesClientSet.ListClusterNodes() - if err != nil { + if err != nil || nodes == nil || nodes.Size() == 0 { return errors.Wrap(err, "failed to discover cluster nodes") } d.ClusterNodes = nodes diff --git a/controllers/providers/kubernetes/nodes.go b/controllers/providers/kubernetes/nodes.go index 4ae2c0c7..7a24ee5b 100644 --- a/controllers/providers/kubernetes/nodes.go +++ b/controllers/providers/kubernetes/nodes.go @@ -33,8 +33,8 @@ import ( func (k *KubernetesClientSet) ListClusterNodes() (*corev1.NodeList, error) { var nodes *corev1.NodeList nodes, err := k.Kubernetes.CoreV1().Nodes().List(context.Background(), metav1.ListOptions{}) - if err != nil { - return nodes, err + if err != nil || nodes == nil { + return &corev1.NodeList{}, err } return nodes, nil } diff --git a/controllers/providers/kubernetes/utils.go b/controllers/providers/kubernetes/utils.go index 6ea58764..45d59ae3 100644 --- a/controllers/providers/kubernetes/utils.go +++ b/controllers/providers/kubernetes/utils.go @@ -20,7 +20,9 @@ import ( "fmt" "os" "os/user" + "reflect" "strings" + "sync" corev1 "k8s.io/api/core/v1" @@ -36,6 +38,12 @@ type KubernetesClientSet struct { Kubernetes kubernetes.Interface } +// DrainManager holds the information to perform drain operation in parallel. +type DrainManagerStruct struct { + DrainErrors chan error + DrainGroup *sync.WaitGroup +} + func GetKubernetesClient() (kubernetes.Interface, error) { var config *rest.Config config, err := GetKubernetesConfig() @@ -99,9 +107,12 @@ func SelectNodeByInstanceID(instanceID string, nodes *corev1.NodeList) corev1.No } func GetNodeInstanceID(node corev1.Node) string { - tokens := strings.Split(node.Spec.ProviderID, "/") - nodeInstanceID := tokens[len(tokens)-1] - return nodeInstanceID + if !reflect.DeepEqual(node, &corev1.Node{}) { + tokens := strings.Split(node.Spec.ProviderID, "/") + nodeInstanceID := tokens[len(tokens)-1] + return nodeInstanceID + } + return "" } func IsNodeReady(node corev1.Node) bool { diff --git a/controllers/rollingupgrade_controller.go b/controllers/rollingupgrade_controller.go index fb61c94b..de417c43 100644 --- a/controllers/rollingupgrade_controller.go +++ b/controllers/rollingupgrade_controller.go @@ -40,14 +40,17 @@ import ( type RollingUpgradeReconciler struct { client.Client logr.Logger - Scheme *runtime.Scheme - AdmissionMap sync.Map - CacheConfig *cache.Config - Auth *RollingUpgradeAuthenticator - Cloud *DiscoveredState - EventWriter *kubeprovider.EventWriter - maxParallel int - ScriptRunner ScriptRunner + Scheme *runtime.Scheme + AdmissionMap sync.Map + CacheConfig *cache.Config + Auth *RollingUpgradeAuthenticator + Cloud *DiscoveredState + EventWriter *kubeprovider.EventWriter + maxParallel int + ScriptRunner ScriptRunner + DrainGroupMapper sync.Map + DrainErrorMapper sync.Map + DrainManager kubeprovider.DrainManagerStruct } type RollingUpgradeAuthenticator struct { @@ -132,6 +135,13 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque return ctrl.Result{}, err } + drainGroup, _ := r.DrainGroupMapper.LoadOrStore(rollingUpgrade.NamespacedName(), &sync.WaitGroup{}) + drainErrs, _ := r.DrainErrorMapper.LoadOrStore(rollingUpgrade.NamespacedName(), make(chan error)) + r.DrainManager = kubeprovider.DrainManagerStruct{ + DrainErrors: drainErrs.(chan error), + DrainGroup: drainGroup.(*sync.WaitGroup), + } + // process node rotation if err := r.RotateNodes(rollingUpgrade); err != nil { rollingUpgrade.SetCurrentStatus(v1alpha1.StatusError) diff --git a/controllers/upgrade.go b/controllers/upgrade.go index f008876c..ea537991 100644 --- a/controllers/upgrade.go +++ b/controllers/upgrade.go @@ -19,6 +19,7 @@ package controllers import ( "reflect" "strings" + "sync" "time" "github.com/aws/aws-sdk-go/aws" @@ -27,11 +28,17 @@ import ( "github.com/keikoproj/upgrade-manager/controllers/common" awsprovider "github.com/keikoproj/upgrade-manager/controllers/providers/aws" kubeprovider "github.com/keikoproj/upgrade-manager/controllers/providers/kubernetes" + "github.com/pkg/errors" corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/intstr" ) +var ( + //DefaultWaitGroupTimeout is the timeout value for DrainGroup + DefaultWaitGroupTimeout = time.Second * 5 +) + // TODO: main node rotation logic func (r *RollingUpgradeReconciler) RotateNodes(rollingUpgrade *v1alpha1.RollingUpgrade) error { var ( @@ -91,25 +98,14 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol switch mode { case v1alpha1.UpdateStrategyModeEager: - - // TODO: THE BELOW LOOP IS A TEMPORARY PLACEHOLDER FOR TESTING PURPOSES - // WE SHOULD SWITCH TO PARALLEL PROCESSING OF TARGETS IN A BATCH VIA GOROUTINES - for _, target := range batch { - var ( - instanceID = aws.StringValue(target.InstanceId) - node = kubeprovider.SelectNodeByInstanceID(instanceID, r.Cloud.ClusterNodes) - nodeName = node.GetName() - scriptTarget = ScriptTarget{ - InstanceID: instanceID, - NodeName: nodeName, - UpgradeObject: rollingUpgrade, - } + instanceID = aws.StringValue(target.InstanceId) + //node = kubeprovider.SelectNodeByInstanceID(instanceID, r.Cloud.ClusterNodes) + //nodeName = node.GetName() ) - //Add statistics - rollingUpgrade.Status.NodeStep(rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationKickoff) + //rollingUpgrade.Status.NodeStep(rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationKickoff) // Add in-progress tag if err := r.Auth.TagEC2instance(instanceID, instanceStateTagKey, inProgressTagValue); err != nil { @@ -119,73 +115,141 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol // Standby if aws.StringValue(target.LifecycleState) == autoscaling.LifecycleStateInService { - r.Info("setting instance to stand-by", "name", rollingUpgrade.NamespacedName(), "instance", instanceID) + r.Info("setting instance to stand-by", "instance", instanceID, "name", rollingUpgrade.NamespacedName()) if err := r.Auth.SetInstanceStandBy(target, rollingUpgrade.Spec.AsgName); err != nil { - r.Error(err, "couldn't set the instance to stand-by", "name", rollingUpgrade.NamespacedName(), "instance", instanceID) - return false, err + // failure to set instance to standby are retryable + r.Info("failed to set instance to stand-by", "instance", instanceID, "message", err.Error(), "name", rollingUpgrade.NamespacedName()) + return true, nil } } // Turns onto desired nodes - rollingUpgrade.Status.NodeStep(rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationDesiredNodeReady) + //rollingUpgrade.Status.NodeStep(rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationDesiredNodeReady) // Wait for desired nodes + r.Info("waiting for desired nodes", "name", rollingUpgrade.NamespacedName()) if !r.DesiredNodesReady(rollingUpgrade) { - r.Info("new node is yet to join the cluster") + r.Info("new node is yet to join the cluster", "name", rollingUpgrade.NamespacedName()) return true, nil } + } - // Turns onto PreDrain script - rollingUpgrade.Status.NodeStep(rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationPredrainScript) + case v1alpha1.UpdateStrategyModeLazy: + for _, target := range batch { + var ( + instanceID = aws.StringValue(target.InstanceId) + //node = kubeprovider.SelectNodeByInstanceID(instanceID, r.Cloud.ClusterNodes) + //nodeName = node.GetName() + ) + //Add statistics + //rollingUpgrade.Status.NodeStep(rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationKickoff) - // Predrain script - if err := r.ScriptRunner.PreDrain(scriptTarget); err != nil { + // Add in-progress tag + if err := r.Auth.TagEC2instance(instanceID, instanceStateTagKey, inProgressTagValue); err != nil { + r.Error(err, "failed to set instance tag", "name", rollingUpgrade.NamespacedName(), "instance", instanceID) return false, err } + } + } + + if reflect.DeepEqual(r.DrainManager.DrainGroup, &sync.WaitGroup{}) { + for _, target := range batch { + var ( + instanceID = aws.StringValue(target.InstanceId) + node = kubeprovider.SelectNodeByInstanceID(instanceID, r.Cloud.ClusterNodes) + nodeName = node.GetName() + scriptTarget = ScriptTarget{ + InstanceID: instanceID, + NodeName: nodeName, + UpgradeObject: rollingUpgrade, + } + ) + r.DrainManager.DrainGroup.Add(1) - // Issue drain concurrently - set lastDrainTime - if node := kubeprovider.SelectNodeByInstanceID(instanceID, r.Cloud.ClusterNodes); !reflect.DeepEqual(node, corev1.Node{}) { - r.Info("draining the node", "name", rollingUpgrade.NamespacedName(), "instance", instanceID, "node name", node.Name) + go func() { + defer r.DrainManager.DrainGroup.Done() - // Turns onto NodeRotationDrain - rollingUpgrade.Status.NodeStep(rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationDrain) + // Turns onto PreDrain script + //rollingUpgrade.Status.NodeStep(rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationPredrainScript) - if err := r.Auth.DrainNode(&node, time.Duration(rollingUpgrade.PostDrainDelaySeconds()), rollingUpgrade.DrainTimeout(), r.Auth.Kubernetes); err != nil { - r.Error(err, "failed to drain node", "name", rollingUpgrade.NamespacedName(), "instance", instanceID, "node name", node.Name) - return false, err + // Predrain script + if err := r.ScriptRunner.PreDrain(scriptTarget); err != nil { + r.DrainManager.DrainErrors <- errors.Errorf("PreDrain failed: instanceID - %v, %v", instanceID, err.Error()) } - } - rollingUpgrade.SetLastNodeDrainTime(metav1.Time{Time: time.Now()}) - // Turns onto NodeRotationPostdrainScript - rollingUpgrade.Status.NodeStep(rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationPostdrainScript) + // Issue drain concurrently - set lastDrainTime + if node := kubeprovider.SelectNodeByInstanceID(instanceID, r.Cloud.ClusterNodes); !reflect.DeepEqual(node, corev1.Node{}) { + r.Info("draining the node", "instance", instanceID, "node name", node.Name, "name", rollingUpgrade.NamespacedName()) - // post drain script - if err := r.ScriptRunner.PostDrain(scriptTarget); err != nil { - return false, err - } + // Turns onto NodeRotationDrain + //rollingUpgrade.Status.NodeStep(rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationDrain) - // Turns onto NodeRotationPostWait - rollingUpgrade.Status.NodeStep(rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationPostWait) + if err := r.Auth.DrainNode(&node, time.Duration(rollingUpgrade.PostDrainDelaySeconds()), rollingUpgrade.DrainTimeout(), r.Auth.Kubernetes); err != nil { + r.DrainManager.DrainErrors <- errors.Errorf("DrainNode failed: instanceID - %v, %v", instanceID, err.Error()) + } + } - // Post Wait Script - if err := r.ScriptRunner.PostWait(scriptTarget); err != nil { - return false, err - } + // Turns onto NodeRotationPostdrainScript + //rollingUpgrade.Status.NodeStep(rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationPostdrainScript) + + // post drain script + if err := r.ScriptRunner.PostDrain(scriptTarget); err != nil { + r.DrainManager.DrainErrors <- errors.Errorf("PostDrain failed: instanceID - %v, %v", instanceID, err.Error()) + } + + // Turns onto NodeRotationPostWait + //rollingUpgrade.Status.NodeStep(rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationPostWait) + + // Post Wait Script + if err := r.ScriptRunner.PostWait(scriptTarget); err != nil { + r.DrainManager.DrainErrors <- errors.Errorf("PostWait failed: instanceID - %v, %v", instanceID, err.Error()) + } + }() + } + } + + timeout := make(chan struct{}) + go func() { + defer close(timeout) + r.DrainManager.DrainGroup.Wait() + }() + + select { + case err := <-r.DrainManager.DrainErrors: + r.Error(err, "failed to rotate the node", "name", rollingUpgrade.NamespacedName()) + return false, err + + case <-timeout: + // goroutines completed, terminate and requeue + rollingUpgrade.SetLastNodeDrainTime(metav1.Time{Time: time.Now()}) + r.Info("instances drained successfully, terminating", "name", rollingUpgrade.NamespacedName()) + for _, target := range batch { + var ( + instanceID = aws.StringValue(target.InstanceId) + node = kubeprovider.SelectNodeByInstanceID(instanceID, r.Cloud.ClusterNodes) + nodeName = node.GetName() + scriptTarget = ScriptTarget{ + InstanceID: instanceID, + NodeName: nodeName, + UpgradeObject: rollingUpgrade, + } + ) // Turns onto NodeRotationTerminate - rollingUpgrade.Status.NodeStep(rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationTerminate) + //rollingUpgrade.Status.NodeStep(rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationTerminate) // Terminate - set lastTerminateTime - r.Info("terminating instance", "name", rollingUpgrade.NamespacedName(), "instance", instanceID) + r.Info("terminating instance", "instance", instanceID, "name", rollingUpgrade.NamespacedName()) + if err := r.Auth.TerminateInstance(target); err != nil { - r.Info("failed to terminate instance", "name", rollingUpgrade.NamespacedName(), "instance", instanceID, "message", err) + // terminate failures are retryable + r.Info("failed to terminate instance", "instance", instanceID, "message", err.Error(), "name", rollingUpgrade.NamespacedName()) return true, nil } rollingUpgrade.SetLastNodeTerminationTime(metav1.Time{Time: time.Now()}) // Turns onto NodeRotationTerminate - rollingUpgrade.Status.NodeStep(rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationPostTerminate) + //rollingUpgrade.Status.NodeStep(rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationPostTerminate) // Post Terminate Script if err := r.ScriptRunner.PostTerminate(scriptTarget); err != nil { @@ -193,19 +257,13 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol } // Turns onto NodeRotationCompleted - rollingUpgrade.Status.NodeStep(rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationCompleted) + //rollingUpgrade.Status.NodeStep(rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationCompleted) } - case v1alpha1.UpdateStrategyModeLazy: - for _, target := range batch { - _ = target - // Add in-progress tag - - // Issue drain/scripts concurrently - set lastDrainTime - // Is drained? - - // Terminate - set lastTerminateTime - } + case <-time.After(DefaultWaitGroupTimeout): + // goroutines timed out - requeue + r.Info("instances are still draining", "name", rollingUpgrade.NamespacedName()) + return true, nil } return true, nil } @@ -226,8 +284,9 @@ func (r *RollingUpgradeReconciler) SelectTargets(rollingUpgrade *v1alpha1.Rollin // first process all in progress instances for _, instance := range r.Cloud.InProgressInstances { - selectedInstance := awsprovider.SelectScalingGroupInstance(instance, scalingGroup) - targets = append(targets, selectedInstance) + if selectedInstance := awsprovider.SelectScalingGroupInstance(instance, scalingGroup); !reflect.DeepEqual(selectedInstance, &autoscaling.Instance{}) { + targets = append(targets, selectedInstance) + } } if len(targets) > 0 { @@ -272,7 +331,6 @@ func (r *RollingUpgradeReconciler) SelectTargets(rollingUpgrade *v1alpha1.Rollin } return AZtargets[:unavailableInt] } - return targets } @@ -296,25 +354,25 @@ func (r *RollingUpgradeReconciler) IsInstanceDrifted(rollingUpgrade *v1alpha1.Ro upgradeCreationTime = rollingUpgrade.CreationTimestamp.Time ) if nodeCreationTime.Before(upgradeCreationTime) { - r.Info("rolling upgrade configured for forced refresh", "name", rollingUpgrade.NamespacedName(), "instance", instanceID) + r.Info("rolling upgrade configured for forced refresh", "instance", instanceID, "name", rollingUpgrade.NamespacedName()) return true } } if scalingGroup.LaunchConfigurationName != nil { if instance.LaunchConfigurationName == nil { - r.Info("launch configuration name differs", "name", rollingUpgrade.NamespacedName(), "instance", instanceID) + r.Info("launch configuration name differs", "instance", instanceID, "name", rollingUpgrade.NamespacedName()) return true } launchConfigName := aws.StringValue(scalingGroup.LaunchConfigurationName) instanceConfigName := aws.StringValue(instance.LaunchConfigurationName) if !strings.EqualFold(launchConfigName, instanceConfigName) { - r.Info("launch configuration name differs", "name", rollingUpgrade.NamespacedName(), "instance", instanceID) + r.Info("launch configuration name differs", "instance", instanceID, "name", rollingUpgrade.NamespacedName()) return true } } else if scalingGroup.LaunchTemplate != nil { if instance.LaunchTemplate == nil { - r.Info("launch template name differs", "name", rollingUpgrade.NamespacedName(), "instance", instanceID) + r.Info("launch template name differs", "instance", instanceID, "name", rollingUpgrade.NamespacedName()) return true } @@ -326,16 +384,16 @@ func (r *RollingUpgradeReconciler) IsInstanceDrifted(rollingUpgrade *v1alpha1.Ro ) if !strings.EqualFold(launchTemplateName, instanceTemplateName) { - r.Info("launch template name differs", "name", rollingUpgrade.NamespacedName(), "instance", instanceID) + r.Info("launch template name differs", "instance", instanceID, "name", rollingUpgrade.NamespacedName()) return true } else if !strings.EqualFold(instanceTemplateVersion, templateVersion) { - r.Info("launch template version differs", "name", rollingUpgrade.NamespacedName(), "instance", instanceID) + r.Info("launch template version differs", "instance", instanceID, "name", rollingUpgrade.NamespacedName()) return true } } else if scalingGroup.MixedInstancesPolicy != nil { if instance.LaunchTemplate == nil { - r.Info("launch template name differs", "name", rollingUpgrade.NamespacedName(), "instance", instanceID) + r.Info("launch template name differs", "instance", instanceID, "name", rollingUpgrade.NamespacedName()) return true } @@ -347,10 +405,10 @@ func (r *RollingUpgradeReconciler) IsInstanceDrifted(rollingUpgrade *v1alpha1.Ro ) if !strings.EqualFold(launchTemplateName, instanceTemplateName) { - r.Info("launch template name differs", "name", rollingUpgrade.NamespacedName(), "instance", instanceID) + r.Info("launch template name differs", "instance", instanceID, "name", rollingUpgrade.NamespacedName()) return true } else if !strings.EqualFold(instanceTemplateVersion, templateVersion) { - r.Info("launch template version differs", "name", rollingUpgrade.NamespacedName(), "instance", instanceID) + r.Info("launch template version differs", "instance", instanceID, "name", rollingUpgrade.NamespacedName()) return true } } @@ -384,10 +442,12 @@ func (r *RollingUpgradeReconciler) DesiredNodesReady(rollingUpgrade *v1alpha1.Ro } // wait for desired nodes - for _, node := range r.Cloud.ClusterNodes.Items { - instanceID := kubeprovider.GetNodeInstanceID(node) - if common.ContainsEqualFold(inServiceInstances, instanceID) && kubeprovider.IsNodeReady(node) && kubeprovider.IsNodePassesReadinessGates(node, rollingUpgrade.Spec.ReadinessGates) { - readyNodes++ + if r.Cloud.ClusterNodes != nil && !reflect.DeepEqual(r.Cloud.ClusterNodes, &corev1.NodeList{}) { + for _, node := range r.Cloud.ClusterNodes.Items { + instanceID := kubeprovider.GetNodeInstanceID(node) + if common.ContainsEqualFold(inServiceInstances, instanceID) && kubeprovider.IsNodeReady(node) && kubeprovider.IsNodePassesReadinessGates(node, rollingUpgrade.Spec.ReadinessGates) { + readyNodes++ + } } } if readyNodes != int(desiredInstances) { From 668c5d86d2e1d19d317131e75f07c381955e1316 Mon Sep 17 00:00:00 2001 From: Shreyas Badiger <7680410+shreyas-badiger@users.noreply.github.com> Date: Thu, 18 Mar 2021 12:11:34 -0700 Subject: [PATCH 19/74] Move the DrainManager within ReplaceBatch(), to access one per RollingUpgrade CR (#195) Signed-off-by: sbadiger --- controllers/providers/kubernetes/utils.go | 7 ----- controllers/rollingupgrade_controller.go | 8 ------ controllers/upgrade.go | 35 ++++++++++++++++------- 3 files changed, 25 insertions(+), 25 deletions(-) diff --git a/controllers/providers/kubernetes/utils.go b/controllers/providers/kubernetes/utils.go index 45d59ae3..997d2709 100644 --- a/controllers/providers/kubernetes/utils.go +++ b/controllers/providers/kubernetes/utils.go @@ -22,7 +22,6 @@ import ( "os/user" "reflect" "strings" - "sync" corev1 "k8s.io/api/core/v1" @@ -38,12 +37,6 @@ type KubernetesClientSet struct { Kubernetes kubernetes.Interface } -// DrainManager holds the information to perform drain operation in parallel. -type DrainManagerStruct struct { - DrainErrors chan error - DrainGroup *sync.WaitGroup -} - func GetKubernetesClient() (kubernetes.Interface, error) { var config *rest.Config config, err := GetKubernetesConfig() diff --git a/controllers/rollingupgrade_controller.go b/controllers/rollingupgrade_controller.go index de417c43..2c1f3d32 100644 --- a/controllers/rollingupgrade_controller.go +++ b/controllers/rollingupgrade_controller.go @@ -50,7 +50,6 @@ type RollingUpgradeReconciler struct { ScriptRunner ScriptRunner DrainGroupMapper sync.Map DrainErrorMapper sync.Map - DrainManager kubeprovider.DrainManagerStruct } type RollingUpgradeAuthenticator struct { @@ -135,13 +134,6 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque return ctrl.Result{}, err } - drainGroup, _ := r.DrainGroupMapper.LoadOrStore(rollingUpgrade.NamespacedName(), &sync.WaitGroup{}) - drainErrs, _ := r.DrainErrorMapper.LoadOrStore(rollingUpgrade.NamespacedName(), make(chan error)) - r.DrainManager = kubeprovider.DrainManagerStruct{ - DrainErrors: drainErrs.(chan error), - DrainGroup: drainGroup.(*sync.WaitGroup), - } - // process node rotation if err := r.RotateNodes(rollingUpgrade); err != nil { rollingUpgrade.SetCurrentStatus(v1alpha1.StatusError) diff --git a/controllers/upgrade.go b/controllers/upgrade.go index ea537991..47737fcd 100644 --- a/controllers/upgrade.go +++ b/controllers/upgrade.go @@ -39,6 +39,12 @@ var ( DefaultWaitGroupTimeout = time.Second * 5 ) +// DrainManager holds the information to perform drain operation in parallel. +type DrainManager struct { + DrainErrors chan error `json:"-"` + DrainGroup *sync.WaitGroup `json:"-"` +} + // TODO: main node rotation logic func (r *RollingUpgradeReconciler) RotateNodes(rollingUpgrade *v1alpha1.RollingUpgrade) error { var ( @@ -91,11 +97,20 @@ func (r *RollingUpgradeReconciler) RotateNodes(rollingUpgrade *v1alpha1.RollingU func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.RollingUpgrade, batch []*autoscaling.Instance) (bool, error) { var ( - mode = rollingUpgrade.StrategyMode() + mode = rollingUpgrade.StrategyMode() + drainManager = &DrainManager{} ) r.Info("rotating batch", "instances", awsprovider.GetInstanceIDs(batch), "name", rollingUpgrade.NamespacedName()) + // load the appropriate waitGroup and Error channel for the DrainManager from reconciler object + drainGroup, _ := r.DrainGroupMapper.LoadOrStore(rollingUpgrade.NamespacedName(), &sync.WaitGroup{}) + drainErrs, _ := r.DrainErrorMapper.LoadOrStore(rollingUpgrade.NamespacedName(), make(chan error)) + drainManager = &DrainManager{ + DrainErrors: drainErrs.(chan error), + DrainGroup: drainGroup.(*sync.WaitGroup), + } + switch mode { case v1alpha1.UpdateStrategyModeEager: for _, target := range batch { @@ -152,7 +167,7 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol } } - if reflect.DeepEqual(r.DrainManager.DrainGroup, &sync.WaitGroup{}) { + if reflect.DeepEqual(drainManager.DrainGroup, &sync.WaitGroup{}) { for _, target := range batch { var ( instanceID = aws.StringValue(target.InstanceId) @@ -164,17 +179,17 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol UpgradeObject: rollingUpgrade, } ) - r.DrainManager.DrainGroup.Add(1) + drainManager.DrainGroup.Add(1) go func() { - defer r.DrainManager.DrainGroup.Done() + defer drainManager.DrainGroup.Done() // Turns onto PreDrain script //rollingUpgrade.Status.NodeStep(rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationPredrainScript) // Predrain script if err := r.ScriptRunner.PreDrain(scriptTarget); err != nil { - r.DrainManager.DrainErrors <- errors.Errorf("PreDrain failed: instanceID - %v, %v", instanceID, err.Error()) + drainManager.DrainErrors <- errors.Errorf("PreDrain failed: instanceID - %v, %v", instanceID, err.Error()) } // Issue drain concurrently - set lastDrainTime @@ -185,7 +200,7 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol //rollingUpgrade.Status.NodeStep(rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationDrain) if err := r.Auth.DrainNode(&node, time.Duration(rollingUpgrade.PostDrainDelaySeconds()), rollingUpgrade.DrainTimeout(), r.Auth.Kubernetes); err != nil { - r.DrainManager.DrainErrors <- errors.Errorf("DrainNode failed: instanceID - %v, %v", instanceID, err.Error()) + drainManager.DrainErrors <- errors.Errorf("DrainNode failed: instanceID - %v, %v", instanceID, err.Error()) } } @@ -194,7 +209,7 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol // post drain script if err := r.ScriptRunner.PostDrain(scriptTarget); err != nil { - r.DrainManager.DrainErrors <- errors.Errorf("PostDrain failed: instanceID - %v, %v", instanceID, err.Error()) + drainManager.DrainErrors <- errors.Errorf("PostDrain failed: instanceID - %v, %v", instanceID, err.Error()) } // Turns onto NodeRotationPostWait @@ -202,7 +217,7 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol // Post Wait Script if err := r.ScriptRunner.PostWait(scriptTarget); err != nil { - r.DrainManager.DrainErrors <- errors.Errorf("PostWait failed: instanceID - %v, %v", instanceID, err.Error()) + drainManager.DrainErrors <- errors.Errorf("PostWait failed: instanceID - %v, %v", instanceID, err.Error()) } }() } @@ -211,11 +226,11 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol timeout := make(chan struct{}) go func() { defer close(timeout) - r.DrainManager.DrainGroup.Wait() + drainManager.DrainGroup.Wait() }() select { - case err := <-r.DrainManager.DrainErrors: + case err := <-drainManager.DrainErrors: r.Error(err, "failed to rotate the node", "name", rollingUpgrade.NamespacedName()) return false, err From 71b310a41292f16ac9e1dc696f5e969bfc88206c Mon Sep 17 00:00:00 2001 From: Sheldon Shao Date: Fri, 26 Mar 2021 10:41:27 -0700 Subject: [PATCH 20/74] Refine metrics implementation to support goroutines (#196) * Refine the metrics status Signed-off-by: xshao * Refine the metrics status Signed-off-by: xshao * Fix test case error Signed-off-by: xshao * Use group instead of ASG Signed-off-by: xshao --- api/v1alpha1/rollingupgrade_types.go | 114 ++++++--- api/v1alpha1/rollingupgrade_types_test.go | 44 ++-- api/v1alpha1/zz_generated.deepcopy.go | 34 +-- ...grademgr.keikoproj.io_rollingupgrades.yaml | 23 +- controllers/common/metrics.go | 12 +- controllers/common/metrics_test.go | 8 +- controllers/upgrade.go | 43 ++-- go.mod | 1 + go.sum | 226 ++++++++++++++++++ 9 files changed, 395 insertions(+), 110 deletions(-) diff --git a/api/v1alpha1/rollingupgrade_types.go b/api/v1alpha1/rollingupgrade_types.go index bc3447a8..42d48a3d 100644 --- a/api/v1alpha1/rollingupgrade_types.go +++ b/api/v1alpha1/rollingupgrade_types.go @@ -54,8 +54,8 @@ type RollingUpgradeStatus struct { LastNodeTerminationTime metav1.Time `json:"lastTerminationTime,omitempty"` LastNodeDrainTime metav1.Time `json:"lastDrainTime,omitempty"` - Statistics []*RollingUpgradeStatistics `json:"statistics,omitempty"` - InProcessingNodes map[string]*NodeInProcessing `json:"inProcessingNodes,omitempty"` + Statistics []*RollingUpgradeStatistics `json:"statistics,omitempty"` + LastBatchNodes []string `json:"lastBatchNodes,omitempty"` } // RollingUpgrade Statistics, includes summary(sum/count) from each step @@ -65,6 +65,14 @@ type RollingUpgradeStatistics struct { DurationCount int32 `json:"durationCount,omitempty"` } +// RollingUpgrade Node step information +type NodeStepDuration struct { + GroupName string `json:"groupName,omitempty"` + NodeName string `json:"nodeName,omitempty"` + StepName RollingUpgradeStep `json:"stepName,omitempty"` + Duration metav1.Duration `json:"duration,omitempty"` +} + // Node In-processing type NodeInProcessing struct { NodeName string `json:"nodeName,omitempty"` @@ -74,66 +82,108 @@ type NodeInProcessing struct { StepEndTime metav1.Time `json:"stepEndTime,omitempty"` } +// Update last batch nodes +func (s *RollingUpgradeStatus) UpdateLastBatchNodes(batchNodes map[string]*NodeInProcessing) { + keys := make([]string, 0, len(batchNodes)) + for k := range batchNodes { + keys = append(keys, k) + } + s.LastBatchNodes = keys +} + +// Update Node Statistics +func (s *RollingUpgradeStatus) UpdateStatistics(nodeSteps map[string][]NodeStepDuration) { + for _, v := range nodeSteps { + for _, step := range v { + s.AddNodeStepDuration(step) + } + } +} + // Add one step duration -func (s *RollingUpgradeStatus) addStepDuration(asgName string, stepName RollingUpgradeStep, duration time.Duration) { +func (s *RollingUpgradeStatus) ToStepDuration(groupName, nodeName string, stepName RollingUpgradeStep, duration time.Duration) NodeStepDuration { + //Add to system level statistics + common.AddStepDuration(groupName, string(stepName), duration) + return NodeStepDuration{ + GroupName: groupName, + NodeName: nodeName, + StepName: stepName, + Duration: metav1.Duration{ + Duration: duration, + }, + } +} + +// Add one step duration +func (s *RollingUpgradeStatus) AddNodeStepDuration(nsd NodeStepDuration) { // if step exists, add count and sum, otherwise append for _, s := range s.Statistics { - if s.StepName == stepName { + if s.StepName == nsd.StepName { s.DurationSum = metav1.Duration{ - Duration: s.DurationSum.Duration + duration, + Duration: s.DurationSum.Duration + nsd.Duration.Duration, } s.DurationCount += 1 return } } s.Statistics = append(s.Statistics, &RollingUpgradeStatistics{ - StepName: stepName, + StepName: nsd.StepName, DurationSum: metav1.Duration{ - Duration: duration, + Duration: nsd.Duration.Duration, }, DurationCount: 1, }) - - //Add to system level statistics - common.AddRollingUpgradeStepDuration(asgName, string(stepName), duration) } // Node turns onto step -func (s *RollingUpgradeStatus) NodeStep(asgName string, nodeName string, stepName RollingUpgradeStep) { - if s.InProcessingNodes == nil { - s.InProcessingNodes = make(map[string]*NodeInProcessing) - } +func (s *RollingUpgradeStatus) NodeStep(InProcessingNodes map[string]*NodeInProcessing, + nodeSteps map[string][]NodeStepDuration, groupName, nodeName string, stepName RollingUpgradeStep) { + var inProcessingNode *NodeInProcessing - if n, ok := s.InProcessingNodes[nodeName]; !ok { + if n, ok := InProcessingNodes[nodeName]; !ok { inProcessingNode = &NodeInProcessing{ NodeName: nodeName, StepName: stepName, UpgradeStartTime: metav1.Now(), StepStartTime: metav1.Now(), } - s.InProcessingNodes[nodeName] = inProcessingNode + InProcessingNodes[nodeName] = inProcessingNode } else { inProcessingNode = n - n.StepEndTime = metav1.Now() - var duration = n.StepEndTime.Sub(n.StepStartTime.Time) - if stepName == NodeRotationCompleted { - //Add overall and remove the node from in-processing map - var total = n.StepEndTime.Sub(n.UpgradeStartTime.Time) - s.addStepDuration(asgName, inProcessingNode.StepName, duration) - s.addStepDuration(asgName, NodeRotationTotal, total) - delete(s.InProcessingNodes, nodeName) - } else if inProcessingNode.StepName != stepName { //Still same step - var oldOrder = NodeRotationStepOrders[inProcessingNode.StepName] - var newOrder = NodeRotationStepOrders[stepName] - if newOrder > oldOrder { //Make sure the steps running in order - s.addStepDuration(asgName, inProcessingNode.StepName, duration) - n.StepStartTime = metav1.Now() - inProcessingNode.StepName = stepName - } + } + + inProcessingNode.StepEndTime = metav1.Now() + var duration = inProcessingNode.StepEndTime.Sub(inProcessingNode.StepStartTime.Time) + if stepName == NodeRotationCompleted { + //Add overall and remove the node from in-processing map + var total = inProcessingNode.StepEndTime.Sub(inProcessingNode.UpgradeStartTime.Time) + duration1 := s.ToStepDuration(groupName, nodeName, inProcessingNode.StepName, duration) + duration2 := s.ToStepDuration(groupName, nodeName, NodeRotationTotal, total) + s.addNodeStepDuration(nodeSteps, nodeName, duration1) + s.addNodeStepDuration(nodeSteps, nodeName, duration2) + } else if inProcessingNode.StepName != stepName { //Still same step + var oldOrder = NodeRotationStepOrders[inProcessingNode.StepName] + var newOrder = NodeRotationStepOrders[stepName] + if newOrder > oldOrder { //Make sure the steps running in order + stepDuration := s.ToStepDuration(groupName, nodeName, inProcessingNode.StepName, duration) + inProcessingNode.StepStartTime = metav1.Now() + inProcessingNode.StepName = stepName + s.addNodeStepDuration(nodeSteps, nodeName, stepDuration) } } } +func (s *RollingUpgradeStatus) addNodeStepDuration(steps map[string][]NodeStepDuration, nodeName string, nsd NodeStepDuration) { + if stepDuration, ok := steps[nodeName]; !ok { + steps[nodeName] = []NodeStepDuration{ + nsd, + } + } else { + stepDuration = append(stepDuration, nsd) + steps[nodeName] = stepDuration + } +} + func (s *RollingUpgradeStatus) SetCondition(cond RollingUpgradeCondition) { // if condition exists, overwrite, otherwise append for ix, c := range s.Conditions { diff --git a/api/v1alpha1/rollingupgrade_types_test.go b/api/v1alpha1/rollingupgrade_types_test.go index 5e752341..292136f5 100644 --- a/api/v1alpha1/rollingupgrade_types_test.go +++ b/api/v1alpha1/rollingupgrade_types_test.go @@ -10,38 +10,40 @@ func TestNodeTurnsOntoStep(t *testing.T) { g := gomega.NewGomegaWithT(t) r := &RollingUpgradeStatus{} + //A map to retain the steps for multiple nodes + nodeSteps := make(map[string][]NodeStepDuration) + inProcessingNodes := make(map[string]*NodeInProcessing) - r.NodeStep("test-asg", "node-1", NodeRotationKickoff) + r.NodeStep(inProcessingNodes, nodeSteps, "test-asg", "node-1", NodeRotationKickoff) - g.Expect(r.InProcessingNodes).NotTo(gomega.BeNil()) - g.Expect(r.Statistics).To(gomega.BeNil()) + g.Expect(inProcessingNodes).NotTo(gomega.BeNil()) + g.Expect(nodeSteps["node-1"]).To(gomega.BeNil()) - r.NodeStep("test-asg", "node-1", NodeRotationDesiredNodeReady) + r.NodeStep(inProcessingNodes, nodeSteps, "test-asg", "node-1", NodeRotationDesiredNodeReady) - g.Expect(r.Statistics).NotTo(gomega.BeNil()) - g.Expect(len(r.Statistics)).To(gomega.Equal(1)) - g.Expect(r.Statistics[0].StepName).To(gomega.Equal(NodeRotationKickoff)) + g.Expect(len(nodeSteps["node-1"])).To(gomega.Equal(1)) + g.Expect(nodeSteps["node-1"][0].StepName).To(gomega.Equal(NodeRotationKickoff)) //Retry desired_node_ready - r.NodeStep("test-asg", "node-1", NodeRotationDesiredNodeReady) - g.Expect(len(r.Statistics)).To(gomega.Equal(1)) - g.Expect(r.Statistics[0].StepName).To(gomega.Equal(NodeRotationKickoff)) + r.NodeStep(inProcessingNodes, nodeSteps, "test-asg", "node-1", NodeRotationDesiredNodeReady) + g.Expect(len(nodeSteps["node-1"])).To(gomega.Equal(1)) + g.Expect(nodeSteps["node-1"][0].StepName).To(gomega.Equal(NodeRotationKickoff)) //Retry desired_node_ready again - r.NodeStep("test-asg", "node-1", NodeRotationDesiredNodeReady) - g.Expect(len(r.Statistics)).To(gomega.Equal(1)) - g.Expect(r.Statistics[0].StepName).To(gomega.Equal(NodeRotationKickoff)) + r.NodeStep(inProcessingNodes, nodeSteps, "test-asg", "node-1", NodeRotationDesiredNodeReady) + g.Expect(len(nodeSteps["node-1"])).To(gomega.Equal(1)) + g.Expect(nodeSteps["node-1"][0].StepName).To(gomega.Equal(NodeRotationKickoff)) //Completed - r.NodeStep("test-asg", "node-1", NodeRotationCompleted) - g.Expect(len(r.Statistics)).To(gomega.Equal(3)) - g.Expect(r.Statistics[1].StepName).To(gomega.Equal(NodeRotationDesiredNodeReady)) - g.Expect(r.Statistics[2].StepName).To(gomega.Equal(NodeRotationTotal)) + r.NodeStep(inProcessingNodes, nodeSteps, "test-asg", "node-1", NodeRotationCompleted) + g.Expect(len(nodeSteps["node-1"])).To(gomega.Equal(3)) + g.Expect(nodeSteps["node-1"][1].StepName).To(gomega.Equal(NodeRotationDesiredNodeReady)) + g.Expect(nodeSteps["node-1"][2].StepName).To(gomega.Equal(NodeRotationTotal)) //Second node - r.NodeStep("test-asg", "node-2", NodeRotationKickoff) - g.Expect(len(r.Statistics)).To(gomega.Equal(3)) + r.NodeStep(inProcessingNodes, nodeSteps, "test-asg", "node-2", NodeRotationKickoff) + g.Expect(len(nodeSteps["node-1"])).To(gomega.Equal(3)) - r.NodeStep("test-asg", "node-2", NodeRotationDesiredNodeReady) - g.Expect(len(r.Statistics)).To(gomega.Equal(3)) + r.NodeStep(inProcessingNodes, nodeSteps, "test-asg", "node-2", NodeRotationDesiredNodeReady) + g.Expect(len(nodeSteps["node-1"])).To(gomega.Equal(3)) } diff --git a/api/v1alpha1/zz_generated.deepcopy.go b/api/v1alpha1/zz_generated.deepcopy.go index 1c76fdc4..16b85e40 100644 --- a/api/v1alpha1/zz_generated.deepcopy.go +++ b/api/v1alpha1/zz_generated.deepcopy.go @@ -64,6 +64,22 @@ func (in *NodeReadinessGate) DeepCopy() *NodeReadinessGate { return out } +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *NodeStepDuration) DeepCopyInto(out *NodeStepDuration) { + *out = *in + out.Duration = in.Duration +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new NodeStepDuration. +func (in *NodeStepDuration) DeepCopy() *NodeStepDuration { + if in == nil { + return nil + } + out := new(NodeStepDuration) + in.DeepCopyInto(out) + return out +} + // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *PostDrainSpec) DeepCopyInto(out *PostDrainSpec) { *out = *in @@ -246,20 +262,10 @@ func (in *RollingUpgradeStatus) DeepCopyInto(out *RollingUpgradeStatus) { } } } - if in.InProcessingNodes != nil { - in, out := &in.InProcessingNodes, &out.InProcessingNodes - *out = make(map[string]*NodeInProcessing, len(*in)) - for key, val := range *in { - var outVal *NodeInProcessing - if val == nil { - (*out)[key] = nil - } else { - in, out := &val, &outVal - *out = new(NodeInProcessing) - (*in).DeepCopyInto(*out) - } - (*out)[key] = outVal - } + if in.LastBatchNodes != nil { + in, out := &in.LastBatchNodes, &out.LastBatchNodes + *out = make([]string, len(*in)) + copy(*out, *in) } } diff --git a/config/crd/bases/upgrademgr.keikoproj.io_rollingupgrades.yaml b/config/crd/bases/upgrademgr.keikoproj.io_rollingupgrades.yaml index 615ffff6..0bc33f02 100644 --- a/config/crd/bases/upgrademgr.keikoproj.io_rollingupgrades.yaml +++ b/config/crd/bases/upgrademgr.keikoproj.io_rollingupgrades.yaml @@ -133,25 +133,10 @@ spec: type: string endTime: type: string - inProcessingNodes: - additionalProperties: - description: Node In-processing - properties: - nodeName: - type: string - stepEndTime: - format: date-time - type: string - stepName: - type: string - stepStartTime: - format: date-time - type: string - upgradeStartTime: - format: date-time - type: string - type: object - type: object + lastBatchNodes: + items: + type: string + type: array lastDrainTime: format: date-time type: string diff --git a/controllers/common/metrics.go b/controllers/common/metrics.go index 649429dc..723457dc 100644 --- a/controllers/common/metrics.go +++ b/controllers/common/metrics.go @@ -36,14 +36,14 @@ func InitMetrics() { } // Add rolling update step duration when the step is completed -func AddRollingUpgradeStepDuration(asgName string, stepName string, duration time.Duration) { +func AddStepDuration(groupName string, stepName string, duration time.Duration) { if strings.EqualFold(stepName, "total") { //Histogram nodeRotationTotal.Observe(duration.Seconds()) } else { //Summary var steps map[string]prometheus.Summary - if m, ok := stepSummaries[asgName]; !ok { + if m, ok := stepSummaries[groupName]; !ok { steps = make(map[string]prometheus.Summary) - stepSummaries[asgName] = steps + stepSummaries[groupName] = steps } else { steps = m } @@ -55,14 +55,14 @@ func AddRollingUpgradeStepDuration(asgName string, stepName string, duration tim Namespace: "node", Name: stepName + "_seconds", Help: "Summary for node " + stepName, - ConstLabels: prometheus.Labels{"asg": asgName}, + ConstLabels: prometheus.Labels{"group": groupName}, }) err := metrics.Registry.Register(summary) if err != nil { if reflect.TypeOf(err).String() == "prometheus.AlreadyRegisteredError" { - log.Warnf("summary was registered again, ASG: %s, step: %s", asgName, stepName) + log.Warnf("summary was registered again, group: %s, step: %s", groupName, stepName) } else { - log.Errorf("register summary error, ASG: %s, step: %s, %v", asgName, stepName, err) + log.Errorf("register summary error, group: %s, step: %s, %v", groupName, stepName, err) } } steps[stepName] = summary diff --git a/controllers/common/metrics_test.go b/controllers/common/metrics_test.go index 296ae336..9437beb6 100644 --- a/controllers/common/metrics_test.go +++ b/controllers/common/metrics_test.go @@ -9,21 +9,21 @@ func TestAddRollingUpgradeStepDuration(t *testing.T) { g := gomega.NewGomegaWithT(t) g.Expect(stepSummaries["test-asg"]).To(gomega.BeNil()) - AddRollingUpgradeStepDuration("test-asg", "kickoff", 1) + AddStepDuration("test-asg", "kickoff", 1) g.Expect(stepSummaries["test-asg"]).NotTo(gomega.BeNil()) g.Expect(stepSummaries["test-asg"]["kickoff"]).NotTo(gomega.BeNil()) //Test duplicate - AddRollingUpgradeStepDuration("test-asg", "kickoff", 1) + AddStepDuration("test-asg", "kickoff", 1) g.Expect(stepSummaries["test-asg"]["kickoff"]).NotTo(gomega.BeNil()) //Test duplicate delete(stepSummaries["test-asg"], "kickoff") - AddRollingUpgradeStepDuration("test-asg", "kickoff", 1) + AddStepDuration("test-asg", "kickoff", 1) g.Expect(stepSummaries["test-asg"]["kickoff"]).NotTo(gomega.BeNil()) //Test total - AddRollingUpgradeStepDuration("test-asg", "total", 1) + AddStepDuration("test-asg", "total", 1) g.Expect(stepSummaries["test-asg"]["kickoff"]).NotTo(gomega.BeNil()) } diff --git a/controllers/upgrade.go b/controllers/upgrade.go index 47737fcd..7e713784 100644 --- a/controllers/upgrade.go +++ b/controllers/upgrade.go @@ -111,16 +111,21 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol DrainGroup: drainGroup.(*sync.WaitGroup), } + //A map to retain the steps for multiple nodes + nodeSteps := make(map[string][]v1alpha1.NodeStepDuration) + + inProcessingNodes := make(map[string]*v1alpha1.NodeInProcessing) + switch mode { case v1alpha1.UpdateStrategyModeEager: for _, target := range batch { var ( instanceID = aws.StringValue(target.InstanceId) - //node = kubeprovider.SelectNodeByInstanceID(instanceID, r.Cloud.ClusterNodes) - //nodeName = node.GetName() + node = kubeprovider.SelectNodeByInstanceID(instanceID, r.Cloud.ClusterNodes) + nodeName = node.GetName() ) //Add statistics - //rollingUpgrade.Status.NodeStep(rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationKickoff) + rollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationKickoff) // Add in-progress tag if err := r.Auth.TagEC2instance(instanceID, instanceStateTagKey, inProgressTagValue); err != nil { @@ -139,7 +144,7 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol } // Turns onto desired nodes - //rollingUpgrade.Status.NodeStep(rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationDesiredNodeReady) + rollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationDesiredNodeReady) // Wait for desired nodes r.Info("waiting for desired nodes", "name", rollingUpgrade.NamespacedName()) @@ -153,11 +158,11 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol for _, target := range batch { var ( instanceID = aws.StringValue(target.InstanceId) - //node = kubeprovider.SelectNodeByInstanceID(instanceID, r.Cloud.ClusterNodes) - //nodeName = node.GetName() + node = kubeprovider.SelectNodeByInstanceID(instanceID, r.Cloud.ClusterNodes) + nodeName = node.GetName() ) //Add statistics - //rollingUpgrade.Status.NodeStep(rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationKickoff) + rollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationKickoff) // Add in-progress tag if err := r.Auth.TagEC2instance(instanceID, instanceStateTagKey, inProgressTagValue); err != nil { @@ -185,7 +190,7 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol defer drainManager.DrainGroup.Done() // Turns onto PreDrain script - //rollingUpgrade.Status.NodeStep(rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationPredrainScript) + rollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationPredrainScript) // Predrain script if err := r.ScriptRunner.PreDrain(scriptTarget); err != nil { @@ -197,7 +202,7 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol r.Info("draining the node", "instance", instanceID, "node name", node.Name, "name", rollingUpgrade.NamespacedName()) // Turns onto NodeRotationDrain - //rollingUpgrade.Status.NodeStep(rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationDrain) + rollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationDrain) if err := r.Auth.DrainNode(&node, time.Duration(rollingUpgrade.PostDrainDelaySeconds()), rollingUpgrade.DrainTimeout(), r.Auth.Kubernetes); err != nil { drainManager.DrainErrors <- errors.Errorf("DrainNode failed: instanceID - %v, %v", instanceID, err.Error()) @@ -205,7 +210,7 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol } // Turns onto NodeRotationPostdrainScript - //rollingUpgrade.Status.NodeStep(rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationPostdrainScript) + rollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationPostdrainScript) // post drain script if err := r.ScriptRunner.PostDrain(scriptTarget); err != nil { @@ -213,7 +218,7 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol } // Turns onto NodeRotationPostWait - //rollingUpgrade.Status.NodeStep(rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationPostWait) + rollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationPostWait) // Post Wait Script if err := r.ScriptRunner.PostWait(scriptTarget); err != nil { @@ -231,6 +236,9 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol select { case err := <-drainManager.DrainErrors: + rollingUpgrade.Status.UpdateStatistics(nodeSteps) + rollingUpgrade.Status.UpdateLastBatchNodes(inProcessingNodes) + r.Error(err, "failed to rotate the node", "name", rollingUpgrade.NamespacedName()) return false, err @@ -251,7 +259,7 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol ) // Turns onto NodeRotationTerminate - //rollingUpgrade.Status.NodeStep(rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationTerminate) + rollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationTerminate) // Terminate - set lastTerminateTime r.Info("terminating instance", "instance", instanceID, "name", rollingUpgrade.NamespacedName()) @@ -264,7 +272,7 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol rollingUpgrade.SetLastNodeTerminationTime(metav1.Time{Time: time.Now()}) // Turns onto NodeRotationTerminate - //rollingUpgrade.Status.NodeStep(rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationPostTerminate) + rollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationPostTerminate) // Post Terminate Script if err := r.ScriptRunner.PostTerminate(scriptTarget); err != nil { @@ -272,11 +280,18 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol } // Turns onto NodeRotationCompleted - //rollingUpgrade.Status.NodeStep(rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationCompleted) + rollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationCompleted) } + rollingUpgrade.Status.UpdateStatistics(nodeSteps) + rollingUpgrade.Status.UpdateLastBatchNodes(inProcessingNodes) + case <-time.After(DefaultWaitGroupTimeout): // goroutines timed out - requeue + + rollingUpgrade.Status.UpdateStatistics(nodeSteps) + rollingUpgrade.Status.UpdateLastBatchNodes(inProcessingNodes) + r.Info("instances are still draining", "name", rollingUpgrade.NamespacedName()) return true, nil } diff --git a/go.mod b/go.mod index b444654f..be0f01f1 100644 --- a/go.mod +++ b/go.mod @@ -11,6 +11,7 @@ require ( github.com/prometheus/client_golang v1.7.1 github.com/sirupsen/logrus v1.6.0 go.uber.org/zap v1.15.0 + golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d // indirect k8s.io/api v0.20.4 k8s.io/apimachinery v0.20.4 k8s.io/client-go v0.20.4 diff --git a/go.sum b/go.sum index 9b81a34f..36cb2d64 100644 --- a/go.sum +++ b/go.sum @@ -5,40 +5,73 @@ cloud.google.com/go v0.44.1/go.mod h1:iSa0KzasP4Uvy3f1mN/7PiObzGgflwredwwASm/v6A cloud.google.com/go v0.44.2/go.mod h1:60680Gw3Yr4ikxnPRS/oxxkBccT6SA1yMk63TGekxKY= cloud.google.com/go v0.45.1/go.mod h1:RpBamKRgapWJb87xiFSdk4g1CME7QZg3uwTez+TSTjc= cloud.google.com/go v0.46.3/go.mod h1:a6bKKbmY7er1mI7TEI4lsAkts/mkhTSZK8w33B4RAg0= +cloud.google.com/go v0.50.0/go.mod h1:r9sluTvynVuxRIOHXQEHMFffphuXHOMZMycpNR5e6To= cloud.google.com/go v0.51.0 h1:PvKAVQWCtlGUSlZkGW3QLelKaWq7KYv/MW1EboG8bfM= cloud.google.com/go v0.51.0/go.mod h1:hWtGJ6gnXH+KgDv+V0zFGDvpi07n3z8ZNj3T1RW0Gcw= +cloud.google.com/go v0.52.0/go.mod h1:pXajvRH/6o3+F9jDHZWQ5PbGhn+o8w9qiu/CffaVdO4= +cloud.google.com/go v0.53.0/go.mod h1:fp/UouUEsRkN6ryDKNW/Upv/JBKnv6WDthjR6+vze6M= +cloud.google.com/go v0.54.0 h1:3ithwDMr7/3vpAMXiH+ZQnYbuIsh+OPhUPMFC9enmn0= +cloud.google.com/go v0.54.0/go.mod h1:1rq2OEkV3YMf6n/9ZvGWI3GWw0VoqH/1x2nd8Is/bPc= cloud.google.com/go/bigquery v1.0.1/go.mod h1:i/xbL2UlR5RvWAURpBYZTtm/cXjCha9lbfbpx4poX+o= +cloud.google.com/go/bigquery v1.3.0/go.mod h1:PjpwJnslEMmckchkHFfq+HTD2DmtT67aNFKH1/VBDHE= +cloud.google.com/go/bigquery v1.4.0/go.mod h1:S8dzgnTigyfTmLBfrtrhyYhwRxG72rYxvftPBK2Dvzc= cloud.google.com/go/datastore v1.0.0/go.mod h1:LXYbyblFSglQ5pkeyhO+Qmw7ukd3C+pD7TKLgZqpHYE= +cloud.google.com/go/datastore v1.1.0/go.mod h1:umbIZjpQpHh4hmRpGhH4tLFup+FVzqBi1b3c64qFpCk= +cloud.google.com/go/firestore v1.1.0/go.mod h1:ulACoGHTpvq5r8rxGJ4ddJZBZqakUQqClKRT5SZwBmk= cloud.google.com/go/pubsub v1.0.1/go.mod h1:R0Gpsv3s54REJCy4fxDixWD93lHJMoZTyQ2kNxGRt3I= +cloud.google.com/go/pubsub v1.1.0/go.mod h1:EwwdRX2sKPjnvnqCa270oGRyludottCI76h+R3AArQw= +cloud.google.com/go/pubsub v1.2.0/go.mod h1:jhfEVHT8odbXTkndysNHCcx0awwzvfOlguIAii9o8iA= cloud.google.com/go/storage v1.0.0/go.mod h1:IhtSnM/ZTZV8YYJWCY8RULGVqBDmpoyjwiyrjsg+URw= +cloud.google.com/go/storage v1.5.0/go.mod h1:tpKbwo567HUNpVclU5sGELwQWBDZ8gh0ZeosJ0Rtdos= +cloud.google.com/go/storage v1.6.0/go.mod h1:N7U0C8pVQ/+NIKOBQyamJIeKQKkZ+mxpohlUTyfDhBk= dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU= +github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78 h1:w+iIsaOQNcT7OZ575w+acHgRric5iCyQh+xv+KJ4HB8= github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78/go.mod h1:LmzpDX56iTiv29bbRTIsUNlaFfuhWRQBWjQdVyAevI8= +github.com/Azure/go-autorest v14.2.0+incompatible h1:V5VMDjClD3GiElqLWO7mz2MxNAK/vTfRHdAubSIPRgs= +github.com/Azure/go-autorest v14.2.0+incompatible/go.mod h1:r+4oMnoxhatjLLJ6zxSWATqVooLgysK6ZNox3g/xq24= github.com/Azure/go-autorest/autorest v0.9.0/go.mod h1:xyHB1BMZT0cuDHU7I0+g046+BFDTQ8rEZB0s4Yfa6bI= github.com/Azure/go-autorest/autorest v0.9.6 h1:5YWtOnckcudzIw8lPPBcWOnmIFWMtHci1ZWAZulMSx0= github.com/Azure/go-autorest/autorest v0.9.6/go.mod h1:/FALq9T/kS7b5J5qsQ+RSTUdAmGFqi0vUdVNNx8q630= +github.com/Azure/go-autorest/autorest v0.11.1 h1:eVvIXUKiTgv++6YnWb42DUA1YL7qDugnKP0HljexdnQ= +github.com/Azure/go-autorest/autorest v0.11.1/go.mod h1:JFgpikqFJ/MleTTxwepExTKnFUKKszPS8UavbQYUMuw= github.com/Azure/go-autorest/autorest/adal v0.5.0/go.mod h1:8Z9fGy2MpX0PvDjB1pEgQTmVqjGhiHBW7RJJEciWzS0= github.com/Azure/go-autorest/autorest/adal v0.8.2 h1:O1X4oexUxnZCaEUGsvMnr8ZGj8HI37tNezwY4npRqA0= github.com/Azure/go-autorest/autorest/adal v0.8.2/go.mod h1:ZjhuQClTqx435SRJ2iMlOxPYt3d2C/T/7TiQCVZSn3Q= +github.com/Azure/go-autorest/autorest/adal v0.9.0/go.mod h1:/c022QCutn2P7uY+/oQWWNcK9YU+MH96NgK+jErpbcg= +github.com/Azure/go-autorest/autorest/adal v0.9.5 h1:Y3bBUV4rTuxenJJs41HU3qmqsb+auo+a3Lz+PlJPpL0= +github.com/Azure/go-autorest/autorest/adal v0.9.5/go.mod h1:B7KF7jKIeC9Mct5spmyCB/A8CG/sEz1vwIRGv/bbw7A= github.com/Azure/go-autorest/autorest/date v0.1.0/go.mod h1:plvfp3oPSKwf2DNjlBjWF/7vwR+cUD/ELuzDCXwHUVA= github.com/Azure/go-autorest/autorest/date v0.2.0 h1:yW+Zlqf26583pE43KhfnhFcdmSWlm5Ew6bxipnr/tbM= github.com/Azure/go-autorest/autorest/date v0.2.0/go.mod h1:vcORJHLJEh643/Ioh9+vPmf1Ij9AEBM5FuBIXLmIy0g= +github.com/Azure/go-autorest/autorest/date v0.3.0 h1:7gUk1U5M/CQbp9WoqinNzJar+8KY+LPI6wiWrP/myHw= +github.com/Azure/go-autorest/autorest/date v0.3.0/go.mod h1:BI0uouVdmngYNUzGWeSYnokU+TrmwEsOqdt8Y6sso74= github.com/Azure/go-autorest/autorest/mocks v0.1.0/go.mod h1:OTyCOPRA2IgIlWxVYxBee2F5Gr4kF2zd2J5cFRaIDN0= github.com/Azure/go-autorest/autorest/mocks v0.2.0/go.mod h1:OTyCOPRA2IgIlWxVYxBee2F5Gr4kF2zd2J5cFRaIDN0= github.com/Azure/go-autorest/autorest/mocks v0.3.0 h1:qJumjCaCudz+OcqE9/XtEPfvtOjOmKaui4EOpFI6zZc= github.com/Azure/go-autorest/autorest/mocks v0.3.0/go.mod h1:a8FDP3DYzQ4RYfVAxAN3SVSiiO77gL2j2ronKKP0syM= +github.com/Azure/go-autorest/autorest/mocks v0.4.0/go.mod h1:LTp+uSrOhSkaKrUy935gNZuuIPPVsHlr9DSOxSayd+k= +github.com/Azure/go-autorest/autorest/mocks v0.4.1/go.mod h1:LTp+uSrOhSkaKrUy935gNZuuIPPVsHlr9DSOxSayd+k= github.com/Azure/go-autorest/logger v0.1.0 h1:ruG4BSDXONFRrZZJ2GUXDiUyVpayPmb1GnWeHDdaNKY= github.com/Azure/go-autorest/logger v0.1.0/go.mod h1:oExouG+K6PryycPJfVSxi/koC6LSNgds39diKLz7Vrc= +github.com/Azure/go-autorest/logger v0.2.0 h1:e4RVHVZKC5p6UANLJHkM4OfR1UKZPj8Wt8Pcx+3oqrE= +github.com/Azure/go-autorest/logger v0.2.0/go.mod h1:T9E3cAhj2VqvPOtCYAvby9aBXkZmbF5NWuPV8+WeEW8= github.com/Azure/go-autorest/tracing v0.5.0 h1:TRn4WjSnkcSy5AEG3pnbtFSwNtwzjr4VYyQflFE619k= github.com/Azure/go-autorest/tracing v0.5.0/go.mod h1:r/s2XiOKccPW3HrqB+W0TQzfbtp2fGCgRFtBroKn4Dk= +github.com/Azure/go-autorest/tracing v0.6.0 h1:TYi4+3m5t6K48TGI9AUdb+IzbnSxvnvUMfuitfgcfuo= +github.com/Azure/go-autorest/tracing v0.6.0/go.mod h1:+vhtPC754Xsa23ID7GlGsrdKBpUA79WCAKPPZVC2DeU= github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ= github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo= +github.com/MakeNowJust/heredoc v0.0.0-20170808103936-bb23615498cd h1:sjQovDkwrZp8u+gxLtPgKGjk5hCxuy2hrRejBTA9xFU= +github.com/MakeNowJust/heredoc v0.0.0-20170808103936-bb23615498cd/go.mod h1:64YHyfSL2R96J44Nlwm39UHepQbyR5q10x7iYa1ks2E= github.com/NYTimes/gziphandler v0.0.0-20170623195520-56545f4a5d46/go.mod h1:3wb06e3pkSAbeQ52E9H9iFoQsEEwGN64994WTCIhntQ= github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU= github.com/PuerkitoBio/purell v1.0.0/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0= github.com/PuerkitoBio/purell v1.1.0/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0= +github.com/PuerkitoBio/purell v1.1.1 h1:WEQqlqaGbrPkxLJWfBwQmfEAE1Z7ONdDLqrN38tNFfI= github.com/PuerkitoBio/purell v1.1.1/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0= github.com/PuerkitoBio/urlesc v0.0.0-20160726150825-5bd2802263f2/go.mod h1:uGdkoq3SwY9Y+13GIhn11/XLaGBb4BfwItxLd5jeuXE= +github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578 h1:d+Bc7a5rLufV/sSk/8dngufqelfh6jnri85riMAaF/M= github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578/go.mod h1:uGdkoq3SwY9Y+13GIhn11/XLaGBb4BfwItxLd5jeuXE= github.com/agnivade/levenshtein v1.0.1/go.mod h1:CURSv5d9Uaml+FovSIICkLbAUZ9S4RqaHDIsdSBg7lM= github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc= @@ -46,7 +79,10 @@ github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuy github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0= github.com/alecthomas/units v0.0.0-20190717042225-c3de453c63f4/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0= github.com/andreyvit/diff v0.0.0-20170406064948-c7f18ee00883/go.mod h1:rCTlJbsFo29Kk6CurOXKm700vrz8f0KW0JNfpkRJY/8= +github.com/armon/circbuf v0.0.0-20150827004946-bbbad097214e/go.mod h1:3U/XgcO3hCbHZ8TKRvWD2dDTCfh9M9ya+I9JpbB7O8o= github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5doyWs3UAsr3K4I6qtAmlQcZDesFNEHPZAzj8= +github.com/armon/go-metrics v0.0.0-20180917152333-f0300d1749da/go.mod h1:Q73ZrmVTwzkszR9V5SSuryQ31EELlFMUz1kKyl939pY= +github.com/armon/go-radix v0.0.0-20180808171621-7fddfc383310/go.mod h1:ufUuZ+zHj4x4TnLV4JWEpy2hxWSpsRywHrMgIH9cCH8= github.com/asaskevich/govalidator v0.0.0-20180720115003-f9ffefc3facf/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY= github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY= github.com/aws/aws-sdk-go v1.35.7/go.mod h1:tlPOdRjfxPBpNIwqDj61rmsnA85v9jc0Ps9+muhnW+k= @@ -57,12 +93,15 @@ github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+Ce github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM= github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw= github.com/bgentry/speakeasy v0.1.0/go.mod h1:+zsyZBPWlz7T6j88CTgSN5bM796AkVf0kBD4zp0CCIs= +github.com/bketelsen/crypt v0.0.3-0.20200106085610-5cbc8cc4026c/go.mod h1:MKsuJmJgSg28kpZDP6UIiPt0e0Oz0kqKNGyRaWEPv84= github.com/blang/semver v3.5.0+incompatible/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnwebNt5EWlYSAyrTnjyyk= +github.com/blang/semver v3.5.1+incompatible/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnwebNt5EWlYSAyrTnjyyk= github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU= github.com/cespare/xxhash v1.1.0 h1:a6HrQnmkObjyL+Gs60czilIUGqrzKutQD6XZog3p+ko= github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc= github.com/cespare/xxhash/v2 v2.1.1 h1:6MnRN8NT7+YBpUIWxHtefFZOKTAPgGjpQSxqLNn0+qY= github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= +github.com/chai2010/gettext-go v0.0.0-20160711120539-c6fed771bfd5/go.mod h1:/iP1qXHoty45bqomnu2LM+VVyAEdWN+vtSHGlQgyxbw= github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI= github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI= github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU= @@ -70,6 +109,7 @@ github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDk github.com/cockroachdb/datadriven v0.0.0-20190809214429-80d97fb3cbaa/go.mod h1:zn76sxSg3SzpJ0PPJaLDCu+Bu0Lg3sKTORVIj19EIF8= github.com/coreos/bbolt v1.3.2/go.mod h1:iRUV2dpdMOn7Bo10OQBFzIJO9kkE559Wcmn+qkEiiKk= github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE= +github.com/coreos/etcd v3.3.13+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE= github.com/coreos/go-oidc v2.1.0+incompatible/go.mod h1:CgnwVTmzoESiwO9qyAFEMiHoZ1nMCKZlZ9V6mm3/LKc= github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk= github.com/coreos/go-semver v0.3.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk= @@ -82,32 +122,45 @@ github.com/creack/pty v1.1.7/go.mod h1:lj5s0c3V2DBrqTV7llrYr5NG6My20zk30Fl46Y7Do github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/daviddengcn/go-colortext v0.0.0-20160507010035-511bcaf42ccd/go.mod h1:dv4zxwHi5C/8AeI+4gX4dCWOIvNi7I6JCSX0HvlKPgE= github.com/dgrijalva/jwt-go v3.2.0+incompatible h1:7qlOGliEKZXTDg6OTjfoBKDXWrumCAMpl/TFQ4/5kLM= github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ= github.com/dgryski/go-sip13 v0.0.0-20181026042036-e10d5fee7954/go.mod h1:vAd38F8PWV+bWy6jNmig1y/TA+kYO4g3RSRF0IAv0no= +github.com/docker/distribution v2.7.1+incompatible/go.mod h1:J2gT2udsDAN96Uj4KfcMRqY0/ypR+oyYUYmja8H+y+w= github.com/docker/go-units v0.3.3/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk= github.com/docker/go-units v0.4.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk= +github.com/docker/spdystream v0.0.0-20160310174837-449fdfce4d96 h1:cenwrSVm+Z7QLSV/BsnenAOcDXdX4cMv4wP0B/5QbPg= github.com/docker/spdystream v0.0.0-20160310174837-449fdfce4d96/go.mod h1:Qh8CwZgvJUkLughtfhJv5dyTYa91l1fOUCrgjqmcifM= github.com/docopt/docopt-go v0.0.0-20180111231733-ee0de3bc6815/go.mod h1:WwZ+bS3ebgob9U8Nd0kOddGdZWjyMGR8Wziv+TBNwSE= github.com/dustin/go-humanize v0.0.0-20171111073723-bb3d318650d4/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk= github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk= github.com/elazarl/goproxy v0.0.0-20180725130230-947c36da3153/go.mod h1:/Zj4wYkgs4iZTTu3o/KG3Itv/qCCa8VVMlb3i9OVuzc= github.com/emicklei/go-restful v0.0.0-20170410110728-ff4f55a20633/go.mod h1:otzb+WCGbkyDHkqmQmT5YD2WR4BBwUdeQoFo8l/7tVs= +github.com/emicklei/go-restful v2.9.5+incompatible h1:spTtZBk5DYEvbxMVutUuTyh1Ao2r4iyvLdACqsl/Ljk= github.com/emicklei/go-restful v2.9.5+incompatible/go.mod h1:otzb+WCGbkyDHkqmQmT5YD2WR4BBwUdeQoFo8l/7tVs= github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4= github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c= github.com/evanphx/json-patch v4.5.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk= github.com/evanphx/json-patch v4.9.0+incompatible h1:kLcOMZeuLAJvL2BPWLMIj5oaZQobrkAqrL+WFZwQses= github.com/evanphx/json-patch v4.9.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk= +github.com/exponent-io/jsonpath v0.0.0-20151013193312-d6023ce2651d h1:105gxyaGwCFad8crR9dcMQWvV9Hvulu6hwUh4tWPJnM= +github.com/exponent-io/jsonpath v0.0.0-20151013193312-d6023ce2651d/go.mod h1:ZZMPRZwes7CROmyNKgQzC3XPs6L/G2EJLHddWejkmf4= +github.com/fatih/camelcase v1.0.0/go.mod h1:yN2Sb0lFhZJUdVvtELVWefmrXpuZESvPmqwoZc+/fpc= github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4= +github.com/form3tech-oss/jwt-go v3.2.2+incompatible h1:TcekIExNqud5crz4xD2pavyTgWiPvpYe4Xau31I0PRk= +github.com/form3tech-oss/jwt-go v3.2.2+incompatible/go.mod h1:pbq4aXjuKjdthFRnoDwaVPLA+WlJuPGy+QneDUgJi2k= github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo= github.com/fsnotify/fsnotify v1.4.9 h1:hsms1Qyu0jgnwNXIxa+/V/PDsU6CfLf6CNO8H7IWoS4= github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ= +github.com/fvbommel/sortorder v1.0.1/go.mod h1:uk88iVf1ovNn1iLfgUVU2F9o5eO30ui720w+kxuqRs0= github.com/ghodss/yaml v0.0.0-20150909031657-73d445a93680/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04= +github.com/ghodss/yaml v1.0.0 h1:wQHKEahhL6wmXdzwWG11gIVCkOv05bNOh+Rxn0yngAk= github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04= github.com/globalsign/mgo v0.0.0-20180905125535-1ca0a4f7cbcb/go.mod h1:xkRDCp4j0OGD1HRkm4kmhM+pmpv3AKq5SU7GMg4oO/Q= github.com/globalsign/mgo v0.0.0-20181015135952-eeefdecb41b8/go.mod h1:xkRDCp4j0OGD1HRkm4kmhM+pmpv3AKq5SU7GMg4oO/Q= +github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU= github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8= +github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8= github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as= github.com/go-kit/kit v0.9.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as= github.com/go-logfmt/logfmt v0.3.0/go.mod h1:Qt1PoO58o5twSAckw1HlFXLmHsOX5/0LbT9GBnD5lWE= @@ -130,11 +183,13 @@ github.com/go-openapi/jsonpointer v0.0.0-20160704185906-46af16f9f7b1/go.mod h1:+ github.com/go-openapi/jsonpointer v0.17.0/go.mod h1:cOnomiV+CVVwFLk0A/MExoFMjwdsUdVpsRhURCKh+3M= github.com/go-openapi/jsonpointer v0.18.0/go.mod h1:cOnomiV+CVVwFLk0A/MExoFMjwdsUdVpsRhURCKh+3M= github.com/go-openapi/jsonpointer v0.19.2/go.mod h1:3akKfEdA7DF1sugOqz1dVQHBcuDBPKZGEoHC/NkiQRg= +github.com/go-openapi/jsonpointer v0.19.3 h1:gihV7YNZK1iK6Tgwwsxo2rJbD1GTbdm72325Bq8FI3w= github.com/go-openapi/jsonpointer v0.19.3/go.mod h1:Pl9vOtqEWErmShwVjC8pYs9cog34VGT37dQOVbmoatg= github.com/go-openapi/jsonreference v0.0.0-20160704190145-13c6e3589ad9/go.mod h1:W3Z9FmVs9qj+KR4zFKmDPGiLdk1D9Rlm7cyMvf57TTg= github.com/go-openapi/jsonreference v0.17.0/go.mod h1:g4xxGn04lDIRh0GJb5QlpE3HfopLOL6uZrK/VgnsK9I= github.com/go-openapi/jsonreference v0.18.0/go.mod h1:g4xxGn04lDIRh0GJb5QlpE3HfopLOL6uZrK/VgnsK9I= github.com/go-openapi/jsonreference v0.19.2/go.mod h1:jMjeRr2HHw6nAVajTXJ4eiUwohSTlpa0o73RUL1owJc= +github.com/go-openapi/jsonreference v0.19.3 h1:5cxNfTy0UVC3X8JL5ymxzyoUZmo8iZb+jeTWn7tUa8o= github.com/go-openapi/jsonreference v0.19.3/go.mod h1:rjx6GuL8TTa9VaixXglHmQmIL98+wF9xc8zWvFonSJ8= github.com/go-openapi/loads v0.17.0/go.mod h1:72tmFy5wsWx89uEVddd0RjRWPZm92WRLhf7AC+0+OOU= github.com/go-openapi/loads v0.18.0/go.mod h1:72tmFy5wsWx89uEVddd0RjRWPZm92WRLhf7AC+0+OOU= @@ -148,6 +203,7 @@ github.com/go-openapi/spec v0.0.0-20160808142527-6aced65f8501/go.mod h1:J8+jY1nA github.com/go-openapi/spec v0.17.0/go.mod h1:XkF/MOi14NmjsfZ8VtAKf8pIlbZzyoTvZsdfssdxcBI= github.com/go-openapi/spec v0.18.0/go.mod h1:XkF/MOi14NmjsfZ8VtAKf8pIlbZzyoTvZsdfssdxcBI= github.com/go-openapi/spec v0.19.2/go.mod h1:sCxk3jxKgioEJikev4fgkNmwS+3kuYdJtcsZsD5zxMY= +github.com/go-openapi/spec v0.19.3 h1:0XRyw8kguri6Yw4SxhsQA/atC88yqrk0+G4YhI2wabc= github.com/go-openapi/spec v0.19.3/go.mod h1:FpwSN1ksY1eteniUU7X0N/BgJ7a4WvBFVA8Lj9mJglo= github.com/go-openapi/strfmt v0.17.0/go.mod h1:P82hnJI0CXkErkXi8IKjPbNBM6lV6+5pLP5l494TcyU= github.com/go-openapi/strfmt v0.18.0/go.mod h1:P82hnJI0CXkErkXi8IKjPbNBM6lV6+5pLP5l494TcyU= @@ -157,6 +213,7 @@ github.com/go-openapi/swag v0.0.0-20160704191624-1d0bd113de87/go.mod h1:DXUve3Dp github.com/go-openapi/swag v0.17.0/go.mod h1:AByQ+nYG6gQg71GINrmuDXCPWdL640yX49/kXLo40Tg= github.com/go-openapi/swag v0.18.0/go.mod h1:AByQ+nYG6gQg71GINrmuDXCPWdL640yX49/kXLo40Tg= github.com/go-openapi/swag v0.19.2/go.mod h1:POnQmlKehdgb5mhVOsnJFsivZCEZ/vjK9gh66Z9tfKk= +github.com/go-openapi/swag v0.19.5 h1:lTz6Ys4CmqqCQmZPBlbQENR1/GucA2bzYTE12Pw4tFY= github.com/go-openapi/swag v0.19.5/go.mod h1:POnQmlKehdgb5mhVOsnJFsivZCEZ/vjK9gh66Z9tfKk= github.com/go-openapi/validate v0.18.0/go.mod h1:Uh4HdOzKt19xGIGm1qHf/ofbX1YQ4Y+MYsct2VUrAJ4= github.com/go-openapi/validate v0.19.2/go.mod h1:1tRCw7m3jtI8eNWEEliiAqUIcBztB2KDnRCRMUi7GTA= @@ -173,13 +230,18 @@ github.com/golang/groupcache v0.0.0-20190129154638-5b532d6fd5ef/go.mod h1:cIg4er github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7 h1:5ZkaAPbicIKTF2I64qf5Fh8Aa83Q/dnOafMYV0OMwjA= github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= +github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e h1:1r7pUrabqp18hOBcwBwiTsbnFeTZHV9eER/QT5JVZxY= +github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= github.com/golang/mock v1.3.1/go.mod h1:sBzyDLLjw3U8JLTeZvSv8jJB+tU5PVekmnlKIyFUx0Y= +github.com/golang/mock v1.4.0/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw= +github.com/golang/mock v1.4.1/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw= github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= github.com/golang/protobuf v1.3.3/go.mod h1:vzj43D7+SQXF/4pzW/hwtAqwc6iTitCiVSaWz5lYuqw= +github.com/golang/protobuf v1.3.4/go.mod h1:vzj43D7+SQXF/4pzW/hwtAqwc6iTitCiVSaWz5lYuqw= github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8= github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA= github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs= @@ -188,12 +250,19 @@ github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvq github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QDs8UjoX8= github.com/golang/protobuf v1.4.2 h1:+Z5KGCizgyZCbGh1KZqA0fcLLkwbsjIzS4aV2v7wJX0= github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI= +github.com/golang/protobuf v1.4.3 h1:JjCZWpVbqXDqFVmTfYWEVTMIYrL/NPdPSCHPJ0T/raM= +github.com/golang/protobuf v1.4.3/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI= +github.com/golangplus/bytes v0.0.0-20160111154220-45c989fe5450/go.mod h1:Bk6SMAONeMXrxql8uvOKuAZSu8aM5RUGv+1C6IJaEho= +github.com/golangplus/fmt v0.0.0-20150411045040-2a5d6d7d2995/go.mod h1:lJgMEyOkYFkPcDKwRXegd+iM6E7matEszMG5HhwytU8= +github.com/golangplus/testing v0.0.0-20180327235837-af21d9c3145e/go.mod h1:0AA//k/eakGydO4jKRoRL2j92ZKSzTgj9tclaCrvXHk= github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= +github.com/google/btree v1.0.0 h1:0udJVsspx3VBr5FwtLhQQtuAsVc79tTq0ocGIPAU6qo= github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M= github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= +github.com/google/go-cmp v0.5.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= github.com/google/go-cmp v0.5.2 h1:X2ev0eStA3AbceY54o37/0PQ/UWqKEiiO2dKL5OPaFM= github.com/google/go-cmp v0.5.2/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= @@ -203,33 +272,58 @@ github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXi github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc= github.com/google/pprof v0.0.0-20190515194954-54271f7e092f/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc= github.com/google/pprof v0.0.0-20191218002539-d4f498aebedc/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM= +github.com/google/pprof v0.0.0-20200212024743-f11f1df84d12/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM= +github.com/google/pprof v0.0.0-20200229191704-1ebb73c60ed3/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM= github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI= github.com/google/uuid v1.0.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/google/uuid v1.1.1 h1:Gkbcsh/GbpXz7lPftLA3P6TYMwjCLYm83jiFQZF/3gY= github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/google/uuid v1.1.2 h1:EVhdT+1Kseyi1/pUmXKaFxYsDNy9RQYkMWRH68J/W7Y= +github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg= github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk= github.com/googleapis/gnostic v0.4.1/go.mod h1:LRhVm6pbyptWbWbuZ38d1eyptfvIytN3ir6b65WBswg= github.com/googleapis/gnostic v0.5.1 h1:A8Yhf6EtqTv9RMsU6MQTyrtV1TjWlR6xU9BsZIwuTCM= github.com/googleapis/gnostic v0.5.1/go.mod h1:6U4PtQXGIEt/Z3h5MAT7FNofLnw9vXk2cUuW7uA/OeU= +github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY= github.com/gorilla/websocket v0.0.0-20170926233335-4201258b820c/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ= github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ= +github.com/gorilla/websocket v1.4.2/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE= +github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7 h1:pdN6V1QBWetyv/0+wjACpqVH+eVULgEjkurDLq3goeM= github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA= github.com/grpc-ecosystem/go-grpc-middleware v1.0.0/go.mod h1:FiyG127CGDf3tlThmgyCl78X/SZQqEOJBCDaAfeWzPs= github.com/grpc-ecosystem/go-grpc-middleware v1.0.1-0.20190118093823-f849b5445de4/go.mod h1:FiyG127CGDf3tlThmgyCl78X/SZQqEOJBCDaAfeWzPs= github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0/go.mod h1:8NvIoxWQoOIhqOTXgfV/d3M/q6VIi02HzZEHgUlZvzk= github.com/grpc-ecosystem/grpc-gateway v1.9.0/go.mod h1:vNeuVxBJEsws4ogUvrchl83t/GYV9WGTSLVdBhOQFDY= github.com/grpc-ecosystem/grpc-gateway v1.9.5/go.mod h1:vNeuVxBJEsws4ogUvrchl83t/GYV9WGTSLVdBhOQFDY= +github.com/hashicorp/consul/api v1.1.0/go.mod h1:VmuI/Lkw1nC05EYQWNKwWGbkg+FbDBtguAZLlVdkD9Q= +github.com/hashicorp/consul/sdk v0.1.1/go.mod h1:VKf9jXwCTEY1QZP2MOLRhb5i/I/ssyNV1vwHyQBF0x8= +github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4= +github.com/hashicorp/go-cleanhttp v0.5.1/go.mod h1:JpRdi6/HCYpAwUzNwuwqhbovhLtngrth3wmdIIUrZ80= +github.com/hashicorp/go-immutable-radix v1.0.0/go.mod h1:0y9vanUI8NX6FsYoO3zeMjhV/C5i9g4Q3DwcSNZ4P60= +github.com/hashicorp/go-msgpack v0.5.3/go.mod h1:ahLV/dePpqEmjfWmKiqvPkv/twdG7iPBM1vqhUKIvfM= +github.com/hashicorp/go-multierror v1.0.0/go.mod h1:dHtQlpGsu+cZNNAkkCN/P3hoUDHhCYQXV3UM06sGGrk= +github.com/hashicorp/go-rootcerts v1.0.0/go.mod h1:K6zTfqpRlCUIjkwsN4Z+hiSfzSTQa6eBIzfwKfwNnHU= +github.com/hashicorp/go-sockaddr v1.0.0/go.mod h1:7Xibr9yA9JjQq1JpNB2Vw7kxv8xerXegt+ozgdvDeDU= +github.com/hashicorp/go-syslog v1.0.0/go.mod h1:qPfqrKkXGihmCqbJM2mZgkZGvKG1dFdvsLplgctolz4= +github.com/hashicorp/go-uuid v1.0.0/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro= +github.com/hashicorp/go-uuid v1.0.1/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro= +github.com/hashicorp/go.net v0.0.1/go.mod h1:hjKkEWcCURg++eb33jQU7oqQcI9XDCnUzHA0oac0k90= github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= github.com/hashicorp/golang-lru v0.5.4 h1:YDjusn29QI/Das2iO9M0BHnIbxPeyuCHsjMW+lJfyTc= github.com/hashicorp/golang-lru v0.5.4/go.mod h1:iADmTwqILo4mZ8BN3D2Q6+9jd8WM5uGBxy+E8yxSoD4= github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ= +github.com/hashicorp/logutils v1.0.0/go.mod h1:QIAnNjmIWmVIIkWDTG1z5v++HQmx9WQRO+LraFDTW64= +github.com/hashicorp/mdns v1.0.0/go.mod h1:tL+uN++7HEJ6SQLQ2/p+z2pH24WQKWjBPkE0mNTz8vQ= +github.com/hashicorp/memberlist v0.1.3/go.mod h1:ajVTdAv/9Im8oMAAj5G31PhhMCZJV2pPBoIllUwCN7I= +github.com/hashicorp/serf v0.8.2/go.mod h1:6hOLApaqBFA1NXqRQAsxw9QxuDEvNxSQRwA/JwenrHc= github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU= github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc= github.com/imdario/mergo v0.3.5/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA= github.com/imdario/mergo v0.3.10 h1:6q5mVkdH/vYmqngx7kZQTjJ5HRsx+ImorDIEQ+beJgc= github.com/imdario/mergo v0.3.10/go.mod h1:jmQim1M+e3UYxmgPu/WyfjB3N3VflVyUjjjwH0dnCYA= +github.com/inconshreveable/mousetrap v1.0.0 h1:Z8tu5sraLXCXIcARxBp/8cbvlwVa7Z1NHg9XEKhtSvM= github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8= github.com/jmespath/go-jmespath v0.4.0 h1:BEgLn5cpjn8UN1mAw4NjwDrS35OdebyEtFe+9YPoQUg= github.com/jmespath/go-jmespath v0.4.0/go.mod h1:T8mJZnbsbmF+m6zOOFylbeCJqk5+pHWvzYPziyZiYoo= @@ -242,6 +336,7 @@ github.com/json-iterator/go v1.1.10 h1:Kz6Cvnvv2wGdaG/V8yMvfkmNiXq9Ya2KUv4rouJJr github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4= github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU= github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk= +github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU= github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w= github.com/karlseguin/ccache v2.0.3+incompatible h1:j68C9tWOROiOLWTS/kCGg9IcJG+ACqn5+0+t8Oh83UU= github.com/karlseguin/ccache v2.0.3+incompatible/go.mod h1:CM9tNPzT6EdRh14+jiW8mEF9mkNZuuE51qmgGYUB93w= @@ -263,21 +358,37 @@ github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= github.com/kr/pty v1.1.5/go.mod h1:9r2w37qlBe7rQ6e1fg1S/9xpWHSnaqNdHD3WcMdbPDA= github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE= github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= +github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de h1:9TO3cAIGXtEhnIaL+V+BEER86oLrvS+kWobKpbJuye0= +github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de/go.mod h1:zAbeS9B/r2mtpb6U+EI2rYA5OAXxsYw6wTamcNW+zcE= +github.com/lithammer/dedent v1.1.0/go.mod h1:jrXYCQtgg0nJiN+StA2KgR7w6CiQNv9Fd/Z9BP0jIOc= github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ= +github.com/magiconair/properties v1.8.1/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ= github.com/mailru/easyjson v0.0.0-20160728113105-d5b7844b561a/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= github.com/mailru/easyjson v0.0.0-20180823135443-60711f1a8329/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= github.com/mailru/easyjson v0.0.0-20190312143242-1de009706dbe/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= github.com/mailru/easyjson v0.0.0-20190614124828-94de47d64c63/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= github.com/mailru/easyjson v0.0.0-20190626092158-b2ccc519800e/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= +github.com/mailru/easyjson v0.7.0 h1:aizVhC/NAAcKWb+5QsU1iNOZb4Yws5UO2I+aIprQITM= github.com/mailru/easyjson v0.7.0/go.mod h1:KAzv3t3aY1NaHWoQz1+4F1ccyAH66Jk7yos7ldAVICs= github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU= +github.com/mattn/go-isatty v0.0.3/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNxMWT7Zi4= github.com/mattn/go-isatty v0.0.4/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNxMWT7Zi4= github.com/mattn/go-runewidth v0.0.2/go.mod h1:LwmH8dsx7+W8Uxz3IHJYH5QSwggIsqBzpuz5H//U1FU= github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0= github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369 h1:I0XW9+e1XWDxdcEniV4rQAIOPUGDq67JSCiRCgGCZLI= github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369/go.mod h1:BSXmuO+STAnVfrANrmjBb36TMTDstsz7MSK+HVaYKv4= +github.com/miekg/dns v1.0.14/go.mod h1:W1PPwlIAgtquWBMBEV9nkV9Cazfe8ScdGz/Lj7v3Nrg= +github.com/mitchellh/cli v1.0.0/go.mod h1:hNIlj7HEI86fIcpObd7a0FcrxTWetlwJDGcceTlRvqc= +github.com/mitchellh/go-homedir v1.0.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0= github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0= +github.com/mitchellh/go-testing-interface v1.0.0/go.mod h1:kRemZodwjscx+RGhAo8eIhFbs2+BFgRtFPeD/KE+zxI= +github.com/mitchellh/go-wordwrap v1.0.0 h1:6GlHJ/LTGMrIJbwgdqdl2eEH8o+Exx/0m8ir9Gns0u4= +github.com/mitchellh/go-wordwrap v1.0.0/go.mod h1:ZXFpozHsX6DPmq2I0TCekCxypsnAUbP2oI0UX1GXzOo= +github.com/mitchellh/gox v0.4.0/go.mod h1:Sd9lOJ0+aimLBi73mGofS1ycjY8lL3uZM3JPS42BGNg= +github.com/mitchellh/iochan v1.0.0/go.mod h1:JwYml1nuB7xOzsp52dPpHFffvOCDupsG0QubkSMEySY= +github.com/mitchellh/mapstructure v0.0.0-20160808181253-ca63d7c062ee/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y= github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y= +github.com/moby/term v0.0.0-20200312100748-672ec06f55cd h1:aY7OQNf2XqY/JQ6qREWamhI/81os/agb2BAGpcx5yWI= github.com/moby/term v0.0.0-20200312100748-672ec06f55cd/go.mod h1:DdlQx2hp0Ss5/fLikoLlEeIYiATotOjgB//nb973jeo= github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg= @@ -305,8 +416,11 @@ github.com/onsi/gomega v1.7.1/go.mod h1:XdKZgCCFLUoM/7CFJVPcG8C1xQ1AJ0vpAezJrB7J github.com/onsi/gomega v1.10.1/go.mod h1:iN09h71vgCQne3DLsj+A5owkum+a2tYe+TOCB1ybHNo= github.com/onsi/gomega v1.10.2 h1:aY/nuoWlKJud2J6U0E3NWsjlg+0GtwXxgEqthRdzlcs= github.com/onsi/gomega v1.10.2/go.mod h1:iN09h71vgCQne3DLsj+A5owkum+a2tYe+TOCB1ybHNo= +github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM= +github.com/pascaldekloe/goe v0.0.0-20180627143212-57f6aae5913c/go.mod h1:lzWF7FIEvWOWxwDKqyGYQf6ZUaNfKdP144TG7ZOy1lc= github.com/pborman/uuid v1.2.0/go.mod h1:X/NO0urCmaxf9VXbdlT7C2Yzkj2IKimNn4k+gtPdI/k= github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic= +github.com/peterbourgon/diskv v2.0.1+incompatible h1:UBdAOUP5p4RWqPBg048CAvpKN+vxiaj6gdUUzhl4XmI= github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU= github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= @@ -314,6 +428,7 @@ github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/posener/complete v1.1.1/go.mod h1:em0nMJCgc9GFtwrmVmEMR/ZL6WyhyjMBndrE9hABlRI= github.com/pquerna/cachecontrol v0.0.0-20171018203845-0dec1b30a021/go.mod h1:prYjPmNq4d1NPVmpShWobRqXY3q7Vp+80DqgxxUrUIA= github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw= github.com/prometheus/client_golang v0.9.3/go.mod h1:/TN21ttK/J9q6uSwhBd54HahCDft0ttaMvbicHlPoso= @@ -337,16 +452,24 @@ github.com/prometheus/procfs v0.0.2/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsT github.com/prometheus/procfs v0.0.8/go.mod h1:7Qr8sr6344vo1JqZ6HhLceV9o3AJ1Ff+GxbHq6oeK9A= github.com/prometheus/procfs v0.1.3 h1:F0+tqvhOksq22sc6iCHF5WGlWjdwj92p0udFh1VFBS8= github.com/prometheus/procfs v0.1.3/go.mod h1:lV6e/gmhEcM9IjHGsFOCxxuZ+z1YqCvr4OA4YeYWdaU= +github.com/prometheus/procfs v0.2.0 h1:wH4vA7pcjKuZzjF7lM8awk4fnuJO6idemZXoKnULUx4= +github.com/prometheus/procfs v0.2.0/go.mod h1:lV6e/gmhEcM9IjHGsFOCxxuZ+z1YqCvr4OA4YeYWdaU= github.com/prometheus/tsdb v0.7.1/go.mod h1:qhTCs0VvXwvX/y3TZrWD7rabWM+ijKTux40TwIPHuXU= github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af/go.mod h1:XWv6SoW27p1b0cqNHllgS5HIMJraePCO15w5zCzIWYg= github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4= +github.com/russross/blackfriday v1.5.2 h1:HyvC0ARfnZBqnXwABFeSZHpKvJHJJfPz81GNueLj0oo= +github.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g= github.com/russross/blackfriday/v2 v2.0.1/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM= +github.com/ryanuber/columnize v0.0.0-20160712163229-9b3edd62028f/go.mod h1:sm1tb6uqfes/u+d4ooFouqFdy9/2g9QGwK3SQygK0Ts= +github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529/go.mod h1:DxrIzT+xaE7yg65j358z/aeFdxmN0P9QXhEzd20vsDc= github.com/sergi/go-diff v1.0.0/go.mod h1:0CfEIISq7TuYL3j771MWULgwwjU+GofnZX9QAmXWZgo= github.com/shurcooL/sanitized_anchor_name v1.0.0/go.mod h1:1NzhyTcUVG4SuEtjjoZeVRXNmyL/1OwPU0+IJeTBvfc= github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo= github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE= github.com/sirupsen/logrus v1.6.0 h1:UBcNElsrwanuuMsnGSlYmtmgbb23qDR5dG+6X6Oo89I= github.com/sirupsen/logrus v1.6.0/go.mod h1:7uNnSEd1DgxDLC74fIahvMZmmYsHGZGEOFrfsX/uA88= +github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc= +github.com/smartystreets/goconvey v1.6.4/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA= github.com/soheilhy/cmux v0.1.4/go.mod h1:IM3LyeVVIOuxMH7sFAkER9+bJ4dT7Ms6E4xg4kGIyLM= github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA= github.com/spf13/afero v1.1.2/go.mod h1:j4pytiNVoe2o6bmDsKpLACNPDBIoEAkihy7loJ1B0CQ= @@ -354,6 +477,8 @@ github.com/spf13/afero v1.2.2/go.mod h1:9ZxEEn6pIJ8Rxe320qSDBk6AsU0r9pR7Q4OcevTd github.com/spf13/cast v1.3.0/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE= github.com/spf13/cobra v0.0.3/go.mod h1:1l0Ry5zgKvJasoi3XT1TypsSe7PqH0Sj9dhYf7v3XqQ= github.com/spf13/cobra v1.0.0/go.mod h1:/6GTrnGXV9HjY+aR4k0oJ5tcvakLuG6EuKReYlHNrgE= +github.com/spf13/cobra v1.1.1 h1:KfztREH0tPxJJ+geloSLaAkaPkr4ki2Er5quFV1TDo4= +github.com/spf13/cobra v1.1.1/go.mod h1:WnodtKOvamDL/PwE2M4iKs8aMDBZ5Q5klgD3qfVJQMI= github.com/spf13/jwalterweatherman v1.0.0/go.mod h1:cQK4TGJAtQXfYWX+Ddv3mKDzgVb68N+wFjFa4jdeBTo= github.com/spf13/pflag v0.0.0-20170130214245-9ff6c6923cff/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4= github.com/spf13/pflag v1.0.1/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4= @@ -361,6 +486,7 @@ github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnIn github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA= github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= github.com/spf13/viper v1.4.0/go.mod h1:PTJ7Z/lr49W6bUbkmS1V3by4uWynFiR9p7+dSq/yZzE= +github.com/spf13/viper v1.7.0/go.mod h1:8WkrPz2fc9jxqZNCJI/76HCieCp4Q8HaLFoCha5qpdg= github.com/stoewer/go-strcase v1.2.0/go.mod h1:IBiWB2sKIp3wVVQ3Y035++gc+knqhUQag1KpM8ahLw8= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= @@ -370,6 +496,8 @@ github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UV github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4= github.com/stretchr/testify v1.5.1 h1:nOGnQDM7FYENwehXlg/kFVnos3rEvtKTjRvOWSzb6H4= github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA= +github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= +github.com/subosito/gotenv v1.2.0/go.mod h1:N0PQaV/YGNqwC0u51sEeR/aUtSLEXKX9iv69rRypqCw= github.com/tidwall/pretty v1.0.0/go.mod h1:XNkn88O1ChpSDQmQeStsy+sBenx6DDtFZJxhVysOjyk= github.com/tmc/grpc-websocket-proxy v0.0.0-20170815181823-89b8d40f7ca8/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U= github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U= @@ -391,6 +519,7 @@ go.mongodb.org/mongo-driver v1.1.2/go.mod h1:u7ryQJ+DOzQmeO7zB6MHyr8jkEQvC8vH7qL go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU= go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8= go.opencensus.io v0.22.2/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw= +go.opencensus.io v0.22.3/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw= go.uber.org/atomic v1.3.2/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE= go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE= go.uber.org/atomic v1.6.0 h1:Ezj3JGmsOnG1MoRWQkPBsKLe9DwWD9QeXzTRzzldNVk= @@ -407,6 +536,7 @@ go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q= go.uber.org/zap v1.15.0 h1:ZZCA22JRF2gQE5FoNmhmrf7jeJJ2uhqDUNRYKm8dvmM= go.uber.org/zap v1.15.0/go.mod h1:Mb2vm2krFEG5DV0W9qcHBYFtp/Wku1cvYaqPsS/WYfc= golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= +golang.org/x/crypto v0.0.0-20181029021203-45a5f77698d3/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= golang.org/x/crypto v0.0.0-20190320223903-b7391e95e576/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= @@ -417,11 +547,18 @@ golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8U golang.org/x/crypto v0.0.0-20191206172530-e9b2fee46413/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9 h1:psW17arqaxU48Z5kZ0CQnkZWQJsqcURM6tKiBApRjXI= golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= +golang.org/x/crypto v0.0.0-20201002170205-7f63de1d35b0 h1:hb9wdF1z5waM+dSIICn1l0DkLVDT3hqhhQsDNUmHPRE= +golang.org/x/crypto v0.0.0-20201002170205-7f63de1d35b0/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8= golang.org/x/exp v0.0.0-20190829153037-c13cbed26979/go.mod h1:86+5VVa7VpoJ4kLfm080zCjGlMRFzhUhsZKEZO7MGek= +golang.org/x/exp v0.0.0-20191030013958-a1ab85dbe136/go.mod h1:JXzH8nQsPlswgeRAPE3MuO9GYsAcnJvJ4vnMwN/5qkY= +golang.org/x/exp v0.0.0-20191129062945-2f5052295587/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4= golang.org/x/exp v0.0.0-20191227195350-da58074b4299/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4= +golang.org/x/exp v0.0.0-20200119233911-0405dc783f0a/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4= +golang.org/x/exp v0.0.0-20200207192155-f17229e696bd/go.mod h1:J/WKrq2StrnmMY6+EHIKF9dgMWnmCNThgcyBT1FY9mM= +golang.org/x/exp v0.0.0-20200224162631-6cc2880d07d6/go.mod h1:3jZMyOhIsHpP37uCMkUooju7aAi5cS1Q23tOzKc+0MU= golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js= golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0= golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= @@ -433,11 +570,14 @@ golang.org/x/lint v0.0.0-20190909230951-414d861bb4ac/go.mod h1:6SW0HCj/g11FgYtHl golang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= golang.org/x/lint v0.0.0-20191125180803-fdd1cda4f05f h1:J5lckAjkw6qYlOZNj90mLYNTEKDvWeuc1yieZ8qUzUE= golang.org/x/lint v0.0.0-20191125180803-fdd1cda4f05f/go.mod h1:5qLYkcX4OjUUV8bRuDixDT3tpyyb+LUpUlRWLxfhWrs= +golang.org/x/lint v0.0.0-20200130185559-910be7a94367/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY= +golang.org/x/lint v0.0.0-20200302205851-738671d3881b/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY= golang.org/x/mobile v0.0.0-20190312151609-d3739f865fa6/go.mod h1:z+o9i4GpDbdi3rU15maQ/Ox0txvL9dWGYEHz965HBQE= golang.org/x/mobile v0.0.0-20190719004257-d2bd2a29d028/go.mod h1:E/iHnbuqvinMTCcRqshq8CkpyQDoeVncDDYHnLhea+o= golang.org/x/mod v0.0.0-20190513183733-4bf6d317e70e/go.mod h1:mXi4GBBbnImb6dmsKGUJ2LatrhH/nqhxcFungHvyanc= golang.org/x/mod v0.1.0/go.mod h1:0QHyrYULN0/3qlju5TqG8bIK38QM8yzMo5ekMj3DlcY= golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg= +golang.org/x/mod v0.1.1-0.20191107180719-034126e5016b/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg= golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.3.0 h1:RM4zey1++hCTbCVQfnWeKs9/IEsaBLA8vTkd0WVtmH4= golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= @@ -445,7 +585,9 @@ golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73r golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20181005035420-146acd28ed58/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20181023162649-9b4f9f5ad519/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20181201002055-351d144fa1fc/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20181220203305-927f97764cc3/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= @@ -458,11 +600,15 @@ golang.org/x/net v0.0.0-20190522155817-f3200d17e092/go.mod h1:HSz+uSET+XFnRR8LxR golang.org/x/net v0.0.0-20190603091049-60506f45cf65/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks= golang.org/x/net v0.0.0-20190613194153-d28f0bde5980/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20190724013045-ca1201d0de80/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20190813141303-74dc4d7220e7/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20190827160401-ba9fcec4b297/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20191209160850-c0dbc17a3553/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20200114155413-6afb5195e5aa/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20200202094626-16171245cfb2/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20200222125558-5a598a2470a0/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20200301022130-244492dfa37a/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20200324143707-d3edc9973b7e/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A= golang.org/x/net v0.0.0-20200520004742-59133d7f0dd7/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A= golang.org/x/net v0.0.0-20200707034311-ab3426394381 h1:VXak5I6aEWmAXeQjA+QSZzlgNrpq9mjcfDemuexIKsU= @@ -474,15 +620,19 @@ golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4Iltr golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6 h1:pE8b58s1HRDMi8RDc79m0HISf9D4TzseP40cEA6IGfs= golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= +golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d h1:TzXSXBo42m9gQenoE3b9BGiEpg5IG2JkU5FkPIawgtw= +golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sys v0.0.0-20180823144017-11551d06cbcc/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20181026203630-95b1ffbd15a5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20181107165924-66b7b1311ac8/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= @@ -495,14 +645,20 @@ golang.org/x/sys v0.0.0-20190507160741-ecd444e8653b/go.mod h1:h1NjWce9XRLGQEsW7w golang.org/x/sys v0.0.0-20190606165138-5da285871e9c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190616124812-15dcb6c0061f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190624142023-c5567b49c5d0/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20190726091711-fc99dfbffb4e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190826190057-c7b8b68b1456/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190904154756-749cb33beabd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20191001151750-bb3f8db39f24/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20191005200804-aed5e4c7ecf9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20191120155948-bd437916bb0e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20191228213918-04cbcbbfeed8/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200106162015-b016eb3dc98e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200113162924-86b910548bc1/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200122134326-e047566fdf82/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200212091648-12a6c2dcc1e4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200302150141-5c8b2ff67527/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200519105757-fe76b779f299/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= @@ -511,11 +667,16 @@ golang.org/x/sys v0.0.0-20200622214017-ed371f2e16b4 h1:5/PjkGUjvEU5Gl6BxmvKRPpqo golang.org/x/sys v0.0.0-20200622214017-ed371f2e16b4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f h1:+Nyd8tzPX9R7BWHguqsrbFdRx3WQ/1ib8I44HXV5yTA= golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20201112073958-5cba982894dd h1:5CtCZbICpIOFdgO940moixOPjc0178IU44m4EjOO5IY= +golang.org/x/sys v0.0.0-20201112073958-5cba982894dd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk= golang.org/x/text v0.3.3 h1:cokOdA+Jmi5PJGXLlLllQSgYigAEfHXJAERHVMaCc2k= golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= +golang.org/x/text v0.3.4 h1:0YWbFKbhXG/wIiuHDSKpS0Iy7FSA+u45VtBMfQcFTTc= +golang.org/x/text v0.3.4/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= @@ -532,6 +693,7 @@ golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3 golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= golang.org/x/tools v0.0.0-20190312151545-0bb0c0a6e846/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= golang.org/x/tools v0.0.0-20190312170243-e65039ee4138/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= +golang.org/x/tools v0.0.0-20190328211700-ab21143f2384/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= golang.org/x/tools v0.0.0-20190425150028-36563e24a262/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= golang.org/x/tools v0.0.0-20190506145303-2d16b83fe98c/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= @@ -547,22 +709,43 @@ golang.org/x/tools v0.0.0-20191012152004-8de300cfc20a/go.mod h1:b+2E5dAYhXwXZwtn golang.org/x/tools v0.0.0-20191029041327-9cc4af7d6b2c/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= golang.org/x/tools v0.0.0-20191029190741-b9c20aec41a5/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= golang.org/x/tools v0.0.0-20191108193012-7d206e10da11/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= +golang.org/x/tools v0.0.0-20191112195655-aa38f8e97acc/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= +golang.org/x/tools v0.0.0-20191113191852-77e3bb0ad9e7/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= +golang.org/x/tools v0.0.0-20191115202509-3a792d9c32b2/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= golang.org/x/tools v0.0.0-20191125144606-a911d9008d1f/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= +golang.org/x/tools v0.0.0-20191130070609-6e064ea0cf2d/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= +golang.org/x/tools v0.0.0-20191216173652-a0e659d51361/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= golang.org/x/tools v0.0.0-20191227053925-7b8e75db28f4/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= +golang.org/x/tools v0.0.0-20200117161641-43d50277825c/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= +golang.org/x/tools v0.0.0-20200122220014-bf1340f18c4a/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= +golang.org/x/tools v0.0.0-20200130002326-2f3ba24bd6e7/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= +golang.org/x/tools v0.0.0-20200204074204-1cc6d1ef6c74/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= +golang.org/x/tools v0.0.0-20200207183749-b753a1ba74fa/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= +golang.org/x/tools v0.0.0-20200212150539-ea181f53ac56/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= +golang.org/x/tools v0.0.0-20200224181240-023911ca70b2/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= +golang.org/x/tools v0.0.0-20200304193943-95d2e580d8eb/go.mod h1:o4KQGtdN14AW+yjsvvwRTJJuXz8XRtIHtEnmAXLyFUw= +golang.org/x/tools v0.0.0-20200505023115-26f46d2f7ef8/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= golang.org/x/tools v0.0.0-20200616133436-c1934b75d054 h1:HHeAlu5H9b71C+Fx0K+1dGgVFN1DM1/wz4aoGOA5qS8= golang.org/x/tools v0.0.0-20200616133436-c1934b75d054/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4= golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE= +golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= gomodules.xyz/jsonpatch/v2 v2.1.0 h1:Phva6wqu+xR//Njw6iorylFFgn/z547tw5Ne3HZPQ+k= gomodules.xyz/jsonpatch/v2 v2.1.0/go.mod h1:IhYNNY4jnS53ZnfE4PAmpKtDpTCj1JFXc+3mwe7XcUU= google.golang.org/api v0.4.0/go.mod h1:8k5glujaEP+g9n7WNsDg8QP6cUVNI86fCNMcbazEtwE= google.golang.org/api v0.7.0/go.mod h1:WtwebWUNSVBH/HAw79HIFXZNqEvBhG+Ra+ax0hx3E3M= google.golang.org/api v0.8.0/go.mod h1:o4eAsZoiT+ibD93RtjEohWalFOjRDx6CVaqeizhEnKg= google.golang.org/api v0.9.0/go.mod h1:o4eAsZoiT+ibD93RtjEohWalFOjRDx6CVaqeizhEnKg= +google.golang.org/api v0.13.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI= +google.golang.org/api v0.14.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI= google.golang.org/api v0.15.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI= +google.golang.org/api v0.17.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE= +google.golang.org/api v0.18.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE= +google.golang.org/api v0.20.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE= google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= @@ -578,7 +761,16 @@ google.golang.org/genproto v0.0.0-20190502173448-54afdca5d873/go.mod h1:VzzqZJRn google.golang.org/genproto v0.0.0-20190801165951-fa694d86fc64/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc= google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc= google.golang.org/genproto v0.0.0-20190911173649-1774047e7e51/go.mod h1:IbNlFCBrqXvoKpeg0TB2l7cyZUmoaFKYIwrEpbDKLA8= +google.golang.org/genproto v0.0.0-20191108220845-16a3f7862a1a/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc= +google.golang.org/genproto v0.0.0-20191115194625-c23dd37a84c9/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc= +google.golang.org/genproto v0.0.0-20191216164720-4f79533eabd1/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc= google.golang.org/genproto v0.0.0-20191230161307-f3c370f40bfb/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc= +google.golang.org/genproto v0.0.0-20200115191322-ca5a22157cba/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc= +google.golang.org/genproto v0.0.0-20200122232147-0452cf42e150/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc= +google.golang.org/genproto v0.0.0-20200204135345-fa8e72b47b90/go.mod h1:GmwEX6Z4W5gMy59cAlVYjN9JhxgbQH6Gn+gFDQe2lzA= +google.golang.org/genproto v0.0.0-20200212174721-66ed5ce911ce/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c= +google.golang.org/genproto v0.0.0-20200224152610-e50cd9704f63/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c= +google.golang.org/genproto v0.0.0-20200305110556-506484158171/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c= google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo= google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c= google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38= @@ -587,6 +779,7 @@ google.golang.org/grpc v1.21.1/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ij google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg= google.golang.org/grpc v1.26.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk= google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk= +google.golang.org/grpc v1.27.1/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk= google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8= google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0= google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM= @@ -597,6 +790,8 @@ google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2 google.golang.org/protobuf v1.23.1-0.20200526195155-81db48ad09cc/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= google.golang.org/protobuf v1.24.0 h1:UhZDfRO8JRQru4/+LlLE0BRKGF8L+PICnvYZmx/fEGA= google.golang.org/protobuf v1.24.0/go.mod h1:r/3tXBNzIEhYS9I1OUVjXDlt8tc493IdKGjtUeSXeh4= +google.golang.org/protobuf v1.25.0 h1:Ejskq+SyPohKW+1uil0JJMtmHCgJPJ/qWTxr8qp+R4c= +google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlbajtzgsN7c= gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= @@ -607,6 +802,7 @@ gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI= gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys= gopkg.in/inf.v0 v0.9.1 h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc= gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw= +gopkg.in/ini.v1 v1.51.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k= gopkg.in/karlseguin/expect.v1 v1.0.1 h1:9u0iUltnhFbJTHaSIH0EP+cuTU5rafIgmcsEsg2JQFw= gopkg.in/karlseguin/expect.v1 v1.0.1/go.mod h1:uB7QIJBcclvYbwlUDkSCsGjAOMis3fP280LyhuDEf2I= gopkg.in/natefinch/lumberjack.v2 v2.0.0/go.mod h1:l0ndWWf7gzL7RNwBG7wST/UCcT4T24xpD6X8LsfU/+k= @@ -622,6 +818,7 @@ gopkg.in/yaml.v2 v2.2.5/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= gopkg.in/yaml.v2 v2.3.0 h1:clyUAQHOM3G0M3f5vQj7LuJrETvjVot3Z5el9nffUtU= gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= +gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.0-20200615113413-eeeca48fe776 h1:tQIYjPdBoyREyB9XMu+nnTclpTYkz2zFM+lzLJFO4gQ= gopkg.in/yaml.v3 v3.0.0-20200615113413-eeeca48fe776/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gotest.tools v2.2.0+incompatible/go.mod h1:DsYFclhRJ6vuDpmuTbkuFWG+y2sxOXAzmJt81HFBacw= @@ -632,34 +829,63 @@ honnef.co/go/tools v0.0.0-20190418001031-e561f6794a2a/go.mod h1:rf3lG4BRIbNafJWh honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= honnef.co/go/tools v0.0.1-2019.2.3 h1:3JgtbtFHMiCmsznwGVTUWbgGov+pVqnlf1dEJTNAXeM= honnef.co/go/tools v0.0.1-2019.2.3/go.mod h1:a3bituU0lyd329TUQxRnasdCoJDkEUEAqEt0JzvZhAg= +honnef.co/go/tools v0.0.1-2020.1.3/go.mod h1:X/FiERA/W4tHapMX5mGpAtMSVEeEUOyHaw9vFzvIQ3k= k8s.io/api v0.19.2 h1:q+/krnHWKsL7OBZg/rxnycsl9569Pud76UJ77MvKXms= k8s.io/api v0.19.2/go.mod h1:IQpK0zFQ1xc5iNIQPqzgoOwuFugaYHK4iCknlAQP9nI= +k8s.io/api v0.20.4 h1:xZjKidCirayzX6tHONRQyTNDVIR55TYVqgATqo6ZULY= +k8s.io/api v0.20.4/go.mod h1:++lNL1AJMkDymriNniQsWRkMDzRaX2Y/POTUi8yvqYQ= k8s.io/apiextensions-apiserver v0.19.2 h1:oG84UwiDsVDu7dlsGQs5GySmQHCzMhknfhFExJMz9tA= k8s.io/apiextensions-apiserver v0.19.2/go.mod h1:EYNjpqIAvNZe+svXVx9j4uBaVhTB4C94HkY3w058qcg= k8s.io/apimachinery v0.19.2 h1:5Gy9vQpAGTKHPVOh5c4plE274X8D/6cuEiTO2zve7tc= k8s.io/apimachinery v0.19.2/go.mod h1:DnPGDnARWFvYa3pMHgSxtbZb7gpzzAZ1pTfaUNDVlmA= +k8s.io/apimachinery v0.20.4 h1:vhxQ0PPUUU2Ns1b9r4/UFp13UPs8cw2iOoTjnY9faa0= +k8s.io/apimachinery v0.20.4/go.mod h1:WlLqWAHZGg07AeltaI0MV5uk1Omp8xaN0JGLY6gkRpU= k8s.io/apiserver v0.19.2/go.mod h1:FreAq0bJ2vtZFj9Ago/X0oNGC51GfubKK/ViOKfVAOA= +k8s.io/cli-runtime v0.20.4 h1:jVU13lBeebHLtarHeHkoIi3uRONFzccmP7hHLzEoQ4w= +k8s.io/cli-runtime v0.20.4/go.mod h1:dz38e1CM4uuIhy8PMFUZv7qsvIdoE3ByZYlmbHNCkt4= k8s.io/client-go v0.19.2 h1:gMJuU3xJZs86L1oQ99R4EViAADUPMHHtS9jFshasHSc= k8s.io/client-go v0.19.2/go.mod h1:S5wPhCqyDNAlzM9CnEdgTGV4OqhsW3jGO1UM1epwfJA= +k8s.io/client-go v0.20.4 h1:85crgh1IotNkLpKYKZHVNI1JT86nr/iDCvq2iWKsql4= +k8s.io/client-go v0.20.4/go.mod h1:LiMv25ND1gLUdBeYxBIwKpkSC5IsozMMmOOeSJboP+k= k8s.io/code-generator v0.19.2/go.mod h1:moqLn7w0t9cMs4+5CQyxnfA/HV8MF6aAVENF+WZZhgk= +k8s.io/code-generator v0.20.4/go.mod h1:UsqdF+VX4PU2g46NC2JRs4gc+IfrctnwHb76RNbWHJg= k8s.io/component-base v0.19.2 h1:jW5Y9RcZTb79liEhW3XDVTW7MuvEGP0tQZnfSX6/+gs= k8s.io/component-base v0.19.2/go.mod h1:g5LrsiTiabMLZ40AR6Hl45f088DevyGY+cCE2agEIVo= +k8s.io/component-base v0.20.4 h1:gdvPs4G11e99meQnW4zN+oYOjH8qkLz1sURrAzvKWqc= +k8s.io/component-base v0.20.4/go.mod h1:t4p9EdiagbVCJKrQ1RsA5/V4rFQNDfRlevJajlGwgjI= +k8s.io/component-helpers v0.20.4/go.mod h1:S7jGg8zQp3kwvSzfuGtNaQAMVmvzomXDioTm5vABn9g= k8s.io/gengo v0.0.0-20200413195148-3a45101e95ac/go.mod h1:ezvh/TsK7cY6rbqRK0oQQ8IAqLxYwwyPxAX1Pzy0ii0= k8s.io/gengo v0.0.0-20200428234225-8167cfdcfc14/go.mod h1:ezvh/TsK7cY6rbqRK0oQQ8IAqLxYwwyPxAX1Pzy0ii0= +k8s.io/gengo v0.0.0-20201113003025-83324d819ded/go.mod h1:FiNAH4ZV3gBg2Kwh89tzAEV2be7d5xI0vBa/VySYy3E= k8s.io/klog/v2 v2.0.0/go.mod h1:PBfzABfn139FHAV07az/IF9Wp1bkk3vpT2XSJ76fSDE= k8s.io/klog/v2 v2.2.0 h1:XRvcwJozkgZ1UQJmfMGpvRthQHOvihEhYtDfAaxMz/A= k8s.io/klog/v2 v2.2.0/go.mod h1:Od+F08eJP+W3HUb4pSrPpgp9DGU4GzlpG/TmITuYh/Y= +k8s.io/klog/v2 v2.4.0 h1:7+X0fUguPyrKEC4WjH8iGDg3laWgMo5tMnRTIGTTxGQ= +k8s.io/klog/v2 v2.4.0/go.mod h1:Od+F08eJP+W3HUb4pSrPpgp9DGU4GzlpG/TmITuYh/Y= k8s.io/kube-openapi v0.0.0-20200805222855-6aeccd4b50c6 h1:+WnxoVtG8TMiudHBSEtrVL1egv36TkkJm+bA8AxicmQ= k8s.io/kube-openapi v0.0.0-20200805222855-6aeccd4b50c6/go.mod h1:UuqjUnNftUyPE5H64/qeyjQoUZhGpeFDVdxjTeEVN2o= +k8s.io/kube-openapi v0.0.0-20201113171705-d219536bb9fd h1:sOHNzJIkytDF6qadMNKhhDRpc6ODik8lVC6nOur7B2c= +k8s.io/kube-openapi v0.0.0-20201113171705-d219536bb9fd/go.mod h1:WOJ3KddDSol4tAGcJo0Tvi+dK12EcqSLqcWsryKMpfM= +k8s.io/kubectl v0.20.4 h1:Y1gUiigiZM+ulcrnWeqSHlTd0/7xWcQIXjuMnjtHyoo= +k8s.io/kubectl v0.20.4/go.mod h1:yCC5lUQyXRmmtwyxfaakryh9ezzp/bT0O14LeoFLbGo= +k8s.io/metrics v0.20.4/go.mod h1:DDXS+Ls+2NAxRcVhXKghRPa3csljyJRjDRjPe6EOg/g= k8s.io/utils v0.0.0-20200729134348-d5654de09c73/go.mod h1:jPW/WVKK9YHAvNhRxK0md/EJ228hCsBRufyofKtW8HA= k8s.io/utils v0.0.0-20200912215256-4140de9c8800 h1:9ZNvfPvVIEsp/T1ez4GQuzCcCTEQWhovSofhqR73A6g= k8s.io/utils v0.0.0-20200912215256-4140de9c8800/go.mod h1:jPW/WVKK9YHAvNhRxK0md/EJ228hCsBRufyofKtW8HA= +k8s.io/utils v0.0.0-20201110183641-67b214c5f920 h1:CbnUZsM497iRC5QMVkHwyl8s2tB3g7yaSHkYPkpgelw= +k8s.io/utils v0.0.0-20201110183641-67b214c5f920/go.mod h1:jPW/WVKK9YHAvNhRxK0md/EJ228hCsBRufyofKtW8HA= rsc.io/binaryregexp v0.2.0/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8= +rsc.io/quote/v3 v3.1.0/go.mod h1:yEA65RcK8LyAZtP9Kv3t0HmxON59tX3rD+tICJqUlj0= +rsc.io/sampler v1.3.0/go.mod h1:T1hPZKmBbMNahiBKFy5HrXp6adAjACjK9JXDnKaTXpA= sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.0.9/go.mod h1:dzAXnQbTRyDlZPJX2SUPEqvnB+j7AJjtlox7PEwigU0= sigs.k8s.io/controller-runtime v0.7.0 h1:bU20IBBEPccWz5+zXpLnpVsgBYxqclaHu1pVDl/gEt8= sigs.k8s.io/controller-runtime v0.7.0/go.mod h1:pJ3YBrJiAqMAZKi6UVGuE98ZrroV1p+pIhoHsMm9wdU= +sigs.k8s.io/kustomize v2.0.3+incompatible h1:JUufWFNlI44MdtnjUqVnvh29rR37PQFzPbLXqhyOyX0= +sigs.k8s.io/kustomize v2.0.3+incompatible/go.mod h1:MkjgH3RdOWrievjo6c9T245dYlB5QeXV4WCbnt/PEpU= sigs.k8s.io/structured-merge-diff/v4 v4.0.1 h1:YXTMot5Qz/X1iBRJhAt+vI+HVttY0WkSqqhKxQ0xVbA= sigs.k8s.io/structured-merge-diff/v4 v4.0.1/go.mod h1:bJZC9H9iH24zzfZ/41RGcq60oK1F7G282QMXDPYydCw= +sigs.k8s.io/structured-merge-diff/v4 v4.0.2 h1:YHQV7Dajm86OuqnIR6zAelnDWBRjo+YhYV9PmGrh1s8= +sigs.k8s.io/structured-merge-diff/v4 v4.0.2/go.mod h1:bJZC9H9iH24zzfZ/41RGcq60oK1F7G282QMXDPYydCw= sigs.k8s.io/yaml v1.1.0/go.mod h1:UJmg0vDUVViEyp3mgSv9WPwZCDxu4rQW1olrI1uml+o= sigs.k8s.io/yaml v1.2.0 h1:kr/MCeFWJWTwyaHoR9c8EjH9OumOmoF9YGiZd7lFm/Q= sigs.k8s.io/yaml v1.2.0/go.mod h1:yfXDCHCao9+ENCvLSE62v9VSji2MKu5jeNfTrofGhJc= From 77f985c87b929fa8d4b42d23337d1c0a90685229 Mon Sep 17 00:00:00 2001 From: Sheldon Shao Date: Mon, 29 Mar 2021 17:16:25 -0700 Subject: [PATCH 21/74] Ignore generated code (#201) * Refine the metrics status Signed-off-by: xshao * Refine the metrics status Signed-off-by: xshao * Fix test case error Signed-off-by: xshao * Use group instead of ASG Signed-off-by: xshao * Ignore generated code Signed-off-by: xshao * Ignore generated code Signed-off-by: xshao --- .gitignore | 1 + api/v1alpha1/zz_generated.deepcopy.go | 296 -------------------------- 2 files changed, 1 insertion(+), 296 deletions(-) delete mode 100644 api/v1alpha1/zz_generated.deepcopy.go diff --git a/.gitignore b/.gitignore index 4ff38461..c055d501 100644 --- a/.gitignore +++ b/.gitignore @@ -16,6 +16,7 @@ bin # Kubernetes Generated files - skip generated files, except for vendored files !vendor/**/zz_generated.* +!api/**/zz_generated.* # editor and IDE paraphernalia .idea diff --git a/api/v1alpha1/zz_generated.deepcopy.go b/api/v1alpha1/zz_generated.deepcopy.go deleted file mode 100644 index 16b85e40..00000000 --- a/api/v1alpha1/zz_generated.deepcopy.go +++ /dev/null @@ -1,296 +0,0 @@ -// +build !ignore_autogenerated - -/* -Copyright 2021 Intuit Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -// Code generated by controller-gen. DO NOT EDIT. - -package v1alpha1 - -import ( - runtime "k8s.io/apimachinery/pkg/runtime" -) - -// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. -func (in *NodeInProcessing) DeepCopyInto(out *NodeInProcessing) { - *out = *in - in.UpgradeStartTime.DeepCopyInto(&out.UpgradeStartTime) - in.StepStartTime.DeepCopyInto(&out.StepStartTime) - in.StepEndTime.DeepCopyInto(&out.StepEndTime) -} - -// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new NodeInProcessing. -func (in *NodeInProcessing) DeepCopy() *NodeInProcessing { - if in == nil { - return nil - } - out := new(NodeInProcessing) - in.DeepCopyInto(out) - return out -} - -// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. -func (in *NodeReadinessGate) DeepCopyInto(out *NodeReadinessGate) { - *out = *in - if in.MatchLabels != nil { - in, out := &in.MatchLabels, &out.MatchLabels - *out = make(map[string]string, len(*in)) - for key, val := range *in { - (*out)[key] = val - } - } -} - -// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new NodeReadinessGate. -func (in *NodeReadinessGate) DeepCopy() *NodeReadinessGate { - if in == nil { - return nil - } - out := new(NodeReadinessGate) - in.DeepCopyInto(out) - return out -} - -// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. -func (in *NodeStepDuration) DeepCopyInto(out *NodeStepDuration) { - *out = *in - out.Duration = in.Duration -} - -// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new NodeStepDuration. -func (in *NodeStepDuration) DeepCopy() *NodeStepDuration { - if in == nil { - return nil - } - out := new(NodeStepDuration) - in.DeepCopyInto(out) - return out -} - -// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. -func (in *PostDrainSpec) DeepCopyInto(out *PostDrainSpec) { - *out = *in -} - -// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new PostDrainSpec. -func (in *PostDrainSpec) DeepCopy() *PostDrainSpec { - if in == nil { - return nil - } - out := new(PostDrainSpec) - in.DeepCopyInto(out) - return out -} - -// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. -func (in *PostTerminateSpec) DeepCopyInto(out *PostTerminateSpec) { - *out = *in -} - -// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new PostTerminateSpec. -func (in *PostTerminateSpec) DeepCopy() *PostTerminateSpec { - if in == nil { - return nil - } - out := new(PostTerminateSpec) - in.DeepCopyInto(out) - return out -} - -// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. -func (in *PreDrainSpec) DeepCopyInto(out *PreDrainSpec) { - *out = *in -} - -// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new PreDrainSpec. -func (in *PreDrainSpec) DeepCopy() *PreDrainSpec { - if in == nil { - return nil - } - out := new(PreDrainSpec) - in.DeepCopyInto(out) - return out -} - -// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. -func (in *RollingUpgrade) DeepCopyInto(out *RollingUpgrade) { - *out = *in - out.TypeMeta = in.TypeMeta - in.ObjectMeta.DeepCopyInto(&out.ObjectMeta) - in.Spec.DeepCopyInto(&out.Spec) - in.Status.DeepCopyInto(&out.Status) -} - -// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RollingUpgrade. -func (in *RollingUpgrade) DeepCopy() *RollingUpgrade { - if in == nil { - return nil - } - out := new(RollingUpgrade) - in.DeepCopyInto(out) - return out -} - -// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. -func (in *RollingUpgrade) DeepCopyObject() runtime.Object { - if c := in.DeepCopy(); c != nil { - return c - } - return nil -} - -// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. -func (in *RollingUpgradeCondition) DeepCopyInto(out *RollingUpgradeCondition) { - *out = *in -} - -// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RollingUpgradeCondition. -func (in *RollingUpgradeCondition) DeepCopy() *RollingUpgradeCondition { - if in == nil { - return nil - } - out := new(RollingUpgradeCondition) - in.DeepCopyInto(out) - return out -} - -// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. -func (in *RollingUpgradeList) DeepCopyInto(out *RollingUpgradeList) { - *out = *in - out.TypeMeta = in.TypeMeta - in.ListMeta.DeepCopyInto(&out.ListMeta) - if in.Items != nil { - in, out := &in.Items, &out.Items - *out = make([]RollingUpgrade, len(*in)) - for i := range *in { - (*in)[i].DeepCopyInto(&(*out)[i]) - } - } -} - -// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RollingUpgradeList. -func (in *RollingUpgradeList) DeepCopy() *RollingUpgradeList { - if in == nil { - return nil - } - out := new(RollingUpgradeList) - in.DeepCopyInto(out) - return out -} - -// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. -func (in *RollingUpgradeList) DeepCopyObject() runtime.Object { - if c := in.DeepCopy(); c != nil { - return c - } - return nil -} - -// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. -func (in *RollingUpgradeSpec) DeepCopyInto(out *RollingUpgradeSpec) { - *out = *in - out.PreDrain = in.PreDrain - out.PostDrain = in.PostDrain - out.PostTerminate = in.PostTerminate - out.Strategy = in.Strategy - if in.ReadinessGates != nil { - in, out := &in.ReadinessGates, &out.ReadinessGates - *out = make([]NodeReadinessGate, len(*in)) - for i := range *in { - (*in)[i].DeepCopyInto(&(*out)[i]) - } - } -} - -// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RollingUpgradeSpec. -func (in *RollingUpgradeSpec) DeepCopy() *RollingUpgradeSpec { - if in == nil { - return nil - } - out := new(RollingUpgradeSpec) - in.DeepCopyInto(out) - return out -} - -// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. -func (in *RollingUpgradeStatistics) DeepCopyInto(out *RollingUpgradeStatistics) { - *out = *in - out.DurationSum = in.DurationSum -} - -// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RollingUpgradeStatistics. -func (in *RollingUpgradeStatistics) DeepCopy() *RollingUpgradeStatistics { - if in == nil { - return nil - } - out := new(RollingUpgradeStatistics) - in.DeepCopyInto(out) - return out -} - -// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. -func (in *RollingUpgradeStatus) DeepCopyInto(out *RollingUpgradeStatus) { - *out = *in - if in.Conditions != nil { - in, out := &in.Conditions, &out.Conditions - *out = make([]RollingUpgradeCondition, len(*in)) - copy(*out, *in) - } - in.LastNodeTerminationTime.DeepCopyInto(&out.LastNodeTerminationTime) - in.LastNodeDrainTime.DeepCopyInto(&out.LastNodeDrainTime) - if in.Statistics != nil { - in, out := &in.Statistics, &out.Statistics - *out = make([]*RollingUpgradeStatistics, len(*in)) - for i := range *in { - if (*in)[i] != nil { - in, out := &(*in)[i], &(*out)[i] - *out = new(RollingUpgradeStatistics) - **out = **in - } - } - } - if in.LastBatchNodes != nil { - in, out := &in.LastBatchNodes, &out.LastBatchNodes - *out = make([]string, len(*in)) - copy(*out, *in) - } -} - -// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RollingUpgradeStatus. -func (in *RollingUpgradeStatus) DeepCopy() *RollingUpgradeStatus { - if in == nil { - return nil - } - out := new(RollingUpgradeStatus) - in.DeepCopyInto(out) - return out -} - -// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. -func (in *UpdateStrategy) DeepCopyInto(out *UpdateStrategy) { - *out = *in - out.MaxUnavailable = in.MaxUnavailable -} - -// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new UpdateStrategy. -func (in *UpdateStrategy) DeepCopy() *UpdateStrategy { - if in == nil { - return nil - } - out := new(UpdateStrategy) - in.DeepCopyInto(out) - return out -} From 665c64b1d8828021abf261a5673da6ab736bd778 Mon Sep 17 00:00:00 2001 From: Shreyas Badiger <7680410+shreyas-badiger@users.noreply.github.com> Date: Mon, 29 Mar 2021 17:26:14 -0700 Subject: [PATCH 22/74] Fix bug in deleting the entry in syncMap (#203) Signed-off-by: sbadiger --- controllers/rollingupgrade_controller.go | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/controllers/rollingupgrade_controller.go b/controllers/rollingupgrade_controller.go index 2c1f3d32..bccc219a 100644 --- a/controllers/rollingupgrade_controller.go +++ b/controllers/rollingupgrade_controller.go @@ -73,7 +73,7 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque err := r.Get(ctx, req.NamespacedName, rollingUpgrade) if err != nil { if kerrors.IsNotFound(err) { - r.AdmissionMap.Delete(req.NamespacedName) + r.AdmissionMap.Delete(rollingUpgrade.NamespacedName()) r.Info("deleted object from admission map", "name", req.NamespacedName) return ctrl.Result{}, nil } @@ -82,7 +82,7 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque // If the resource is being deleted, remove it from the admissionMap if !rollingUpgrade.DeletionTimestamp.IsZero() { - r.AdmissionMap.Delete(req.NamespacedName) + r.AdmissionMap.Delete(rollingUpgrade.NamespacedName()) r.Info("rolling upgrade deleted", "name", req.NamespacedName) return reconcile.Result{}, nil } @@ -90,7 +90,7 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque // Stop processing upgrades which are in finite state currentStatus := rollingUpgrade.CurrentStatus() if common.ContainsEqualFold(v1alpha1.FiniteStates, currentStatus) { - r.AdmissionMap.Delete(req.NamespacedName) + r.AdmissionMap.Delete(rollingUpgrade.NamespacedName()) r.Info("rolling upgrade ended", "name", req.NamespacedName, "status", currentStatus) return reconcile.Result{}, nil } From d5935e3dc9e1cbd03b8fec30da9efffcfe61738b Mon Sep 17 00:00:00 2001 From: Shreyas Badiger <7680410+shreyas-badiger@users.noreply.github.com> Date: Mon, 19 Apr 2021 11:01:42 -0700 Subject: [PATCH 23/74] Unit tests for controller-v2 (#215) * Unit tests Signed-off-by: sbadiger * minor change in accessing the namespace name Signed-off-by: sbadiger * move helper functions to a differnt file Signed-off-by: sbadiger --- controllers/helpers_test.go | 177 +++++++++++ controllers/rollingupgrade_controller.go | 8 +- controllers/upgrade_test.go | 361 +++++++++++++++++++++++ 3 files changed, 542 insertions(+), 4 deletions(-) create mode 100644 controllers/helpers_test.go create mode 100644 controllers/upgrade_test.go diff --git a/controllers/helpers_test.go b/controllers/helpers_test.go new file mode 100644 index 00000000..948e6f91 --- /dev/null +++ b/controllers/helpers_test.go @@ -0,0 +1,177 @@ +package controllers + +import ( + "testing" + + "k8s.io/client-go/kubernetes/fake" + ctrl "sigs.k8s.io/controller-runtime" + + "github.com/keikoproj/upgrade-manager/api/v1alpha1" + awsprovider "github.com/keikoproj/upgrade-manager/controllers/providers/aws" + kubeprovider "github.com/keikoproj/upgrade-manager/controllers/providers/kubernetes" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + + //AWS + "github.com/aws/aws-sdk-go/aws" + "github.com/aws/aws-sdk-go/aws/awserr" + "github.com/aws/aws-sdk-go/service/autoscaling" + "github.com/aws/aws-sdk-go/service/autoscaling/autoscalingiface" + "github.com/aws/aws-sdk-go/service/ec2" + "github.com/aws/aws-sdk-go/service/ec2/ec2iface" +) + +// K8s +func createRollingUpgradeReconciler(t *testing.T) *RollingUpgradeReconciler { + // amazon client + amazonClient := createAmazonClient(t) + + // k8s client (fake client) + kubeClient := &kubeprovider.KubernetesClientSet{ + Kubernetes: fake.NewSimpleClientset(createNodeList()), + } + + // logger + logger := ctrl.Log.WithName("controllers").WithName("RollingUpgrade") + + // authenticator + auth := &RollingUpgradeAuthenticator{ + KubernetesClientSet: kubeClient, + AmazonClientSet: amazonClient, + } + + // reconciler object + reconciler := &RollingUpgradeReconciler{ + Logger: logger, + Auth: auth, + EventWriter: kubeprovider.NewEventWriter(kubeClient, logger), + ScriptRunner: ScriptRunner{ + Logger: logger, + }, + Cloud: NewDiscoveredState(auth, logger), + } + return reconciler + +} + +func createRollingUpgrade() *v1alpha1.RollingUpgrade { + return &v1alpha1.RollingUpgrade{ + ObjectMeta: metav1.ObjectMeta{Name: "0", Namespace: "default"}, + Spec: v1alpha1.RollingUpgradeSpec{ + AsgName: "mock-asg-1", + PostDrainDelaySeconds: 30, + Strategy: v1alpha1.UpdateStrategy{ + Type: v1alpha1.RandomUpdateStrategy, + DrainTimeout: 30, + }, + }, + } +} + +func createNodeList() *corev1.NodeList { + return &corev1.NodeList{ + Items: []corev1.Node{ + corev1.Node{ + ObjectMeta: metav1.ObjectMeta{Name: "mock-node-1"}, + Spec: corev1.NodeSpec{ProviderID: "foo-bar/mock-instance-1"}, + Status: corev1.NodeStatus{ + Conditions: []corev1.NodeCondition{ + {Type: corev1.NodeReady, Status: corev1.ConditionTrue}, + }, + }, + }, + corev1.Node{ + ObjectMeta: metav1.ObjectMeta{Name: "mock-node-2"}, + Spec: corev1.NodeSpec{ProviderID: "foo-bar/mock-instance-2"}, + Status: corev1.NodeStatus{ + Conditions: []corev1.NodeCondition{ + {Type: corev1.NodeReady, Status: corev1.ConditionTrue}, + }, + }, + }, + corev1.Node{ + ObjectMeta: metav1.ObjectMeta{Name: "mock-node-3"}, + Spec: corev1.NodeSpec{ProviderID: "foo-bar/mock-instance-3"}, + Status: corev1.NodeStatus{ + Conditions: []corev1.NodeCondition{ + {Type: corev1.NodeReady, Status: corev1.ConditionTrue}, + }, + }, + }, + }, + } +} + +func createNode() *corev1.Node { + return &corev1.Node{ + ObjectMeta: metav1.ObjectMeta{Name: "mock-node-1", Namespace: "default"}, + Spec: corev1.NodeSpec{ProviderID: "foo-bar/mock-instance-1"}, + Status: corev1.NodeStatus{ + Conditions: []corev1.NodeCondition{ + {Type: corev1.NodeReady, Status: corev1.ConditionTrue}, + }, + }, + } +} + +// AWS +type MockAutoscalingGroup struct { + autoscalingiface.AutoScalingAPI + errorFlag bool + awsErr awserr.Error + errorInstanceId string + autoScalingGroups []*autoscaling.Group +} + +type MockEC2 struct { + ec2iface.EC2API + awsErr awserr.Error + reservations []*ec2.Reservation +} + +func createASGInstance(instanceID string, launchConfigName string) *autoscaling.Instance { + return &autoscaling.Instance{ + InstanceId: &instanceID, + LaunchConfigurationName: &launchConfigName, + AvailabilityZone: aws.String("az-1"), + LifecycleState: aws.String("InService"), + } +} + +func createASG(asgName string, launchConfigName string) *autoscaling.Group { + return &autoscaling.Group{ + AutoScalingGroupName: &asgName, + LaunchConfigurationName: &launchConfigName, + Instances: []*autoscaling.Instance{ + createASGInstance("mock-instance-1", launchConfigName), + createASGInstance("mock-instance-2", launchConfigName), + createASGInstance("mock-instance-3", launchConfigName), + }, + DesiredCapacity: func(x int) *int64 { i := int64(x); return &i }(3), + } +} + +func createASGs() []*autoscaling.Group { + return []*autoscaling.Group{ + createASG("mock-asg-1", "mock-launch-config-1"), + createASG("mock-asg-2", "mock-launch-config-2"), + createASG("mock-asg-3", "mock-launch-config-3"), + } +} + +func createASGClient() *MockAutoscalingGroup { + return &MockAutoscalingGroup{ + autoScalingGroups: createASGs(), + } +} + +func createEc2Client() MockEC2 { + return MockEC2{} +} + +func createAmazonClient(t *testing.T) *awsprovider.AmazonClientSet { + return &awsprovider.AmazonClientSet{ + AsgClient: createASGClient(), + Ec2Client: createEc2Client(), + } +} diff --git a/controllers/rollingupgrade_controller.go b/controllers/rollingupgrade_controller.go index bccc219a..af18c678 100644 --- a/controllers/rollingupgrade_controller.go +++ b/controllers/rollingupgrade_controller.go @@ -73,7 +73,7 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque err := r.Get(ctx, req.NamespacedName, rollingUpgrade) if err != nil { if kerrors.IsNotFound(err) { - r.AdmissionMap.Delete(rollingUpgrade.NamespacedName()) + r.AdmissionMap.Delete(req.NamespacedName) r.Info("deleted object from admission map", "name", req.NamespacedName) return ctrl.Result{}, nil } @@ -83,7 +83,7 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque // If the resource is being deleted, remove it from the admissionMap if !rollingUpgrade.DeletionTimestamp.IsZero() { r.AdmissionMap.Delete(rollingUpgrade.NamespacedName()) - r.Info("rolling upgrade deleted", "name", req.NamespacedName) + r.Info("rolling upgrade deleted", "name", rollingUpgrade.NamespacedName()) return reconcile.Result{}, nil } @@ -91,7 +91,7 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque currentStatus := rollingUpgrade.CurrentStatus() if common.ContainsEqualFold(v1alpha1.FiniteStates, currentStatus) { r.AdmissionMap.Delete(rollingUpgrade.NamespacedName()) - r.Info("rolling upgrade ended", "name", req.NamespacedName, "status", currentStatus) + r.Info("rolling upgrade ended", "name", rollingUpgrade.NamespacedName(), "status", currentStatus) return reconcile.Result{}, nil } @@ -124,7 +124,7 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque return ctrl.Result{RequeueAfter: time.Second * 30}, nil } - r.Info("admitted new rollingupgrade", "name", req.NamespacedName, "scalingGroup", scalingGroupName) + r.Info("admitted new rollingupgrade", "name", rollingUpgrade.NamespacedName(), "scalingGroup", scalingGroupName) r.AdmissionMap.Store(rollingUpgrade.NamespacedName(), scalingGroupName) rollingUpgrade.SetCurrentStatus(v1alpha1.StatusInit) diff --git a/controllers/upgrade_test.go b/controllers/upgrade_test.go new file mode 100644 index 00000000..5bdc5e53 --- /dev/null +++ b/controllers/upgrade_test.go @@ -0,0 +1,361 @@ +package controllers + +import ( + "os" + "testing" + + drain "k8s.io/kubectl/pkg/drain" + + "reflect" + "time" + + "github.com/keikoproj/upgrade-manager/api/v1alpha1" + corev1 "k8s.io/api/core/v1" + + //AWS + "github.com/aws/aws-sdk-go/aws" + "github.com/aws/aws-sdk-go/service/autoscaling" +) + +func TestListClusterNodes(t *testing.T) { + var tests = []struct { + TestDescription string + Reconciler *RollingUpgradeReconciler + RollingUpgrade *v1alpha1.RollingUpgrade + Node *corev1.Node + ExpectError bool + }{ + { + "List cluster should succeed", + createRollingUpgradeReconciler(t), + createRollingUpgrade(), + createNode(), + false, + }, + } + + for _, test := range tests { + actual, err := test.Reconciler.Auth.ListClusterNodes() + expected := createNodeList() + if err != nil || !reflect.DeepEqual(actual, expected) { + t.Errorf("ListClusterNodes fail %v", err) + } + } +} + +// This test checks implementation of our DrainNode which does both cordon + drain +func TestDrainNode(t *testing.T) { + var tests = []struct { + TestDescription string + Reconciler *RollingUpgradeReconciler + RollingUpgrade *v1alpha1.RollingUpgrade + Node *corev1.Node + ExpectError bool + }{ + { + "Drain should succeed as node is registered with fakeClient", + createRollingUpgradeReconciler(t), + createRollingUpgrade(), + createNode(), + false, + }, + { + "Drain should fail as node is not registered with fakeClient", + createRollingUpgradeReconciler(t), + createRollingUpgrade(), + &corev1.Node{}, + true, + }, + } + + for _, test := range tests { + err := test.Reconciler.Auth.DrainNode( + test.Node, + time.Duration(test.RollingUpgrade.PostDrainDelaySeconds()), + test.RollingUpgrade.DrainTimeout(), + test.Reconciler.Auth.Kubernetes, + ) + if (test.ExpectError && err == nil) || (!test.ExpectError && err != nil) { + t.Errorf("Test Description: %s \n expected error(bool): %v, Actual err: %v", test.TestDescription, test.ExpectError, err) + } + } + +} + +// This test checks implementation of the package provided Cordon/Uncordon function +func TestRunCordonOrUncordon(t *testing.T) { + var tests = []struct { + TestDescription string + Reconciler *RollingUpgradeReconciler + RollingUpgrade *v1alpha1.RollingUpgrade + Node *corev1.Node + Cordon bool + ExpectError bool + }{ + { + "Cordon should succeed as node is registered with fakeClient", + createRollingUpgradeReconciler(t), + createRollingUpgrade(), + createNode(), + true, + false, + }, + { + "Cordon should fail as node is not registered with fakeClient", + createRollingUpgradeReconciler(t), + createRollingUpgrade(), + &corev1.Node{}, + true, + true, + }, + { + "Uncordon should succeed as node is registered with fakeClient", + createRollingUpgradeReconciler(t), + createRollingUpgrade(), + func() *corev1.Node { + node := createNode() + node.Spec.Unschedulable = true + return node + }(), + false, + false, + }, + { + "Uncordon should fail as node is not registered with fakeClient", + createRollingUpgradeReconciler(t), + createRollingUpgrade(), + func() *corev1.Node { + node := &corev1.Node{} + node.Spec.Unschedulable = true + return node + }(), + false, + true, + }, + } + + for _, test := range tests { + helper := &drain.Helper{ + Client: test.Reconciler.Auth.Kubernetes, + Force: true, + GracePeriodSeconds: -1, + IgnoreAllDaemonSets: true, + Out: os.Stdout, + ErrOut: os.Stdout, + DeleteEmptyDirData: true, + Timeout: time.Duration(test.RollingUpgrade.Spec.Strategy.DrainTimeout) * time.Second, + } + err := drain.RunCordonOrUncordon(helper, test.Node, test.Cordon) + if (test.ExpectError && err == nil) || (!test.ExpectError && err != nil) { + t.Errorf("Test Description: %s \n expected error(bool): %v, Actual err: %v", test.TestDescription, test.ExpectError, err) + } + //check if the node is actually cordoned/uncordoned. + if test.Cordon && test.Node != nil && !test.Node.Spec.Unschedulable { + t.Errorf("Test Description: %s \n expected the node to be cordoned but it is uncordoned", test.TestDescription) + } + if !test.Cordon && test.Node != nil && test.Node.Spec.Unschedulable { + t.Errorf("Test Description: %s \n expected the node to be uncordoned but it is cordoned", test.TestDescription) + } + + } + +} + +// This test checks implementation of the package provided Drain function +func TestRunDrainNode(t *testing.T) { + var tests = []struct { + TestDescription string + Reconciler *RollingUpgradeReconciler + RollingUpgrade *v1alpha1.RollingUpgrade + Node *corev1.Node + ExpectError bool + }{ + { + "Drain should succeed as node is registered with fakeClient", + createRollingUpgradeReconciler(t), + createRollingUpgrade(), + createNode(), + false, + }, + // This test should fail, create an upstream ticket. + // https://github.com/kubernetes/kubectl/blob/d5b32e7f3c0260abb5b1cd5a62d4eb1de287bc93/pkg/drain/default.go#L33 + { + "Drain should fail as node is not registered with fakeClient", + createRollingUpgradeReconciler(t), + createRollingUpgrade(), + &corev1.Node{}, + true, + }, + } + for _, test := range tests { + helper := &drain.Helper{ + Client: test.Reconciler.Auth.Kubernetes, + Force: true, + GracePeriodSeconds: -1, + IgnoreAllDaemonSets: true, + Out: os.Stdout, + ErrOut: os.Stdout, + DeleteEmptyDirData: true, + Timeout: time.Duration(test.RollingUpgrade.Spec.Strategy.DrainTimeout) * time.Second, + } + err := drain.RunNodeDrain(helper, test.Node.Name) + if (test.ExpectError && err == nil) || (!test.ExpectError && err != nil) { + t.Errorf("Test Description: %s \n expected error(bool): %v, Actual err: %v", test.TestDescription, test.ExpectError, err) + } + } + +} + +func TestIsInstanceDrifted(t *testing.T) { + var tests = []struct { + TestDescription string + Reconciler *RollingUpgradeReconciler + RollingUpgrade *v1alpha1.RollingUpgrade + Instance *autoscaling.Instance + ExpectedValue bool + }{ + { + "Instance has the same launch config as the ASG, expect false from IsInstanceDrifted", + createRollingUpgradeReconciler(t), + createRollingUpgrade(), + createASGInstance("mock-instance-1", "mock-launch-config-1"), + false, + }, + { + "Instance has different launch config from the ASG, expect true from IsInstanceDrifted", + createRollingUpgradeReconciler(t), + createRollingUpgrade(), + createASGInstance("mock-instance-1", "different-launch-config"), + true, + }, + { + "Instance has no launch config, expect true from IsInstanceDrifted", + createRollingUpgradeReconciler(t), + createRollingUpgrade(), + createASGInstance("mock-instance-1", ""), + true, + }, + } + for _, test := range tests { + test.Reconciler.Cloud.ScalingGroups = createASGs() + actualValue := test.Reconciler.IsInstanceDrifted(test.RollingUpgrade, test.Instance) + if actualValue != test.ExpectedValue { + t.Errorf("Test Description: %s \n expected value: %v, actual value: %v", test.TestDescription, test.ExpectedValue, actualValue) + } + } +} + +func TestIsScalingGroupDrifted(t *testing.T) { + var tests = []struct { + TestDescription string + Reconciler *RollingUpgradeReconciler + RollingUpgrade *v1alpha1.RollingUpgrade + AsgClient *MockAutoscalingGroup + ExpectedValue bool + }{ + { + "All instances have the same launch config as the ASG, expect false from IsScalingGroupDrifted", + createRollingUpgradeReconciler(t), + createRollingUpgrade(), + createASGClient(), + false, + }, + { + "All instances have different launch config as the ASG, expect false from IsScalingGroupDrifted", + createRollingUpgradeReconciler(t), + createRollingUpgrade(), + func() *MockAutoscalingGroup { + newAsgClient := createASGClient() + newAsgClient.autoScalingGroups[0].LaunchConfigurationName = aws.String("different-launch-config") + return newAsgClient + }(), + true, + }, + } + for _, test := range tests { + test.Reconciler.Cloud.ScalingGroups = test.AsgClient.autoScalingGroups + test.Reconciler.Auth.AmazonClientSet.AsgClient = test.AsgClient + + actualValue := test.Reconciler.IsScalingGroupDrifted(test.RollingUpgrade) + if actualValue != test.ExpectedValue { + t.Errorf("Test Description: %s \n expected value: %v, actual value: %v", test.TestDescription, test.ExpectedValue, actualValue) + } + } + +} + +func TestDesiredNodesReady(t *testing.T) { + var tests = []struct { + TestDescription string + Reconciler *RollingUpgradeReconciler + RollingUpgrade *v1alpha1.RollingUpgrade + AsgClient *MockAutoscalingGroup + ClusterNodes *corev1.NodeList + ExpectedValue bool + }{ + { + "Desired nodes are ready", + createRollingUpgradeReconciler(t), + createRollingUpgrade(), + createASGClient(), + createNodeList(), + true, + }, + { + "Desired instances are not ready (desiredCount != inServiceCount)", + createRollingUpgradeReconciler(t), + createRollingUpgrade(), + func() *MockAutoscalingGroup { + newAsgClient := createASGClient() + newAsgClient.autoScalingGroups[0].DesiredCapacity = func(x int) *int64 { i := int64(x); return &i }(4) + return newAsgClient + }(), + createNodeList(), + false, + }, + { + "None of the nodes are ready (desiredCount != readyCount)", + createRollingUpgradeReconciler(t), + createRollingUpgrade(), + createASGClient(), + func() *corev1.NodeList { + var nodeList = &corev1.NodeList{Items: []corev1.Node{}} + for i := 0; i < 3; i++ { + node := createNode() + node.Status.Conditions = []corev1.NodeCondition{ + {Type: corev1.NodeReady, Status: corev1.ConditionFalse}, + } + nodeList.Items = append(nodeList.Items, *node) + } + return nodeList + }(), + false, + }, + { + "None of the instances are InService (desiredCount != inServiceCount)", + createRollingUpgradeReconciler(t), + createRollingUpgrade(), + func() *MockAutoscalingGroup { + newAsgClient := createASGClient() + newAsgClient.autoScalingGroups[0].Instances = []*autoscaling.Instance{ + &autoscaling.Instance{InstanceId: aws.String("mock-instance-1"), LifecycleState: aws.String("Pending")}, + &autoscaling.Instance{InstanceId: aws.String("mock-instance-2"), LifecycleState: aws.String("Terminating")}, + &autoscaling.Instance{InstanceId: aws.String("mock-instance-3"), LifecycleState: aws.String("Terminating")}, + } + return newAsgClient + }(), + createNodeList(), + false, + }, + } + for _, test := range tests { + test.Reconciler.Cloud.ScalingGroups = test.AsgClient.autoScalingGroups + test.Reconciler.Cloud.ClusterNodes = test.ClusterNodes + test.Reconciler.Auth.AmazonClientSet.AsgClient = test.AsgClient + + actualValue := test.Reconciler.DesiredNodesReady(test.RollingUpgrade) + if actualValue != test.ExpectedValue { + t.Errorf("Test Description: %s \n expected value: %v, actual value: %v", test.TestDescription, test.ExpectedValue, actualValue) + } + } +} From 1f0f075feb7896cb3162be03c114a5e66aee6be6 Mon Sep 17 00:00:00 2001 From: Sahil Badla Date: Wed, 21 Apr 2021 12:09:10 -0700 Subject: [PATCH 24/74] #2285: rollup CR statistic metrics in v2 (#218) * #2285: rollup CR statistic metrics in v2 Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 --- controllers/common/metrics.go | 77 ++++++++++++++++-------- controllers/common/metrics_test.go | 22 ++++++- controllers/rollingupgrade_controller.go | 5 ++ controllers/upgrade.go | 3 + controllers/upgrade_test.go | 47 ++++++++++++++- 5 files changed, 128 insertions(+), 26 deletions(-) diff --git a/controllers/common/metrics.go b/controllers/common/metrics.go index 723457dc..272afe82 100644 --- a/controllers/common/metrics.go +++ b/controllers/common/metrics.go @@ -1,38 +1,55 @@ package common import ( - "github.com/keikoproj/upgrade-manager/controllers/common/log" - "github.com/prometheus/client_golang/prometheus" "reflect" - "sigs.k8s.io/controller-runtime/pkg/metrics" "strings" "time" + + "github.com/keikoproj/upgrade-manager/controllers/common/log" + "github.com/prometheus/client_golang/prometheus" + "sigs.k8s.io/controller-runtime/pkg/metrics" ) -//All cluster level node upgrade statistics +var ( + metricNamespace = "upgrade_manager_v2" -var nodeRotationTotal = prometheus.NewHistogram( - prometheus.HistogramOpts{ - Namespace: "node", - Name: "rotation_total_seconds", - Help: "Node rotation total", - Buckets: []float64{ - 10.0, - 30.0, - 60.0, - 90.0, - 120.0, - 180.0, - 300.0, - 600.0, - 900.0, - }, - }) + //All cluster level node upgrade statistics + nodeRotationTotal = prometheus.NewHistogram( + prometheus.HistogramOpts{ + Namespace: metricNamespace, + Name: "node_rotation_total_seconds", + Help: "Node rotation total", + Buckets: []float64{ + 10.0, + 30.0, + 60.0, + 90.0, + 120.0, + 180.0, + 300.0, + 600.0, + 900.0, + }, + }) + + stepSummaries = make(map[string]map[string]prometheus.Summary) -var stepSummaries = make(map[string]map[string]prometheus.Summary) + CRStatus = prometheus.NewGaugeVec( + prometheus.GaugeOpts{ + Namespace: metricNamespace, + Name: "resource_status", + Help: "Rollup CR statistics, partitioned by name.", + }, + []string{ + // name of the CR + "resource_name", + }, + ) +) func InitMetrics() { metrics.Registry.MustRegister(nodeRotationTotal) + metrics.Registry.MustRegister(CRStatus) } // Add rolling update step duration when the step is completed @@ -52,8 +69,8 @@ func AddStepDuration(groupName string, stepName string, duration time.Duration) if s, ok := steps[stepName]; !ok { summary = prometheus.NewSummary( prometheus.SummaryOpts{ - Namespace: "node", - Name: stepName + "_seconds", + Namespace: metricNamespace, + Name: "node_" + stepName + "_seconds", Help: "Summary for node " + stepName, ConstLabels: prometheus.Labels{"group": groupName}, }) @@ -72,3 +89,15 @@ func AddStepDuration(groupName string, stepName string, duration time.Duration) summary.Observe(duration.Seconds()) } } + +func SetRollupInitOrRunningStatus(ruName string) { + CRStatus.WithLabelValues(ruName).Set(0) +} + +func SetRollupCompletedStatus(ruName string) { + CRStatus.WithLabelValues(ruName).Set(1) +} + +func SetRollupFailedStatus(ruName string) { + CRStatus.WithLabelValues(ruName).Set(-1) +} diff --git a/controllers/common/metrics_test.go b/controllers/common/metrics_test.go index 9437beb6..29d14f25 100644 --- a/controllers/common/metrics_test.go +++ b/controllers/common/metrics_test.go @@ -1,8 +1,9 @@ package common import ( - "github.com/onsi/gomega" "testing" + + "github.com/onsi/gomega" ) func TestAddRollingUpgradeStepDuration(t *testing.T) { @@ -27,3 +28,22 @@ func TestAddRollingUpgradeStepDuration(t *testing.T) { AddStepDuration("test-asg", "total", 1) g.Expect(stepSummaries["test-asg"]["kickoff"]).NotTo(gomega.BeNil()) } + +func TestCRStatusCompleted(t *testing.T) { + g := gomega.NewGomegaWithT(t) + + SetRollupInitOrRunningStatus("cr_test_1") + gauage, err := CRStatus.GetMetricWithLabelValues("cr_test_1") + g.Expect(err).To(gomega.BeNil()) + g.Expect(gauage).ToNot(gomega.BeNil()) + + SetRollupCompletedStatus("cr_test_2") + gauage, err = CRStatus.GetMetricWithLabelValues("cr_test_2") + g.Expect(err).To(gomega.BeNil()) + g.Expect(gauage).ToNot(gomega.BeNil()) + + SetRollupFailedStatus("cr_test_3") + gauage, err = CRStatus.GetMetricWithLabelValues("cr_test_3") + g.Expect(err).To(gomega.BeNil()) + g.Expect(gauage).ToNot(gomega.BeNil()) +} diff --git a/controllers/rollingupgrade_controller.go b/controllers/rollingupgrade_controller.go index af18c678..3ee2f841 100644 --- a/controllers/rollingupgrade_controller.go +++ b/controllers/rollingupgrade_controller.go @@ -127,16 +127,21 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque r.Info("admitted new rollingupgrade", "name", rollingUpgrade.NamespacedName(), "scalingGroup", scalingGroupName) r.AdmissionMap.Store(rollingUpgrade.NamespacedName(), scalingGroupName) rollingUpgrade.SetCurrentStatus(v1alpha1.StatusInit) + common.SetRollupInitOrRunningStatus(rollingUpgrade.Name) r.Cloud = NewDiscoveredState(r.Auth, r.Logger) if err := r.Cloud.Discover(); err != nil { rollingUpgrade.SetCurrentStatus(v1alpha1.StatusError) + // Set prometheus metric cr_status_failed + common.SetRollupFailedStatus(rollingUpgrade.Name) return ctrl.Result{}, err } // process node rotation if err := r.RotateNodes(rollingUpgrade); err != nil { rollingUpgrade.SetCurrentStatus(v1alpha1.StatusError) + // Set prometheus metric cr_status_failed + common.SetRollupFailedStatus(rollingUpgrade.Name) return ctrl.Result{}, err } diff --git a/controllers/upgrade.go b/controllers/upgrade.go index 7e713784..36783fbc 100644 --- a/controllers/upgrade.go +++ b/controllers/upgrade.go @@ -54,6 +54,7 @@ func (r *RollingUpgradeReconciler) RotateNodes(rollingUpgrade *v1alpha1.RollingU drainInterval = rollingUpgrade.PostDrainDelaySeconds() ) rollingUpgrade.SetCurrentStatus(v1alpha1.StatusRunning) + common.SetRollupInitOrRunningStatus(rollingUpgrade.Name) // set status start time if rollingUpgrade.StartTime() == "" { @@ -84,6 +85,8 @@ func (r *RollingUpgradeReconciler) RotateNodes(rollingUpgrade *v1alpha1.RollingU // check if all instances are rotated. if !r.IsScalingGroupDrifted(rollingUpgrade) { rollingUpgrade.SetCurrentStatus(v1alpha1.StatusComplete) + // Set prometheus metric cr_status_completed + common.SetRollupCompletedStatus(rollingUpgrade.Name) return nil } diff --git a/controllers/upgrade_test.go b/controllers/upgrade_test.go index 5bdc5e53..2572be5a 100644 --- a/controllers/upgrade_test.go +++ b/controllers/upgrade_test.go @@ -261,7 +261,7 @@ func TestIsScalingGroupDrifted(t *testing.T) { false, }, { - "All instances have different launch config as the ASG, expect false from IsScalingGroupDrifted", + "All instances have different launch config as the ASG, expect true from IsScalingGroupDrifted", createRollingUpgradeReconciler(t), createRollingUpgrade(), func() *MockAutoscalingGroup { @@ -284,6 +284,51 @@ func TestIsScalingGroupDrifted(t *testing.T) { } +func TestRotateNodes(t *testing.T) { + var tests = []struct { + TestDescription string + Reconciler *RollingUpgradeReconciler + RollingUpgrade *v1alpha1.RollingUpgrade + AsgClient *MockAutoscalingGroup + ExpectedValue bool + ExpectedStatusValue string + }{ + { + "All instances have different launch config as the ASG, expect true from IsScalingGroupDrifted", + createRollingUpgradeReconciler(t), + createRollingUpgrade(), + func() *MockAutoscalingGroup { + newAsgClient := createASGClient() + newAsgClient.autoScalingGroups[0].LaunchConfigurationName = aws.String("different-launch-config") + return newAsgClient + }(), + true, + v1alpha1.StatusRunning, + }, + { + "All instances have the same launch config as the ASG, expect false from IsScalingGroupDrifted", + createRollingUpgradeReconciler(t), + createRollingUpgrade(), + createASGClient(), + false, + v1alpha1.StatusComplete, + }, + } + for _, test := range tests { + test.Reconciler.Cloud.ScalingGroups = test.AsgClient.autoScalingGroups + test.Reconciler.Auth.AmazonClientSet.AsgClient = test.AsgClient + + err := test.Reconciler.RotateNodes(test.RollingUpgrade) + if err != nil { + t.Errorf("Test Description: \n expected value: nil, actual value: %v", err) + } + if test.RollingUpgrade.CurrentStatus() != test.ExpectedStatusValue { + t.Errorf("Test Description: %s \n expected value: %s, actual value: %s", test.TestDescription, test.ExpectedStatusValue, test.RollingUpgrade.CurrentStatus()) + } + } + +} + func TestDesiredNodesReady(t *testing.T) { var tests = []struct { TestDescription string From c445af91eb6bd7d5916d4586976e74503440e615 Mon Sep 17 00:00:00 2001 From: Sahil Badla Date: Thu, 29 Apr 2021 14:28:23 -0700 Subject: [PATCH 25/74] #2285: renamed some methods related to metrics (#224) Signed-off-by: sbadla1 --- controllers/common/metrics.go | 6 +++--- controllers/common/metrics_test.go | 6 +++--- controllers/rollingupgrade_controller.go | 6 +++--- controllers/upgrade.go | 4 ++-- 4 files changed, 11 insertions(+), 11 deletions(-) diff --git a/controllers/common/metrics.go b/controllers/common/metrics.go index 272afe82..74ca9e12 100644 --- a/controllers/common/metrics.go +++ b/controllers/common/metrics.go @@ -90,14 +90,14 @@ func AddStepDuration(groupName string, stepName string, duration time.Duration) } } -func SetRollupInitOrRunningStatus(ruName string) { +func SetMetricRollupInitOrRunning(ruName string) { CRStatus.WithLabelValues(ruName).Set(0) } -func SetRollupCompletedStatus(ruName string) { +func SetMetricRollupCompleted(ruName string) { CRStatus.WithLabelValues(ruName).Set(1) } -func SetRollupFailedStatus(ruName string) { +func SetMetricRollupFailed(ruName string) { CRStatus.WithLabelValues(ruName).Set(-1) } diff --git a/controllers/common/metrics_test.go b/controllers/common/metrics_test.go index 29d14f25..8d96626b 100644 --- a/controllers/common/metrics_test.go +++ b/controllers/common/metrics_test.go @@ -32,17 +32,17 @@ func TestAddRollingUpgradeStepDuration(t *testing.T) { func TestCRStatusCompleted(t *testing.T) { g := gomega.NewGomegaWithT(t) - SetRollupInitOrRunningStatus("cr_test_1") + SetMetricRollupInitOrRunning("cr_test_1") gauage, err := CRStatus.GetMetricWithLabelValues("cr_test_1") g.Expect(err).To(gomega.BeNil()) g.Expect(gauage).ToNot(gomega.BeNil()) - SetRollupCompletedStatus("cr_test_2") + SetMetricRollupCompleted("cr_test_2") gauage, err = CRStatus.GetMetricWithLabelValues("cr_test_2") g.Expect(err).To(gomega.BeNil()) g.Expect(gauage).ToNot(gomega.BeNil()) - SetRollupFailedStatus("cr_test_3") + SetMetricRollupFailed("cr_test_3") gauage, err = CRStatus.GetMetricWithLabelValues("cr_test_3") g.Expect(err).To(gomega.BeNil()) g.Expect(gauage).ToNot(gomega.BeNil()) diff --git a/controllers/rollingupgrade_controller.go b/controllers/rollingupgrade_controller.go index 3ee2f841..84237abd 100644 --- a/controllers/rollingupgrade_controller.go +++ b/controllers/rollingupgrade_controller.go @@ -127,13 +127,13 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque r.Info("admitted new rollingupgrade", "name", rollingUpgrade.NamespacedName(), "scalingGroup", scalingGroupName) r.AdmissionMap.Store(rollingUpgrade.NamespacedName(), scalingGroupName) rollingUpgrade.SetCurrentStatus(v1alpha1.StatusInit) - common.SetRollupInitOrRunningStatus(rollingUpgrade.Name) + common.SetMetricRollupInitOrRunning(rollingUpgrade.Name) r.Cloud = NewDiscoveredState(r.Auth, r.Logger) if err := r.Cloud.Discover(); err != nil { rollingUpgrade.SetCurrentStatus(v1alpha1.StatusError) // Set prometheus metric cr_status_failed - common.SetRollupFailedStatus(rollingUpgrade.Name) + common.SetMetricRollupFailed(rollingUpgrade.Name) return ctrl.Result{}, err } @@ -141,7 +141,7 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque if err := r.RotateNodes(rollingUpgrade); err != nil { rollingUpgrade.SetCurrentStatus(v1alpha1.StatusError) // Set prometheus metric cr_status_failed - common.SetRollupFailedStatus(rollingUpgrade.Name) + common.SetMetricRollupFailed(rollingUpgrade.Name) return ctrl.Result{}, err } diff --git a/controllers/upgrade.go b/controllers/upgrade.go index 36783fbc..d6586e36 100644 --- a/controllers/upgrade.go +++ b/controllers/upgrade.go @@ -54,7 +54,7 @@ func (r *RollingUpgradeReconciler) RotateNodes(rollingUpgrade *v1alpha1.RollingU drainInterval = rollingUpgrade.PostDrainDelaySeconds() ) rollingUpgrade.SetCurrentStatus(v1alpha1.StatusRunning) - common.SetRollupInitOrRunningStatus(rollingUpgrade.Name) + common.SetMetricRollupInitOrRunning(rollingUpgrade.Name) // set status start time if rollingUpgrade.StartTime() == "" { @@ -86,7 +86,7 @@ func (r *RollingUpgradeReconciler) RotateNodes(rollingUpgrade *v1alpha1.RollingU if !r.IsScalingGroupDrifted(rollingUpgrade) { rollingUpgrade.SetCurrentStatus(v1alpha1.StatusComplete) // Set prometheus metric cr_status_completed - common.SetRollupCompletedStatus(rollingUpgrade.Name) + common.SetMetricRollupCompleted(rollingUpgrade.Name) return nil } From b8d0e7290c523dc636da140e8e219089c099a4db Mon Sep 17 00:00:00 2001 From: Sahil Badla Date: Mon, 3 May 2021 10:49:24 -0700 Subject: [PATCH 26/74] #2286: removed version from metric namespace (#227) Signed-off-by: sbadla1 --- controllers/common/metrics.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/controllers/common/metrics.go b/controllers/common/metrics.go index 74ca9e12..694571eb 100644 --- a/controllers/common/metrics.go +++ b/controllers/common/metrics.go @@ -11,7 +11,7 @@ import ( ) var ( - metricNamespace = "upgrade_manager_v2" + metricNamespace = "upgrade_manager" //All cluster level node upgrade statistics nodeRotationTotal = prometheus.NewHistogram( From b664fdd65177824003e528bd388c46f6398a3fc6 Mon Sep 17 00:00:00 2001 From: Shreyas Badiger <7680410+shreyas-badiger@users.noreply.github.com> Date: Thu, 13 May 2021 14:15:08 -0700 Subject: [PATCH 27/74] Create RollingUpgradeContext (#234) * #2285: rollup CR statistic metrics in v2 (#218) * #2285: rollup CR statistic metrics in v2 Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 Signed-off-by: sbadiger * log cloud discovery failure Signed-off-by: sbadiger * Create RollingUpgrade Context Signed-off-by: sbadiger * rollingupgrade context Signed-off-by: sbadiger Co-authored-by: Sahil Badla --- controllers/helpers_test.go | 12 +- controllers/rollingupgrade_controller.go | 30 ++-- controllers/upgrade.go | 182 ++++++++++++----------- controllers/upgrade_test.go | 95 +++++------- 4 files changed, 163 insertions(+), 156 deletions(-) diff --git a/controllers/helpers_test.go b/controllers/helpers_test.go index 948e6f91..7e0335c4 100644 --- a/controllers/helpers_test.go +++ b/controllers/helpers_test.go @@ -48,12 +48,22 @@ func createRollingUpgradeReconciler(t *testing.T) *RollingUpgradeReconciler { ScriptRunner: ScriptRunner{ Logger: logger, }, - Cloud: NewDiscoveredState(auth, logger), } return reconciler } +func createRollingUpgradeContext(r *RollingUpgradeReconciler) *RollingUpgradeContext { + return &RollingUpgradeContext{ + Logger: r.Logger, + Auth: r.Auth, + ScriptRunner: r.ScriptRunner, + Cloud: NewDiscoveredState(r.Auth, r.Logger), + RollingUpgrade: createRollingUpgrade(), + } + +} + func createRollingUpgrade() *v1alpha1.RollingUpgrade { return &v1alpha1.RollingUpgrade{ ObjectMeta: metav1.ObjectMeta{Name: "0", Namespace: "default"}, diff --git a/controllers/rollingupgrade_controller.go b/controllers/rollingupgrade_controller.go index 84237abd..93505bec 100644 --- a/controllers/rollingupgrade_controller.go +++ b/controllers/rollingupgrade_controller.go @@ -40,16 +40,13 @@ import ( type RollingUpgradeReconciler struct { client.Client logr.Logger - Scheme *runtime.Scheme - AdmissionMap sync.Map - CacheConfig *cache.Config - Auth *RollingUpgradeAuthenticator - Cloud *DiscoveredState - EventWriter *kubeprovider.EventWriter - maxParallel int - ScriptRunner ScriptRunner - DrainGroupMapper sync.Map - DrainErrorMapper sync.Map + Scheme *runtime.Scheme + AdmissionMap sync.Map + CacheConfig *cache.Config + EventWriter *kubeprovider.EventWriter + maxParallel int + ScriptRunner ScriptRunner + Auth *RollingUpgradeAuthenticator } type RollingUpgradeAuthenticator struct { @@ -129,8 +126,15 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque rollingUpgrade.SetCurrentStatus(v1alpha1.StatusInit) common.SetMetricRollupInitOrRunning(rollingUpgrade.Name) - r.Cloud = NewDiscoveredState(r.Auth, r.Logger) - if err := r.Cloud.Discover(); err != nil { + rollupCtx := &RollingUpgradeContext{ + Logger: r.Logger, + Auth: r.Auth, + ScriptRunner: r.ScriptRunner, + RollingUpgrade: rollingUpgrade, + } + rollupCtx.Cloud = NewDiscoveredState(rollupCtx.Auth, rollupCtx.Logger) + if err := rollupCtx.Cloud.Discover(); err != nil { + r.Info("failed to discover the cloud", "name", rollingUpgrade.NamespacedName(), "scalingGroup", scalingGroupName) rollingUpgrade.SetCurrentStatus(v1alpha1.StatusError) // Set prometheus metric cr_status_failed common.SetMetricRollupFailed(rollingUpgrade.Name) @@ -138,7 +142,7 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque } // process node rotation - if err := r.RotateNodes(rollingUpgrade); err != nil { + if err := rollupCtx.RotateNodes(); err != nil { rollingUpgrade.SetCurrentStatus(v1alpha1.StatusError) // Set prometheus metric cr_status_failed common.SetMetricRollupFailed(rollingUpgrade.Name) diff --git a/controllers/upgrade.go b/controllers/upgrade.go index d6586e36..6efa3acb 100644 --- a/controllers/upgrade.go +++ b/controllers/upgrade.go @@ -24,6 +24,7 @@ import ( "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/service/autoscaling" + "github.com/go-logr/logr" "github.com/keikoproj/upgrade-manager/api/v1alpha1" "github.com/keikoproj/upgrade-manager/controllers/common" awsprovider "github.com/keikoproj/upgrade-manager/controllers/providers/aws" @@ -45,70 +46,80 @@ type DrainManager struct { DrainGroup *sync.WaitGroup `json:"-"` } +type RollingUpgradeContext struct { + logr.Logger + ScriptRunner ScriptRunner + Auth *RollingUpgradeAuthenticator + Cloud *DiscoveredState + DrainGroupMapper sync.Map + DrainErrorMapper sync.Map + RollingUpgrade *v1alpha1.RollingUpgrade +} + // TODO: main node rotation logic -func (r *RollingUpgradeReconciler) RotateNodes(rollingUpgrade *v1alpha1.RollingUpgrade) error { +func (r *RollingUpgradeContext) RotateNodes() error { var ( - lastTerminationTime = rollingUpgrade.LastNodeTerminationTime() - nodeInterval = rollingUpgrade.NodeIntervalSeconds() - lastDrainTime = rollingUpgrade.LastNodeDrainTime() - drainInterval = rollingUpgrade.PostDrainDelaySeconds() + lastTerminationTime = r.RollingUpgrade.LastNodeTerminationTime() + nodeInterval = r.RollingUpgrade.NodeIntervalSeconds() + lastDrainTime = r.RollingUpgrade.LastNodeDrainTime() + drainInterval = r.RollingUpgrade.PostDrainDelaySeconds() ) - rollingUpgrade.SetCurrentStatus(v1alpha1.StatusRunning) - common.SetMetricRollupInitOrRunning(rollingUpgrade.Name) + r.RollingUpgrade.SetCurrentStatus(v1alpha1.StatusRunning) + common.SetRollupInitOrRunningStatus(r.RollingUpgrade.Name) // set status start time - if rollingUpgrade.StartTime() == "" { - rollingUpgrade.SetStartTime(time.Now().Format(time.RFC3339)) + if r.RollingUpgrade.StartTime() == "" { + r.RollingUpgrade.SetStartTime(time.Now().Format(time.RFC3339)) } if !lastTerminationTime.IsZero() || !lastDrainTime.IsZero() { // Check if we are still waiting on a termination delay if time.Since(lastTerminationTime.Time).Seconds() < float64(nodeInterval) { - r.Info("reconcile requeue due to termination interval wait", "name", rollingUpgrade.NamespacedName()) + r.Info("reconcile requeue due to termination interval wait", "name", r.RollingUpgrade.NamespacedName()) return nil } // Check if we are still waiting on a drain delay if time.Since(lastDrainTime.Time).Seconds() < float64(drainInterval) { - r.Info("reconcile requeue due to drain interval wait", "name", rollingUpgrade.NamespacedName()) + r.Info("reconcile requeue due to drain interval wait", "name", r.RollingUpgrade.NamespacedName()) return nil } } var ( - scalingGroup = awsprovider.SelectScalingGroup(rollingUpgrade.ScalingGroupName(), r.Cloud.ScalingGroups) + scalingGroup = awsprovider.SelectScalingGroup(r.RollingUpgrade.ScalingGroupName(), r.Cloud.ScalingGroups) ) - rollingUpgrade.SetTotalNodes(len(scalingGroup.Instances)) + r.RollingUpgrade.SetTotalNodes(len(scalingGroup.Instances)) // check if all instances are rotated. - if !r.IsScalingGroupDrifted(rollingUpgrade) { - rollingUpgrade.SetCurrentStatus(v1alpha1.StatusComplete) + if !r.IsScalingGroupDrifted() { + r.RollingUpgrade.SetCurrentStatus(v1alpha1.StatusComplete) // Set prometheus metric cr_status_completed - common.SetMetricRollupCompleted(rollingUpgrade.Name) + common.SetRollupCompletedStatus(r.RollingUpgrade.Name) return nil } - rotationTargets := r.SelectTargets(rollingUpgrade, scalingGroup) - if ok, err := r.ReplaceNodeBatch(rollingUpgrade, rotationTargets); !ok { + rotationTargets := r.SelectTargets(scalingGroup) + if ok, err := r.ReplaceNodeBatch(rotationTargets); !ok { return err } return nil } -func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.RollingUpgrade, batch []*autoscaling.Instance) (bool, error) { +func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) (bool, error) { var ( - mode = rollingUpgrade.StrategyMode() + mode = r.RollingUpgrade.StrategyMode() drainManager = &DrainManager{} ) - r.Info("rotating batch", "instances", awsprovider.GetInstanceIDs(batch), "name", rollingUpgrade.NamespacedName()) + r.Info("rotating batch", "instances", awsprovider.GetInstanceIDs(batch), "name", r.RollingUpgrade.NamespacedName()) // load the appropriate waitGroup and Error channel for the DrainManager from reconciler object - drainGroup, _ := r.DrainGroupMapper.LoadOrStore(rollingUpgrade.NamespacedName(), &sync.WaitGroup{}) - drainErrs, _ := r.DrainErrorMapper.LoadOrStore(rollingUpgrade.NamespacedName(), make(chan error)) + drainGroup, _ := r.DrainGroupMapper.LoadOrStore(r.RollingUpgrade.NamespacedName(), &sync.WaitGroup{}) + drainErrs, _ := r.DrainErrorMapper.LoadOrStore(r.RollingUpgrade.NamespacedName(), make(chan error)) drainManager = &DrainManager{ DrainErrors: drainErrs.(chan error), DrainGroup: drainGroup.(*sync.WaitGroup), @@ -128,31 +139,31 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol nodeName = node.GetName() ) //Add statistics - rollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationKickoff) + r.RollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationKickoff) // Add in-progress tag if err := r.Auth.TagEC2instance(instanceID, instanceStateTagKey, inProgressTagValue); err != nil { - r.Error(err, "failed to set instance tag", "name", rollingUpgrade.NamespacedName(), "instance", instanceID) + r.Error(err, "failed to set instance tag", "name", r.RollingUpgrade.NamespacedName(), "instance", instanceID) return false, err } // Standby if aws.StringValue(target.LifecycleState) == autoscaling.LifecycleStateInService { - r.Info("setting instance to stand-by", "instance", instanceID, "name", rollingUpgrade.NamespacedName()) - if err := r.Auth.SetInstanceStandBy(target, rollingUpgrade.Spec.AsgName); err != nil { + r.Info("setting instance to stand-by", "instance", instanceID, "name", r.RollingUpgrade.NamespacedName()) + if err := r.Auth.SetInstanceStandBy(target, r.RollingUpgrade.Spec.AsgName); err != nil { // failure to set instance to standby are retryable - r.Info("failed to set instance to stand-by", "instance", instanceID, "message", err.Error(), "name", rollingUpgrade.NamespacedName()) + r.Info("failed to set instance to stand-by", "instance", instanceID, "message", err.Error(), "name", r.RollingUpgrade.NamespacedName()) return true, nil } } // Turns onto desired nodes - rollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationDesiredNodeReady) + r.RollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationDesiredNodeReady) // Wait for desired nodes - r.Info("waiting for desired nodes", "name", rollingUpgrade.NamespacedName()) - if !r.DesiredNodesReady(rollingUpgrade) { - r.Info("new node is yet to join the cluster", "name", rollingUpgrade.NamespacedName()) + r.Info("waiting for desired nodes", "name", r.RollingUpgrade.NamespacedName()) + if !r.DesiredNodesReady() { + r.Info("new node is yet to join the cluster", "name", r.RollingUpgrade.NamespacedName()) return true, nil } } @@ -165,11 +176,11 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol nodeName = node.GetName() ) //Add statistics - rollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationKickoff) + r.RollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationKickoff) // Add in-progress tag if err := r.Auth.TagEC2instance(instanceID, instanceStateTagKey, inProgressTagValue); err != nil { - r.Error(err, "failed to set instance tag", "name", rollingUpgrade.NamespacedName(), "instance", instanceID) + r.Error(err, "failed to set instance tag", "name", r.RollingUpgrade.NamespacedName(), "instance", instanceID) return false, err } } @@ -184,7 +195,7 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol scriptTarget = ScriptTarget{ InstanceID: instanceID, NodeName: nodeName, - UpgradeObject: rollingUpgrade, + UpgradeObject: r.RollingUpgrade, } ) drainManager.DrainGroup.Add(1) @@ -193,7 +204,7 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol defer drainManager.DrainGroup.Done() // Turns onto PreDrain script - rollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationPredrainScript) + r.RollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationPredrainScript) // Predrain script if err := r.ScriptRunner.PreDrain(scriptTarget); err != nil { @@ -202,18 +213,18 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol // Issue drain concurrently - set lastDrainTime if node := kubeprovider.SelectNodeByInstanceID(instanceID, r.Cloud.ClusterNodes); !reflect.DeepEqual(node, corev1.Node{}) { - r.Info("draining the node", "instance", instanceID, "node name", node.Name, "name", rollingUpgrade.NamespacedName()) + r.Info("draining the node", "instance", instanceID, "node name", node.Name, "name", r.RollingUpgrade.NamespacedName()) // Turns onto NodeRotationDrain - rollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationDrain) + r.RollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationDrain) - if err := r.Auth.DrainNode(&node, time.Duration(rollingUpgrade.PostDrainDelaySeconds()), rollingUpgrade.DrainTimeout(), r.Auth.Kubernetes); err != nil { + if err := r.Auth.DrainNode(&node, time.Duration(r.RollingUpgrade.PostDrainDelaySeconds()), r.RollingUpgrade.DrainTimeout(), r.Auth.Kubernetes); err != nil { drainManager.DrainErrors <- errors.Errorf("DrainNode failed: instanceID - %v, %v", instanceID, err.Error()) } } // Turns onto NodeRotationPostdrainScript - rollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationPostdrainScript) + r.RollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationPostdrainScript) // post drain script if err := r.ScriptRunner.PostDrain(scriptTarget); err != nil { @@ -221,7 +232,7 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol } // Turns onto NodeRotationPostWait - rollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationPostWait) + r.RollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationPostWait) // Post Wait Script if err := r.ScriptRunner.PostWait(scriptTarget); err != nil { @@ -239,16 +250,16 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol select { case err := <-drainManager.DrainErrors: - rollingUpgrade.Status.UpdateStatistics(nodeSteps) - rollingUpgrade.Status.UpdateLastBatchNodes(inProcessingNodes) + r.RollingUpgrade.Status.UpdateStatistics(nodeSteps) + r.RollingUpgrade.Status.UpdateLastBatchNodes(inProcessingNodes) - r.Error(err, "failed to rotate the node", "name", rollingUpgrade.NamespacedName()) + r.Error(err, "failed to rotate the node", "name", r.RollingUpgrade.NamespacedName()) return false, err case <-timeout: // goroutines completed, terminate and requeue - rollingUpgrade.SetLastNodeDrainTime(metav1.Time{Time: time.Now()}) - r.Info("instances drained successfully, terminating", "name", rollingUpgrade.NamespacedName()) + r.RollingUpgrade.SetLastNodeDrainTime(metav1.Time{Time: time.Now()}) + r.Info("instances drained successfully, terminating", "name", r.RollingUpgrade.NamespacedName()) for _, target := range batch { var ( instanceID = aws.StringValue(target.InstanceId) @@ -257,25 +268,25 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol scriptTarget = ScriptTarget{ InstanceID: instanceID, NodeName: nodeName, - UpgradeObject: rollingUpgrade, + UpgradeObject: r.RollingUpgrade, } ) // Turns onto NodeRotationTerminate - rollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationTerminate) + r.RollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationTerminate) // Terminate - set lastTerminateTime - r.Info("terminating instance", "instance", instanceID, "name", rollingUpgrade.NamespacedName()) + r.Info("terminating instance", "instance", instanceID, "name", r.RollingUpgrade.NamespacedName()) if err := r.Auth.TerminateInstance(target); err != nil { // terminate failures are retryable - r.Info("failed to terminate instance", "instance", instanceID, "message", err.Error(), "name", rollingUpgrade.NamespacedName()) + r.Info("failed to terminate instance", "instance", instanceID, "message", err.Error(), "name", r.RollingUpgrade.NamespacedName()) return true, nil } - rollingUpgrade.SetLastNodeTerminationTime(metav1.Time{Time: time.Now()}) + r.RollingUpgrade.SetLastNodeTerminationTime(metav1.Time{Time: time.Now()}) // Turns onto NodeRotationTerminate - rollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationPostTerminate) + r.RollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationPostTerminate) // Post Terminate Script if err := r.ScriptRunner.PostTerminate(scriptTarget); err != nil { @@ -283,27 +294,27 @@ func (r *RollingUpgradeReconciler) ReplaceNodeBatch(rollingUpgrade *v1alpha1.Rol } // Turns onto NodeRotationCompleted - rollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, rollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationCompleted) + r.RollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationCompleted) } - rollingUpgrade.Status.UpdateStatistics(nodeSteps) - rollingUpgrade.Status.UpdateLastBatchNodes(inProcessingNodes) + r.RollingUpgrade.Status.UpdateStatistics(nodeSteps) + r.RollingUpgrade.Status.UpdateLastBatchNodes(inProcessingNodes) case <-time.After(DefaultWaitGroupTimeout): // goroutines timed out - requeue - rollingUpgrade.Status.UpdateStatistics(nodeSteps) - rollingUpgrade.Status.UpdateLastBatchNodes(inProcessingNodes) + r.RollingUpgrade.Status.UpdateStatistics(nodeSteps) + r.RollingUpgrade.Status.UpdateLastBatchNodes(inProcessingNodes) - r.Info("instances are still draining", "name", rollingUpgrade.NamespacedName()) + r.Info("instances are still draining", "name", r.RollingUpgrade.NamespacedName()) return true, nil } return true, nil } -func (r *RollingUpgradeReconciler) SelectTargets(rollingUpgrade *v1alpha1.RollingUpgrade, scalingGroup *autoscaling.Group) []*autoscaling.Instance { +func (r *RollingUpgradeContext) SelectTargets(scalingGroup *autoscaling.Group) []*autoscaling.Instance { var ( - batchSize = rollingUpgrade.MaxUnavailable() + batchSize = r.RollingUpgrade.MaxUnavailable() totalNodes = len(scalingGroup.Instances) targets = make([]*autoscaling.Instance, 0) ) @@ -330,9 +341,9 @@ func (r *RollingUpgradeReconciler) SelectTargets(rollingUpgrade *v1alpha1.Rollin } // select via strategy if there are no in-progress instances - if rollingUpgrade.UpdateStrategyType() == v1alpha1.RandomUpdateStrategy { + if r.RollingUpgrade.UpdateStrategyType() == v1alpha1.RandomUpdateStrategy { for _, instance := range scalingGroup.Instances { - if r.IsInstanceDrifted(rollingUpgrade, instance) { + if r.IsInstanceDrifted(instance) { targets = append(targets, instance) } } @@ -341,9 +352,9 @@ func (r *RollingUpgradeReconciler) SelectTargets(rollingUpgrade *v1alpha1.Rollin } return targets[:unavailableInt] - } else if rollingUpgrade.UpdateStrategyType() == v1alpha1.UniformAcrossAzUpdateStrategy { + } else if r.RollingUpgrade.UpdateStrategyType() == v1alpha1.UniformAcrossAzUpdateStrategy { for _, instance := range scalingGroup.Instances { - if r.IsInstanceDrifted(rollingUpgrade, instance) { + if r.IsInstanceDrifted(instance) { targets = append(targets, instance) } } @@ -367,10 +378,10 @@ func (r *RollingUpgradeReconciler) SelectTargets(rollingUpgrade *v1alpha1.Rollin return targets } -func (r *RollingUpgradeReconciler) IsInstanceDrifted(rollingUpgrade *v1alpha1.RollingUpgrade, instance *autoscaling.Instance) bool { +func (r *RollingUpgradeContext) IsInstanceDrifted(instance *autoscaling.Instance) bool { var ( - scalingGroupName = rollingUpgrade.ScalingGroupName() + scalingGroupName = r.RollingUpgrade.ScalingGroupName() scalingGroup = awsprovider.SelectScalingGroup(scalingGroupName, r.Cloud.ScalingGroups) instanceID = aws.StringValue(instance.InstanceId) ) @@ -380,32 +391,32 @@ func (r *RollingUpgradeReconciler) IsInstanceDrifted(rollingUpgrade *v1alpha1.Ro return false } // check if there is atleast one node that meets the force-referesh criteria - if rollingUpgrade.IsForceRefresh() { + if r.RollingUpgrade.IsForceRefresh() { var ( node = kubeprovider.SelectNodeByInstanceID(instanceID, r.Cloud.ClusterNodes) nodeCreationTime = node.CreationTimestamp.Time - upgradeCreationTime = rollingUpgrade.CreationTimestamp.Time + upgradeCreationTime = r.RollingUpgrade.CreationTimestamp.Time ) if nodeCreationTime.Before(upgradeCreationTime) { - r.Info("rolling upgrade configured for forced refresh", "instance", instanceID, "name", rollingUpgrade.NamespacedName()) + r.Info("rolling upgrade configured for forced refresh", "instance", instanceID, "name", r.RollingUpgrade.NamespacedName()) return true } } if scalingGroup.LaunchConfigurationName != nil { if instance.LaunchConfigurationName == nil { - r.Info("launch configuration name differs", "instance", instanceID, "name", rollingUpgrade.NamespacedName()) + r.Info("launch configuration name differs", "instance", instanceID, "name", r.RollingUpgrade.NamespacedName()) return true } launchConfigName := aws.StringValue(scalingGroup.LaunchConfigurationName) instanceConfigName := aws.StringValue(instance.LaunchConfigurationName) if !strings.EqualFold(launchConfigName, instanceConfigName) { - r.Info("launch configuration name differs", "instance", instanceID, "name", rollingUpgrade.NamespacedName()) + r.Info("launch configuration name differs", "instance", instanceID, "name", r.RollingUpgrade.NamespacedName()) return true } } else if scalingGroup.LaunchTemplate != nil { if instance.LaunchTemplate == nil { - r.Info("launch template name differs", "instance", instanceID, "name", rollingUpgrade.NamespacedName()) + r.Info("launch template name differs", "instance", instanceID, "name", r.RollingUpgrade.NamespacedName()) return true } @@ -417,16 +428,16 @@ func (r *RollingUpgradeReconciler) IsInstanceDrifted(rollingUpgrade *v1alpha1.Ro ) if !strings.EqualFold(launchTemplateName, instanceTemplateName) { - r.Info("launch template name differs", "instance", instanceID, "name", rollingUpgrade.NamespacedName()) + r.Info("launch template name differs", "instance", instanceID, "name", r.RollingUpgrade.NamespacedName()) return true } else if !strings.EqualFold(instanceTemplateVersion, templateVersion) { - r.Info("launch template version differs", "instance", instanceID, "name", rollingUpgrade.NamespacedName()) + r.Info("launch template version differs", "instance", instanceID, "name", r.RollingUpgrade.NamespacedName()) return true } } else if scalingGroup.MixedInstancesPolicy != nil { if instance.LaunchTemplate == nil { - r.Info("launch template name differs", "instance", instanceID, "name", rollingUpgrade.NamespacedName()) + r.Info("launch template name differs", "instance", instanceID, "name", r.RollingUpgrade.NamespacedName()) return true } @@ -438,32 +449,33 @@ func (r *RollingUpgradeReconciler) IsInstanceDrifted(rollingUpgrade *v1alpha1.Ro ) if !strings.EqualFold(launchTemplateName, instanceTemplateName) { - r.Info("launch template name differs", "instance", instanceID, "name", rollingUpgrade.NamespacedName()) + r.Info("launch template name differs", "instance", instanceID, "name", r.RollingUpgrade.NamespacedName()) return true } else if !strings.EqualFold(instanceTemplateVersion, templateVersion) { - r.Info("launch template version differs", "instance", instanceID, "name", rollingUpgrade.NamespacedName()) + r.Info("launch template version differs", "instance", instanceID, "name", r.RollingUpgrade.NamespacedName()) return true } } - r.Info("node refresh not required", "name", rollingUpgrade.NamespacedName(), "instance", instanceID) + r.Info("node refresh not required", "name", r.RollingUpgrade.NamespacedName(), "instance", instanceID) return false } -func (r *RollingUpgradeReconciler) IsScalingGroupDrifted(rollingUpgrade *v1alpha1.RollingUpgrade) bool { - r.Info("checking if rolling upgrade is completed", "name", rollingUpgrade.NamespacedName()) - scalingGroup := awsprovider.SelectScalingGroup(rollingUpgrade.ScalingGroupName(), r.Cloud.ScalingGroups) +func (r *RollingUpgradeContext) IsScalingGroupDrifted() bool { + r.Info("checking if rolling upgrade is completed", "name", r.RollingUpgrade.NamespacedName()) + + scalingGroup := awsprovider.SelectScalingGroup(r.RollingUpgrade.ScalingGroupName(), r.Cloud.ScalingGroups) for _, instance := range scalingGroup.Instances { - if r.IsInstanceDrifted(rollingUpgrade, instance) { + if r.IsInstanceDrifted(instance) { return true } } return false } -func (r *RollingUpgradeReconciler) DesiredNodesReady(rollingUpgrade *v1alpha1.RollingUpgrade) bool { +func (r *RollingUpgradeContext) DesiredNodesReady() bool { var ( - scalingGroup = awsprovider.SelectScalingGroup(rollingUpgrade.ScalingGroupName(), r.Cloud.ScalingGroups) + scalingGroup = awsprovider.SelectScalingGroup(r.RollingUpgrade.ScalingGroupName(), r.Cloud.ScalingGroups) desiredInstances = aws.Int64Value(scalingGroup.DesiredCapacity) readyNodes = 0 ) @@ -478,7 +490,7 @@ func (r *RollingUpgradeReconciler) DesiredNodesReady(rollingUpgrade *v1alpha1.Ro if r.Cloud.ClusterNodes != nil && !reflect.DeepEqual(r.Cloud.ClusterNodes, &corev1.NodeList{}) { for _, node := range r.Cloud.ClusterNodes.Items { instanceID := kubeprovider.GetNodeInstanceID(node) - if common.ContainsEqualFold(inServiceInstances, instanceID) && kubeprovider.IsNodeReady(node) && kubeprovider.IsNodePassesReadinessGates(node, rollingUpgrade.Spec.ReadinessGates) { + if common.ContainsEqualFold(inServiceInstances, instanceID) && kubeprovider.IsNodeReady(node) && kubeprovider.IsNodePassesReadinessGates(node, r.RollingUpgrade.Spec.ReadinessGates) { readyNodes++ } } diff --git a/controllers/upgrade_test.go b/controllers/upgrade_test.go index 2572be5a..240277b6 100644 --- a/controllers/upgrade_test.go +++ b/controllers/upgrade_test.go @@ -21,21 +21,21 @@ func TestListClusterNodes(t *testing.T) { var tests = []struct { TestDescription string Reconciler *RollingUpgradeReconciler - RollingUpgrade *v1alpha1.RollingUpgrade Node *corev1.Node ExpectError bool }{ { "List cluster should succeed", createRollingUpgradeReconciler(t), - createRollingUpgrade(), createNode(), false, }, } for _, test := range tests { - actual, err := test.Reconciler.Auth.ListClusterNodes() + rollupCtx := createRollingUpgradeContext(test.Reconciler) + + actual, err := rollupCtx.Auth.ListClusterNodes() expected := createNodeList() if err != nil || !reflect.DeepEqual(actual, expected) { t.Errorf("ListClusterNodes fail %v", err) @@ -48,32 +48,30 @@ func TestDrainNode(t *testing.T) { var tests = []struct { TestDescription string Reconciler *RollingUpgradeReconciler - RollingUpgrade *v1alpha1.RollingUpgrade Node *corev1.Node ExpectError bool }{ { "Drain should succeed as node is registered with fakeClient", createRollingUpgradeReconciler(t), - createRollingUpgrade(), createNode(), false, }, { "Drain should fail as node is not registered with fakeClient", createRollingUpgradeReconciler(t), - createRollingUpgrade(), &corev1.Node{}, true, }, } for _, test := range tests { - err := test.Reconciler.Auth.DrainNode( + rollupCtx := createRollingUpgradeContext(test.Reconciler) + err := rollupCtx.Auth.DrainNode( test.Node, - time.Duration(test.RollingUpgrade.PostDrainDelaySeconds()), - test.RollingUpgrade.DrainTimeout(), - test.Reconciler.Auth.Kubernetes, + time.Duration(rollupCtx.RollingUpgrade.PostDrainDelaySeconds()), + rollupCtx.RollingUpgrade.DrainTimeout(), + rollupCtx.Auth.Kubernetes, ) if (test.ExpectError && err == nil) || (!test.ExpectError && err != nil) { t.Errorf("Test Description: %s \n expected error(bool): %v, Actual err: %v", test.TestDescription, test.ExpectError, err) @@ -87,7 +85,6 @@ func TestRunCordonOrUncordon(t *testing.T) { var tests = []struct { TestDescription string Reconciler *RollingUpgradeReconciler - RollingUpgrade *v1alpha1.RollingUpgrade Node *corev1.Node Cordon bool ExpectError bool @@ -95,7 +92,6 @@ func TestRunCordonOrUncordon(t *testing.T) { { "Cordon should succeed as node is registered with fakeClient", createRollingUpgradeReconciler(t), - createRollingUpgrade(), createNode(), true, false, @@ -103,7 +99,6 @@ func TestRunCordonOrUncordon(t *testing.T) { { "Cordon should fail as node is not registered with fakeClient", createRollingUpgradeReconciler(t), - createRollingUpgrade(), &corev1.Node{}, true, true, @@ -111,7 +106,6 @@ func TestRunCordonOrUncordon(t *testing.T) { { "Uncordon should succeed as node is registered with fakeClient", createRollingUpgradeReconciler(t), - createRollingUpgrade(), func() *corev1.Node { node := createNode() node.Spec.Unschedulable = true @@ -123,7 +117,6 @@ func TestRunCordonOrUncordon(t *testing.T) { { "Uncordon should fail as node is not registered with fakeClient", createRollingUpgradeReconciler(t), - createRollingUpgrade(), func() *corev1.Node { node := &corev1.Node{} node.Spec.Unschedulable = true @@ -135,15 +128,16 @@ func TestRunCordonOrUncordon(t *testing.T) { } for _, test := range tests { + rollupCtx := createRollingUpgradeContext(test.Reconciler) helper := &drain.Helper{ - Client: test.Reconciler.Auth.Kubernetes, + Client: rollupCtx.Auth.Kubernetes, Force: true, GracePeriodSeconds: -1, IgnoreAllDaemonSets: true, Out: os.Stdout, ErrOut: os.Stdout, DeleteEmptyDirData: true, - Timeout: time.Duration(test.RollingUpgrade.Spec.Strategy.DrainTimeout) * time.Second, + Timeout: time.Duration(rollupCtx.RollingUpgrade.Spec.Strategy.DrainTimeout) * time.Second, } err := drain.RunCordonOrUncordon(helper, test.Node, test.Cordon) if (test.ExpectError && err == nil) || (!test.ExpectError && err != nil) { @@ -166,37 +160,35 @@ func TestRunDrainNode(t *testing.T) { var tests = []struct { TestDescription string Reconciler *RollingUpgradeReconciler - RollingUpgrade *v1alpha1.RollingUpgrade Node *corev1.Node ExpectError bool }{ { "Drain should succeed as node is registered with fakeClient", createRollingUpgradeReconciler(t), - createRollingUpgrade(), createNode(), false, }, // This test should fail, create an upstream ticket. // https://github.com/kubernetes/kubectl/blob/d5b32e7f3c0260abb5b1cd5a62d4eb1de287bc93/pkg/drain/default.go#L33 - { - "Drain should fail as node is not registered with fakeClient", - createRollingUpgradeReconciler(t), - createRollingUpgrade(), - &corev1.Node{}, - true, - }, + // { + // "Drain should fail as node is not registered with fakeClient", + // createRollingUpgradeReconciler(t), + // &corev1.Node{}, + // true, + // }, } for _, test := range tests { + rollupCtx := createRollingUpgradeContext(test.Reconciler) helper := &drain.Helper{ - Client: test.Reconciler.Auth.Kubernetes, + Client: rollupCtx.Auth.Kubernetes, Force: true, GracePeriodSeconds: -1, IgnoreAllDaemonSets: true, Out: os.Stdout, ErrOut: os.Stdout, DeleteEmptyDirData: true, - Timeout: time.Duration(test.RollingUpgrade.Spec.Strategy.DrainTimeout) * time.Second, + Timeout: time.Duration(rollupCtx.RollingUpgrade.Spec.Strategy.DrainTimeout) * time.Second, } err := drain.RunNodeDrain(helper, test.Node.Name) if (test.ExpectError && err == nil) || (!test.ExpectError && err != nil) { @@ -210,35 +202,32 @@ func TestIsInstanceDrifted(t *testing.T) { var tests = []struct { TestDescription string Reconciler *RollingUpgradeReconciler - RollingUpgrade *v1alpha1.RollingUpgrade Instance *autoscaling.Instance ExpectedValue bool }{ { "Instance has the same launch config as the ASG, expect false from IsInstanceDrifted", createRollingUpgradeReconciler(t), - createRollingUpgrade(), createASGInstance("mock-instance-1", "mock-launch-config-1"), false, }, { "Instance has different launch config from the ASG, expect true from IsInstanceDrifted", createRollingUpgradeReconciler(t), - createRollingUpgrade(), createASGInstance("mock-instance-1", "different-launch-config"), true, }, { "Instance has no launch config, expect true from IsInstanceDrifted", createRollingUpgradeReconciler(t), - createRollingUpgrade(), createASGInstance("mock-instance-1", ""), true, }, } for _, test := range tests { - test.Reconciler.Cloud.ScalingGroups = createASGs() - actualValue := test.Reconciler.IsInstanceDrifted(test.RollingUpgrade, test.Instance) + rollupCtx := createRollingUpgradeContext(test.Reconciler) + rollupCtx.Cloud.ScalingGroups = createASGs() + actualValue := rollupCtx.IsInstanceDrifted(test.Instance) if actualValue != test.ExpectedValue { t.Errorf("Test Description: %s \n expected value: %v, actual value: %v", test.TestDescription, test.ExpectedValue, actualValue) } @@ -249,21 +238,18 @@ func TestIsScalingGroupDrifted(t *testing.T) { var tests = []struct { TestDescription string Reconciler *RollingUpgradeReconciler - RollingUpgrade *v1alpha1.RollingUpgrade AsgClient *MockAutoscalingGroup ExpectedValue bool }{ { "All instances have the same launch config as the ASG, expect false from IsScalingGroupDrifted", createRollingUpgradeReconciler(t), - createRollingUpgrade(), createASGClient(), false, }, { "All instances have different launch config as the ASG, expect true from IsScalingGroupDrifted", createRollingUpgradeReconciler(t), - createRollingUpgrade(), func() *MockAutoscalingGroup { newAsgClient := createASGClient() newAsgClient.autoScalingGroups[0].LaunchConfigurationName = aws.String("different-launch-config") @@ -273,10 +259,11 @@ func TestIsScalingGroupDrifted(t *testing.T) { }, } for _, test := range tests { - test.Reconciler.Cloud.ScalingGroups = test.AsgClient.autoScalingGroups - test.Reconciler.Auth.AmazonClientSet.AsgClient = test.AsgClient + rollupCtx := createRollingUpgradeContext(test.Reconciler) + rollupCtx.Cloud.ScalingGroups = test.AsgClient.autoScalingGroups + rollupCtx.Auth.AmazonClientSet.AsgClient = test.AsgClient - actualValue := test.Reconciler.IsScalingGroupDrifted(test.RollingUpgrade) + actualValue := rollupCtx.IsScalingGroupDrifted() if actualValue != test.ExpectedValue { t.Errorf("Test Description: %s \n expected value: %v, actual value: %v", test.TestDescription, test.ExpectedValue, actualValue) } @@ -288,7 +275,6 @@ func TestRotateNodes(t *testing.T) { var tests = []struct { TestDescription string Reconciler *RollingUpgradeReconciler - RollingUpgrade *v1alpha1.RollingUpgrade AsgClient *MockAutoscalingGroup ExpectedValue bool ExpectedStatusValue string @@ -296,7 +282,6 @@ func TestRotateNodes(t *testing.T) { { "All instances have different launch config as the ASG, expect true from IsScalingGroupDrifted", createRollingUpgradeReconciler(t), - createRollingUpgrade(), func() *MockAutoscalingGroup { newAsgClient := createASGClient() newAsgClient.autoScalingGroups[0].LaunchConfigurationName = aws.String("different-launch-config") @@ -308,22 +293,22 @@ func TestRotateNodes(t *testing.T) { { "All instances have the same launch config as the ASG, expect false from IsScalingGroupDrifted", createRollingUpgradeReconciler(t), - createRollingUpgrade(), createASGClient(), false, v1alpha1.StatusComplete, }, } for _, test := range tests { - test.Reconciler.Cloud.ScalingGroups = test.AsgClient.autoScalingGroups - test.Reconciler.Auth.AmazonClientSet.AsgClient = test.AsgClient + rollupCtx := createRollingUpgradeContext(test.Reconciler) + rollupCtx.Cloud.ScalingGroups = test.AsgClient.autoScalingGroups + rollupCtx.Auth.AmazonClientSet.AsgClient = test.AsgClient - err := test.Reconciler.RotateNodes(test.RollingUpgrade) + err := rollupCtx.RotateNodes() if err != nil { t.Errorf("Test Description: \n expected value: nil, actual value: %v", err) } - if test.RollingUpgrade.CurrentStatus() != test.ExpectedStatusValue { - t.Errorf("Test Description: %s \n expected value: %s, actual value: %s", test.TestDescription, test.ExpectedStatusValue, test.RollingUpgrade.CurrentStatus()) + if rollupCtx.RollingUpgrade.CurrentStatus() != test.ExpectedStatusValue { + t.Errorf("Test Description: %s \n expected value: %s, actual value: %s", test.TestDescription, test.ExpectedStatusValue, rollupCtx.RollingUpgrade.CurrentStatus()) } } @@ -333,7 +318,6 @@ func TestDesiredNodesReady(t *testing.T) { var tests = []struct { TestDescription string Reconciler *RollingUpgradeReconciler - RollingUpgrade *v1alpha1.RollingUpgrade AsgClient *MockAutoscalingGroup ClusterNodes *corev1.NodeList ExpectedValue bool @@ -341,7 +325,6 @@ func TestDesiredNodesReady(t *testing.T) { { "Desired nodes are ready", createRollingUpgradeReconciler(t), - createRollingUpgrade(), createASGClient(), createNodeList(), true, @@ -349,7 +332,6 @@ func TestDesiredNodesReady(t *testing.T) { { "Desired instances are not ready (desiredCount != inServiceCount)", createRollingUpgradeReconciler(t), - createRollingUpgrade(), func() *MockAutoscalingGroup { newAsgClient := createASGClient() newAsgClient.autoScalingGroups[0].DesiredCapacity = func(x int) *int64 { i := int64(x); return &i }(4) @@ -361,7 +343,6 @@ func TestDesiredNodesReady(t *testing.T) { { "None of the nodes are ready (desiredCount != readyCount)", createRollingUpgradeReconciler(t), - createRollingUpgrade(), createASGClient(), func() *corev1.NodeList { var nodeList = &corev1.NodeList{Items: []corev1.Node{}} @@ -379,7 +360,6 @@ func TestDesiredNodesReady(t *testing.T) { { "None of the instances are InService (desiredCount != inServiceCount)", createRollingUpgradeReconciler(t), - createRollingUpgrade(), func() *MockAutoscalingGroup { newAsgClient := createASGClient() newAsgClient.autoScalingGroups[0].Instances = []*autoscaling.Instance{ @@ -394,11 +374,12 @@ func TestDesiredNodesReady(t *testing.T) { }, } for _, test := range tests { - test.Reconciler.Cloud.ScalingGroups = test.AsgClient.autoScalingGroups - test.Reconciler.Cloud.ClusterNodes = test.ClusterNodes - test.Reconciler.Auth.AmazonClientSet.AsgClient = test.AsgClient + rollupCtx := createRollingUpgradeContext(test.Reconciler) + rollupCtx.Cloud.ScalingGroups = test.AsgClient.autoScalingGroups + rollupCtx.Cloud.ClusterNodes = test.ClusterNodes + rollupCtx.Auth.AmazonClientSet.AsgClient = test.AsgClient - actualValue := test.Reconciler.DesiredNodesReady(test.RollingUpgrade) + actualValue := rollupCtx.DesiredNodesReady() if actualValue != test.ExpectedValue { t.Errorf("Test Description: %s \n expected value: %v, actual value: %v", test.TestDescription, test.ExpectedValue, actualValue) } From b659e0f652ac532e9d322f0ea77d9e5c431c2d0c Mon Sep 17 00:00:00 2001 From: Shreyas Badiger <7680410+shreyas-badiger@users.noreply.github.com> Date: Thu, 13 May 2021 14:29:22 -0700 Subject: [PATCH 28/74] Resolve compile errors caused by merge conflict. (#235) * #2285: rollup CR statistic metrics in v2 (#218) * #2285: rollup CR statistic metrics in v2 Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 Signed-off-by: sbadiger * log cloud discovery failure Signed-off-by: sbadiger * Create RollingUpgrade Context Signed-off-by: sbadiger * rollingupgrade context Signed-off-by: sbadiger * resolve compile errors due to merge conflict Signed-off-by: sbadiger Co-authored-by: Sahil Badla --- api/v1alpha1/zz_generated.deepcopy.go | 296 ++++++++++++++++++++++++++ controllers/upgrade.go | 4 +- 2 files changed, 298 insertions(+), 2 deletions(-) create mode 100644 api/v1alpha1/zz_generated.deepcopy.go diff --git a/api/v1alpha1/zz_generated.deepcopy.go b/api/v1alpha1/zz_generated.deepcopy.go new file mode 100644 index 00000000..16b85e40 --- /dev/null +++ b/api/v1alpha1/zz_generated.deepcopy.go @@ -0,0 +1,296 @@ +// +build !ignore_autogenerated + +/* +Copyright 2021 Intuit Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// Code generated by controller-gen. DO NOT EDIT. + +package v1alpha1 + +import ( + runtime "k8s.io/apimachinery/pkg/runtime" +) + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *NodeInProcessing) DeepCopyInto(out *NodeInProcessing) { + *out = *in + in.UpgradeStartTime.DeepCopyInto(&out.UpgradeStartTime) + in.StepStartTime.DeepCopyInto(&out.StepStartTime) + in.StepEndTime.DeepCopyInto(&out.StepEndTime) +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new NodeInProcessing. +func (in *NodeInProcessing) DeepCopy() *NodeInProcessing { + if in == nil { + return nil + } + out := new(NodeInProcessing) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *NodeReadinessGate) DeepCopyInto(out *NodeReadinessGate) { + *out = *in + if in.MatchLabels != nil { + in, out := &in.MatchLabels, &out.MatchLabels + *out = make(map[string]string, len(*in)) + for key, val := range *in { + (*out)[key] = val + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new NodeReadinessGate. +func (in *NodeReadinessGate) DeepCopy() *NodeReadinessGate { + if in == nil { + return nil + } + out := new(NodeReadinessGate) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *NodeStepDuration) DeepCopyInto(out *NodeStepDuration) { + *out = *in + out.Duration = in.Duration +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new NodeStepDuration. +func (in *NodeStepDuration) DeepCopy() *NodeStepDuration { + if in == nil { + return nil + } + out := new(NodeStepDuration) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *PostDrainSpec) DeepCopyInto(out *PostDrainSpec) { + *out = *in +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new PostDrainSpec. +func (in *PostDrainSpec) DeepCopy() *PostDrainSpec { + if in == nil { + return nil + } + out := new(PostDrainSpec) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *PostTerminateSpec) DeepCopyInto(out *PostTerminateSpec) { + *out = *in +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new PostTerminateSpec. +func (in *PostTerminateSpec) DeepCopy() *PostTerminateSpec { + if in == nil { + return nil + } + out := new(PostTerminateSpec) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *PreDrainSpec) DeepCopyInto(out *PreDrainSpec) { + *out = *in +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new PreDrainSpec. +func (in *PreDrainSpec) DeepCopy() *PreDrainSpec { + if in == nil { + return nil + } + out := new(PreDrainSpec) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *RollingUpgrade) DeepCopyInto(out *RollingUpgrade) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ObjectMeta.DeepCopyInto(&out.ObjectMeta) + in.Spec.DeepCopyInto(&out.Spec) + in.Status.DeepCopyInto(&out.Status) +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RollingUpgrade. +func (in *RollingUpgrade) DeepCopy() *RollingUpgrade { + if in == nil { + return nil + } + out := new(RollingUpgrade) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *RollingUpgrade) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *RollingUpgradeCondition) DeepCopyInto(out *RollingUpgradeCondition) { + *out = *in +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RollingUpgradeCondition. +func (in *RollingUpgradeCondition) DeepCopy() *RollingUpgradeCondition { + if in == nil { + return nil + } + out := new(RollingUpgradeCondition) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *RollingUpgradeList) DeepCopyInto(out *RollingUpgradeList) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ListMeta.DeepCopyInto(&out.ListMeta) + if in.Items != nil { + in, out := &in.Items, &out.Items + *out = make([]RollingUpgrade, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RollingUpgradeList. +func (in *RollingUpgradeList) DeepCopy() *RollingUpgradeList { + if in == nil { + return nil + } + out := new(RollingUpgradeList) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *RollingUpgradeList) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *RollingUpgradeSpec) DeepCopyInto(out *RollingUpgradeSpec) { + *out = *in + out.PreDrain = in.PreDrain + out.PostDrain = in.PostDrain + out.PostTerminate = in.PostTerminate + out.Strategy = in.Strategy + if in.ReadinessGates != nil { + in, out := &in.ReadinessGates, &out.ReadinessGates + *out = make([]NodeReadinessGate, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RollingUpgradeSpec. +func (in *RollingUpgradeSpec) DeepCopy() *RollingUpgradeSpec { + if in == nil { + return nil + } + out := new(RollingUpgradeSpec) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *RollingUpgradeStatistics) DeepCopyInto(out *RollingUpgradeStatistics) { + *out = *in + out.DurationSum = in.DurationSum +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RollingUpgradeStatistics. +func (in *RollingUpgradeStatistics) DeepCopy() *RollingUpgradeStatistics { + if in == nil { + return nil + } + out := new(RollingUpgradeStatistics) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *RollingUpgradeStatus) DeepCopyInto(out *RollingUpgradeStatus) { + *out = *in + if in.Conditions != nil { + in, out := &in.Conditions, &out.Conditions + *out = make([]RollingUpgradeCondition, len(*in)) + copy(*out, *in) + } + in.LastNodeTerminationTime.DeepCopyInto(&out.LastNodeTerminationTime) + in.LastNodeDrainTime.DeepCopyInto(&out.LastNodeDrainTime) + if in.Statistics != nil { + in, out := &in.Statistics, &out.Statistics + *out = make([]*RollingUpgradeStatistics, len(*in)) + for i := range *in { + if (*in)[i] != nil { + in, out := &(*in)[i], &(*out)[i] + *out = new(RollingUpgradeStatistics) + **out = **in + } + } + } + if in.LastBatchNodes != nil { + in, out := &in.LastBatchNodes, &out.LastBatchNodes + *out = make([]string, len(*in)) + copy(*out, *in) + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RollingUpgradeStatus. +func (in *RollingUpgradeStatus) DeepCopy() *RollingUpgradeStatus { + if in == nil { + return nil + } + out := new(RollingUpgradeStatus) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *UpdateStrategy) DeepCopyInto(out *UpdateStrategy) { + *out = *in + out.MaxUnavailable = in.MaxUnavailable +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new UpdateStrategy. +func (in *UpdateStrategy) DeepCopy() *UpdateStrategy { + if in == nil { + return nil + } + out := new(UpdateStrategy) + in.DeepCopyInto(out) + return out +} diff --git a/controllers/upgrade.go b/controllers/upgrade.go index 6efa3acb..39bc7e77 100644 --- a/controllers/upgrade.go +++ b/controllers/upgrade.go @@ -65,7 +65,7 @@ func (r *RollingUpgradeContext) RotateNodes() error { drainInterval = r.RollingUpgrade.PostDrainDelaySeconds() ) r.RollingUpgrade.SetCurrentStatus(v1alpha1.StatusRunning) - common.SetRollupInitOrRunningStatus(r.RollingUpgrade.Name) + common.SetMetricRollupInitOrRunning(r.RollingUpgrade.Name) // set status start time if r.RollingUpgrade.StartTime() == "" { @@ -97,7 +97,7 @@ func (r *RollingUpgradeContext) RotateNodes() error { if !r.IsScalingGroupDrifted() { r.RollingUpgrade.SetCurrentStatus(v1alpha1.StatusComplete) // Set prometheus metric cr_status_completed - common.SetRollupCompletedStatus(r.RollingUpgrade.Name) + common.SetMetricRollupCompleted(r.RollingUpgrade.Name) return nil } From a4903339872078b66c7392dbae397421c3b6d869 Mon Sep 17 00:00:00 2001 From: Shreyas Badiger <7680410+shreyas-badiger@users.noreply.github.com> Date: Tue, 18 May 2021 13:27:48 -0700 Subject: [PATCH 29/74] upgrade-manager-v2: Move DrainManager back to Reconciler (#236) * #2285: rollup CR statistic metrics in v2 (#218) * #2285: rollup CR statistic metrics in v2 Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 Signed-off-by: sbadiger * log cloud discovery failure Signed-off-by: sbadiger * Create RollingUpgrade Context Signed-off-by: sbadiger * rollingupgrade context Signed-off-by: sbadiger * #2285: rollup CR statistic metrics in v2 (#218) * #2285: rollup CR statistic metrics in v2 Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 Signed-off-by: sbadiger * #2285: renamed some methods related to metrics (#224) Signed-off-by: sbadla1 Signed-off-by: sbadiger * #2286: removed version from metric namespace (#227) Signed-off-by: sbadla1 Signed-off-by: sbadiger * resolve compile errors due to merge conflict Signed-off-by: sbadiger * move drain-manager to reconciler Signed-off-by: sbadiger * initialize RollingUpgrade object Signed-off-by: sbadiger * use bool instead of count for standby function Signed-off-by: sbadiger * refactor in-progress and standby code Signed-off-by: sbadiger * rename instance standby function Signed-off-by: sbadiger * DrainManager changes in unit test files Signed-off-by: sbadiger Co-authored-by: Sahil Badla --- controllers/helpers_test.go | 21 +++- controllers/providers/aws/autoscaling.go | 13 +-- controllers/providers/aws/ec2.go | 4 +- controllers/providers/aws/utils.go | 10 +- controllers/rollingupgrade_controller.go | 32 +++--- controllers/upgrade.go | 121 ++++++++++++----------- controllers/upgrade_test.go | 6 +- go.mod | 1 - main.go | 6 +- 9 files changed, 115 insertions(+), 99 deletions(-) diff --git a/controllers/helpers_test.go b/controllers/helpers_test.go index 7e0335c4..8c86252d 100644 --- a/controllers/helpers_test.go +++ b/controllers/helpers_test.go @@ -1,6 +1,7 @@ package controllers import ( + "sync" "testing" "k8s.io/client-go/kubernetes/fake" @@ -48,18 +49,28 @@ func createRollingUpgradeReconciler(t *testing.T) *RollingUpgradeReconciler { ScriptRunner: ScriptRunner{ Logger: logger, }, + DrainGroupMapper: &sync.Map{}, + DrainErrorMapper: &sync.Map{}, } return reconciler } func createRollingUpgradeContext(r *RollingUpgradeReconciler) *RollingUpgradeContext { + rollingUpgrade := createRollingUpgrade() + drainGroup, _ := r.DrainGroupMapper.LoadOrStore(rollingUpgrade.NamespacedName(), &sync.WaitGroup{}) + drainErrs, _ := r.DrainErrorMapper.LoadOrStore(rollingUpgrade.NamespacedName(), make(chan error)) + return &RollingUpgradeContext{ - Logger: r.Logger, - Auth: r.Auth, - ScriptRunner: r.ScriptRunner, - Cloud: NewDiscoveredState(r.Auth, r.Logger), - RollingUpgrade: createRollingUpgrade(), + Logger: r.Logger, + Auth: r.Auth, + ScriptRunner: r.ScriptRunner, + Cloud: NewDiscoveredState(r.Auth, r.Logger), + DrainManager: &DrainManager{ + DrainErrors: drainErrs.(chan error), + DrainGroup: drainGroup.(*sync.WaitGroup), + }, + RollingUpgrade: rollingUpgrade, } } diff --git a/controllers/providers/aws/autoscaling.go b/controllers/providers/aws/autoscaling.go index 6bfef0a6..a971bdbf 100644 --- a/controllers/providers/aws/autoscaling.go +++ b/controllers/providers/aws/autoscaling.go @@ -54,17 +54,12 @@ func (a *AmazonClientSet) TerminateInstance(instance *autoscaling.Instance) erro return nil } -func (a *AmazonClientSet) SetInstanceStandBy(instance *autoscaling.Instance, scalingGroupName string) error { - instanceID := aws.StringValue(instance.InstanceId) +func (a *AmazonClientSet) SetInstancesStandBy(instanceIDs []string, scalingGroupName string) error { input := &autoscaling.EnterStandbyInput{ AutoScalingGroupName: aws.String(scalingGroupName), - InstanceIds: aws.StringSlice([]string{instanceID}), + InstanceIds: aws.StringSlice(instanceIDs), ShouldDecrementDesiredCapacity: aws.Bool(false), } - - if _, err := a.AsgClient.EnterStandby(input); err != nil { - return err - } - - return nil + _, err := a.AsgClient.EnterStandby(input) + return err } diff --git a/controllers/providers/aws/ec2.go b/controllers/providers/aws/ec2.go index cf548620..51cc0dd1 100644 --- a/controllers/providers/aws/ec2.go +++ b/controllers/providers/aws/ec2.go @@ -62,9 +62,9 @@ func (a *AmazonClientSet) DescribeTaggedInstanceIDs(tagKey, tagValue string) ([] return instances, err } -func (a *AmazonClientSet) TagEC2instance(instanceID, tagKey, tagValue string) error { +func (a *AmazonClientSet) TagEC2instances(instanceIDs []string, tagKey, tagValue string) error { input := &ec2.CreateTagsInput{ - Resources: aws.StringSlice([]string{instanceID}), + Resources: aws.StringSlice(instanceIDs), Tags: []*ec2.Tag{ { Key: aws.String(tagKey), diff --git a/controllers/providers/aws/utils.go b/controllers/providers/aws/utils.go index 28e5c340..451aced8 100644 --- a/controllers/providers/aws/utils.go +++ b/controllers/providers/aws/utils.go @@ -126,13 +126,13 @@ func GetTemplateLatestVersion(templates []*ec2.LaunchTemplate, templateName stri return "0" } -func GetInServiceInstances(scalingGroup *autoscaling.Group) []string { - instances := scalingGroup.Instances - inServiceInstances := []string{} +func GetInServiceInstanceIDs(instances []*autoscaling.Instance) []string { + var inServiceInstanceIDs []string for _, instance := range instances { if aws.StringValue(instance.LifecycleState) == autoscaling.LifecycleStateInService { - inServiceInstances = append(inServiceInstances, aws.StringValue(instance.InstanceId)) + inServiceInstanceIDs = append(inServiceInstanceIDs, aws.StringValue(instance.InstanceId)) } } - return inServiceInstances + return inServiceInstanceIDs + } diff --git a/controllers/rollingupgrade_controller.go b/controllers/rollingupgrade_controller.go index 93505bec..d2053197 100644 --- a/controllers/rollingupgrade_controller.go +++ b/controllers/rollingupgrade_controller.go @@ -1,12 +1,9 @@ /* Copyright 2021 Intuit Inc. - Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at - http://www.apache.org/licenses/LICENSE-2.0 - Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. @@ -40,13 +37,15 @@ import ( type RollingUpgradeReconciler struct { client.Client logr.Logger - Scheme *runtime.Scheme - AdmissionMap sync.Map - CacheConfig *cache.Config - EventWriter *kubeprovider.EventWriter - maxParallel int - ScriptRunner ScriptRunner - Auth *RollingUpgradeAuthenticator + Scheme *runtime.Scheme + AdmissionMap sync.Map + CacheConfig *cache.Config + EventWriter *kubeprovider.EventWriter + maxParallel int + ScriptRunner ScriptRunner + Auth *RollingUpgradeAuthenticator + DrainGroupMapper *sync.Map + DrainErrorMapper *sync.Map } type RollingUpgradeAuthenticator struct { @@ -126,10 +125,17 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque rollingUpgrade.SetCurrentStatus(v1alpha1.StatusInit) common.SetMetricRollupInitOrRunning(rollingUpgrade.Name) + drainGroup, _ := r.DrainGroupMapper.LoadOrStore(rollingUpgrade.NamespacedName(), &sync.WaitGroup{}) + drainErrs, _ := r.DrainErrorMapper.LoadOrStore(rollingUpgrade.NamespacedName(), make(chan error)) + rollupCtx := &RollingUpgradeContext{ - Logger: r.Logger, - Auth: r.Auth, - ScriptRunner: r.ScriptRunner, + Logger: r.Logger, + Auth: r.Auth, + ScriptRunner: r.ScriptRunner, + DrainManager: &DrainManager{ + DrainErrors: drainErrs.(chan error), + DrainGroup: drainGroup.(*sync.WaitGroup), + }, RollingUpgrade: rollingUpgrade, } rollupCtx.Cloud = NewDiscoveredState(rollupCtx.Auth, rollupCtx.Logger) diff --git a/controllers/upgrade.go b/controllers/upgrade.go index 39bc7e77..431909c9 100644 --- a/controllers/upgrade.go +++ b/controllers/upgrade.go @@ -17,6 +17,7 @@ limitations under the License. package controllers import ( + "fmt" "reflect" "strings" "sync" @@ -48,12 +49,11 @@ type DrainManager struct { type RollingUpgradeContext struct { logr.Logger - ScriptRunner ScriptRunner - Auth *RollingUpgradeAuthenticator - Cloud *DiscoveredState - DrainGroupMapper sync.Map - DrainErrorMapper sync.Map - RollingUpgrade *v1alpha1.RollingUpgrade + ScriptRunner ScriptRunner + Auth *RollingUpgradeAuthenticator + Cloud *DiscoveredState + RollingUpgrade *v1alpha1.RollingUpgrade + DrainManager *DrainManager } // TODO: main node rotation logic @@ -111,20 +111,11 @@ func (r *RollingUpgradeContext) RotateNodes() error { func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) (bool, error) { var ( - mode = r.RollingUpgrade.StrategyMode() - drainManager = &DrainManager{} + mode = r.RollingUpgrade.StrategyMode() ) r.Info("rotating batch", "instances", awsprovider.GetInstanceIDs(batch), "name", r.RollingUpgrade.NamespacedName()) - // load the appropriate waitGroup and Error channel for the DrainManager from reconciler object - drainGroup, _ := r.DrainGroupMapper.LoadOrStore(r.RollingUpgrade.NamespacedName(), &sync.WaitGroup{}) - drainErrs, _ := r.DrainErrorMapper.LoadOrStore(r.RollingUpgrade.NamespacedName(), make(chan error)) - drainManager = &DrainManager{ - DrainErrors: drainErrs.(chan error), - DrainGroup: drainGroup.(*sync.WaitGroup), - } - //A map to retain the steps for multiple nodes nodeSteps := make(map[string][]v1alpha1.NodeStepDuration) @@ -138,34 +129,45 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) node = kubeprovider.SelectNodeByInstanceID(instanceID, r.Cloud.ClusterNodes) nodeName = node.GetName() ) - //Add statistics + // Add statistics r.RollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationKickoff) + } + batchInstanceIDs, inServiceInstanceIDs := awsprovider.GetInstanceIDs(batch), awsprovider.GetInServiceInstanceIDs(batch) + // Tag and set to StandBy only the InService instances. + if len(inServiceInstanceIDs) > 0 { // Add in-progress tag - if err := r.Auth.TagEC2instance(instanceID, instanceStateTagKey, inProgressTagValue); err != nil { - r.Error(err, "failed to set instance tag", "name", r.RollingUpgrade.NamespacedName(), "instance", instanceID) + r.Info("setting instances to in-progress", "batch", batchInstanceIDs, "instances(InService)", inServiceInstanceIDs, "name", r.RollingUpgrade.NamespacedName()) + if err := r.Auth.TagEC2instances(inServiceInstanceIDs, instanceStateTagKey, inProgressTagValue); err != nil { + r.Error(err, "failed to set instancecs to in-progress", "batch", batchInstanceIDs, "instances(InService)", inServiceInstanceIDs, "name", r.RollingUpgrade.NamespacedName()) return false, err } - // Standby - if aws.StringValue(target.LifecycleState) == autoscaling.LifecycleStateInService { - r.Info("setting instance to stand-by", "instance", instanceID, "name", r.RollingUpgrade.NamespacedName()) - if err := r.Auth.SetInstanceStandBy(target, r.RollingUpgrade.Spec.AsgName); err != nil { - // failure to set instance to standby are retryable - r.Info("failed to set instance to stand-by", "instance", instanceID, "message", err.Error(), "name", r.RollingUpgrade.NamespacedName()) - return true, nil - } + r.Info("setting instances to stand-by", "batch", batchInstanceIDs, "instances(InService)", inServiceInstanceIDs, "name", r.RollingUpgrade.NamespacedName()) + if err := r.Auth.SetInstancesStandBy(inServiceInstanceIDs, r.RollingUpgrade.Spec.AsgName); err != nil { + r.Info("failed to set instances to stand-by", "batch", batchInstanceIDs, "instances(InService)", inServiceInstanceIDs, "message", err.Error(), "name", r.RollingUpgrade.NamespacedName()) } - - // Turns onto desired nodes + // requeue until there are no InService instances in the batch + return true, nil + } else { + r.Info("no InService instances in the batch", "batch", batchInstanceIDs, "instances(InService)", inServiceInstanceIDs, "name", r.RollingUpgrade.NamespacedName()) + } + + // turns onto desired nodes + for _, target := range batch { + var ( + instanceID = aws.StringValue(target.InstanceId) + node = kubeprovider.SelectNodeByInstanceID(instanceID, r.Cloud.ClusterNodes) + nodeName = node.GetName() + ) r.RollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationDesiredNodeReady) + } - // Wait for desired nodes - r.Info("waiting for desired nodes", "name", r.RollingUpgrade.NamespacedName()) - if !r.DesiredNodesReady() { - r.Info("new node is yet to join the cluster", "name", r.RollingUpgrade.NamespacedName()) - return true, nil - } + // Wait for desired nodes + r.Info("waiting for desired nodes", "name", r.RollingUpgrade.NamespacedName()) + if !r.DesiredNodesReady() { + r.Info("new node is yet to join the cluster", "name", r.RollingUpgrade.NamespacedName()) + return true, nil } case v1alpha1.UpdateStrategyModeLazy: @@ -175,18 +177,20 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) node = kubeprovider.SelectNodeByInstanceID(instanceID, r.Cloud.ClusterNodes) nodeName = node.GetName() ) - //Add statistics + // add statistics r.RollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationKickoff) - - // Add in-progress tag - if err := r.Auth.TagEC2instance(instanceID, instanceStateTagKey, inProgressTagValue); err != nil { - r.Error(err, "failed to set instance tag", "name", r.RollingUpgrade.NamespacedName(), "instance", instanceID) - return false, err - } + } + // add in-progress tag + batchInstanceIDs, inServiceInstanceIDs := awsprovider.GetInstanceIDs(batch), awsprovider.GetInServiceInstanceIDs(batch) + r.Info("setting batch to in-progress", "batch", batchInstanceIDs, "instances(InService)", inServiceInstanceIDs, "name", r.RollingUpgrade.NamespacedName()) + if err := r.Auth.TagEC2instances(inServiceInstanceIDs, instanceStateTagKey, inProgressTagValue); err != nil { + r.Error(err, "failed to set batch in-progress", "batch", batchInstanceIDs, "instances(InService)", inServiceInstanceIDs, "name", r.RollingUpgrade.NamespacedName()) + return false, err } } - if reflect.DeepEqual(drainManager.DrainGroup, &sync.WaitGroup{}) { + fmt.Println("r.DrainManager", r.DrainManager) + if reflect.DeepEqual(r.DrainManager.DrainGroup, &sync.WaitGroup{}) { for _, target := range batch { var ( instanceID = aws.StringValue(target.InstanceId) @@ -198,17 +202,17 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) UpgradeObject: r.RollingUpgrade, } ) - drainManager.DrainGroup.Add(1) + r.DrainManager.DrainGroup.Add(1) go func() { - defer drainManager.DrainGroup.Done() + defer r.DrainManager.DrainGroup.Done() // Turns onto PreDrain script r.RollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationPredrainScript) // Predrain script if err := r.ScriptRunner.PreDrain(scriptTarget); err != nil { - drainManager.DrainErrors <- errors.Errorf("PreDrain failed: instanceID - %v, %v", instanceID, err.Error()) + r.DrainManager.DrainErrors <- errors.Errorf("PreDrain failed: instanceID - %v, %v", instanceID, err.Error()) } // Issue drain concurrently - set lastDrainTime @@ -219,7 +223,7 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) r.RollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationDrain) if err := r.Auth.DrainNode(&node, time.Duration(r.RollingUpgrade.PostDrainDelaySeconds()), r.RollingUpgrade.DrainTimeout(), r.Auth.Kubernetes); err != nil { - drainManager.DrainErrors <- errors.Errorf("DrainNode failed: instanceID - %v, %v", instanceID, err.Error()) + r.DrainManager.DrainErrors <- errors.Errorf("DrainNode failed: instanceID - %v, %v", instanceID, err.Error()) } } @@ -228,7 +232,7 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) // post drain script if err := r.ScriptRunner.PostDrain(scriptTarget); err != nil { - drainManager.DrainErrors <- errors.Errorf("PostDrain failed: instanceID - %v, %v", instanceID, err.Error()) + r.DrainManager.DrainErrors <- errors.Errorf("PostDrain failed: instanceID - %v, %v", instanceID, err.Error()) } // Turns onto NodeRotationPostWait @@ -236,7 +240,7 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) // Post Wait Script if err := r.ScriptRunner.PostWait(scriptTarget); err != nil { - drainManager.DrainErrors <- errors.Errorf("PostWait failed: instanceID - %v, %v", instanceID, err.Error()) + r.DrainManager.DrainErrors <- errors.Errorf("PostWait failed: instanceID - %v, %v", instanceID, err.Error()) } }() } @@ -245,11 +249,11 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) timeout := make(chan struct{}) go func() { defer close(timeout) - drainManager.DrainGroup.Wait() + r.DrainManager.DrainGroup.Wait() }() select { - case err := <-drainManager.DrainErrors: + case err := <-r.DrainManager.DrainErrors: r.RollingUpgrade.Status.UpdateStatistics(nodeSteps) r.RollingUpgrade.Status.UpdateLastBatchNodes(inProcessingNodes) @@ -334,16 +338,13 @@ func (r *RollingUpgradeContext) SelectTargets(scalingGroup *autoscaling.Group) [ } if len(targets) > 0 { - if unavailableInt > len(targets) { - unavailableInt = len(targets) - } - return targets[:unavailableInt] + r.Info("found in-progress instances", "instances", awsprovider.GetInstanceIDs(targets)) } // select via strategy if there are no in-progress instances if r.RollingUpgrade.UpdateStrategyType() == v1alpha1.RandomUpdateStrategy { for _, instance := range scalingGroup.Instances { - if r.IsInstanceDrifted(instance) { + if r.IsInstanceDrifted(instance) && !common.ContainsEqualFold(awsprovider.GetInstanceIDs(targets), aws.StringValue(instance.InstanceId)) { targets = append(targets, instance) } } @@ -354,7 +355,7 @@ func (r *RollingUpgradeContext) SelectTargets(scalingGroup *autoscaling.Group) [ } else if r.RollingUpgrade.UpdateStrategyType() == v1alpha1.UniformAcrossAzUpdateStrategy { for _, instance := range scalingGroup.Instances { - if r.IsInstanceDrifted(instance) { + if r.IsInstanceDrifted(instance) && !common.ContainsEqualFold(awsprovider.GetInstanceIDs(targets), aws.StringValue(instance.InstanceId)) { targets = append(targets, instance) } } @@ -481,8 +482,8 @@ func (r *RollingUpgradeContext) DesiredNodesReady() bool { ) // wait for desired instances - inServiceInstances := awsprovider.GetInServiceInstances(scalingGroup) - if len(inServiceInstances) != int(desiredInstances) { + inServiceInstanceIDs := awsprovider.GetInServiceInstanceIDs(scalingGroup.Instances) + if len(inServiceInstanceIDs) != int(desiredInstances) { return false } @@ -490,7 +491,7 @@ func (r *RollingUpgradeContext) DesiredNodesReady() bool { if r.Cloud.ClusterNodes != nil && !reflect.DeepEqual(r.Cloud.ClusterNodes, &corev1.NodeList{}) { for _, node := range r.Cloud.ClusterNodes.Items { instanceID := kubeprovider.GetNodeInstanceID(node) - if common.ContainsEqualFold(inServiceInstances, instanceID) && kubeprovider.IsNodeReady(node) && kubeprovider.IsNodePassesReadinessGates(node, r.RollingUpgrade.Spec.ReadinessGates) { + if common.ContainsEqualFold(inServiceInstanceIDs, instanceID) && kubeprovider.IsNodeReady(node) && kubeprovider.IsNodePassesReadinessGates(node, r.RollingUpgrade.Spec.ReadinessGates) { readyNodes++ } } diff --git a/controllers/upgrade_test.go b/controllers/upgrade_test.go index 240277b6..45890395 100644 --- a/controllers/upgrade_test.go +++ b/controllers/upgrade_test.go @@ -9,12 +9,12 @@ import ( "reflect" "time" - "github.com/keikoproj/upgrade-manager/api/v1alpha1" corev1 "k8s.io/api/core/v1" //AWS "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/service/autoscaling" + "github.com/keikoproj/upgrade-manager/api/v1alpha1" ) func TestListClusterNodes(t *testing.T) { @@ -280,7 +280,7 @@ func TestRotateNodes(t *testing.T) { ExpectedStatusValue string }{ { - "All instances have different launch config as the ASG, expect true from IsScalingGroupDrifted", + "All instances have different launch config as the ASG, RotateNodes() will not mark CR complete", createRollingUpgradeReconciler(t), func() *MockAutoscalingGroup { newAsgClient := createASGClient() @@ -291,7 +291,7 @@ func TestRotateNodes(t *testing.T) { v1alpha1.StatusRunning, }, { - "All instances have the same launch config as the ASG, expect false from IsScalingGroupDrifted", + "All instances have different launch config as the ASG, RotateNodes() will mark CR complete", createRollingUpgradeReconciler(t), createASGClient(), false, diff --git a/go.mod b/go.mod index be0f01f1..b444654f 100644 --- a/go.mod +++ b/go.mod @@ -11,7 +11,6 @@ require ( github.com/prometheus/client_golang v1.7.1 github.com/sirupsen/logrus v1.6.0 go.uber.org/zap v1.15.0 - golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d // indirect k8s.io/api v0.20.4 k8s.io/apimachinery v0.20.4 k8s.io/client-go v0.20.4 diff --git a/main.go b/main.go index c6fd6fbc..f6efe702 100644 --- a/main.go +++ b/main.go @@ -18,10 +18,12 @@ package main import ( "flag" - "github.com/keikoproj/upgrade-manager/controllers/common" "os" + "sync" "time" + "github.com/keikoproj/upgrade-manager/controllers/common" + // Import all Kubernetes client auth plugins (e.g. Azure, GCP, OIDC, etc.) // to ensure that exec-entrypoint and run can make use of them. _ "k8s.io/client-go/plugin/pkg/client/auth" @@ -195,6 +197,8 @@ func main() { ScriptRunner: controllers.ScriptRunner{ Logger: logger, }, + DrainGroupMapper: &sync.Map{}, + DrainErrorMapper: &sync.Map{}, } reconciler.SetMaxParallel(maxParallel) From 6fef5fdceed77821fc5ee2c400a255941e899453 Mon Sep 17 00:00:00 2001 From: Sheldon Shao Date: Tue, 18 May 2021 13:59:59 -0700 Subject: [PATCH 30/74] V2 controller metrics concurrency fix (#231) * Refine the metrics status Signed-off-by: xshao * Refine the metrics status Signed-off-by: xshao * Fix test case error Signed-off-by: xshao * Use group instead of ASG Signed-off-by: xshao * Ignore generated code Signed-off-by: xshao * Ignore generated code Signed-off-by: xshao * Fix the concurrent issue Signed-off-by: xshao * Fix the concurrent issue Signed-off-by: xshao * Move metrics related functions into RollingUpgradeContext Signed-off-by: xshao * Move metrics related functions into RollingUpgradeContext Signed-off-by: xshao * Move metrics related functions into upgrade_metrics.go Signed-off-by: xshao * Move metrics related functions into metrics.go Signed-off-by: xshao --- .gitignore | 2 +- api/v1alpha1/rollingupgrade_types.go | 108 +---------------- controllers/common/metrics.go | 7 ++ controllers/helpers_test.go | 1 + controllers/metrics.go | 113 ++++++++++++++++++ .../metrics_test.go | 38 +++--- controllers/upgrade.go | 37 +++--- 7 files changed, 164 insertions(+), 142 deletions(-) create mode 100644 controllers/metrics.go rename api/v1alpha1/rollingupgrade_types_test.go => controllers/metrics_test.go (53%) diff --git a/.gitignore b/.gitignore index c055d501..cebb35f8 100644 --- a/.gitignore +++ b/.gitignore @@ -27,4 +27,4 @@ vendor/ config/default/manager_image_patch.yaml-e .vscode/ cover.html -.DS_Store \ No newline at end of file +.DS_Store diff --git a/api/v1alpha1/rollingupgrade_types.go b/api/v1alpha1/rollingupgrade_types.go index 42d48a3d..af461c6b 100644 --- a/api/v1alpha1/rollingupgrade_types.go +++ b/api/v1alpha1/rollingupgrade_types.go @@ -18,14 +18,12 @@ package v1alpha1 import ( "fmt" - "strconv" - "strings" - "time" - "github.com/keikoproj/upgrade-manager/controllers/common" corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/intstr" + "strconv" + "strings" ) // RollingUpgradeSpec defines the desired state of RollingUpgrade @@ -82,108 +80,6 @@ type NodeInProcessing struct { StepEndTime metav1.Time `json:"stepEndTime,omitempty"` } -// Update last batch nodes -func (s *RollingUpgradeStatus) UpdateLastBatchNodes(batchNodes map[string]*NodeInProcessing) { - keys := make([]string, 0, len(batchNodes)) - for k := range batchNodes { - keys = append(keys, k) - } - s.LastBatchNodes = keys -} - -// Update Node Statistics -func (s *RollingUpgradeStatus) UpdateStatistics(nodeSteps map[string][]NodeStepDuration) { - for _, v := range nodeSteps { - for _, step := range v { - s.AddNodeStepDuration(step) - } - } -} - -// Add one step duration -func (s *RollingUpgradeStatus) ToStepDuration(groupName, nodeName string, stepName RollingUpgradeStep, duration time.Duration) NodeStepDuration { - //Add to system level statistics - common.AddStepDuration(groupName, string(stepName), duration) - return NodeStepDuration{ - GroupName: groupName, - NodeName: nodeName, - StepName: stepName, - Duration: metav1.Duration{ - Duration: duration, - }, - } -} - -// Add one step duration -func (s *RollingUpgradeStatus) AddNodeStepDuration(nsd NodeStepDuration) { - // if step exists, add count and sum, otherwise append - for _, s := range s.Statistics { - if s.StepName == nsd.StepName { - s.DurationSum = metav1.Duration{ - Duration: s.DurationSum.Duration + nsd.Duration.Duration, - } - s.DurationCount += 1 - return - } - } - s.Statistics = append(s.Statistics, &RollingUpgradeStatistics{ - StepName: nsd.StepName, - DurationSum: metav1.Duration{ - Duration: nsd.Duration.Duration, - }, - DurationCount: 1, - }) -} - -// Node turns onto step -func (s *RollingUpgradeStatus) NodeStep(InProcessingNodes map[string]*NodeInProcessing, - nodeSteps map[string][]NodeStepDuration, groupName, nodeName string, stepName RollingUpgradeStep) { - - var inProcessingNode *NodeInProcessing - if n, ok := InProcessingNodes[nodeName]; !ok { - inProcessingNode = &NodeInProcessing{ - NodeName: nodeName, - StepName: stepName, - UpgradeStartTime: metav1.Now(), - StepStartTime: metav1.Now(), - } - InProcessingNodes[nodeName] = inProcessingNode - } else { - inProcessingNode = n - } - - inProcessingNode.StepEndTime = metav1.Now() - var duration = inProcessingNode.StepEndTime.Sub(inProcessingNode.StepStartTime.Time) - if stepName == NodeRotationCompleted { - //Add overall and remove the node from in-processing map - var total = inProcessingNode.StepEndTime.Sub(inProcessingNode.UpgradeStartTime.Time) - duration1 := s.ToStepDuration(groupName, nodeName, inProcessingNode.StepName, duration) - duration2 := s.ToStepDuration(groupName, nodeName, NodeRotationTotal, total) - s.addNodeStepDuration(nodeSteps, nodeName, duration1) - s.addNodeStepDuration(nodeSteps, nodeName, duration2) - } else if inProcessingNode.StepName != stepName { //Still same step - var oldOrder = NodeRotationStepOrders[inProcessingNode.StepName] - var newOrder = NodeRotationStepOrders[stepName] - if newOrder > oldOrder { //Make sure the steps running in order - stepDuration := s.ToStepDuration(groupName, nodeName, inProcessingNode.StepName, duration) - inProcessingNode.StepStartTime = metav1.Now() - inProcessingNode.StepName = stepName - s.addNodeStepDuration(nodeSteps, nodeName, stepDuration) - } - } -} - -func (s *RollingUpgradeStatus) addNodeStepDuration(steps map[string][]NodeStepDuration, nodeName string, nsd NodeStepDuration) { - if stepDuration, ok := steps[nodeName]; !ok { - steps[nodeName] = []NodeStepDuration{ - nsd, - } - } else { - stepDuration = append(stepDuration, nsd) - steps[nodeName] = stepDuration - } -} - func (s *RollingUpgradeStatus) SetCondition(cond RollingUpgradeCondition) { // if condition exists, overwrite, otherwise append for ix, c := range s.Conditions { diff --git a/controllers/common/metrics.go b/controllers/common/metrics.go index 694571eb..2d5cd829 100644 --- a/controllers/common/metrics.go +++ b/controllers/common/metrics.go @@ -3,6 +3,7 @@ package common import ( "reflect" "strings" + "sync" "time" "github.com/keikoproj/upgrade-manager/controllers/common/log" @@ -34,6 +35,8 @@ var ( stepSummaries = make(map[string]map[string]prometheus.Summary) + stepSumMutex = sync.Mutex{} + CRStatus = prometheus.NewGaugeVec( prometheus.GaugeOpts{ Namespace: metricNamespace, @@ -60,7 +63,9 @@ func AddStepDuration(groupName string, stepName string, duration time.Duration) var steps map[string]prometheus.Summary if m, ok := stepSummaries[groupName]; !ok { steps = make(map[string]prometheus.Summary) + stepSumMutex.Lock() stepSummaries[groupName] = steps + stepSumMutex.Unlock() } else { steps = m } @@ -82,7 +87,9 @@ func AddStepDuration(groupName string, stepName string, duration time.Duration) log.Errorf("register summary error, group: %s, step: %s, %v", groupName, stepName, err) } } + stepSumMutex.Lock() steps[stepName] = summary + stepSumMutex.Unlock() } else { summary = s } diff --git a/controllers/helpers_test.go b/controllers/helpers_test.go index 8c86252d..097cfa16 100644 --- a/controllers/helpers_test.go +++ b/controllers/helpers_test.go @@ -71,6 +71,7 @@ func createRollingUpgradeContext(r *RollingUpgradeReconciler) *RollingUpgradeCon DrainGroup: drainGroup.(*sync.WaitGroup), }, RollingUpgrade: rollingUpgrade, + metricsMutex: &sync.Mutex{}, } } diff --git a/controllers/metrics.go b/controllers/metrics.go new file mode 100644 index 00000000..95964733 --- /dev/null +++ b/controllers/metrics.go @@ -0,0 +1,113 @@ +package controllers + +import ( + "time" + + "github.com/keikoproj/upgrade-manager/api/v1alpha1" + "github.com/keikoproj/upgrade-manager/controllers/common" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" +) + +// Update last batch nodes +func (s *RollingUpgradeContext) UpdateLastBatchNodes(batchNodes map[string]*v1alpha1.NodeInProcessing) { + keys := make([]string, 0, len(batchNodes)) + for k := range batchNodes { + keys = append(keys, k) + } + s.RollingUpgrade.Status.LastBatchNodes = keys +} + +// Update Node Statistics +func (s *RollingUpgradeContext) UpdateStatistics(nodeSteps map[string][]v1alpha1.NodeStepDuration) { + for _, v := range nodeSteps { + for _, step := range v { + s.AddNodeStepDuration(step) + } + } +} + +// Add one step duration +func (s *RollingUpgradeContext) AddNodeStepDuration(nsd v1alpha1.NodeStepDuration) { + // if step exists, add count and sum, otherwise append + for _, s := range s.RollingUpgrade.Status.Statistics { + if s.StepName == nsd.StepName { + s.DurationSum = metav1.Duration{ + Duration: s.DurationSum.Duration + nsd.Duration.Duration, + } + s.DurationCount += 1 + return + } + } + s.RollingUpgrade.Status.Statistics = append(s.RollingUpgrade.Status.Statistics, &v1alpha1.RollingUpgradeStatistics{ + StepName: nsd.StepName, + DurationSum: metav1.Duration{ + Duration: nsd.Duration.Duration, + }, + DurationCount: 1, + }) +} + +// Add one step duration +func (s *RollingUpgradeContext) ToStepDuration(groupName, nodeName string, stepName v1alpha1.RollingUpgradeStep, duration time.Duration) v1alpha1.NodeStepDuration { + //Add to system level statistics + common.AddStepDuration(groupName, string(stepName), duration) + return v1alpha1.NodeStepDuration{ + GroupName: groupName, + NodeName: nodeName, + StepName: stepName, + Duration: metav1.Duration{ + Duration: duration, + }, + } +} + +// Node turns onto step +func (s *RollingUpgradeContext) NodeStep(InProcessingNodes map[string]*v1alpha1.NodeInProcessing, + nodeSteps map[string][]v1alpha1.NodeStepDuration, groupName, nodeName string, stepName v1alpha1.RollingUpgradeStep) { + + var inProcessingNode *v1alpha1.NodeInProcessing + if n, ok := InProcessingNodes[nodeName]; !ok { + inProcessingNode = &v1alpha1.NodeInProcessing{ + NodeName: nodeName, + StepName: stepName, + UpgradeStartTime: metav1.Now(), + StepStartTime: metav1.Now(), + } + InProcessingNodes[nodeName] = inProcessingNode + } else { + inProcessingNode = n + } + + inProcessingNode.StepEndTime = metav1.Now() + var duration = inProcessingNode.StepEndTime.Sub(inProcessingNode.StepStartTime.Time) + if stepName == v1alpha1.NodeRotationCompleted { + //Add overall and remove the node from in-processing map + var total = inProcessingNode.StepEndTime.Sub(inProcessingNode.UpgradeStartTime.Time) + duration1 := s.ToStepDuration(groupName, nodeName, inProcessingNode.StepName, duration) + duration2 := s.ToStepDuration(groupName, nodeName, v1alpha1.NodeRotationTotal, total) + s.addNodeStepDuration(nodeSteps, nodeName, duration1) + s.addNodeStepDuration(nodeSteps, nodeName, duration2) + } else if inProcessingNode.StepName != stepName { //Still same step + var oldOrder = v1alpha1.NodeRotationStepOrders[inProcessingNode.StepName] + var newOrder = v1alpha1.NodeRotationStepOrders[stepName] + if newOrder > oldOrder { //Make sure the steps running in order + stepDuration := s.ToStepDuration(groupName, nodeName, inProcessingNode.StepName, duration) + inProcessingNode.StepStartTime = metav1.Now() + inProcessingNode.StepName = stepName + s.addNodeStepDuration(nodeSteps, nodeName, stepDuration) + } + } +} + +func (s *RollingUpgradeContext) addNodeStepDuration(steps map[string][]v1alpha1.NodeStepDuration, nodeName string, nsd v1alpha1.NodeStepDuration) { + s.metricsMutex.Lock() + if stepDuration, ok := steps[nodeName]; !ok { + steps[nodeName] = []v1alpha1.NodeStepDuration{ + nsd, + } + } else { + stepDuration = append(stepDuration, nsd) + steps[nodeName] = stepDuration + } + s.metricsMutex.Unlock() +} diff --git a/api/v1alpha1/rollingupgrade_types_test.go b/controllers/metrics_test.go similarity index 53% rename from api/v1alpha1/rollingupgrade_types_test.go rename to controllers/metrics_test.go index 292136f5..a55d7025 100644 --- a/api/v1alpha1/rollingupgrade_types_test.go +++ b/controllers/metrics_test.go @@ -1,49 +1,53 @@ -package v1alpha1 +package controllers import ( - "github.com/onsi/gomega" "testing" + + "github.com/keikoproj/upgrade-manager/api/v1alpha1" + "github.com/onsi/gomega" ) // Test func TestNodeTurnsOntoStep(t *testing.T) { g := gomega.NewGomegaWithT(t) - r := &RollingUpgradeStatus{} + reconsiler := createRollingUpgradeReconciler(t) + r := createRollingUpgradeContext(reconsiler) + //A map to retain the steps for multiple nodes - nodeSteps := make(map[string][]NodeStepDuration) - inProcessingNodes := make(map[string]*NodeInProcessing) + nodeSteps := make(map[string][]v1alpha1.NodeStepDuration) + inProcessingNodes := make(map[string]*v1alpha1.NodeInProcessing) - r.NodeStep(inProcessingNodes, nodeSteps, "test-asg", "node-1", NodeRotationKickoff) + r.NodeStep(inProcessingNodes, nodeSteps, "test-asg", "node-1", v1alpha1.NodeRotationKickoff) g.Expect(inProcessingNodes).NotTo(gomega.BeNil()) g.Expect(nodeSteps["node-1"]).To(gomega.BeNil()) - r.NodeStep(inProcessingNodes, nodeSteps, "test-asg", "node-1", NodeRotationDesiredNodeReady) + r.NodeStep(inProcessingNodes, nodeSteps, "test-asg", "node-1", v1alpha1.NodeRotationDesiredNodeReady) g.Expect(len(nodeSteps["node-1"])).To(gomega.Equal(1)) - g.Expect(nodeSteps["node-1"][0].StepName).To(gomega.Equal(NodeRotationKickoff)) + g.Expect(nodeSteps["node-1"][0].StepName).To(gomega.Equal(v1alpha1.NodeRotationKickoff)) //Retry desired_node_ready - r.NodeStep(inProcessingNodes, nodeSteps, "test-asg", "node-1", NodeRotationDesiredNodeReady) + r.NodeStep(inProcessingNodes, nodeSteps, "test-asg", "node-1", v1alpha1.NodeRotationDesiredNodeReady) g.Expect(len(nodeSteps["node-1"])).To(gomega.Equal(1)) - g.Expect(nodeSteps["node-1"][0].StepName).To(gomega.Equal(NodeRotationKickoff)) + g.Expect(nodeSteps["node-1"][0].StepName).To(gomega.Equal(v1alpha1.NodeRotationKickoff)) //Retry desired_node_ready again - r.NodeStep(inProcessingNodes, nodeSteps, "test-asg", "node-1", NodeRotationDesiredNodeReady) + r.NodeStep(inProcessingNodes, nodeSteps, "test-asg", "node-1", v1alpha1.NodeRotationDesiredNodeReady) g.Expect(len(nodeSteps["node-1"])).To(gomega.Equal(1)) - g.Expect(nodeSteps["node-1"][0].StepName).To(gomega.Equal(NodeRotationKickoff)) + g.Expect(nodeSteps["node-1"][0].StepName).To(gomega.Equal(v1alpha1.NodeRotationKickoff)) //Completed - r.NodeStep(inProcessingNodes, nodeSteps, "test-asg", "node-1", NodeRotationCompleted) + r.NodeStep(inProcessingNodes, nodeSteps, "test-asg", "node-1", v1alpha1.NodeRotationCompleted) g.Expect(len(nodeSteps["node-1"])).To(gomega.Equal(3)) - g.Expect(nodeSteps["node-1"][1].StepName).To(gomega.Equal(NodeRotationDesiredNodeReady)) - g.Expect(nodeSteps["node-1"][2].StepName).To(gomega.Equal(NodeRotationTotal)) + g.Expect(nodeSteps["node-1"][1].StepName).To(gomega.Equal(v1alpha1.NodeRotationDesiredNodeReady)) + g.Expect(nodeSteps["node-1"][2].StepName).To(gomega.Equal(v1alpha1.NodeRotationTotal)) //Second node - r.NodeStep(inProcessingNodes, nodeSteps, "test-asg", "node-2", NodeRotationKickoff) + r.NodeStep(inProcessingNodes, nodeSteps, "test-asg", "node-2", v1alpha1.NodeRotationKickoff) g.Expect(len(nodeSteps["node-1"])).To(gomega.Equal(3)) - r.NodeStep(inProcessingNodes, nodeSteps, "test-asg", "node-2", NodeRotationDesiredNodeReady) + r.NodeStep(inProcessingNodes, nodeSteps, "test-asg", "node-2", v1alpha1.NodeRotationDesiredNodeReady) g.Expect(len(nodeSteps["node-1"])).To(gomega.Equal(3)) } diff --git a/controllers/upgrade.go b/controllers/upgrade.go index 431909c9..5dbda496 100644 --- a/controllers/upgrade.go +++ b/controllers/upgrade.go @@ -54,6 +54,7 @@ type RollingUpgradeContext struct { Cloud *DiscoveredState RollingUpgrade *v1alpha1.RollingUpgrade DrainManager *DrainManager + metricsMutex *sync.Mutex } // TODO: main node rotation logic @@ -129,9 +130,8 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) node = kubeprovider.SelectNodeByInstanceID(instanceID, r.Cloud.ClusterNodes) nodeName = node.GetName() ) - // Add statistics - r.RollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationKickoff) - } + //Add statistics + r.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationKickoff) batchInstanceIDs, inServiceInstanceIDs := awsprovider.GetInstanceIDs(batch), awsprovider.GetInServiceInstanceIDs(batch) // Tag and set to StandBy only the InService instances. @@ -160,9 +160,10 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) node = kubeprovider.SelectNodeByInstanceID(instanceID, r.Cloud.ClusterNodes) nodeName = node.GetName() ) - r.RollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationDesiredNodeReady) + r.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationDesiredNodeReady) } + // Wait for desired nodes r.Info("waiting for desired nodes", "name", r.RollingUpgrade.NamespacedName()) if !r.DesiredNodesReady() { @@ -178,7 +179,7 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) nodeName = node.GetName() ) // add statistics - r.RollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationKickoff) + r.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationKickoff) } // add in-progress tag batchInstanceIDs, inServiceInstanceIDs := awsprovider.GetInstanceIDs(batch), awsprovider.GetInServiceInstanceIDs(batch) @@ -208,7 +209,7 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) defer r.DrainManager.DrainGroup.Done() // Turns onto PreDrain script - r.RollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationPredrainScript) + r.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationPredrainScript) // Predrain script if err := r.ScriptRunner.PreDrain(scriptTarget); err != nil { @@ -220,7 +221,7 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) r.Info("draining the node", "instance", instanceID, "node name", node.Name, "name", r.RollingUpgrade.NamespacedName()) // Turns onto NodeRotationDrain - r.RollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationDrain) + r.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationDrain) if err := r.Auth.DrainNode(&node, time.Duration(r.RollingUpgrade.PostDrainDelaySeconds()), r.RollingUpgrade.DrainTimeout(), r.Auth.Kubernetes); err != nil { r.DrainManager.DrainErrors <- errors.Errorf("DrainNode failed: instanceID - %v, %v", instanceID, err.Error()) @@ -228,7 +229,7 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) } // Turns onto NodeRotationPostdrainScript - r.RollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationPostdrainScript) + r.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationPostdrainScript) // post drain script if err := r.ScriptRunner.PostDrain(scriptTarget); err != nil { @@ -236,7 +237,7 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) } // Turns onto NodeRotationPostWait - r.RollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationPostWait) + r.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationPostWait) // Post Wait Script if err := r.ScriptRunner.PostWait(scriptTarget); err != nil { @@ -254,8 +255,8 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) select { case err := <-r.DrainManager.DrainErrors: - r.RollingUpgrade.Status.UpdateStatistics(nodeSteps) - r.RollingUpgrade.Status.UpdateLastBatchNodes(inProcessingNodes) + r.UpdateStatistics(nodeSteps) + r.UpdateLastBatchNodes(inProcessingNodes) r.Error(err, "failed to rotate the node", "name", r.RollingUpgrade.NamespacedName()) return false, err @@ -277,7 +278,7 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) ) // Turns onto NodeRotationTerminate - r.RollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationTerminate) + r.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationTerminate) // Terminate - set lastTerminateTime r.Info("terminating instance", "instance", instanceID, "name", r.RollingUpgrade.NamespacedName()) @@ -290,7 +291,7 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) r.RollingUpgrade.SetLastNodeTerminationTime(metav1.Time{Time: time.Now()}) // Turns onto NodeRotationTerminate - r.RollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationPostTerminate) + r.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationPostTerminate) // Post Terminate Script if err := r.ScriptRunner.PostTerminate(scriptTarget); err != nil { @@ -298,17 +299,17 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) } // Turns onto NodeRotationCompleted - r.RollingUpgrade.Status.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationCompleted) + r.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationCompleted) } - r.RollingUpgrade.Status.UpdateStatistics(nodeSteps) - r.RollingUpgrade.Status.UpdateLastBatchNodes(inProcessingNodes) + r.UpdateStatistics(nodeSteps) + r.UpdateLastBatchNodes(inProcessingNodes) case <-time.After(DefaultWaitGroupTimeout): // goroutines timed out - requeue - r.RollingUpgrade.Status.UpdateStatistics(nodeSteps) - r.RollingUpgrade.Status.UpdateLastBatchNodes(inProcessingNodes) + r.UpdateStatistics(nodeSteps) + r.UpdateLastBatchNodes(inProcessingNodes) r.Info("instances are still draining", "name", r.RollingUpgrade.NamespacedName()) return true, nil From a9ac50f64a71ffd46991be2b67a71e2caae6d189 Mon Sep 17 00:00:00 2001 From: Shreyas Badiger <7680410+shreyas-badiger@users.noreply.github.com> Date: Tue, 18 May 2021 16:20:28 -0700 Subject: [PATCH 31/74] add missing parenthesis (#239) --- controllers/upgrade.go | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/controllers/upgrade.go b/controllers/upgrade.go index 5dbda496..f6116af1 100644 --- a/controllers/upgrade.go +++ b/controllers/upgrade.go @@ -54,7 +54,7 @@ type RollingUpgradeContext struct { Cloud *DiscoveredState RollingUpgrade *v1alpha1.RollingUpgrade DrainManager *DrainManager - metricsMutex *sync.Mutex + metricsMutex *sync.Mutex } // TODO: main node rotation logic @@ -132,6 +132,7 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) ) //Add statistics r.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationKickoff) + } batchInstanceIDs, inServiceInstanceIDs := awsprovider.GetInstanceIDs(batch), awsprovider.GetInServiceInstanceIDs(batch) // Tag and set to StandBy only the InService instances. @@ -152,7 +153,7 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) } else { r.Info("no InService instances in the batch", "batch", batchInstanceIDs, "instances(InService)", inServiceInstanceIDs, "name", r.RollingUpgrade.NamespacedName()) } - + // turns onto desired nodes for _, target := range batch { var ( @@ -163,7 +164,6 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) r.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationDesiredNodeReady) } - // Wait for desired nodes r.Info("waiting for desired nodes", "name", r.RollingUpgrade.NamespacedName()) if !r.DesiredNodesReady() { From 1fc584715e6900da5587bfe2a1b402e395ae0652 Mon Sep 17 00:00:00 2001 From: Sheldon Shao Date: Tue, 18 May 2021 16:59:44 -0700 Subject: [PATCH 32/74] metricsMutex should be initialized (#240) Signed-off-by: xshao --- controllers/rollingupgrade_controller.go | 1 + 1 file changed, 1 insertion(+) diff --git a/controllers/rollingupgrade_controller.go b/controllers/rollingupgrade_controller.go index d2053197..3166812a 100644 --- a/controllers/rollingupgrade_controller.go +++ b/controllers/rollingupgrade_controller.go @@ -137,6 +137,7 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque DrainGroup: drainGroup.(*sync.WaitGroup), }, RollingUpgrade: rollingUpgrade, + metricsMutex: &sync.Mutex{}, } rollupCtx.Cloud = NewDiscoveredState(rollupCtx.Auth, rollupCtx.Logger) if err := rollupCtx.Cloud.Discover(); err != nil { From 18e0e75d54080712098e6347903c8461e199ca5c Mon Sep 17 00:00:00 2001 From: Shreyas Badiger <7680410+shreyas-badiger@users.noreply.github.com> Date: Wed, 2 Jun 2021 12:45:50 -0700 Subject: [PATCH 33/74] upgrade-manager-v2: Load test fixes (#245) * upgrade-manager-v2: Move DrainManager back to Reconciler (#236) * #2285: rollup CR statistic metrics in v2 (#218) * #2285: rollup CR statistic metrics in v2 Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 Signed-off-by: sbadiger * log cloud discovery failure Signed-off-by: sbadiger * Create RollingUpgrade Context Signed-off-by: sbadiger * rollingupgrade context Signed-off-by: sbadiger * #2285: rollup CR statistic metrics in v2 (#218) * #2285: rollup CR statistic metrics in v2 Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 Signed-off-by: sbadiger * #2285: renamed some methods related to metrics (#224) Signed-off-by: sbadla1 Signed-off-by: sbadiger * #2286: removed version from metric namespace (#227) Signed-off-by: sbadla1 Signed-off-by: sbadiger * resolve compile errors due to merge conflict Signed-off-by: sbadiger * move drain-manager to reconciler Signed-off-by: sbadiger * initialize RollingUpgrade object Signed-off-by: sbadiger * use bool instead of count for standby function Signed-off-by: sbadiger * refactor in-progress and standby code Signed-off-by: sbadiger * rename instance standby function Signed-off-by: sbadiger * DrainManager changes in unit test files Signed-off-by: sbadiger Co-authored-by: Sahil Badla Signed-off-by: sbadiger * V2 controller metrics concurrency fix (#231) * Refine the metrics status Signed-off-by: xshao * Refine the metrics status Signed-off-by: xshao * Fix test case error Signed-off-by: xshao * Use group instead of ASG Signed-off-by: xshao * Ignore generated code Signed-off-by: xshao * Ignore generated code Signed-off-by: xshao * Fix the concurrent issue Signed-off-by: xshao * Fix the concurrent issue Signed-off-by: xshao * Move metrics related functions into RollingUpgradeContext Signed-off-by: xshao * Move metrics related functions into RollingUpgradeContext Signed-off-by: xshao * Move metrics related functions into upgrade_metrics.go Signed-off-by: xshao * Move metrics related functions into metrics.go Signed-off-by: xshao Signed-off-by: sbadiger * add missing parenthesis Signed-off-by: sbadiger * load test fixes Signed-off-by: sbadiger * handle scaling group not found Signed-off-by: sbadiger * Update upgrade.go Signed-off-by: sbadiger * log one level up * remove double logging Signed-off-by: sbadiger --- controllers/helpers_test.go | 2 +- controllers/providers/aws/utils.go | 19 ----------- controllers/rollingupgrade_controller.go | 21 +++++++----- controllers/upgrade.go | 42 ++++++++++++++---------- 4 files changed, 39 insertions(+), 45 deletions(-) diff --git a/controllers/helpers_test.go b/controllers/helpers_test.go index 097cfa16..c9e98dca 100644 --- a/controllers/helpers_test.go +++ b/controllers/helpers_test.go @@ -71,7 +71,7 @@ func createRollingUpgradeContext(r *RollingUpgradeReconciler) *RollingUpgradeCon DrainGroup: drainGroup.(*sync.WaitGroup), }, RollingUpgrade: rollingUpgrade, - metricsMutex: &sync.Mutex{}, + metricsMutex: &sync.Mutex{}, } } diff --git a/controllers/providers/aws/utils.go b/controllers/providers/aws/utils.go index 451aced8..b5b46874 100644 --- a/controllers/providers/aws/utils.go +++ b/controllers/providers/aws/utils.go @@ -96,25 +96,6 @@ func GetInstanceIDs(instances []*autoscaling.Instance) []string { return IDs } -// func SelectInstancesByAZ(instances []*autoscaling.Group) *autoscaling.Instance { -// for _, instance := range group.Instances { -// selectedID := aws.StringValue(instance.InstanceId) -// if strings.EqualFold(instanceID, selectedID) { -// return instance -// } -// } -// return &autoscaling.Instance{} -// } - -// func ListScalingInstanceIDs(group *autoscaling.Group) []string { -// instanceIDs := make([]string, 0) -// for _, instance := range group.Instances { -// instanceID := aws.StringValue(instance.InstanceId) -// instanceIDs = append(instanceIDs, instanceID) -// } -// return instanceIDs -// } - func GetTemplateLatestVersion(templates []*ec2.LaunchTemplate, templateName string) string { for _, template := range templates { name := aws.StringValue(template.LaunchTemplateName) diff --git a/controllers/rollingupgrade_controller.go b/controllers/rollingupgrade_controller.go index 3166812a..35bbe53f 100644 --- a/controllers/rollingupgrade_controller.go +++ b/controllers/rollingupgrade_controller.go @@ -15,6 +15,7 @@ package controllers import ( "context" + "fmt" "strings" "sync" "time" @@ -65,12 +66,13 @@ type RollingUpgradeAuthenticator struct { // Reconcile reads that state of the cluster for a RollingUpgrade object and makes changes based on the state read // and the details in the RollingUpgrade.Spec func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { + r.Info("***Reconciling***") rollingUpgrade := &v1alpha1.RollingUpgrade{} err := r.Get(ctx, req.NamespacedName, rollingUpgrade) if err != nil { if kerrors.IsNotFound(err) { - r.AdmissionMap.Delete(req.NamespacedName) - r.Info("deleted object from admission map", "name", req.NamespacedName) + r.AdmissionMap.Delete(fmt.Sprintf("%s", req.NamespacedName)) + r.Info("rolling upgrade resource not found, deleted object from admission map", "name", req.NamespacedName) return ctrl.Result{}, nil } return ctrl.Result{}, err @@ -108,7 +110,7 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque val := v.(string) resource := k.(string) if strings.EqualFold(val, scalingGroupName) && !strings.EqualFold(resource, rollingUpgrade.NamespacedName()) { - r.Info("object already being processed by existing resource", "resource", resource, "scalingGroup", scalingGroupName) + r.Info("object already being processed by existing resource", "resource", resource, "scalingGroup", scalingGroupName, "name", rollingUpgrade.NamespacedName()) inProgress = true return false } @@ -120,8 +122,13 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque return ctrl.Result{RequeueAfter: time.Second * 30}, nil } - r.Info("admitted new rollingupgrade", "name", rollingUpgrade.NamespacedName(), "scalingGroup", scalingGroupName) - r.AdmissionMap.Store(rollingUpgrade.NamespacedName(), scalingGroupName) + // store the rolling upgrade in admission map + if _, present := r.AdmissionMap.LoadOrStore(rollingUpgrade.NamespacedName(), scalingGroupName); present == false { + r.Info("admitted new rolling upgrade", "scalingGroup", scalingGroupName, "update strategy", rollingUpgrade.Spec.Strategy, "name", rollingUpgrade.NamespacedName()) + r.CacheConfig.FlushCache("autoscaling") + } else { + r.Info("operating on existing rolling upgrade", "scalingGroup", scalingGroupName, "update strategy", rollingUpgrade.Spec.Strategy, "name", rollingUpgrade.NamespacedName()) + } rollingUpgrade.SetCurrentStatus(v1alpha1.StatusInit) common.SetMetricRollupInitOrRunning(rollingUpgrade.Name) @@ -141,9 +148,8 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque } rollupCtx.Cloud = NewDiscoveredState(rollupCtx.Auth, rollupCtx.Logger) if err := rollupCtx.Cloud.Discover(); err != nil { - r.Info("failed to discover the cloud", "name", rollingUpgrade.NamespacedName(), "scalingGroup", scalingGroupName) + r.Info("failed to discover the cloud", "scalingGroup", scalingGroupName, "name", rollingUpgrade.NamespacedName()) rollingUpgrade.SetCurrentStatus(v1alpha1.StatusError) - // Set prometheus metric cr_status_failed common.SetMetricRollupFailed(rollingUpgrade.Name) return ctrl.Result{}, err } @@ -151,7 +157,6 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque // process node rotation if err := rollupCtx.RotateNodes(); err != nil { rollingUpgrade.SetCurrentStatus(v1alpha1.StatusError) - // Set prometheus metric cr_status_failed common.SetMetricRollupFailed(rollingUpgrade.Name) return ctrl.Result{}, err } diff --git a/controllers/upgrade.go b/controllers/upgrade.go index f6116af1..c64b0df8 100644 --- a/controllers/upgrade.go +++ b/controllers/upgrade.go @@ -17,8 +17,8 @@ limitations under the License. package controllers import ( - "fmt" "reflect" + "strconv" "strings" "sync" "time" @@ -91,13 +91,21 @@ func (r *RollingUpgradeContext) RotateNodes() error { var ( scalingGroup = awsprovider.SelectScalingGroup(r.RollingUpgrade.ScalingGroupName(), r.Cloud.ScalingGroups) ) + if reflect.DeepEqual(scalingGroup, &autoscaling.Group{}) { + return errors.Errorf("scaling group not found, scalingGroupName: %v", r.RollingUpgrade.ScalingGroupName()) + } + r.Info( + "scaling group details", + "scalingGroup", r.RollingUpgrade.ScalingGroupName(), + "launchConfig", aws.StringValue(scalingGroup.LaunchConfigurationName), + "name", r.RollingUpgrade.NamespacedName(), + ) r.RollingUpgrade.SetTotalNodes(len(scalingGroup.Instances)) // check if all instances are rotated. if !r.IsScalingGroupDrifted() { r.RollingUpgrade.SetCurrentStatus(v1alpha1.StatusComplete) - // Set prometheus metric cr_status_completed common.SetMetricRollupCompleted(r.RollingUpgrade.Name) return nil } @@ -140,7 +148,7 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) // Add in-progress tag r.Info("setting instances to in-progress", "batch", batchInstanceIDs, "instances(InService)", inServiceInstanceIDs, "name", r.RollingUpgrade.NamespacedName()) if err := r.Auth.TagEC2instances(inServiceInstanceIDs, instanceStateTagKey, inProgressTagValue); err != nil { - r.Error(err, "failed to set instancecs to in-progress", "batch", batchInstanceIDs, "instances(InService)", inServiceInstanceIDs, "name", r.RollingUpgrade.NamespacedName()) + r.Error(err, "failed to set instances to in-progress", "batch", batchInstanceIDs, "instances(InService)", inServiceInstanceIDs, "name", r.RollingUpgrade.NamespacedName()) return false, err } // Standby @@ -170,6 +178,7 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) r.Info("new node is yet to join the cluster", "name", r.RollingUpgrade.NamespacedName()) return true, nil } + r.Info("desired nodes are ready", "name", r.RollingUpgrade.NamespacedName()) case v1alpha1.UpdateStrategyModeLazy: for _, target := range batch { @@ -190,7 +199,6 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) } } - fmt.Println("r.DrainManager", r.DrainManager) if reflect.DeepEqual(r.DrainManager.DrainGroup, &sync.WaitGroup{}) { for _, target := range batch { var ( @@ -247,9 +255,9 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) } } - timeout := make(chan struct{}) + done := make(chan struct{}) go func() { - defer close(timeout) + defer close(done) r.DrainManager.DrainGroup.Wait() }() @@ -261,7 +269,7 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) r.Error(err, "failed to rotate the node", "name", r.RollingUpgrade.NamespacedName()) return false, err - case <-timeout: + case <-done: // goroutines completed, terminate and requeue r.RollingUpgrade.SetLastNodeDrainTime(metav1.Time{Time: time.Now()}) r.Info("instances drained successfully, terminating", "name", r.RollingUpgrade.NamespacedName()) @@ -326,12 +334,16 @@ func (r *RollingUpgradeContext) SelectTargets(scalingGroup *autoscaling.Group) [ var unavailableInt int if batchSize.Type == intstr.String { - unavailableInt, _ = intstr.GetValueFromIntOrPercent(&batchSize, totalNodes, true) + if strings.Contains(batchSize.StrVal, "%") { + unavailableInt, _ = intstr.GetValueFromIntOrPercent(&batchSize, totalNodes, true) + } + unavailableInt, _ = strconv.Atoi(batchSize.StrVal) } else { unavailableInt = batchSize.IntValue() } // first process all in progress instances + r.Info("selecting batch for rotation", "batch size", batchSize, "name", r.RollingUpgrade.NamespacedName()) for _, instance := range r.Cloud.InProgressInstances { if selectedInstance := awsprovider.SelectScalingGroupInstance(instance, scalingGroup); !reflect.DeepEqual(selectedInstance, &autoscaling.Instance{}) { targets = append(targets, selectedInstance) @@ -392,6 +404,7 @@ func (r *RollingUpgradeContext) IsInstanceDrifted(instance *autoscaling.Instance if common.ContainsEqualFold(awsprovider.TerminatingInstanceStates, aws.StringValue(instance.LifecycleState)) { return false } + // check if there is atleast one node that meets the force-referesh criteria if r.RollingUpgrade.IsForceRefresh() { var ( @@ -407,18 +420,15 @@ func (r *RollingUpgradeContext) IsInstanceDrifted(instance *autoscaling.Instance if scalingGroup.LaunchConfigurationName != nil { if instance.LaunchConfigurationName == nil { - r.Info("launch configuration name differs", "instance", instanceID, "name", r.RollingUpgrade.NamespacedName()) return true } launchConfigName := aws.StringValue(scalingGroup.LaunchConfigurationName) instanceConfigName := aws.StringValue(instance.LaunchConfigurationName) if !strings.EqualFold(launchConfigName, instanceConfigName) { - r.Info("launch configuration name differs", "instance", instanceID, "name", r.RollingUpgrade.NamespacedName()) return true } } else if scalingGroup.LaunchTemplate != nil { if instance.LaunchTemplate == nil { - r.Info("launch template name differs", "instance", instanceID, "name", r.RollingUpgrade.NamespacedName()) return true } @@ -430,16 +440,13 @@ func (r *RollingUpgradeContext) IsInstanceDrifted(instance *autoscaling.Instance ) if !strings.EqualFold(launchTemplateName, instanceTemplateName) { - r.Info("launch template name differs", "instance", instanceID, "name", r.RollingUpgrade.NamespacedName()) return true } else if !strings.EqualFold(instanceTemplateVersion, templateVersion) { - r.Info("launch template version differs", "instance", instanceID, "name", r.RollingUpgrade.NamespacedName()) return true } } else if scalingGroup.MixedInstancesPolicy != nil { if instance.LaunchTemplate == nil { - r.Info("launch template name differs", "instance", instanceID, "name", r.RollingUpgrade.NamespacedName()) return true } @@ -451,15 +458,12 @@ func (r *RollingUpgradeContext) IsInstanceDrifted(instance *autoscaling.Instance ) if !strings.EqualFold(launchTemplateName, instanceTemplateName) { - r.Info("launch template name differs", "instance", instanceID, "name", r.RollingUpgrade.NamespacedName()) return true } else if !strings.EqualFold(instanceTemplateVersion, templateVersion) { - r.Info("launch template version differs", "instance", instanceID, "name", r.RollingUpgrade.NamespacedName()) return true } } - r.Info("node refresh not required", "name", r.RollingUpgrade.NamespacedName(), "instance", instanceID) return false } @@ -469,9 +473,11 @@ func (r *RollingUpgradeContext) IsScalingGroupDrifted() bool { scalingGroup := awsprovider.SelectScalingGroup(r.RollingUpgrade.ScalingGroupName(), r.Cloud.ScalingGroups) for _, instance := range scalingGroup.Instances { if r.IsInstanceDrifted(instance) { + r.Info("launch definition differs", "instance", aws.StringValue(instance.InstanceId), "name", r.RollingUpgrade.NamespacedName()) return true } } + r.Info("no drift in scaling group", "name", r.RollingUpgrade.NamespacedName()) return false } @@ -485,6 +491,7 @@ func (r *RollingUpgradeContext) DesiredNodesReady() bool { // wait for desired instances inServiceInstanceIDs := awsprovider.GetInServiceInstanceIDs(scalingGroup.Instances) if len(inServiceInstanceIDs) != int(desiredInstances) { + r.Info("desired number of instances are not InService", "desired", int(desiredInstances), "inServiceCount", len(inServiceInstanceIDs), "name", r.RollingUpgrade.NamespacedName()) return false } @@ -498,6 +505,7 @@ func (r *RollingUpgradeContext) DesiredNodesReady() bool { } } if readyNodes != int(desiredInstances) { + r.Info("desired number of nodes are not ready", "desired", int(desiredInstances), "readyNodesCount", readyNodes, "name", r.RollingUpgrade.NamespacedName()) return false } From 066731d9f0edb1b46a1238c120be7ff3f9fecbf0 Mon Sep 17 00:00:00 2001 From: Shreyas Badiger <7680410+shreyas-badiger@users.noreply.github.com> Date: Tue, 15 Jun 2021 20:49:30 -0700 Subject: [PATCH 34/74] final push before RC release. (#254) * support IgnoreDrainFailures flag Signed-off-by: sbadiger * add else condition Signed-off-by: sbadiger * set min for maxUnavailable Signed-off-by: sbadiger * calculateMaxUnavailable function Signed-off-by: sbadiger * add a new coloumn (completePercentage) Signed-off-by: sbadiger * disable debug logs by default Signed-off-by: sbadiger --- api/v1alpha1/rollingupgrade_types.go | 25 +++++-- api/v1alpha1/zz_generated.deepcopy.go | 10 ++- ...grademgr.keikoproj.io_rollingupgrades.yaml | 6 ++ controllers/rollingupgrade_controller.go | 2 +- controllers/upgrade.go | 75 ++++++++++++++----- 5 files changed, 89 insertions(+), 29 deletions(-) diff --git a/api/v1alpha1/rollingupgrade_types.go b/api/v1alpha1/rollingupgrade_types.go index af461c6b..b296b79b 100644 --- a/api/v1alpha1/rollingupgrade_types.go +++ b/api/v1alpha1/rollingupgrade_types.go @@ -18,12 +18,13 @@ package v1alpha1 import ( "fmt" + "strconv" + "strings" + "github.com/keikoproj/upgrade-manager/controllers/common" corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/intstr" - "strconv" - "strings" ) // RollingUpgradeSpec defines the desired state of RollingUpgrade @@ -48,9 +49,10 @@ type RollingUpgradeStatus struct { TotalProcessingTime string `json:"totalProcessingTime,omitempty"` NodesProcessed int `json:"nodesProcessed,omitempty"` TotalNodes int `json:"totalNodes,omitempty"` + CompletePercentage string `json:"completePercentage,omitempty"` Conditions []RollingUpgradeCondition `json:"conditions,omitempty"` - LastNodeTerminationTime metav1.Time `json:"lastTerminationTime,omitempty"` - LastNodeDrainTime metav1.Time `json:"lastDrainTime,omitempty"` + LastNodeTerminationTime *metav1.Time `json:"lastTerminationTime,omitempty"` + LastNodeDrainTime *metav1.Time `json:"lastDrainTime,omitempty"` Statistics []*RollingUpgradeStatistics `json:"statistics,omitempty"` LastBatchNodes []string `json:"lastBatchNodes,omitempty"` @@ -97,6 +99,7 @@ func (s *RollingUpgradeStatus) SetCondition(cond RollingUpgradeCondition) { // +kubebuilder:printcolumn:name="Status",type="string",JSONPath=".status.currentStatus",description="current status of the rollingupgarde" // +kubebuilder:printcolumn:name="TotalNodes",type="string",JSONPath=".status.totalNodes",description="total nodes involved in the rollingupgarde" // +kubebuilder:printcolumn:name="NodesProcessed",type="string",JSONPath=".status.nodesProcessed",description="current number of nodes processed in the rollingupgarde" +// +kubebuilder:printcolumn:name="Complete",type="string",JSONPath=".status.completePercentage",description="percentage of completion for the rollingupgrade CR" // RollingUpgrade is the Schema for the rollingupgrades API type RollingUpgrade struct { @@ -255,19 +258,19 @@ func (r *RollingUpgrade) MaxUnavailable() intstr.IntOrString { return r.Spec.Strategy.MaxUnavailable } -func (r *RollingUpgrade) LastNodeTerminationTime() metav1.Time { +func (r *RollingUpgrade) LastNodeTerminationTime() *metav1.Time { return r.Status.LastNodeTerminationTime } -func (r *RollingUpgrade) SetLastNodeTerminationTime(t metav1.Time) { +func (r *RollingUpgrade) SetLastNodeTerminationTime(t *metav1.Time) { r.Status.LastNodeTerminationTime = t } -func (r *RollingUpgrade) LastNodeDrainTime() metav1.Time { +func (r *RollingUpgrade) LastNodeDrainTime() *metav1.Time { return r.Status.LastNodeDrainTime } -func (r *RollingUpgrade) SetLastNodeDrainTime(t metav1.Time) { +func (r *RollingUpgrade) SetLastNodeDrainTime(t *metav1.Time) { r.Status.LastNodeDrainTime = t } @@ -307,6 +310,9 @@ func (r *RollingUpgrade) SetNodesProcessed(n int) { r.Status.NodesProcessed = n } +func (r *RollingUpgrade) SetCompletePercentage(n int) { + r.Status.CompletePercentage = fmt.Sprintf("%s%%", strconv.Itoa(n)) +} func (r *RollingUpgrade) GetStatus() RollingUpgradeStatus { return r.Status } @@ -315,6 +321,9 @@ func (r *RollingUpgrade) IsForceRefresh() bool { return r.Spec.ForceRefresh } +func (r *RollingUpgrade) IsIgnoreDrainFailures() bool { + return r.Spec.IgnoreDrainFailures +} func (r *RollingUpgrade) StrategyMode() UpdateStrategyMode { return r.Spec.Strategy.Mode } diff --git a/api/v1alpha1/zz_generated.deepcopy.go b/api/v1alpha1/zz_generated.deepcopy.go index 16b85e40..5e13ec90 100644 --- a/api/v1alpha1/zz_generated.deepcopy.go +++ b/api/v1alpha1/zz_generated.deepcopy.go @@ -249,8 +249,14 @@ func (in *RollingUpgradeStatus) DeepCopyInto(out *RollingUpgradeStatus) { *out = make([]RollingUpgradeCondition, len(*in)) copy(*out, *in) } - in.LastNodeTerminationTime.DeepCopyInto(&out.LastNodeTerminationTime) - in.LastNodeDrainTime.DeepCopyInto(&out.LastNodeDrainTime) + if in.LastNodeTerminationTime != nil { + in, out := &in.LastNodeTerminationTime, &out.LastNodeTerminationTime + *out = (*in).DeepCopy() + } + if in.LastNodeDrainTime != nil { + in, out := &in.LastNodeDrainTime, &out.LastNodeDrainTime + *out = (*in).DeepCopy() + } if in.Statistics != nil { in, out := &in.Statistics, &out.Statistics *out = make([]*RollingUpgradeStatistics, len(*in)) diff --git a/config/crd/bases/upgrademgr.keikoproj.io_rollingupgrades.yaml b/config/crd/bases/upgrademgr.keikoproj.io_rollingupgrades.yaml index 0bc33f02..03fc5394 100644 --- a/config/crd/bases/upgrademgr.keikoproj.io_rollingupgrades.yaml +++ b/config/crd/bases/upgrademgr.keikoproj.io_rollingupgrades.yaml @@ -31,6 +31,10 @@ spec: jsonPath: .status.nodesProcessed name: NodesProcessed type: string + - description: percentage of completion for the rollingupgrade CR + jsonPath: .status.completePercentage + name: Complete + type: string name: v1alpha1 schema: openAPIV3Schema: @@ -118,6 +122,8 @@ spec: status: description: RollingUpgradeStatus defines the observed state of RollingUpgrade properties: + completePercentage: + type: string conditions: items: description: RollingUpgradeCondition describes the state of the diff --git a/controllers/rollingupgrade_controller.go b/controllers/rollingupgrade_controller.go index 35bbe53f..e485e10e 100644 --- a/controllers/rollingupgrade_controller.go +++ b/controllers/rollingupgrade_controller.go @@ -181,6 +181,6 @@ func (r *RollingUpgradeReconciler) SetMaxParallel(n int) { func (r *RollingUpgradeReconciler) UpdateStatus(rollingUpgrade *v1alpha1.RollingUpgrade) { if err := r.Status().Update(context.Background(), rollingUpgrade); err != nil { - r.Error(err, "failed to update status", "name", rollingUpgrade.NamespacedName()) + r.Info("failed to update status", "message", err.Error(), "name", rollingUpgrade.NamespacedName()) } } diff --git a/controllers/upgrade.go b/controllers/upgrade.go index c64b0df8..4ffd35a5 100644 --- a/controllers/upgrade.go +++ b/controllers/upgrade.go @@ -17,6 +17,8 @@ limitations under the License. package controllers import ( + "fmt" + "math" "reflect" "strconv" "strings" @@ -97,6 +99,7 @@ func (r *RollingUpgradeContext) RotateNodes() error { r.Info( "scaling group details", "scalingGroup", r.RollingUpgrade.ScalingGroupName(), + "desiredInstances", aws.Int64Value(scalingGroup.DesiredCapacity), "launchConfig", aws.StringValue(scalingGroup.LaunchConfigurationName), "name", r.RollingUpgrade.NamespacedName(), ) @@ -232,7 +235,9 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) r.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationDrain) if err := r.Auth.DrainNode(&node, time.Duration(r.RollingUpgrade.PostDrainDelaySeconds()), r.RollingUpgrade.DrainTimeout(), r.Auth.Kubernetes); err != nil { - r.DrainManager.DrainErrors <- errors.Errorf("DrainNode failed: instanceID - %v, %v", instanceID, err.Error()) + if !r.RollingUpgrade.IsIgnoreDrainFailures() { + r.DrainManager.DrainErrors <- errors.Errorf("DrainNode failed: instanceID - %v, %v", instanceID, err.Error()) + } } } @@ -271,7 +276,7 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) case <-done: // goroutines completed, terminate and requeue - r.RollingUpgrade.SetLastNodeDrainTime(metav1.Time{Time: time.Now()}) + r.RollingUpgrade.SetLastNodeDrainTime(&metav1.Time{Time: time.Now()}) r.Info("instances drained successfully, terminating", "name", r.RollingUpgrade.NamespacedName()) for _, target := range batch { var ( @@ -296,7 +301,7 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) r.Info("failed to terminate instance", "instance", instanceID, "message", err.Error(), "name", r.RollingUpgrade.NamespacedName()) return true, nil } - r.RollingUpgrade.SetLastNodeTerminationTime(metav1.Time{Time: time.Now()}) + r.RollingUpgrade.SetLastNodeTerminationTime(&metav1.Time{Time: time.Now()}) // Turns onto NodeRotationTerminate r.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationPostTerminate) @@ -331,19 +336,10 @@ func (r *RollingUpgradeContext) SelectTargets(scalingGroup *autoscaling.Group) [ totalNodes = len(scalingGroup.Instances) targets = make([]*autoscaling.Instance, 0) ) - - var unavailableInt int - if batchSize.Type == intstr.String { - if strings.Contains(batchSize.StrVal, "%") { - unavailableInt, _ = intstr.GetValueFromIntOrPercent(&batchSize, totalNodes, true) - } - unavailableInt, _ = strconv.Atoi(batchSize.StrVal) - } else { - unavailableInt = batchSize.IntValue() - } + unavailableInt := CalculateMaxUnavailable(batchSize, totalNodes) // first process all in progress instances - r.Info("selecting batch for rotation", "batch size", batchSize, "name", r.RollingUpgrade.NamespacedName()) + r.Info("selecting batch for rotation", "batch size", unavailableInt, "name", r.RollingUpgrade.NamespacedName()) for _, instance := range r.Cloud.InProgressInstances { if selectedInstance := awsprovider.SelectScalingGroupInstance(instance, scalingGroup); !reflect.DeepEqual(selectedInstance, &autoscaling.Instance{}) { targets = append(targets, selectedInstance) @@ -369,7 +365,9 @@ func (r *RollingUpgradeContext) SelectTargets(scalingGroup *autoscaling.Group) [ } else if r.RollingUpgrade.UpdateStrategyType() == v1alpha1.UniformAcrossAzUpdateStrategy { for _, instance := range scalingGroup.Instances { if r.IsInstanceDrifted(instance) && !common.ContainsEqualFold(awsprovider.GetInstanceIDs(targets), aws.StringValue(instance.InstanceId)) { - targets = append(targets, instance) + if !common.ContainsEqualFold(awsprovider.TerminatingInstanceStates, aws.StringValue(instance.LifecycleState)) { + targets = append(targets, instance) + } } } @@ -468,15 +466,24 @@ func (r *RollingUpgradeContext) IsInstanceDrifted(instance *autoscaling.Instance } func (r *RollingUpgradeContext) IsScalingGroupDrifted() bool { + var ( + driftCount = 0 + scalingGroup = awsprovider.SelectScalingGroup(r.RollingUpgrade.ScalingGroupName(), r.Cloud.ScalingGroups) + desiredCapacity = int(aws.Int64Value(scalingGroup.DesiredCapacity)) + ) r.Info("checking if rolling upgrade is completed", "name", r.RollingUpgrade.NamespacedName()) - scalingGroup := awsprovider.SelectScalingGroup(r.RollingUpgrade.ScalingGroupName(), r.Cloud.ScalingGroups) for _, instance := range scalingGroup.Instances { if r.IsInstanceDrifted(instance) { - r.Info("launch definition differs", "instance", aws.StringValue(instance.InstanceId), "name", r.RollingUpgrade.NamespacedName()) - return true + driftCount++ } } + if driftCount != 0 { + r.Info("drift detected in scaling group", "driftedInstancesCount/DesiredInstancesCount", fmt.Sprintf("(%v/%v)", driftCount, desiredCapacity), "name", r.RollingUpgrade.NamespacedName()) + r.SetProgress(desiredCapacity-driftCount, desiredCapacity) + return true + } + r.SetProgress(desiredCapacity, desiredCapacity) r.Info("no drift in scaling group", "name", r.RollingUpgrade.NamespacedName()) return false } @@ -511,3 +518,35 @@ func (r *RollingUpgradeContext) DesiredNodesReady() bool { return true } + +func CalculateMaxUnavailable(batchSize intstr.IntOrString, totalNodes int) int { + var unavailableInt int + if batchSize.Type == intstr.String { + if strings.Contains(batchSize.StrVal, "%") { + unavailableInt, _ = intstr.GetValueFromIntOrPercent(&batchSize, totalNodes, true) + } else { + unavailableInt, _ = strconv.Atoi(batchSize.StrVal) + } + } else { + unavailableInt = batchSize.IntValue() + } + + // batch size should be atleast 1 + if unavailableInt == 0 { + unavailableInt = 1 + } + + // batch size should be atmost the number of nodes + if unavailableInt > totalNodes { + unavailableInt = totalNodes + } + + return unavailableInt +} + +func (r *RollingUpgradeContext) SetProgress(nodesProcessed int, totalNodes int) { + completePercentage := int(math.Round(float64(nodesProcessed) / float64(totalNodes) * 100)) + r.RollingUpgrade.SetTotalNodes(totalNodes) + r.RollingUpgrade.SetNodesProcessed(nodesProcessed) + r.RollingUpgrade.SetCompletePercentage(completePercentage) +} From f5dd1cb5f76f2b78cb15c53daed14032a2a4c6ec Mon Sep 17 00:00:00 2001 From: Sheldon Shao Date: Tue, 15 Jun 2021 20:50:47 -0700 Subject: [PATCH 35/74] Fix metrics collecting issue (#249) * metricsMutex should be initialized Signed-off-by: xshao * Use InProcessingNode instead of Stringp[] so that it can have the status of steps Signed-off-by: xshao --- api/v1alpha1/rollingupgrade_types.go | 4 ++-- controllers/metrics.go | 6 +----- controllers/upgrade.go | 5 ++++- 3 files changed, 7 insertions(+), 8 deletions(-) diff --git a/api/v1alpha1/rollingupgrade_types.go b/api/v1alpha1/rollingupgrade_types.go index b296b79b..3258c1df 100644 --- a/api/v1alpha1/rollingupgrade_types.go +++ b/api/v1alpha1/rollingupgrade_types.go @@ -54,8 +54,8 @@ type RollingUpgradeStatus struct { LastNodeTerminationTime *metav1.Time `json:"lastTerminationTime,omitempty"` LastNodeDrainTime *metav1.Time `json:"lastDrainTime,omitempty"` - Statistics []*RollingUpgradeStatistics `json:"statistics,omitempty"` - LastBatchNodes []string `json:"lastBatchNodes,omitempty"` + Statistics []*RollingUpgradeStatistics `json:"statistics,omitempty"` + LastBatchNodes map[string]*NodeInProcessing `json:"lastBatchNodes,omitempty"` } // RollingUpgrade Statistics, includes summary(sum/count) from each step diff --git a/controllers/metrics.go b/controllers/metrics.go index 95964733..8b4c69bf 100644 --- a/controllers/metrics.go +++ b/controllers/metrics.go @@ -10,11 +10,7 @@ import ( // Update last batch nodes func (s *RollingUpgradeContext) UpdateLastBatchNodes(batchNodes map[string]*v1alpha1.NodeInProcessing) { - keys := make([]string, 0, len(batchNodes)) - for k := range batchNodes { - keys = append(keys, k) - } - s.RollingUpgrade.Status.LastBatchNodes = keys + s.RollingUpgrade.Status.LastBatchNodes = batchNodes } // Update Node Statistics diff --git a/controllers/upgrade.go b/controllers/upgrade.go index 4ffd35a5..9cdeccaf 100644 --- a/controllers/upgrade.go +++ b/controllers/upgrade.go @@ -131,7 +131,10 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) //A map to retain the steps for multiple nodes nodeSteps := make(map[string][]v1alpha1.NodeStepDuration) - inProcessingNodes := make(map[string]*v1alpha1.NodeInProcessing) + inProcessingNodes := r.RollingUpgrade.Status.LastBatchNodes + if inProcessingNodes == nil { + inProcessingNodes = make(map[string]*v1alpha1.NodeInProcessing) + } switch mode { case v1alpha1.UpdateStrategyModeEager: From 376657f34506cf1b012da1814f8d114e09db5151 Mon Sep 17 00:00:00 2001 From: Shreyas Badiger <7680410+shreyas-badiger@users.noreply.github.com> Date: Wed, 16 Jun 2021 00:18:41 -0700 Subject: [PATCH 36/74] Revert "Fix metrics collecting issue (#249)" (#256) This reverts commit f5dd1cb5f76f2b78cb15c53daed14032a2a4c6ec. --- api/v1alpha1/rollingupgrade_types.go | 4 ++-- controllers/metrics.go | 6 +++++- controllers/upgrade.go | 5 +---- 3 files changed, 8 insertions(+), 7 deletions(-) diff --git a/api/v1alpha1/rollingupgrade_types.go b/api/v1alpha1/rollingupgrade_types.go index 3258c1df..b296b79b 100644 --- a/api/v1alpha1/rollingupgrade_types.go +++ b/api/v1alpha1/rollingupgrade_types.go @@ -54,8 +54,8 @@ type RollingUpgradeStatus struct { LastNodeTerminationTime *metav1.Time `json:"lastTerminationTime,omitempty"` LastNodeDrainTime *metav1.Time `json:"lastDrainTime,omitempty"` - Statistics []*RollingUpgradeStatistics `json:"statistics,omitempty"` - LastBatchNodes map[string]*NodeInProcessing `json:"lastBatchNodes,omitempty"` + Statistics []*RollingUpgradeStatistics `json:"statistics,omitempty"` + LastBatchNodes []string `json:"lastBatchNodes,omitempty"` } // RollingUpgrade Statistics, includes summary(sum/count) from each step diff --git a/controllers/metrics.go b/controllers/metrics.go index 8b4c69bf..95964733 100644 --- a/controllers/metrics.go +++ b/controllers/metrics.go @@ -10,7 +10,11 @@ import ( // Update last batch nodes func (s *RollingUpgradeContext) UpdateLastBatchNodes(batchNodes map[string]*v1alpha1.NodeInProcessing) { - s.RollingUpgrade.Status.LastBatchNodes = batchNodes + keys := make([]string, 0, len(batchNodes)) + for k := range batchNodes { + keys = append(keys, k) + } + s.RollingUpgrade.Status.LastBatchNodes = keys } // Update Node Statistics diff --git a/controllers/upgrade.go b/controllers/upgrade.go index 9cdeccaf..4ffd35a5 100644 --- a/controllers/upgrade.go +++ b/controllers/upgrade.go @@ -131,10 +131,7 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) //A map to retain the steps for multiple nodes nodeSteps := make(map[string][]v1alpha1.NodeStepDuration) - inProcessingNodes := r.RollingUpgrade.Status.LastBatchNodes - if inProcessingNodes == nil { - inProcessingNodes = make(map[string]*v1alpha1.NodeInProcessing) - } + inProcessingNodes := make(map[string]*v1alpha1.NodeInProcessing) switch mode { case v1alpha1.UpdateStrategyModeEager: From 3eafd005751d71379600efcda73a0afb24b943c4 Mon Sep 17 00:00:00 2001 From: Sheldon Shao Date: Wed, 16 Jun 2021 16:16:40 -0700 Subject: [PATCH 37/74] Fix metrics calculation issue (#258) * metricsMutex should be initialized Signed-off-by: xshao * Use InProcessingNode instead of Stringp[] so that it can have the status of steps Signed-off-by: xshao * Make the change backward compatible Signed-off-by: xshao * Make the change backward compatible Signed-off-by: xshao * Add mutex for InProcessingNode deleting Signed-off-by: xshao --- api/v1alpha1/rollingupgrade_types.go | 6 ++++-- ...grademgr.keikoproj.io_rollingupgrades.yaml | 20 +++++++++++++++++++ controllers/metrics.go | 4 ++++ controllers/upgrade.go | 5 ++++- 4 files changed, 32 insertions(+), 3 deletions(-) diff --git a/api/v1alpha1/rollingupgrade_types.go b/api/v1alpha1/rollingupgrade_types.go index b296b79b..3fdf089e 100644 --- a/api/v1alpha1/rollingupgrade_types.go +++ b/api/v1alpha1/rollingupgrade_types.go @@ -54,8 +54,10 @@ type RollingUpgradeStatus struct { LastNodeTerminationTime *metav1.Time `json:"lastTerminationTime,omitempty"` LastNodeDrainTime *metav1.Time `json:"lastDrainTime,omitempty"` - Statistics []*RollingUpgradeStatistics `json:"statistics,omitempty"` - LastBatchNodes []string `json:"lastBatchNodes,omitempty"` + Statistics []*RollingUpgradeStatistics `json:"statistics,omitempty"` + // For backward compatibility + LastBatchNodes []string `json:"lastBatchNodes,omitempty"` + NodeInProcessing map[string]*NodeInProcessing `json:"nodeInProcessing,omitempty"` } // RollingUpgrade Statistics, includes summary(sum/count) from each step diff --git a/config/crd/bases/upgrademgr.keikoproj.io_rollingupgrades.yaml b/config/crd/bases/upgrademgr.keikoproj.io_rollingupgrades.yaml index 03fc5394..3e1012ed 100644 --- a/config/crd/bases/upgrademgr.keikoproj.io_rollingupgrades.yaml +++ b/config/crd/bases/upgrademgr.keikoproj.io_rollingupgrades.yaml @@ -140,6 +140,7 @@ spec: endTime: type: string lastBatchNodes: + description: For backward compatibility items: type: string type: array @@ -149,6 +150,25 @@ spec: lastTerminationTime: format: date-time type: string + nodeInProcessing: + additionalProperties: + description: Node In-processing + properties: + nodeName: + type: string + stepEndTime: + format: date-time + type: string + stepName: + type: string + stepStartTime: + format: date-time + type: string + upgradeStartTime: + format: date-time + type: string + type: object + type: object nodesProcessed: type: integer startTime: diff --git a/controllers/metrics.go b/controllers/metrics.go index 95964733..e3986111 100644 --- a/controllers/metrics.go +++ b/controllers/metrics.go @@ -10,6 +10,7 @@ import ( // Update last batch nodes func (s *RollingUpgradeContext) UpdateLastBatchNodes(batchNodes map[string]*v1alpha1.NodeInProcessing) { + s.RollingUpgrade.Status.NodeInProcessing = batchNodes keys := make([]string, 0, len(batchNodes)) for k := range batchNodes { keys = append(keys, k) @@ -87,6 +88,9 @@ func (s *RollingUpgradeContext) NodeStep(InProcessingNodes map[string]*v1alpha1. duration2 := s.ToStepDuration(groupName, nodeName, v1alpha1.NodeRotationTotal, total) s.addNodeStepDuration(nodeSteps, nodeName, duration1) s.addNodeStepDuration(nodeSteps, nodeName, duration2) + s.metricsMutex.Lock() + delete(InProcessingNodes, nodeName) + s.metricsMutex.Unlock() } else if inProcessingNode.StepName != stepName { //Still same step var oldOrder = v1alpha1.NodeRotationStepOrders[inProcessingNode.StepName] var newOrder = v1alpha1.NodeRotationStepOrders[stepName] diff --git a/controllers/upgrade.go b/controllers/upgrade.go index 4ffd35a5..e27df803 100644 --- a/controllers/upgrade.go +++ b/controllers/upgrade.go @@ -131,7 +131,10 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) //A map to retain the steps for multiple nodes nodeSteps := make(map[string][]v1alpha1.NodeStepDuration) - inProcessingNodes := make(map[string]*v1alpha1.NodeInProcessing) + inProcessingNodes := r.RollingUpgrade.Status.NodeInProcessing + if inProcessingNodes == nil { + inProcessingNodes = make(map[string]*v1alpha1.NodeInProcessing) + } switch mode { case v1alpha1.UpdateStrategyModeEager: From 79db022205ed7566dfc5804d2ed3be0abd169e16 Mon Sep 17 00:00:00 2001 From: Shreyas Badiger <7680410+shreyas-badiger@users.noreply.github.com> Date: Mon, 21 Jun 2021 17:01:36 -0700 Subject: [PATCH 38/74] Add a mock for test and update version in Makefile (#262) Signed-off-by: sbadiger --- Makefile | 2 +- api/v1alpha1/zz_generated.deepcopy.go | 15 +++++++++++++++ controllers/helpers_test.go | 16 ++++++++++++++++ controllers/providers/aws/autoscaling.go | 5 +++++ controllers/upgrade.go | 9 +++++---- controllers/upgrade_test.go | 4 ++-- go.mod | 3 ++- go.sum | 2 ++ 8 files changed, 48 insertions(+), 8 deletions(-) diff --git a/Makefile b/Makefile index c4f78a33..72bcec3d 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ - +VERSION=1.0.0-RC1 # Image URL to use all building/pushing image targets IMG ?= controller:latest # Produce CRDs that work back to Kubernetes 1.11 (no version conversion) diff --git a/api/v1alpha1/zz_generated.deepcopy.go b/api/v1alpha1/zz_generated.deepcopy.go index 5e13ec90..62489a6f 100644 --- a/api/v1alpha1/zz_generated.deepcopy.go +++ b/api/v1alpha1/zz_generated.deepcopy.go @@ -273,6 +273,21 @@ func (in *RollingUpgradeStatus) DeepCopyInto(out *RollingUpgradeStatus) { *out = make([]string, len(*in)) copy(*out, *in) } + if in.NodeInProcessing != nil { + in, out := &in.NodeInProcessing, &out.NodeInProcessing + *out = make(map[string]*NodeInProcessing, len(*in)) + for key, val := range *in { + var outVal *NodeInProcessing + if val == nil { + (*out)[key] = nil + } else { + in, out := &val, &outVal + *out = new(NodeInProcessing) + (*in).DeepCopyInto(*out) + } + (*out)[key] = outVal + } + } } // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new RollingUpgradeStatus. diff --git a/controllers/helpers_test.go b/controllers/helpers_test.go index c9e98dca..3faeae5b 100644 --- a/controllers/helpers_test.go +++ b/controllers/helpers_test.go @@ -3,6 +3,7 @@ package controllers import ( "sync" "testing" + "time" "k8s.io/client-go/kubernetes/fake" ctrl "sigs.k8s.io/controller-runtime" @@ -197,3 +198,18 @@ func createAmazonClient(t *testing.T) *awsprovider.AmazonClientSet { Ec2Client: createEc2Client(), } } + +func (mockAutoscalingGroup MockAutoscalingGroup) TerminateInstanceInAutoScalingGroup(input *autoscaling.TerminateInstanceInAutoScalingGroupInput) (*autoscaling.TerminateInstanceInAutoScalingGroupOutput, error) { + output := &autoscaling.TerminateInstanceInAutoScalingGroupOutput{} + if mockAutoscalingGroup.errorFlag { + if mockAutoscalingGroup.awsErr != nil { + if len(mockAutoscalingGroup.errorInstanceId) <= 0 || + mockAutoscalingGroup.errorInstanceId == *input.InstanceId { + return output, mockAutoscalingGroup.awsErr + } + } + } + asgChange := autoscaling.Activity{ActivityId: aws.String("xxx"), AutoScalingGroupName: aws.String("sss"), Cause: aws.String("xxx"), StartTime: aws.Time(time.Now()), StatusCode: aws.String("200"), StatusMessage: aws.String("success")} + output.Activity = &asgChange + return output, nil +} diff --git a/controllers/providers/aws/autoscaling.go b/controllers/providers/aws/autoscaling.go index a971bdbf..3d66cee9 100644 --- a/controllers/providers/aws/autoscaling.go +++ b/controllers/providers/aws/autoscaling.go @@ -26,6 +26,11 @@ var ( autoscaling.LifecycleStateTerminating, autoscaling.LifecycleStateTerminatingWait, autoscaling.LifecycleStateTerminatingProceed, + autoscaling.LifecycleStateTerminated, + autoscaling.LifecycleStateWarmedTerminating, + autoscaling.LifecycleStateWarmedTerminatingWait, + autoscaling.LifecycleStateWarmedTerminatingProceed, + autoscaling.LifecycleStateWarmedTerminated, } ) diff --git a/controllers/upgrade.go b/controllers/upgrade.go index e27df803..77954e6d 100644 --- a/controllers/upgrade.go +++ b/controllers/upgrade.go @@ -357,7 +357,10 @@ func (r *RollingUpgradeContext) SelectTargets(scalingGroup *autoscaling.Group) [ if r.RollingUpgrade.UpdateStrategyType() == v1alpha1.RandomUpdateStrategy { for _, instance := range scalingGroup.Instances { if r.IsInstanceDrifted(instance) && !common.ContainsEqualFold(awsprovider.GetInstanceIDs(targets), aws.StringValue(instance.InstanceId)) { - targets = append(targets, instance) + //In-progress instances shouldn't be considered if they are in terminating state. + if !common.ContainsEqualFold(awsprovider.TerminatingInstanceStates, aws.StringValue(instance.LifecycleState)) { + targets = append(targets, instance) + } } } if unavailableInt > len(targets) { @@ -368,9 +371,7 @@ func (r *RollingUpgradeContext) SelectTargets(scalingGroup *autoscaling.Group) [ } else if r.RollingUpgrade.UpdateStrategyType() == v1alpha1.UniformAcrossAzUpdateStrategy { for _, instance := range scalingGroup.Instances { if r.IsInstanceDrifted(instance) && !common.ContainsEqualFold(awsprovider.GetInstanceIDs(targets), aws.StringValue(instance.InstanceId)) { - if !common.ContainsEqualFold(awsprovider.TerminatingInstanceStates, aws.StringValue(instance.LifecycleState)) { - targets = append(targets, instance) - } + targets = append(targets, instance) } } diff --git a/controllers/upgrade_test.go b/controllers/upgrade_test.go index 45890395..ee96189b 100644 --- a/controllers/upgrade_test.go +++ b/controllers/upgrade_test.go @@ -291,7 +291,7 @@ func TestRotateNodes(t *testing.T) { v1alpha1.StatusRunning, }, { - "All instances have different launch config as the ASG, RotateNodes() will mark CR complete", + "All instances have same launch config as the ASG, RotateNodes() will mark CR complete", createRollingUpgradeReconciler(t), createASGClient(), false, @@ -305,7 +305,7 @@ func TestRotateNodes(t *testing.T) { err := rollupCtx.RotateNodes() if err != nil { - t.Errorf("Test Description: \n expected value: nil, actual value: %v", err) + t.Errorf("Test Description: %s \n error: %v", test.TestDescription, err) } if rollupCtx.RollingUpgrade.CurrentStatus() != test.ExpectedStatusValue { t.Errorf("Test Description: %s \n expected value: %s, actual value: %s", test.TestDescription, test.ExpectedStatusValue, rollupCtx.RollingUpgrade.CurrentStatus()) diff --git a/go.mod b/go.mod index b444654f..f114b375 100644 --- a/go.mod +++ b/go.mod @@ -3,7 +3,7 @@ module github.com/keikoproj/upgrade-manager go 1.15 require ( - github.com/aws/aws-sdk-go v1.36.24 + github.com/aws/aws-sdk-go v1.38.24 github.com/go-logr/logr v0.3.0 github.com/keikoproj/aws-sdk-go-cache v0.0.0-20201118182730-f6f418a4e2df github.com/onsi/gomega v1.10.2 @@ -16,4 +16,5 @@ require ( k8s.io/client-go v0.20.4 k8s.io/kubectl v0.20.4 sigs.k8s.io/controller-runtime v0.7.0 + ) diff --git a/go.sum b/go.sum index 36cb2d64..b2aebc80 100644 --- a/go.sum +++ b/go.sum @@ -88,6 +88,8 @@ github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a/go.mod h1:l github.com/aws/aws-sdk-go v1.35.7/go.mod h1:tlPOdRjfxPBpNIwqDj61rmsnA85v9jc0Ps9+muhnW+k= github.com/aws/aws-sdk-go v1.36.24 h1:uVuio0zA5ideP3DGZDpIoExQJd0WcoNUVlNZaKwBnf8= github.com/aws/aws-sdk-go v1.36.24/go.mod h1:hcU610XS61/+aQV88ixoOzUoG7v3b31pl2zKMmprdro= +github.com/aws/aws-sdk-go v1.38.24 h1:zbKHDxFepE77ihVMZ+wZ62Ci646zkorN8rB5s4fj4kU= +github.com/aws/aws-sdk-go v1.38.24/go.mod h1:hcU610XS61/+aQV88ixoOzUoG7v3b31pl2zKMmprdro= github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q= github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8= github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM= From 2390ea03e738b44287bf4b5912d84b4ba170473f Mon Sep 17 00:00:00 2001 From: Shreyas Badiger <7680410+shreyas-badiger@users.noreply.github.com> Date: Wed, 23 Jun 2021 15:28:39 -0700 Subject: [PATCH 39/74] and CR end time (#264) Signed-off-by: sbadiger --- api/v1alpha1/rollingupgrade_types.go | 4 ++++ controllers/upgrade.go | 16 ++++++++++++++++ 2 files changed, 20 insertions(+) diff --git a/api/v1alpha1/rollingupgrade_types.go b/api/v1alpha1/rollingupgrade_types.go index 3fdf089e..177d9edf 100644 --- a/api/v1alpha1/rollingupgrade_types.go +++ b/api/v1alpha1/rollingupgrade_types.go @@ -304,6 +304,10 @@ func (r *RollingUpgrade) EndTime() string { return r.Status.EndTime } +func (r *RollingUpgrade) SetTotalProcessingTime(t string) { + r.Status.TotalProcessingTime = t +} + func (r *RollingUpgrade) SetTotalNodes(n int) { r.Status.TotalNodes = n } diff --git a/controllers/upgrade.go b/controllers/upgrade.go index 77954e6d..d770e5c9 100644 --- a/controllers/upgrade.go +++ b/controllers/upgrade.go @@ -110,6 +110,7 @@ func (r *RollingUpgradeContext) RotateNodes() error { if !r.IsScalingGroupDrifted() { r.RollingUpgrade.SetCurrentStatus(v1alpha1.StatusComplete) common.SetMetricRollupCompleted(r.RollingUpgrade.Name) + r.endTimeUpdate() return nil } @@ -554,3 +555,18 @@ func (r *RollingUpgradeContext) SetProgress(nodesProcessed int, totalNodes int) r.RollingUpgrade.SetNodesProcessed(nodesProcessed) r.RollingUpgrade.SetCompletePercentage(completePercentage) } + +func (r *RollingUpgradeContext) endTimeUpdate() { + //set end time + r.RollingUpgrade.SetEndTime(time.Now().Format(time.RFC3339)) + + //set total processing time + startTime, err1 := time.Parse(time.RFC3339, r.RollingUpgrade.StartTime()) + endTime, err2 := time.Parse(time.RFC3339, r.RollingUpgrade.EndTime()) + if err1 != nil || err2 != nil { + r.Info("failed to calculate totalProcessingTime") + } else { + var totalProcessingTime = endTime.Sub(startTime) + r.RollingUpgrade.SetTotalProcessingTime(totalProcessingTime.String()) + } +} From a4e0e84a049f370084e63deccc8edddc941ba08c Mon Sep 17 00:00:00 2001 From: Shreyas Badiger <7680410+shreyas-badiger@users.noreply.github.com> Date: Wed, 23 Jun 2021 16:35:24 -0700 Subject: [PATCH 40/74] upgrade-manager-v2: expose totalProcessing time and other metrics (#265) * and CR end time Signed-off-by: sbadiger * expose totalProcessing time and other metrics Signed-off-by: sbadiger * addressing review comments Signed-off-by: sbadiger --- controllers/common/metrics.go | 69 +++++++++++++++++++++++++++++++++++ controllers/upgrade.go | 23 ++++++++++++ 2 files changed, 92 insertions(+) diff --git a/controllers/common/metrics.go b/controllers/common/metrics.go index 2d5cd829..f1909cc0 100644 --- a/controllers/common/metrics.go +++ b/controllers/common/metrics.go @@ -33,6 +33,12 @@ var ( }, }) + totalProcessingTime = make(map[string]prometheus.Summary) + + totalNodesMetrics = make(map[string]prometheus.Gauge) + + nodesProcessedMetrics = make(map[string]prometheus.Gauge) + stepSummaries = make(map[string]map[string]prometheus.Summary) stepSumMutex = sync.Mutex{} @@ -55,6 +61,69 @@ func InitMetrics() { metrics.Registry.MustRegister(CRStatus) } +// observe total processing time +func TotalProcessingTime(groupName string, duration time.Duration) { + var summary prometheus.Summary + if s, ok := totalProcessingTime[groupName]; !ok { + summary = prometheus.NewSummary( + prometheus.SummaryOpts{ + Namespace: metricNamespace, + Name: "total_processing_time_seconds", + Help: "Total processing time for all nodes in the group", + ConstLabels: prometheus.Labels{"group": groupName}, + }) + err := metrics.Registry.Register(summary) + if err != nil { + if reflect.TypeOf(err).String() == "prometheus.AlreadyRegisteredError" { + log.Warnf("summary was registered again, group: %s", groupName) + } else { + log.Errorf("register summary error, group: %s, %v", groupName, err) + } + } + stepSumMutex.Lock() + totalProcessingTime[groupName] = summary + stepSumMutex.Unlock() + } else { + summary = s + } + summary.Observe(duration.Seconds()) +} + +func addGaugeOrUpdate(gaugeMap map[string]prometheus.Gauge, groupName string, count int, metricName, help string) { + var gauge prometheus.Gauge + if c, ok := gaugeMap[groupName]; !ok { + gauge = prometheus.NewGauge( + prometheus.GaugeOpts{ + Namespace: metricNamespace, + Name: metricName, + Help: help, + ConstLabels: prometheus.Labels{"group": groupName}, + }) + err := metrics.Registry.Register(gauge) + if err != nil { + if reflect.TypeOf(err).String() == "prometheus.AlreadyRegisteredError" { + log.Warnf("gauge was registered again, group: %s", groupName) + } else { + log.Errorf("register gauge error, group: %s, %v", groupName, err) + } + } + stepSumMutex.Lock() + gaugeMap[groupName] = gauge + stepSumMutex.Unlock() + } else { + gauge = c + } + gauge.Set(float64(count)) +} + +func SetTotalNodesMetric(groupName string, nodes int) { + addGaugeOrUpdate(totalNodesMetrics, groupName, nodes, "total_nodes", "Total nodes in the group") +} + +func SetNodesProcessedMetric(groupName string, nodesProcessed int) { + addGaugeOrUpdate(nodesProcessedMetrics, groupName, nodesProcessed, "nodes_processed", "Nodes processed in the group") +} + // Add rolling update step duration when the step is completed func AddStepDuration(groupName string, stepName string, duration time.Duration) { if strings.EqualFold(stepName, "total") { //Histogram diff --git a/controllers/upgrade.go b/controllers/upgrade.go index d770e5c9..a5c12695 100644 --- a/controllers/upgrade.go +++ b/controllers/upgrade.go @@ -554,6 +554,29 @@ func (r *RollingUpgradeContext) SetProgress(nodesProcessed int, totalNodes int) r.RollingUpgrade.SetTotalNodes(totalNodes) r.RollingUpgrade.SetNodesProcessed(nodesProcessed) r.RollingUpgrade.SetCompletePercentage(completePercentage) + + // expose total nodes and nodes processed to prometheus + common.SetTotalNodesMetric(r.RollingUpgrade.ScalingGroupName(), totalNodes) + common.SetNodesProcessedMetric(r.RollingUpgrade.ScalingGroupName(), nodesProcessed) + +} + +func (r *RollingUpgradeContext) endTimeUpdate() { + // set end time + r.RollingUpgrade.SetEndTime(time.Now().Format(time.RFC3339)) + + // set total processing time + startTime, err1 := time.Parse(time.RFC3339, r.RollingUpgrade.StartTime()) + endTime, err2 := time.Parse(time.RFC3339, r.RollingUpgrade.EndTime()) + if err1 != nil || err2 != nil { + r.Info("failed to calculate totalProcessingTime") + } else { + var totalProcessingTime = endTime.Sub(startTime) + r.RollingUpgrade.SetTotalProcessingTime(totalProcessingTime.String()) + + // expose total processing time to prometheus + common.TotalProcessingTime(r.RollingUpgrade.ScalingGroupName(), totalProcessingTime) + } } func (r *RollingUpgradeContext) endTimeUpdate() { From 610f45435350f59cfb848f6047b381d63711e010 Mon Sep 17 00:00:00 2001 From: Shreyas Badiger <7680410+shreyas-badiger@users.noreply.github.com> Date: Wed, 23 Jun 2021 19:25:09 -0700 Subject: [PATCH 41/74] upgrade-manager-v2: remove function duplicate declaration. (#266) * and CR end time Signed-off-by: sbadiger * expose totalProcessing time and other metrics Signed-off-by: sbadiger * addressing review comments Signed-off-by: sbadiger * remove function duplication Signed-off-by: sbadiger --- controllers/upgrade.go | 15 --------------- 1 file changed, 15 deletions(-) diff --git a/controllers/upgrade.go b/controllers/upgrade.go index a5c12695..5312c6a5 100644 --- a/controllers/upgrade.go +++ b/controllers/upgrade.go @@ -578,18 +578,3 @@ func (r *RollingUpgradeContext) endTimeUpdate() { common.TotalProcessingTime(r.RollingUpgrade.ScalingGroupName(), totalProcessingTime) } } - -func (r *RollingUpgradeContext) endTimeUpdate() { - //set end time - r.RollingUpgrade.SetEndTime(time.Now().Format(time.RFC3339)) - - //set total processing time - startTime, err1 := time.Parse(time.RFC3339, r.RollingUpgrade.StartTime()) - endTime, err2 := time.Parse(time.RFC3339, r.RollingUpgrade.EndTime()) - if err1 != nil || err2 != nil { - r.Info("failed to calculate totalProcessingTime") - } else { - var totalProcessingTime = endTime.Sub(startTime) - r.RollingUpgrade.SetTotalProcessingTime(totalProcessingTime.String()) - } -} From b15838e670692ceb793f5168564ef684e491a10c Mon Sep 17 00:00:00 2001 From: Sheldon Shao Date: Thu, 24 Jun 2021 00:18:28 -0700 Subject: [PATCH 42/74] Carry the metrics status in RollingUpgrade CR (#267) * Update metrics status at same time Signed-off-by: xshao * Update metrics status when terminating instance Signed-off-by: xshao * Add terminated step Signed-off-by: xshao * Add terminated step Signed-off-by: xshao * Add terminated step Signed-off-by: xshao --- api/v1alpha1/rollingupgrade_types.go | 2 ++ controllers/metrics.go | 21 ++++++++++++++++----- controllers/upgrade.go | 22 ++++++++++++++-------- 3 files changed, 32 insertions(+), 13 deletions(-) diff --git a/api/v1alpha1/rollingupgrade_types.go b/api/v1alpha1/rollingupgrade_types.go index 177d9edf..2563964f 100644 --- a/api/v1alpha1/rollingupgrade_types.go +++ b/api/v1alpha1/rollingupgrade_types.go @@ -168,6 +168,7 @@ const ( NodeRotationPostWait RollingUpgradeStep = "post_wait" NodeRotationTerminate RollingUpgradeStep = "terminate" NodeRotationPostTerminate RollingUpgradeStep = "post_terminate" + NodeRotationTerminated RollingUpgradeStep = "terminated" NodeRotationCompleted RollingUpgradeStep = "completed" ) @@ -180,6 +181,7 @@ var NodeRotationStepOrders = map[RollingUpgradeStep]int{ NodeRotationPostWait: 60, NodeRotationTerminate: 70, NodeRotationPostTerminate: 80, + NodeRotationTerminated: 90, NodeRotationCompleted: 1000, } diff --git a/controllers/metrics.go b/controllers/metrics.go index e3986111..337b91d2 100644 --- a/controllers/metrics.go +++ b/controllers/metrics.go @@ -8,6 +8,12 @@ import ( metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" ) +// Update metrics status UpdateMetricsStatus +func (s *RollingUpgradeContext) UpdateMetricsStatus(batchNodes map[string]*v1alpha1.NodeInProcessing, nodeSteps map[string][]v1alpha1.NodeStepDuration) { + s.UpdateLastBatchNodes(batchNodes) + s.UpdateStatistics(nodeSteps) +} + // Update last batch nodes func (s *RollingUpgradeContext) UpdateLastBatchNodes(batchNodes map[string]*v1alpha1.NodeInProcessing) { s.RollingUpgrade.Status.NodeInProcessing = batchNodes @@ -62,9 +68,8 @@ func (s *RollingUpgradeContext) ToStepDuration(groupName, nodeName string, stepN } } -// Node turns onto step -func (s *RollingUpgradeContext) NodeStep(InProcessingNodes map[string]*v1alpha1.NodeInProcessing, - nodeSteps map[string][]v1alpha1.NodeStepDuration, groupName, nodeName string, stepName v1alpha1.RollingUpgradeStep) { +func (s *RollingUpgradeContext) DoNodeStep(InProcessingNodes map[string]*v1alpha1.NodeInProcessing, + nodeSteps map[string][]v1alpha1.NodeStepDuration, groupName, nodeName string, stepName v1alpha1.RollingUpgradeStep, endTime metav1.Time) { var inProcessingNode *v1alpha1.NodeInProcessing if n, ok := InProcessingNodes[nodeName]; !ok { @@ -79,7 +84,7 @@ func (s *RollingUpgradeContext) NodeStep(InProcessingNodes map[string]*v1alpha1. inProcessingNode = n } - inProcessingNode.StepEndTime = metav1.Now() + inProcessingNode.StepEndTime = endTime var duration = inProcessingNode.StepEndTime.Sub(inProcessingNode.StepStartTime.Time) if stepName == v1alpha1.NodeRotationCompleted { //Add overall and remove the node from in-processing map @@ -96,13 +101,19 @@ func (s *RollingUpgradeContext) NodeStep(InProcessingNodes map[string]*v1alpha1. var newOrder = v1alpha1.NodeRotationStepOrders[stepName] if newOrder > oldOrder { //Make sure the steps running in order stepDuration := s.ToStepDuration(groupName, nodeName, inProcessingNode.StepName, duration) - inProcessingNode.StepStartTime = metav1.Now() + inProcessingNode.StepStartTime = endTime inProcessingNode.StepName = stepName s.addNodeStepDuration(nodeSteps, nodeName, stepDuration) } } } +// Node turns onto step +func (s *RollingUpgradeContext) NodeStep(InProcessingNodes map[string]*v1alpha1.NodeInProcessing, + nodeSteps map[string][]v1alpha1.NodeStepDuration, groupName, nodeName string, stepName v1alpha1.RollingUpgradeStep) { + s.DoNodeStep(InProcessingNodes, nodeSteps, groupName, nodeName, stepName, metav1.Now()) +} + func (s *RollingUpgradeContext) addNodeStepDuration(steps map[string][]v1alpha1.NodeStepDuration, nodeName string, nsd v1alpha1.NodeStepDuration) { s.metricsMutex.Lock() if stepDuration, ok := steps[nodeName]; !ok { diff --git a/controllers/upgrade.go b/controllers/upgrade.go index 5312c6a5..a959702d 100644 --- a/controllers/upgrade.go +++ b/controllers/upgrade.go @@ -156,6 +156,7 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) r.Info("setting instances to in-progress", "batch", batchInstanceIDs, "instances(InService)", inServiceInstanceIDs, "name", r.RollingUpgrade.NamespacedName()) if err := r.Auth.TagEC2instances(inServiceInstanceIDs, instanceStateTagKey, inProgressTagValue); err != nil { r.Error(err, "failed to set instances to in-progress", "batch", batchInstanceIDs, "instances(InService)", inServiceInstanceIDs, "name", r.RollingUpgrade.NamespacedName()) + r.UpdateMetricsStatus(inProcessingNodes, nodeSteps) return false, err } // Standby @@ -164,6 +165,7 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) r.Info("failed to set instances to stand-by", "batch", batchInstanceIDs, "instances(InService)", inServiceInstanceIDs, "message", err.Error(), "name", r.RollingUpgrade.NamespacedName()) } // requeue until there are no InService instances in the batch + r.UpdateMetricsStatus(inProcessingNodes, nodeSteps) return true, nil } else { r.Info("no InService instances in the batch", "batch", batchInstanceIDs, "instances(InService)", inServiceInstanceIDs, "name", r.RollingUpgrade.NamespacedName()) @@ -183,6 +185,7 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) r.Info("waiting for desired nodes", "name", r.RollingUpgrade.NamespacedName()) if !r.DesiredNodesReady() { r.Info("new node is yet to join the cluster", "name", r.RollingUpgrade.NamespacedName()) + r.UpdateMetricsStatus(inProcessingNodes, nodeSteps) return true, nil } r.Info("desired nodes are ready", "name", r.RollingUpgrade.NamespacedName()) @@ -202,6 +205,7 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) r.Info("setting batch to in-progress", "batch", batchInstanceIDs, "instances(InService)", inServiceInstanceIDs, "name", r.RollingUpgrade.NamespacedName()) if err := r.Auth.TagEC2instances(inServiceInstanceIDs, instanceStateTagKey, inProgressTagValue); err != nil { r.Error(err, "failed to set batch in-progress", "batch", batchInstanceIDs, "instances(InService)", inServiceInstanceIDs, "name", r.RollingUpgrade.NamespacedName()) + r.UpdateMetricsStatus(inProcessingNodes, nodeSteps) return false, err } } @@ -272,8 +276,7 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) select { case err := <-r.DrainManager.DrainErrors: - r.UpdateStatistics(nodeSteps) - r.UpdateLastBatchNodes(inProcessingNodes) + r.UpdateMetricsStatus(inProcessingNodes, nodeSteps) r.Error(err, "failed to rotate the node", "name", r.RollingUpgrade.NamespacedName()) return false, err @@ -303,6 +306,7 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) if err := r.Auth.TerminateInstance(target); err != nil { // terminate failures are retryable r.Info("failed to terminate instance", "instance", instanceID, "message", err.Error(), "name", r.RollingUpgrade.NamespacedName()) + r.UpdateMetricsStatus(inProcessingNodes, nodeSteps) return true, nil } r.RollingUpgrade.SetLastNodeTerminationTime(&metav1.Time{Time: time.Now()}) @@ -315,18 +319,20 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) return false, err } - // Turns onto NodeRotationCompleted - r.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationCompleted) + //Calculate the terminating time, + terminatedTime := metav1.Time{ + Time: metav1.Now().Add(time.Duration(r.RollingUpgrade.NodeIntervalSeconds()) * time.Second), + } + r.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationTerminated) + r.DoNodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationCompleted, terminatedTime) } - r.UpdateStatistics(nodeSteps) - r.UpdateLastBatchNodes(inProcessingNodes) + r.UpdateMetricsStatus(inProcessingNodes, nodeSteps) case <-time.After(DefaultWaitGroupTimeout): // goroutines timed out - requeue - r.UpdateStatistics(nodeSteps) - r.UpdateLastBatchNodes(inProcessingNodes) + r.UpdateMetricsStatus(inProcessingNodes, nodeSteps) r.Info("instances are still draining", "name", r.RollingUpgrade.NamespacedName()) return true, nil From c0a163b8916aee002e3ac1c4e329d6b5e2c4a9c0 Mon Sep 17 00:00:00 2001 From: Shreyas Badiger <7680410+shreyas-badiger@users.noreply.github.com> Date: Tue, 29 Jun 2021 14:31:38 -0700 Subject: [PATCH 43/74] move cloud discovery after nodeInterval / drainInterval wait (#270) Signed-off-by: sbadiger --- controllers/rollingupgrade_controller.go | 7 ------ controllers/upgrade.go | 27 +++++++++++++++--------- 2 files changed, 17 insertions(+), 17 deletions(-) diff --git a/controllers/rollingupgrade_controller.go b/controllers/rollingupgrade_controller.go index e485e10e..48babd4d 100644 --- a/controllers/rollingupgrade_controller.go +++ b/controllers/rollingupgrade_controller.go @@ -146,13 +146,6 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque RollingUpgrade: rollingUpgrade, metricsMutex: &sync.Mutex{}, } - rollupCtx.Cloud = NewDiscoveredState(rollupCtx.Auth, rollupCtx.Logger) - if err := rollupCtx.Cloud.Discover(); err != nil { - r.Info("failed to discover the cloud", "scalingGroup", scalingGroupName, "name", rollingUpgrade.NamespacedName()) - rollingUpgrade.SetCurrentStatus(v1alpha1.StatusError) - common.SetMetricRollupFailed(rollingUpgrade.Name) - return ctrl.Result{}, err - } // process node rotation if err := rollupCtx.RotateNodes(); err != nil { diff --git a/controllers/upgrade.go b/controllers/upgrade.go index a959702d..4894d0de 100644 --- a/controllers/upgrade.go +++ b/controllers/upgrade.go @@ -59,7 +59,6 @@ type RollingUpgradeContext struct { metricsMutex *sync.Mutex } -// TODO: main node rotation logic func (r *RollingUpgradeContext) RotateNodes() error { var ( lastTerminationTime = r.RollingUpgrade.LastNodeTerminationTime() @@ -67,16 +66,8 @@ func (r *RollingUpgradeContext) RotateNodes() error { lastDrainTime = r.RollingUpgrade.LastNodeDrainTime() drainInterval = r.RollingUpgrade.PostDrainDelaySeconds() ) - r.RollingUpgrade.SetCurrentStatus(v1alpha1.StatusRunning) - common.SetMetricRollupInitOrRunning(r.RollingUpgrade.Name) - - // set status start time - if r.RollingUpgrade.StartTime() == "" { - r.RollingUpgrade.SetStartTime(time.Now().Format(time.RFC3339)) - } if !lastTerminationTime.IsZero() || !lastDrainTime.IsZero() { - // Check if we are still waiting on a termination delay if time.Since(lastTerminationTime.Time).Seconds() < float64(nodeInterval) { r.Info("reconcile requeue due to termination interval wait", "name", r.RollingUpgrade.NamespacedName()) @@ -90,6 +81,22 @@ func (r *RollingUpgradeContext) RotateNodes() error { } } + // set status start time + if r.RollingUpgrade.StartTime() == "" { + r.RollingUpgrade.SetStartTime(time.Now().Format(time.RFC3339)) + } + r.RollingUpgrade.SetCurrentStatus(v1alpha1.StatusRunning) + common.SetMetricRollupInitOrRunning(r.RollingUpgrade.Name) + + // discover the state of AWS and K8s cluster. + r.Cloud = NewDiscoveredState(r.Auth, r.Logger) + if err := r.Cloud.Discover(); err != nil { + r.Info("failed to discover the cloud", "scalingGroup", r.RollingUpgrade.ScalingGroupName(), "name", r.RollingUpgrade.NamespacedName()) + r.RollingUpgrade.SetCurrentStatus(v1alpha1.StatusError) + common.SetMetricRollupFailed(r.RollingUpgrade.Name) + return err + } + var ( scalingGroup = awsprovider.SelectScalingGroup(r.RollingUpgrade.ScalingGroupName(), r.Cloud.ScalingGroups) ) @@ -245,6 +252,7 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) if err := r.Auth.DrainNode(&node, time.Duration(r.RollingUpgrade.PostDrainDelaySeconds()), r.RollingUpgrade.DrainTimeout(), r.Auth.Kubernetes); err != nil { if !r.RollingUpgrade.IsIgnoreDrainFailures() { r.DrainManager.DrainErrors <- errors.Errorf("DrainNode failed: instanceID - %v, %v", instanceID, err.Error()) + //TODO: BREAK AFTER ERRORS? } } } @@ -326,7 +334,6 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) r.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationTerminated) r.DoNodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationCompleted, terminatedTime) } - r.UpdateMetricsStatus(inProcessingNodes, nodeSteps) case <-time.After(DefaultWaitGroupTimeout): From b2b39a0654d6de1b969e965968ab689a3c1f4bf5 Mon Sep 17 00:00:00 2001 From: Shreyas Badiger <7680410+shreyas-badiger@users.noreply.github.com> Date: Mon, 12 Jul 2021 12:22:13 -0700 Subject: [PATCH 44/74] upgrade-manager-v2: Add nodeEvents handler instead of a watch handler (#272) * upgrade-manager-v2: remove function duplicate declaration. (#266) * and CR end time Signed-off-by: sbadiger * expose totalProcessing time and other metrics Signed-off-by: sbadiger * addressing review comments Signed-off-by: sbadiger * remove function duplication Signed-off-by: sbadiger * Carry the metrics status in RollingUpgrade CR (#267) * Update metrics status at same time Signed-off-by: xshao * Update metrics status when terminating instance Signed-off-by: xshao * Add terminated step Signed-off-by: xshao * Add terminated step Signed-off-by: xshao * Add terminated step Signed-off-by: xshao Signed-off-by: sbadiger * move cloud discovery after nodeInterval / drainInterval wait Signed-off-by: sbadiger * Add watch event for cluster nodes instead of API calls Signed-off-by: sbadiger * upon node deletion, remove it from syncMap as well Signed-off-by: sbadiger * Add nodeEvents handler instead of watch handler Signed-off-by: sbadiger * Ignore Reconciles on nodeEvents Signed-off-by: sbadiger * Add comments Signed-off-by: sbadiger Co-authored-by: Sheldon Shao --- config/rbac/role.yaml | 1 + controllers/cloud.go | 8 +- controllers/helpers_test.go | 128 ++++++++++++++++++++-- controllers/providers/kubernetes/utils.go | 15 ++- controllers/rollingupgrade_controller.go | 67 ++++++++++- controllers/upgrade.go | 75 ++++++++----- controllers/upgrade_test.go | 53 +++------ main.go | 1 + 8 files changed, 257 insertions(+), 91 deletions(-) diff --git a/config/rbac/role.yaml b/config/rbac/role.yaml index 5c4ea57f..b01753b3 100644 --- a/config/rbac/role.yaml +++ b/config/rbac/role.yaml @@ -35,6 +35,7 @@ rules: - get - list - patch + - watch - apiGroups: - "" resources: diff --git a/controllers/cloud.go b/controllers/cloud.go index 2fea2bd7..dc19f72d 100644 --- a/controllers/cloud.go +++ b/controllers/cloud.go @@ -34,7 +34,7 @@ var ( type DiscoveredState struct { *RollingUpgradeAuthenticator logr.Logger - ClusterNodes *corev1.NodeList + ClusterNodes []*corev1.Node LaunchTemplates []*ec2.LaunchTemplate ScalingGroups []*autoscaling.Group InProgressInstances []string @@ -67,11 +67,5 @@ func (d *DiscoveredState) Discover() error { } d.InProgressInstances = inProgressInstances - nodes, err := d.KubernetesClientSet.ListClusterNodes() - if err != nil || nodes == nil || nodes.Size() == 0 { - return errors.Wrap(err, "failed to discover cluster nodes") - } - d.ClusterNodes = nodes - return nil } diff --git a/controllers/helpers_test.go b/controllers/helpers_test.go index 3faeae5b..50fe6cfa 100644 --- a/controllers/helpers_test.go +++ b/controllers/helpers_test.go @@ -1,6 +1,7 @@ package controllers import ( + "strings" "sync" "testing" "time" @@ -125,9 +126,17 @@ func createNodeList() *corev1.NodeList { } } -func createNode() *corev1.Node { +func createNodeSlice() []*corev1.Node { + return []*corev1.Node{ + createNode("mock-node-1"), + createNode("mock-node-2"), + createNode("mock-node-3"), + } +} + +func createNode(name string) *corev1.Node { return &corev1.Node{ - ObjectMeta: metav1.ObjectMeta{Name: "mock-node-1", Namespace: "default"}, + ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: "default"}, Spec: corev1.NodeSpec{ProviderID: "foo-bar/mock-instance-1"}, Status: corev1.NodeStatus{ Conditions: []corev1.NodeCondition{ @@ -140,18 +149,27 @@ func createNode() *corev1.Node { // AWS type MockAutoscalingGroup struct { autoscalingiface.AutoScalingAPI - errorFlag bool - awsErr awserr.Error - errorInstanceId string - autoScalingGroups []*autoscaling.Group + errorFlag bool + awsErr awserr.Error + errorInstanceId string + autoScalingGroups []*autoscaling.Group + Groups map[string]*autoscaling.Group + LaunchConfigurations map[string]*autoscaling.LaunchConfiguration } +type launchTemplateInfo struct { + data *ec2.ResponseLaunchTemplateData + name *string +} type MockEC2 struct { ec2iface.EC2API - awsErr awserr.Error - reservations []*ec2.Reservation + awsErr awserr.Error + reservations []*ec2.Reservation + LaunchTemplates map[string]*launchTemplateInfo } +var _ ec2iface.EC2API = &MockEC2{} + func createASGInstance(instanceID string, launchConfigName string) *autoscaling.Instance { return &autoscaling.Instance{ InstanceId: &instanceID, @@ -188,8 +206,8 @@ func createASGClient() *MockAutoscalingGroup { } } -func createEc2Client() MockEC2 { - return MockEC2{} +func createEc2Client() *MockEC2 { + return &MockEC2{} } func createAmazonClient(t *testing.T) *awsprovider.AmazonClientSet { @@ -199,6 +217,8 @@ func createAmazonClient(t *testing.T) *awsprovider.AmazonClientSet { } } +/******************************* AWS MOCKS *******************************/ + func (mockAutoscalingGroup MockAutoscalingGroup) TerminateInstanceInAutoScalingGroup(input *autoscaling.TerminateInstanceInAutoScalingGroupInput) (*autoscaling.TerminateInstanceInAutoScalingGroupOutput, error) { output := &autoscaling.TerminateInstanceInAutoScalingGroupOutput{} if mockAutoscalingGroup.errorFlag { @@ -213,3 +233,91 @@ func (mockAutoscalingGroup MockAutoscalingGroup) TerminateInstanceInAutoScalingG output.Activity = &asgChange return output, nil } + +// DescribeLaunchTemplatesPages mocks the describing the launch templates +func (m *MockEC2) DescribeLaunchTemplatesPages(request *ec2.DescribeLaunchTemplatesInput, callback func(*ec2.DescribeLaunchTemplatesOutput, bool) bool) error { + page, err := m.DescribeLaunchTemplates(request) + if err != nil { + return err + } + + callback(page, false) + + return nil +} + +// DescribeLaunchTemplates mocks the describing the launch templates +func (m *MockEC2) DescribeLaunchTemplates(request *ec2.DescribeLaunchTemplatesInput) (*ec2.DescribeLaunchTemplatesOutput, error) { + + o := &ec2.DescribeLaunchTemplatesOutput{} + + if m.LaunchTemplates == nil { + return o, nil + } + + for id, ltInfo := range m.LaunchTemplates { + launchTemplatetName := aws.StringValue(ltInfo.name) + + allFiltersMatch := true + for _, filter := range request.Filters { + filterName := aws.StringValue(filter.Name) + filterValue := aws.StringValue(filter.Values[0]) + + filterMatches := false + if filterName == "tag:Name" && filterValue == launchTemplatetName { + filterMatches = true + } + if strings.HasPrefix(filterName, "tag:kubernetes.io/cluster/") { + filterMatches = true + } + + if !filterMatches { + allFiltersMatch = false + break + } + } + + if allFiltersMatch { + o.LaunchTemplates = append(o.LaunchTemplates, &ec2.LaunchTemplate{ + LaunchTemplateName: aws.String(launchTemplatetName), + LaunchTemplateId: aws.String(id), + }) + } + } + + return o, nil +} + +func (m *MockAutoscalingGroup) DescribeAutoScalingGroupsPages(request *autoscaling.DescribeAutoScalingGroupsInput, callback func(*autoscaling.DescribeAutoScalingGroupsOutput, bool) bool) error { + // For the mock, we just send everything in one page + page, err := m.DescribeAutoScalingGroups(request) + if err != nil { + return err + } + + callback(page, false) + + return nil +} + +func (m *MockAutoscalingGroup) DescribeAutoScalingGroups(input *autoscaling.DescribeAutoScalingGroupsInput) (*autoscaling.DescribeAutoScalingGroupsOutput, error) { + return &autoscaling.DescribeAutoScalingGroupsOutput{ + AutoScalingGroups: createASGs(), + }, nil +} + +func (m *MockEC2) DescribeInstancesPages(request *ec2.DescribeInstancesInput, callback func(*ec2.DescribeInstancesOutput, bool) bool) error { + // For the mock, we just send everything in one page + page, err := m.DescribeInstances(request) + if err != nil { + return err + } + + callback(page, false) + + return nil +} + +func (m *MockEC2) DescribeInstances(*ec2.DescribeInstancesInput) (*ec2.DescribeInstancesOutput, error) { + return &ec2.DescribeInstancesOutput{}, nil +} diff --git a/controllers/providers/kubernetes/utils.go b/controllers/providers/kubernetes/utils.go index 997d2709..2e557416 100644 --- a/controllers/providers/kubernetes/utils.go +++ b/controllers/providers/kubernetes/utils.go @@ -20,7 +20,6 @@ import ( "fmt" "os" "os/user" - "reflect" "strings" corev1 "k8s.io/api/core/v1" @@ -87,20 +86,20 @@ func GetKubernetesLocalConfig() (*rest.Config, error) { return config, nil } -func SelectNodeByInstanceID(instanceID string, nodes *corev1.NodeList) corev1.Node { +func SelectNodeByInstanceID(instanceID string, nodes []*corev1.Node) *corev1.Node { if nodes != nil { - for _, node := range nodes.Items { + for _, node := range nodes { nodeID := GetNodeInstanceID(node) if strings.EqualFold(instanceID, nodeID) { return node } } } - return corev1.Node{} + return nil } -func GetNodeInstanceID(node corev1.Node) string { - if !reflect.DeepEqual(node, &corev1.Node{}) { +func GetNodeInstanceID(node *corev1.Node) string { + if node != nil { tokens := strings.Split(node.Spec.ProviderID, "/") nodeInstanceID := tokens[len(tokens)-1] return nodeInstanceID @@ -108,7 +107,7 @@ func GetNodeInstanceID(node corev1.Node) string { return "" } -func IsNodeReady(node corev1.Node) bool { +func IsNodeReady(node *corev1.Node) bool { for _, condition := range node.Status.Conditions { if condition.Type == corev1.NodeReady && condition.Status == corev1.ConditionTrue { return true @@ -117,7 +116,7 @@ func IsNodeReady(node corev1.Node) bool { return false } -func IsNodePassesReadinessGates(node corev1.Node, requiredReadinessGates []v1alpha1.NodeReadinessGate) bool { +func IsNodePassesReadinessGates(node *corev1.Node, requiredReadinessGates []v1alpha1.NodeReadinessGate) bool { if len(requiredReadinessGates) == 0 { return true } diff --git a/controllers/rollingupgrade_controller.go b/controllers/rollingupgrade_controller.go index 48babd4d..63e7f70d 100644 --- a/controllers/rollingupgrade_controller.go +++ b/controllers/rollingupgrade_controller.go @@ -24,14 +24,19 @@ import ( "github.com/keikoproj/aws-sdk-go-cache/cache" "github.com/keikoproj/upgrade-manager/api/v1alpha1" "github.com/keikoproj/upgrade-manager/controllers/common" + "github.com/keikoproj/upgrade-manager/controllers/common/log" awsprovider "github.com/keikoproj/upgrade-manager/controllers/providers/aws" kubeprovider "github.com/keikoproj/upgrade-manager/controllers/providers/kubernetes" + corev1 "k8s.io/api/core/v1" kerrors "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/runtime" ctrl "sigs.k8s.io/controller-runtime" "sigs.k8s.io/controller-runtime/pkg/client" "sigs.k8s.io/controller-runtime/pkg/controller" + "sigs.k8s.io/controller-runtime/pkg/event" + "sigs.k8s.io/controller-runtime/pkg/predicate" "sigs.k8s.io/controller-runtime/pkg/reconcile" + "sigs.k8s.io/controller-runtime/pkg/source" ) // RollingUpgradeReconciler reconciles a RollingUpgrade object @@ -47,6 +52,7 @@ type RollingUpgradeReconciler struct { Auth *RollingUpgradeAuthenticator DrainGroupMapper *sync.Map DrainErrorMapper *sync.Map + ClusterNodesMap *sync.Map } type RollingUpgradeAuthenticator struct { @@ -56,7 +62,7 @@ type RollingUpgradeAuthenticator struct { // +kubebuilder:rbac:groups=upgrademgr.keikoproj.io,resources=rollingupgrades,verbs=get;list;watch;create;update;patch;delete // +kubebuilder:rbac:groups=upgrademgr.keikoproj.io,resources=rollingupgrades/status,verbs=get;update;patch -// +kubebuilder:rbac:groups=core,resources=nodes,verbs=get;list;patch +// +kubebuilder:rbac:groups=core,resources=nodes,verbs=get;list;patch;watch // +kubebuilder:rbac:groups=core,resources=pods,verbs=list // +kubebuilder:rbac:groups=core,resources=events,verbs=create // +kubebuilder:rbac:groups=core,resources=pods/eviction,verbs=create @@ -145,6 +151,13 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque }, RollingUpgrade: rollingUpgrade, metricsMutex: &sync.Mutex{}, + + // discover the K8s cluster at controller level through watch + Cloud: func() *DiscoveredState { + var c = NewDiscoveredState(r.Auth, r.Logger) + c.ClusterNodes = r.getClusterNodes() + return c + }(), } // process node rotation @@ -161,10 +174,48 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque func (r *RollingUpgradeReconciler) SetupWithManager(mgr ctrl.Manager) error { return ctrl.NewControllerManagedBy(mgr). For(&v1alpha1.RollingUpgrade{}). + Watches(&source.Kind{Type: &corev1.Node{}}, nil). + WithEventFilter(r.nodeEventsHandler()). WithOptions(controller.Options{MaxConcurrentReconciles: r.maxParallel}). Complete(r) } +// nodesEventHandler will fetch us the nodes on corresponding events, an alternative to doing explicit API calls. +func (r *RollingUpgradeReconciler) nodeEventsHandler() predicate.Predicate { + return predicate.Funcs{ + CreateFunc: func(e event.CreateEvent) bool { + nodeObj, ok := e.Object.(*corev1.Node) + if ok { + nodeName := e.Object.GetName() + log.Debug("nodeEventsHandler[create] nodeObj created, stored in sync map", "nodeName", nodeName) + r.ClusterNodesMap.Store(nodeName, nodeObj) + return false + } + return true + }, + UpdateFunc: func(e event.UpdateEvent) bool { + nodeObj, ok := e.ObjectNew.(*corev1.Node) + if ok { + nodeName := e.ObjectNew.GetName() + log.Debug("nodeEventsHandler[update] nodeObj updated, updated in sync map", "nodeName", nodeName) + r.ClusterNodesMap.Store(nodeName, nodeObj) + return false + } + return true + }, + DeleteFunc: func(e event.DeleteEvent) bool { + _, ok := e.Object.(*corev1.Node) + if ok { + nodeName := e.Object.GetName() + r.ClusterNodesMap.Delete(nodeName) + log.Debug("nodeEventsHandler[delete] - nodeObj not found, deleted from sync map", "name", nodeName) + return false + } + return true + }, + } +} + func (r *RollingUpgradeReconciler) SetMaxParallel(n int) { if n >= 1 { r.Info("setting max parallel reconcile", "value", n) @@ -177,3 +228,17 @@ func (r *RollingUpgradeReconciler) UpdateStatus(rollingUpgrade *v1alpha1.Rolling r.Info("failed to update status", "message", err.Error(), "name", rollingUpgrade.NamespacedName()) } } + +func (r *RollingUpgradeReconciler) getClusterNodes() []*corev1.Node { + var clusterNodes []*corev1.Node + + m := map[string]interface{}{} + r.ClusterNodesMap.Range(func(key, value interface{}) bool { + m[fmt.Sprint(key)] = value + return true + }) + for _, value := range m { + clusterNodes = append(clusterNodes, value.(*corev1.Node)) + } + return clusterNodes +} diff --git a/controllers/upgrade.go b/controllers/upgrade.go index 4894d0de..ea09f825 100644 --- a/controllers/upgrade.go +++ b/controllers/upgrade.go @@ -60,6 +60,14 @@ type RollingUpgradeContext struct { } func (r *RollingUpgradeContext) RotateNodes() error { + + // set status start time + if r.RollingUpgrade.StartTime() == "" { + r.RollingUpgrade.SetStartTime(time.Now().Format(time.RFC3339)) + } + r.RollingUpgrade.SetCurrentStatus(v1alpha1.StatusRunning) + common.SetMetricRollupInitOrRunning(r.RollingUpgrade.Name) + var ( lastTerminationTime = r.RollingUpgrade.LastNodeTerminationTime() nodeInterval = r.RollingUpgrade.NodeIntervalSeconds() @@ -81,15 +89,7 @@ func (r *RollingUpgradeContext) RotateNodes() error { } } - // set status start time - if r.RollingUpgrade.StartTime() == "" { - r.RollingUpgrade.SetStartTime(time.Now().Format(time.RFC3339)) - } - r.RollingUpgrade.SetCurrentStatus(v1alpha1.StatusRunning) - common.SetMetricRollupInitOrRunning(r.RollingUpgrade.Name) - // discover the state of AWS and K8s cluster. - r.Cloud = NewDiscoveredState(r.Auth, r.Logger) if err := r.Cloud.Discover(); err != nil { r.Info("failed to discover the cloud", "scalingGroup", r.RollingUpgrade.ScalingGroupName(), "name", r.RollingUpgrade.NamespacedName()) r.RollingUpgrade.SetCurrentStatus(v1alpha1.StatusError) @@ -147,10 +147,15 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) switch mode { case v1alpha1.UpdateStrategyModeEager: for _, target := range batch { + instanceID := aws.StringValue(target.InstanceId) + node := kubeprovider.SelectNodeByInstanceID(instanceID, r.Cloud.ClusterNodes) + if node == nil { + r.Info("node object not found in clusterNodes, skipping this node for now", "instanceID", instanceID, "name", r.RollingUpgrade.NamespacedName()) + continue + } + var ( - instanceID = aws.StringValue(target.InstanceId) - node = kubeprovider.SelectNodeByInstanceID(instanceID, r.Cloud.ClusterNodes) - nodeName = node.GetName() + nodeName = node.GetName() ) //Add statistics r.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationKickoff) @@ -180,10 +185,14 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) // turns onto desired nodes for _, target := range batch { + instanceID := aws.StringValue(target.InstanceId) + node := kubeprovider.SelectNodeByInstanceID(instanceID, r.Cloud.ClusterNodes) + if node == nil { + r.Info("node object not found in clusterNodes, skipping this node for now", "instanceID", instanceID, "name", r.RollingUpgrade.NamespacedName()) + continue + } var ( - instanceID = aws.StringValue(target.InstanceId) - node = kubeprovider.SelectNodeByInstanceID(instanceID, r.Cloud.ClusterNodes) - nodeName = node.GetName() + nodeName = node.GetName() ) r.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationDesiredNodeReady) } @@ -199,10 +208,14 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) case v1alpha1.UpdateStrategyModeLazy: for _, target := range batch { + instanceID := aws.StringValue(target.InstanceId) + node := kubeprovider.SelectNodeByInstanceID(instanceID, r.Cloud.ClusterNodes) + if node == nil { + r.Info("node object not found in clusterNodes, skipping this node for now", "instanceID", instanceID, "name", r.RollingUpgrade.NamespacedName()) + continue + } var ( - instanceID = aws.StringValue(target.InstanceId) - node = kubeprovider.SelectNodeByInstanceID(instanceID, r.Cloud.ClusterNodes) - nodeName = node.GetName() + nodeName = node.GetName() ) // add statistics r.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationKickoff) @@ -219,9 +232,13 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) if reflect.DeepEqual(r.DrainManager.DrainGroup, &sync.WaitGroup{}) { for _, target := range batch { + instanceID := aws.StringValue(target.InstanceId) + node := kubeprovider.SelectNodeByInstanceID(instanceID, r.Cloud.ClusterNodes) + if node == nil { + r.Info("node object not found in clusterNodes, skipping this node for now", "instanceID", instanceID, "name", r.RollingUpgrade.NamespacedName()) + continue + } var ( - instanceID = aws.StringValue(target.InstanceId) - node = kubeprovider.SelectNodeByInstanceID(instanceID, r.Cloud.ClusterNodes) nodeName = node.GetName() scriptTarget = ScriptTarget{ InstanceID: instanceID, @@ -243,13 +260,13 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) } // Issue drain concurrently - set lastDrainTime - if node := kubeprovider.SelectNodeByInstanceID(instanceID, r.Cloud.ClusterNodes); !reflect.DeepEqual(node, corev1.Node{}) { + if node := kubeprovider.SelectNodeByInstanceID(instanceID, r.Cloud.ClusterNodes); node != nil { r.Info("draining the node", "instance", instanceID, "node name", node.Name, "name", r.RollingUpgrade.NamespacedName()) // Turns onto NodeRotationDrain r.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationDrain) - if err := r.Auth.DrainNode(&node, time.Duration(r.RollingUpgrade.PostDrainDelaySeconds()), r.RollingUpgrade.DrainTimeout(), r.Auth.Kubernetes); err != nil { + if err := r.Auth.DrainNode(node, time.Duration(r.RollingUpgrade.PostDrainDelaySeconds()), r.RollingUpgrade.DrainTimeout(), r.Auth.Kubernetes); err != nil { if !r.RollingUpgrade.IsIgnoreDrainFailures() { r.DrainManager.DrainErrors <- errors.Errorf("DrainNode failed: instanceID - %v, %v", instanceID, err.Error()) //TODO: BREAK AFTER ERRORS? @@ -294,9 +311,13 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) r.RollingUpgrade.SetLastNodeDrainTime(&metav1.Time{Time: time.Now()}) r.Info("instances drained successfully, terminating", "name", r.RollingUpgrade.NamespacedName()) for _, target := range batch { + instanceID := aws.StringValue(target.InstanceId) + node := kubeprovider.SelectNodeByInstanceID(instanceID, r.Cloud.ClusterNodes) + if node == nil { + r.Info("node object not found in clusterNodes, skipping this node for now", "instanceID", instanceID, "name", r.RollingUpgrade.NamespacedName()) + continue + } var ( - instanceID = aws.StringValue(target.InstanceId) - node = kubeprovider.SelectNodeByInstanceID(instanceID, r.Cloud.ClusterNodes) nodeName = node.GetName() scriptTarget = ScriptTarget{ InstanceID: instanceID, @@ -423,8 +444,12 @@ func (r *RollingUpgradeContext) IsInstanceDrifted(instance *autoscaling.Instance // check if there is atleast one node that meets the force-referesh criteria if r.RollingUpgrade.IsForceRefresh() { + node := kubeprovider.SelectNodeByInstanceID(instanceID, r.Cloud.ClusterNodes) + if node == nil { + r.Info("node object not found in clusterNodes, skipping this node for now", "instanceID", instanceID, "name", r.RollingUpgrade.NamespacedName()) + return false + } var ( - node = kubeprovider.SelectNodeByInstanceID(instanceID, r.Cloud.ClusterNodes) nodeCreationTime = node.CreationTimestamp.Time upgradeCreationTime = r.RollingUpgrade.CreationTimestamp.Time ) @@ -522,7 +547,7 @@ func (r *RollingUpgradeContext) DesiredNodesReady() bool { // wait for desired nodes if r.Cloud.ClusterNodes != nil && !reflect.DeepEqual(r.Cloud.ClusterNodes, &corev1.NodeList{}) { - for _, node := range r.Cloud.ClusterNodes.Items { + for _, node := range r.Cloud.ClusterNodes { instanceID := kubeprovider.GetNodeInstanceID(node) if common.ContainsEqualFold(inServiceInstanceIDs, instanceID) && kubeprovider.IsNodeReady(node) && kubeprovider.IsNodePassesReadinessGates(node, r.RollingUpgrade.Spec.ReadinessGates) { readyNodes++ diff --git a/controllers/upgrade_test.go b/controllers/upgrade_test.go index ee96189b..794ff68d 100644 --- a/controllers/upgrade_test.go +++ b/controllers/upgrade_test.go @@ -6,7 +6,6 @@ import ( drain "k8s.io/kubectl/pkg/drain" - "reflect" "time" corev1 "k8s.io/api/core/v1" @@ -17,32 +16,6 @@ import ( "github.com/keikoproj/upgrade-manager/api/v1alpha1" ) -func TestListClusterNodes(t *testing.T) { - var tests = []struct { - TestDescription string - Reconciler *RollingUpgradeReconciler - Node *corev1.Node - ExpectError bool - }{ - { - "List cluster should succeed", - createRollingUpgradeReconciler(t), - createNode(), - false, - }, - } - - for _, test := range tests { - rollupCtx := createRollingUpgradeContext(test.Reconciler) - - actual, err := rollupCtx.Auth.ListClusterNodes() - expected := createNodeList() - if err != nil || !reflect.DeepEqual(actual, expected) { - t.Errorf("ListClusterNodes fail %v", err) - } - } -} - // This test checks implementation of our DrainNode which does both cordon + drain func TestDrainNode(t *testing.T) { var tests = []struct { @@ -54,7 +27,7 @@ func TestDrainNode(t *testing.T) { { "Drain should succeed as node is registered with fakeClient", createRollingUpgradeReconciler(t), - createNode(), + createNode("mock-node-1"), false, }, { @@ -92,7 +65,7 @@ func TestRunCordonOrUncordon(t *testing.T) { { "Cordon should succeed as node is registered with fakeClient", createRollingUpgradeReconciler(t), - createNode(), + createNode("mock-node-1"), true, false, }, @@ -107,7 +80,7 @@ func TestRunCordonOrUncordon(t *testing.T) { "Uncordon should succeed as node is registered with fakeClient", createRollingUpgradeReconciler(t), func() *corev1.Node { - node := createNode() + node := createNode("mock-node-1") node.Spec.Unschedulable = true return node }(), @@ -166,7 +139,7 @@ func TestRunDrainNode(t *testing.T) { { "Drain should succeed as node is registered with fakeClient", createRollingUpgradeReconciler(t), - createNode(), + createNode("mock-node-1"), false, }, // This test should fail, create an upstream ticket. @@ -319,14 +292,14 @@ func TestDesiredNodesReady(t *testing.T) { TestDescription string Reconciler *RollingUpgradeReconciler AsgClient *MockAutoscalingGroup - ClusterNodes *corev1.NodeList + ClusterNodes []*corev1.Node ExpectedValue bool }{ { "Desired nodes are ready", createRollingUpgradeReconciler(t), createASGClient(), - createNodeList(), + createNodeSlice(), true, }, { @@ -337,23 +310,23 @@ func TestDesiredNodesReady(t *testing.T) { newAsgClient.autoScalingGroups[0].DesiredCapacity = func(x int) *int64 { i := int64(x); return &i }(4) return newAsgClient }(), - createNodeList(), + createNodeSlice(), false, }, { "None of the nodes are ready (desiredCount != readyCount)", createRollingUpgradeReconciler(t), createASGClient(), - func() *corev1.NodeList { - var nodeList = &corev1.NodeList{Items: []corev1.Node{}} + func() []*corev1.Node { + var nodeSlice []*corev1.Node for i := 0; i < 3; i++ { - node := createNode() + node := createNode("mock-node-1") node.Status.Conditions = []corev1.NodeCondition{ {Type: corev1.NodeReady, Status: corev1.ConditionFalse}, } - nodeList.Items = append(nodeList.Items, *node) + nodeSlice = append(nodeSlice, node) } - return nodeList + return nodeSlice }(), false, }, @@ -369,7 +342,7 @@ func TestDesiredNodesReady(t *testing.T) { } return newAsgClient }(), - createNodeList(), + createNodeSlice(), false, }, } diff --git a/main.go b/main.go index f6efe702..88a0e58f 100644 --- a/main.go +++ b/main.go @@ -199,6 +199,7 @@ func main() { }, DrainGroupMapper: &sync.Map{}, DrainErrorMapper: &sync.Map{}, + ClusterNodesMap: &sync.Map{}, } reconciler.SetMaxParallel(maxParallel) From 0e64929729112cc69ba3105392abdcafea6b30f4 Mon Sep 17 00:00:00 2001 From: Shreyas Badiger <7680410+shreyas-badiger@users.noreply.github.com> Date: Tue, 20 Jul 2021 11:19:36 -0700 Subject: [PATCH 45/74] upgrade-manager-v2: Process next batch while waiting on nodeInterval period. (#273) * upgrade-manager-v2: remove function duplicate declaration. (#266) * and CR end time Signed-off-by: sbadiger * expose totalProcessing time and other metrics Signed-off-by: sbadiger * addressing review comments Signed-off-by: sbadiger * remove function duplication Signed-off-by: sbadiger * Carry the metrics status in RollingUpgrade CR (#267) * Update metrics status at same time Signed-off-by: xshao * Update metrics status when terminating instance Signed-off-by: xshao * Add terminated step Signed-off-by: xshao * Add terminated step Signed-off-by: xshao * Add terminated step Signed-off-by: xshao Signed-off-by: sbadiger * move cloud discovery after nodeInterval / drainInterval wait Signed-off-by: sbadiger * Add watch event for cluster nodes instead of API calls Signed-off-by: sbadiger * upon node deletion, remove it from syncMap as well Signed-off-by: sbadiger * Add nodeEvents handler instead of watch handler Signed-off-by: sbadiger * Ignore Reconciles on nodeEvents Signed-off-by: sbadiger * Add comments Signed-off-by: sbadiger * Set nextbatch to standBy while waiting for terminate * Avoid parallel reconcile operation per ASG * add default requeue time Co-authored-by: Sheldon Shao --- api/v1alpha1/rollingupgrade_types.go | 2 + controllers/rollingupgrade_controller.go | 34 +++++++++----- controllers/upgrade.go | 59 ++++++++++-------------- main.go | 1 + 4 files changed, 51 insertions(+), 45 deletions(-) diff --git a/api/v1alpha1/rollingupgrade_types.go b/api/v1alpha1/rollingupgrade_types.go index 2563964f..078b1c3e 100644 --- a/api/v1alpha1/rollingupgrade_types.go +++ b/api/v1alpha1/rollingupgrade_types.go @@ -20,6 +20,7 @@ import ( "fmt" "strconv" "strings" + "time" "github.com/keikoproj/upgrade-manager/controllers/common" corev1 "k8s.io/api/core/v1" @@ -189,6 +190,7 @@ var ( FiniteStates = []string{StatusComplete, StatusError} AllowedStrategyType = []string{string(RandomUpdateStrategy), string(UniformAcrossAzUpdateStrategy)} AllowedStrategyMode = []string{string(UpdateStrategyModeLazy), string(UpdateStrategyModeEager)} + DefaultRequeueTime = time.Second * 30 ) // RollingUpgradeCondition describes the state of the RollingUpgrade diff --git a/controllers/rollingupgrade_controller.go b/controllers/rollingupgrade_controller.go index 63e7f70d..37d196ef 100644 --- a/controllers/rollingupgrade_controller.go +++ b/controllers/rollingupgrade_controller.go @@ -18,7 +18,6 @@ import ( "fmt" "strings" "sync" - "time" "github.com/go-logr/logr" "github.com/keikoproj/aws-sdk-go-cache/cache" @@ -53,8 +52,10 @@ type RollingUpgradeReconciler struct { DrainGroupMapper *sync.Map DrainErrorMapper *sync.Map ClusterNodesMap *sync.Map + ReconcileMap *sync.Map } +// RollingUpgradeAuthenticator has the clients for providers type RollingUpgradeAuthenticator struct { *awsprovider.AmazonClientSet *kubeprovider.KubernetesClientSet @@ -69,7 +70,7 @@ type RollingUpgradeAuthenticator struct { // +kubebuilder:rbac:groups=extensions;apps,resources=daemonsets;replicasets;statefulsets,verbs=get // +kubebuilder:rbac:groups=batch,resources=jobs,verbs=get -// Reconcile reads that state of the cluster for a RollingUpgrade object and makes changes based on the state read +// reconcile reads that state of the cluster for a RollingUpgrade object and makes changes based on the state read // and the details in the RollingUpgrade.Spec func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { r.Info("***Reconciling***") @@ -84,14 +85,14 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque return ctrl.Result{}, err } - // If the resource is being deleted, remove it from the admissionMap + // if the resource is being deleted, remove it from the admissionMap if !rollingUpgrade.DeletionTimestamp.IsZero() { r.AdmissionMap.Delete(rollingUpgrade.NamespacedName()) r.Info("rolling upgrade deleted", "name", rollingUpgrade.NamespacedName()) return reconcile.Result{}, nil } - // Stop processing upgrades which are in finite state + // stop processing upgrades which are in finite state currentStatus := rollingUpgrade.CurrentStatus() if common.ContainsEqualFold(v1alpha1.FiniteStates, currentStatus) { r.AdmissionMap.Delete(rollingUpgrade.NamespacedName()) @@ -103,13 +104,19 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque return reconcile.Result{}, err } + // defer a status update on the resource + defer r.UpdateStatus(rollingUpgrade) + var ( scalingGroupName = rollingUpgrade.ScalingGroupName() inProgress bool ) - // Defer a status update on the resource - defer r.UpdateStatus(rollingUpgrade) + // at any given point in time, there should be only one reconcile operation running per ASG + if _, present := r.ReconcileMap.LoadOrStore(rollingUpgrade.NamespacedName(), scalingGroupName); present == true { + r.Info("a reconcile operation is already in progress for this ASG, requeuing", "scalingGroup", scalingGroupName, "name", rollingUpgrade.NamespacedName()) + return ctrl.Result{RequeueAfter: v1alpha1.DefaultRequeueTime}, nil + } // handle condition where multiple resources submitted targeting the same scaling group by requeing r.AdmissionMap.Range(func(k, v interface{}) bool { @@ -125,7 +132,7 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque if inProgress { // requeue any resources which are already being processed by a different resource, until the resource is completed/deleted - return ctrl.Result{RequeueAfter: time.Second * 30}, nil + return ctrl.Result{RequeueAfter: v1alpha1.DefaultRequeueTime}, nil } // store the rolling upgrade in admission map @@ -138,6 +145,7 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque rollingUpgrade.SetCurrentStatus(v1alpha1.StatusInit) common.SetMetricRollupInitOrRunning(rollingUpgrade.Name) + // setup the RollingUpgradeContext needed for node rotations. drainGroup, _ := r.DrainGroupMapper.LoadOrStore(rollingUpgrade.NamespacedName(), &sync.WaitGroup{}) drainErrs, _ := r.DrainErrorMapper.LoadOrStore(rollingUpgrade.NamespacedName(), make(chan error)) @@ -167,7 +175,7 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque return ctrl.Result{}, err } - return reconcile.Result{RequeueAfter: time.Second * 10}, nil + return reconcile.Result{RequeueAfter: v1alpha1.DefaultRequeueTime}, nil } // SetupWithManager sets up the controller with the Manager. @@ -175,13 +183,13 @@ func (r *RollingUpgradeReconciler) SetupWithManager(mgr ctrl.Manager) error { return ctrl.NewControllerManagedBy(mgr). For(&v1alpha1.RollingUpgrade{}). Watches(&source.Kind{Type: &corev1.Node{}}, nil). - WithEventFilter(r.nodeEventsHandler()). + WithEventFilter(r.NodeEventsHandler()). WithOptions(controller.Options{MaxConcurrentReconciles: r.maxParallel}). Complete(r) } -// nodesEventHandler will fetch us the nodes on corresponding events, an alternative to doing explicit API calls. -func (r *RollingUpgradeReconciler) nodeEventsHandler() predicate.Predicate { +// NodesEventHandler will fetch us the nodes on corresponding events, an alternative to doing explicit API calls. +func (r *RollingUpgradeReconciler) NodeEventsHandler() predicate.Predicate { return predicate.Funcs{ CreateFunc: func(e event.CreateEvent) bool { nodeObj, ok := e.Object.(*corev1.Node) @@ -216,6 +224,7 @@ func (r *RollingUpgradeReconciler) nodeEventsHandler() predicate.Predicate { } } +// number of reconciles the upgrade-manager should handle in parallel func (r *RollingUpgradeReconciler) SetMaxParallel(n int) { if n >= 1 { r.Info("setting max parallel reconcile", "value", n) @@ -223,12 +232,15 @@ func (r *RollingUpgradeReconciler) SetMaxParallel(n int) { } } +// at the end of every reconcile, update the RollingUpgrade object func (r *RollingUpgradeReconciler) UpdateStatus(rollingUpgrade *v1alpha1.RollingUpgrade) { + r.ReconcileMap.LoadAndDelete(rollingUpgrade.NamespacedName()) if err := r.Status().Update(context.Background(), rollingUpgrade); err != nil { r.Info("failed to update status", "message", err.Error(), "name", rollingUpgrade.NamespacedName()) } } +// extract node objects from syncMap to a slice func (r *RollingUpgradeReconciler) getClusterNodes() []*corev1.Node { var clusterNodes []*corev1.Node diff --git a/controllers/upgrade.go b/controllers/upgrade.go index ea09f825..4b5ddc19 100644 --- a/controllers/upgrade.go +++ b/controllers/upgrade.go @@ -60,35 +60,6 @@ type RollingUpgradeContext struct { } func (r *RollingUpgradeContext) RotateNodes() error { - - // set status start time - if r.RollingUpgrade.StartTime() == "" { - r.RollingUpgrade.SetStartTime(time.Now().Format(time.RFC3339)) - } - r.RollingUpgrade.SetCurrentStatus(v1alpha1.StatusRunning) - common.SetMetricRollupInitOrRunning(r.RollingUpgrade.Name) - - var ( - lastTerminationTime = r.RollingUpgrade.LastNodeTerminationTime() - nodeInterval = r.RollingUpgrade.NodeIntervalSeconds() - lastDrainTime = r.RollingUpgrade.LastNodeDrainTime() - drainInterval = r.RollingUpgrade.PostDrainDelaySeconds() - ) - - if !lastTerminationTime.IsZero() || !lastDrainTime.IsZero() { - // Check if we are still waiting on a termination delay - if time.Since(lastTerminationTime.Time).Seconds() < float64(nodeInterval) { - r.Info("reconcile requeue due to termination interval wait", "name", r.RollingUpgrade.NamespacedName()) - return nil - } - - // Check if we are still waiting on a drain delay - if time.Since(lastDrainTime.Time).Seconds() < float64(drainInterval) { - r.Info("reconcile requeue due to drain interval wait", "name", r.RollingUpgrade.NamespacedName()) - return nil - } - } - // discover the state of AWS and K8s cluster. if err := r.Cloud.Discover(); err != nil { r.Info("failed to discover the cloud", "scalingGroup", r.RollingUpgrade.ScalingGroupName(), "name", r.RollingUpgrade.NamespacedName()) @@ -230,6 +201,24 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) } } + var ( + lastTerminationTime = r.RollingUpgrade.LastNodeTerminationTime() + nodeInterval = r.RollingUpgrade.NodeIntervalSeconds() + lastDrainTime = r.RollingUpgrade.LastNodeDrainTime() + drainInterval = r.RollingUpgrade.PostDrainDelaySeconds() + ) + + // check if we are still waiting on a termination delay + if lastTerminationTime != nil && !lastTerminationTime.IsZero() && time.Since(lastTerminationTime.Time).Seconds() < float64(nodeInterval) { + r.Info("reconcile requeue due to termination interval wait", "name", r.RollingUpgrade.NamespacedName()) + return true, nil + } + // check if we are still waiting on a drain delay + if lastDrainTime != nil && !lastDrainTime.IsZero() && time.Since(lastDrainTime.Time).Seconds() < float64(drainInterval) { + r.Info("reconcile requeue due to drain interval wait", "name", r.RollingUpgrade.NamespacedName()) + return true, nil + } + if reflect.DeepEqual(r.DrainManager.DrainGroup, &sync.WaitGroup{}) { for _, target := range batch { instanceID := aws.StringValue(target.InstanceId) @@ -338,6 +327,7 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) r.UpdateMetricsStatus(inProcessingNodes, nodeSteps) return true, nil } + r.RollingUpgrade.SetLastNodeTerminationTime(&metav1.Time{Time: time.Now()}) // Turns onto NodeRotationTerminate @@ -354,6 +344,7 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) } r.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationTerminated) r.DoNodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationCompleted, terminatedTime) + } r.UpdateMetricsStatus(inProcessingNodes, nodeSteps) @@ -380,7 +371,10 @@ func (r *RollingUpgradeContext) SelectTargets(scalingGroup *autoscaling.Group) [ r.Info("selecting batch for rotation", "batch size", unavailableInt, "name", r.RollingUpgrade.NamespacedName()) for _, instance := range r.Cloud.InProgressInstances { if selectedInstance := awsprovider.SelectScalingGroupInstance(instance, scalingGroup); !reflect.DeepEqual(selectedInstance, &autoscaling.Instance{}) { - targets = append(targets, selectedInstance) + //In-progress instances shouldn't be considered if they are in terminating state. + if !common.ContainsEqualFold(awsprovider.TerminatingInstanceStates, aws.StringValue(selectedInstance.LifecycleState)) { + targets = append(targets, selectedInstance) + } } } @@ -392,10 +386,7 @@ func (r *RollingUpgradeContext) SelectTargets(scalingGroup *autoscaling.Group) [ if r.RollingUpgrade.UpdateStrategyType() == v1alpha1.RandomUpdateStrategy { for _, instance := range scalingGroup.Instances { if r.IsInstanceDrifted(instance) && !common.ContainsEqualFold(awsprovider.GetInstanceIDs(targets), aws.StringValue(instance.InstanceId)) { - //In-progress instances shouldn't be considered if they are in terminating state. - if !common.ContainsEqualFold(awsprovider.TerminatingInstanceStates, aws.StringValue(instance.LifecycleState)) { - targets = append(targets, instance) - } + targets = append(targets, instance) } } if unavailableInt > len(targets) { diff --git a/main.go b/main.go index 88a0e58f..1cd35a51 100644 --- a/main.go +++ b/main.go @@ -200,6 +200,7 @@ func main() { DrainGroupMapper: &sync.Map{}, DrainErrorMapper: &sync.Map{}, ClusterNodesMap: &sync.Map{}, + ReconcileMap: &sync.Map{}, } reconciler.SetMaxParallel(maxParallel) From 00f7e89fc595da0766ca24bdd8190bced90dd8cf Mon Sep 17 00:00:00 2001 From: Shreyas Badiger <7680410+shreyas-badiger@users.noreply.github.com> Date: Tue, 20 Jul 2021 18:14:02 -0700 Subject: [PATCH 46/74] upgrade-manager-v2: Fix unit tests (#275) * Delete README.md Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * delete all Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * scaffolding Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * add API Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * initial code Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * add more scaffolding Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * Add kubernetes API calls Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * aws API calls Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * AWS API calls & Drift detection Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * initial rotation logic Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * Implemented RollingUpgrade object validation. (#176) * Validation step to check Nodes and ASG launch configs Signed-off-by: shreyas-badiger * Validating launch definition after a rolling upgrade Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Fix all the "make vet" errors in Controller V2 branch. (#177) * Validation step to check Nodes and ASG launch configs Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Validating launch definition after a rolling upgrade Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Resolve error log message and return statement Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Adding Functional Test (#113) * Adding BDD, workflow and badge * Changing CI workflow job name * Adding make manifests * Clarifying cron time zone comment Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * release 0.13 (#115) * release 0.13 * Update CHANGELOG.md Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * bump version (#116) Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Repo selection for CI and BDD workflows & CI step for releases (#117) * CI-BDD not on forks & Step for releases (#2) * Testing CI-BDD not on forks & Step for releases * Adding step for image with tag git-tag Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Terminate unjoined nodes Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Resolving PR comments Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Set version and update CHANGELOG for version 0.14. (#121) Co-authored-by: Shri Javadekar Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Bump version to 0.15-dev. Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Fix typo in README.md. (#125) Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Ignore the terminated instance during upgrade Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Added WARNING prefix in the logging Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Apply suggestions from code review Co-authored-by: Kevin Downey Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Capitalize sprintf to Sprintf Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Upgrade to Go 1.15 (#128) Signed-off-by: Oleg Atamanenko Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Fix few typos and simplify error returns, remove redundant types (#131) Signed-off-by: Oleg Atamanenko Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Readiness gates implementation for eager mode (#130) Signed-off-by: Oleg Atamanenko Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Adding Functional Test (#113) * Adding BDD, workflow and badge * Changing CI workflow job name * Adding make manifests * Clarifying cron time zone comment Signed-off-by: sbadiger * Validation step to check Nodes and ASG launch configs (#112) * Validation step to check Nodes and ASG launch configs * Validating launch definition after a rolling upgrade * Resolve error log message and return statement Co-authored-by: Eytan Avisror Signed-off-by: sbadiger * release 0.13 (#115) * release 0.13 * Update CHANGELOG.md Signed-off-by: sbadiger * bump version (#116) Signed-off-by: sbadiger * Repo selection for CI and BDD workflows & CI step for releases (#117) * CI-BDD not on forks & Step for releases (#2) * Testing CI-BDD not on forks & Step for releases * Adding step for image with tag git-tag Signed-off-by: sbadiger * Terminate unjoined nodes (#120) * Validation step to check Nodes and ASG launch configs * Validating launch definition after a rolling upgrade * Resolve error log message and return statement * Terminate unjoined nodes * Resolving PR comments Co-authored-by: Eytan Avisror Signed-off-by: sbadiger * Set version and update CHANGELOG for version 0.14. (#121) Co-authored-by: Shri Javadekar Signed-off-by: sbadiger * Bump version to 0.15-dev. Signed-off-by: sbadiger * Fix bug when switching to launch templates (#136) * Update rollingupgrade_controller.go * Update rollingupgrade_controller.go Signed-off-by: Eytan Avisror * spacing fixes Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * Extract script runner to a separate type; fix work with env. variables (#132) Signed-off-by: Oleg Atamanenko Signed-off-by: sbadiger * Set version and update CHANGELOG for version v0.15 (#137) Signed-off-by: Shri Javadekar Signed-off-by: sbadiger * Bump version to v0.16-dev. Signed-off-by: Shri Javadekar Signed-off-by: sbadiger * Propagate parent env variables to allow to talk with API Server (#144) Signed-off-by: Oleg Atamanenko Signed-off-by: sbadiger * Bump Golang CI action to fix failed CI run (#146) Signed-off-by: Oleg Atamanenko Signed-off-by: sbadiger * Simplify (#145) Signed-off-by: Oleg Atamanenko Signed-off-by: sbadiger * Add Expiration to cache and do not refresh ASG if cache is not expired (#143) Signed-off-by: Oleg Atamanenko Co-authored-by: Shri Javadekar Signed-off-by: sbadiger * Fix documentation for uniform across AZ Update strategy and fix typos (#147) Signed-off-by: Oleg Atamanenko Signed-off-by: sbadiger * Move cluster state from package level to a cluster state impl (#148) Signed-off-by: Oleg Atamanenko Signed-off-by: sbadiger * Simplify work with intstr type. (#149) Signed-off-by: Oleg Atamanenko Signed-off-by: sbadiger * If instance is in standby mode already, just return (#138) Signed-off-by: Oleg Atamanenko Co-authored-by: Shri Javadekar Signed-off-by: sbadiger * Handle terminated instances gracefully. (#150) Signed-off-by: Oleg Atamanenko Signed-off-by: sbadiger * Template version comparison fix (#155) * get template version Signed-off-by: Eytan Avisror * fix tests Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * release 0.16 (#157) Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * bump version to 0.17-dev (#158) Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * Don't uncordon node on failure to run postDrain script when IgnoreDrainFailures set (#151) * Don't uncordon node on failure to run postDrain script when IgnoreDrainFailures set Signed-off-by: Adam Malcontenti-Wilson * Test node uncordon when postDrain / postDrainWait script fails Signed-off-by: Adam Malcontenti-Wilson Signed-off-by: sbadiger * Abort on strategy failure instead of continuing (#152) * Abort on strategy failure instead of continuing Signed-off-by: Adam Malcontenti-Wilson * Remove unformatted error message placeholder Signed-off-by: Adam Malcontenti-Wilson * Explictly specify strategy for tests Signed-off-by: Adam Malcontenti-Wilson Signed-off-by: sbadiger * use NamespacedName (#160) Signed-off-by: Eytan Avisror Co-authored-by: Shri Javadekar Signed-off-by: sbadiger * Set version and update CHANGELOG for version v0.17 (#161) Signed-off-by: Shri Javadekar Signed-off-by: sbadiger * Bump version to v0.18-dev (#162) Signed-off-by: Shri Javadekar Signed-off-by: sbadiger * Move constants to types so that they can be reused (#167) Signed-off-by: Oleg Atamanenko Signed-off-by: sbadiger * Remove separate module for pkg/log (#168) Signed-off-by: Oleg Atamanenko Co-authored-by: Shri Javadekar Signed-off-by: sbadiger * Bump dependencies. (#169) Signed-off-by: Oleg Atamanenko Signed-off-by: sbadiger * use standard fmt.Errorf to format error message; unify error format (#171) Signed-off-by: Oleg Atamanenko Signed-off-by: sbadiger * Fix namespaced name order (#170) Signed-off-by: Oleg Atamanenko Signed-off-by: sbadiger * Add instance id to the logs (#173) Signed-off-by: Oleg Atamanenko Co-authored-by: Shri Javadekar Signed-off-by: sbadiger * Bump golang and busybox (#172) Signed-off-by: Oleg Atamanenko Co-authored-by: Shri Javadekar Signed-off-by: sbadiger * Expose template list and other execution errors to logs (#166) * Log and return wrapped launchtemplate error Signed-off-by: Adam Malcontenti-Wilson * Expose execution error in logs Signed-off-by: Adam Malcontenti-Wilson Signed-off-by: sbadiger * output can contain other messages from API Server, so be more relaxed (#174) Signed-off-by: Oleg Atamanenko Signed-off-by: sbadiger * Delete README.md Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * delete all Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * scaffolding Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * add API Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * initial code Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * add more scaffolding Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * Add kubernetes API calls Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * aws API calls Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * AWS API calls & Drift detection Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * validate() function Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * modified validate() Signed-off-by: sbadiger * modified validate() Signed-off-by: sbadiger * initial rotation logic Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * basic script_runner without any modifications Signed-off-by: sbadiger * Fix all the vet related errors Signed-off-by: sbadiger Co-authored-by: Alfredo Garo <44888596+garomonegro@users.noreply.github.com> Co-authored-by: Eytan Avisror Co-authored-by: Shri Javadekar Co-authored-by: Shri Javadekar Co-authored-by: Shri Javadekar Co-authored-by: Craig Robson Co-authored-by: Kevin Downey Co-authored-by: Oleg Atamanenko Co-authored-by: Shreyas Badiger <7680410+hard-fault@users.noreply.github.com> Co-authored-by: Adam Malcontenti-Wilson Co-authored-by: Adam Malcontenti-Wilson Co-authored-by: Eytan Avisror Signed-off-by: sbadiger * Controller v2: Implementation of Instance termination (#178) * fix make vet errors. Signed-off-by: sbadiger * Terminate instances and run v2 for first time. Signed-off-by: sbadiger * Addressing review comments Signed-off-by: sbadiger * addressing more review comments Signed-off-by: sbadiger * Log error message Signed-off-by: sbadiger * error handling for instance tagging Signed-off-by: sbadiger * Migrate Script Runner (#179) * Basic script runner Signed-off-by: Eytan Avisror * Update upgrade.go Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * Implemented node drain. (#181) Signed-off-by: sbadiger * Eager mode implementation (#183) * Eager mode implementation Signed-off-by: sbadiger * Metrics features (#189) Signed-off-by: xshao Signed-off-by: sbadiger * Process the batch rotation in parallel (#192) * Process the batch rotation in parallel Signed-off-by: sbadiger * addressing review comments Signed-off-by: sbadiger * Move the DrainManager within ReplaceBatch(), to access one per RollingUpgrade CR (#195) Signed-off-by: sbadiger * Refine metrics implementation to support goroutines (#196) * Refine the metrics status Signed-off-by: xshao * Refine the metrics status Signed-off-by: xshao * Fix test case error Signed-off-by: xshao * Use group instead of ASG Signed-off-by: xshao Signed-off-by: sbadiger * Ignore generated code (#201) * Refine the metrics status Signed-off-by: xshao * Refine the metrics status Signed-off-by: xshao * Fix test case error Signed-off-by: xshao * Use group instead of ASG Signed-off-by: xshao * Ignore generated code Signed-off-by: xshao * Ignore generated code Signed-off-by: xshao Signed-off-by: sbadiger * Fix bug in deleting the entry in syncMap (#203) Signed-off-by: sbadiger * Unit tests for controller-v2 (#215) * Unit tests Signed-off-by: sbadiger * minor change in accessing the namespace name Signed-off-by: sbadiger * move helper functions to a differnt file Signed-off-by: sbadiger * #2285: rollup CR statistic metrics in v2 (#218) * #2285: rollup CR statistic metrics in v2 Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 Signed-off-by: sbadiger * #2285: renamed some methods related to metrics (#224) Signed-off-by: sbadla1 Signed-off-by: sbadiger * #2286: removed version from metric namespace (#227) Signed-off-by: sbadla1 Signed-off-by: sbadiger * Create RollingUpgradeContext (#234) * #2285: rollup CR statistic metrics in v2 (#218) * #2285: rollup CR statistic metrics in v2 Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 Signed-off-by: sbadiger * log cloud discovery failure Signed-off-by: sbadiger * Create RollingUpgrade Context Signed-off-by: sbadiger * rollingupgrade context Signed-off-by: sbadiger Co-authored-by: Sahil Badla Signed-off-by: sbadiger * Resolve compile errors caused by merge conflict. (#235) * #2285: rollup CR statistic metrics in v2 (#218) * #2285: rollup CR statistic metrics in v2 Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 Signed-off-by: sbadiger * log cloud discovery failure Signed-off-by: sbadiger * Create RollingUpgrade Context Signed-off-by: sbadiger * rollingupgrade context Signed-off-by: sbadiger * resolve compile errors due to merge conflict Signed-off-by: sbadiger Co-authored-by: Sahil Badla Signed-off-by: sbadiger * upgrade-manager-v2: Move DrainManager back to Reconciler (#236) * #2285: rollup CR statistic metrics in v2 (#218) * #2285: rollup CR statistic metrics in v2 Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 Signed-off-by: sbadiger * log cloud discovery failure Signed-off-by: sbadiger * Create RollingUpgrade Context Signed-off-by: sbadiger * rollingupgrade context Signed-off-by: sbadiger * #2285: rollup CR statistic metrics in v2 (#218) * #2285: rollup CR statistic metrics in v2 Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 Signed-off-by: sbadiger * #2285: renamed some methods related to metrics (#224) Signed-off-by: sbadla1 Signed-off-by: sbadiger * #2286: removed version from metric namespace (#227) Signed-off-by: sbadla1 Signed-off-by: sbadiger * resolve compile errors due to merge conflict Signed-off-by: sbadiger * move drain-manager to reconciler Signed-off-by: sbadiger * initialize RollingUpgrade object Signed-off-by: sbadiger * use bool instead of count for standby function Signed-off-by: sbadiger * refactor in-progress and standby code Signed-off-by: sbadiger * rename instance standby function Signed-off-by: sbadiger * DrainManager changes in unit test files Signed-off-by: sbadiger Co-authored-by: Sahil Badla Signed-off-by: sbadiger * V2 controller metrics concurrency fix (#231) * Refine the metrics status Signed-off-by: xshao * Refine the metrics status Signed-off-by: xshao * Fix test case error Signed-off-by: xshao * Use group instead of ASG Signed-off-by: xshao * Ignore generated code Signed-off-by: xshao * Ignore generated code Signed-off-by: xshao * Fix the concurrent issue Signed-off-by: xshao * Fix the concurrent issue Signed-off-by: xshao * Move metrics related functions into RollingUpgradeContext Signed-off-by: xshao * Move metrics related functions into RollingUpgradeContext Signed-off-by: xshao * Move metrics related functions into upgrade_metrics.go Signed-off-by: xshao * Move metrics related functions into metrics.go Signed-off-by: xshao Signed-off-by: sbadiger * add missing parenthesis (#239) Signed-off-by: sbadiger * metricsMutex should be initialized (#240) Signed-off-by: xshao Signed-off-by: sbadiger * upgrade-manager-v2: Load test fixes (#245) * upgrade-manager-v2: Move DrainManager back to Reconciler (#236) * #2285: rollup CR statistic metrics in v2 (#218) * #2285: rollup CR statistic metrics in v2 Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 Signed-off-by: sbadiger * log cloud discovery failure Signed-off-by: sbadiger * Create RollingUpgrade Context Signed-off-by: sbadiger * rollingupgrade context Signed-off-by: sbadiger * #2285: rollup CR statistic metrics in v2 (#218) * #2285: rollup CR statistic metrics in v2 Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 Signed-off-by: sbadiger * #2285: renamed some methods related to metrics (#224) Signed-off-by: sbadla1 Signed-off-by: sbadiger * #2286: removed version from metric namespace (#227) Signed-off-by: sbadla1 Signed-off-by: sbadiger * resolve compile errors due to merge conflict Signed-off-by: sbadiger * move drain-manager to reconciler Signed-off-by: sbadiger * initialize RollingUpgrade object Signed-off-by: sbadiger * use bool instead of count for standby function Signed-off-by: sbadiger * refactor in-progress and standby code Signed-off-by: sbadiger * rename instance standby function Signed-off-by: sbadiger * DrainManager changes in unit test files Signed-off-by: sbadiger Co-authored-by: Sahil Badla Signed-off-by: sbadiger * V2 controller metrics concurrency fix (#231) * Refine the metrics status Signed-off-by: xshao * Refine the metrics status Signed-off-by: xshao * Fix test case error Signed-off-by: xshao * Use group instead of ASG Signed-off-by: xshao * Ignore generated code Signed-off-by: xshao * Ignore generated code Signed-off-by: xshao * Fix the concurrent issue Signed-off-by: xshao * Fix the concurrent issue Signed-off-by: xshao * Move metrics related functions into RollingUpgradeContext Signed-off-by: xshao * Move metrics related functions into RollingUpgradeContext Signed-off-by: xshao * Move metrics related functions into upgrade_metrics.go Signed-off-by: xshao * Move metrics related functions into metrics.go Signed-off-by: xshao Signed-off-by: sbadiger * add missing parenthesis Signed-off-by: sbadiger * load test fixes Signed-off-by: sbadiger * handle scaling group not found Signed-off-by: sbadiger * Update upgrade.go Signed-off-by: sbadiger * log one level up * remove double logging Signed-off-by: sbadiger * final push before RC release. (#254) * support IgnoreDrainFailures flag Signed-off-by: sbadiger * add else condition Signed-off-by: sbadiger * set min for maxUnavailable Signed-off-by: sbadiger * calculateMaxUnavailable function Signed-off-by: sbadiger * add a new coloumn (completePercentage) Signed-off-by: sbadiger * disable debug logs by default Signed-off-by: sbadiger * Fix metrics collecting issue (#249) * metricsMutex should be initialized Signed-off-by: xshao * Use InProcessingNode instead of Stringp[] so that it can have the status of steps Signed-off-by: xshao Signed-off-by: sbadiger * Revert "Fix metrics collecting issue (#249)" (#256) This reverts commit f5dd1cb5f76f2b78cb15c53daed14032a2a4c6ec. Signed-off-by: sbadiger * Fix metrics calculation issue (#258) * metricsMutex should be initialized Signed-off-by: xshao * Use InProcessingNode instead of Stringp[] so that it can have the status of steps Signed-off-by: xshao * Make the change backward compatible Signed-off-by: xshao * Make the change backward compatible Signed-off-by: xshao * Add mutex for InProcessingNode deleting Signed-off-by: xshao Signed-off-by: sbadiger * Add a mock for test and update version in Makefile (#262) Signed-off-by: sbadiger * and CR end time (#264) Signed-off-by: sbadiger * upgrade-manager-v2: expose totalProcessing time and other metrics (#265) * and CR end time Signed-off-by: sbadiger * expose totalProcessing time and other metrics Signed-off-by: sbadiger * addressing review comments Signed-off-by: sbadiger * upgrade-manager-v2: remove function duplicate declaration. (#266) * and CR end time Signed-off-by: sbadiger * expose totalProcessing time and other metrics Signed-off-by: sbadiger * addressing review comments Signed-off-by: sbadiger * remove function duplication Signed-off-by: sbadiger * Carry the metrics status in RollingUpgrade CR (#267) * Update metrics status at same time Signed-off-by: xshao * Update metrics status when terminating instance Signed-off-by: xshao * Add terminated step Signed-off-by: xshao * Add terminated step Signed-off-by: xshao * Add terminated step Signed-off-by: xshao Signed-off-by: sbadiger * move cloud discovery after nodeInterval / drainInterval wait (#270) Signed-off-by: sbadiger * upgrade-manager-v2: Add nodeEvents handler instead of a watch handler (#272) * upgrade-manager-v2: remove function duplicate declaration. (#266) * and CR end time Signed-off-by: sbadiger * expose totalProcessing time and other metrics Signed-off-by: sbadiger * addressing review comments Signed-off-by: sbadiger * remove function duplication Signed-off-by: sbadiger * Carry the metrics status in RollingUpgrade CR (#267) * Update metrics status at same time Signed-off-by: xshao * Update metrics status when terminating instance Signed-off-by: xshao * Add terminated step Signed-off-by: xshao * Add terminated step Signed-off-by: xshao * Add terminated step Signed-off-by: xshao Signed-off-by: sbadiger * move cloud discovery after nodeInterval / drainInterval wait Signed-off-by: sbadiger * Add watch event for cluster nodes instead of API calls Signed-off-by: sbadiger * upon node deletion, remove it from syncMap as well Signed-off-by: sbadiger * Add nodeEvents handler instead of watch handler Signed-off-by: sbadiger * Ignore Reconciles on nodeEvents Signed-off-by: sbadiger * Add comments Signed-off-by: sbadiger Co-authored-by: Sheldon Shao Signed-off-by: sbadiger * upgrade-manager-v2: Process next batch while waiting on nodeInterval period. (#273) * upgrade-manager-v2: remove function duplicate declaration. (#266) * and CR end time Signed-off-by: sbadiger * expose totalProcessing time and other metrics Signed-off-by: sbadiger * addressing review comments Signed-off-by: sbadiger * remove function duplication Signed-off-by: sbadiger * Carry the metrics status in RollingUpgrade CR (#267) * Update metrics status at same time Signed-off-by: xshao * Update metrics status when terminating instance Signed-off-by: xshao * Add terminated step Signed-off-by: xshao * Add terminated step Signed-off-by: xshao * Add terminated step Signed-off-by: xshao Signed-off-by: sbadiger * move cloud discovery after nodeInterval / drainInterval wait Signed-off-by: sbadiger * Add watch event for cluster nodes instead of API calls Signed-off-by: sbadiger * upon node deletion, remove it from syncMap as well Signed-off-by: sbadiger * Add nodeEvents handler instead of watch handler Signed-off-by: sbadiger * Ignore Reconciles on nodeEvents Signed-off-by: sbadiger * Add comments Signed-off-by: sbadiger * Set nextbatch to standBy while waiting for terminate * Avoid parallel reconcile operation per ASG * add default requeue time Co-authored-by: Sheldon Shao Signed-off-by: sbadiger * fix unit tests Signed-off-by: sbadiger Co-authored-by: Eytan Avisror Co-authored-by: Alfredo Garo <44888596+garomonegro@users.noreply.github.com> Co-authored-by: Eytan Avisror Co-authored-by: Shri Javadekar Co-authored-by: Shri Javadekar Co-authored-by: Shri Javadekar Co-authored-by: Craig Robson Co-authored-by: Kevin Downey Co-authored-by: Oleg Atamanenko Co-authored-by: Shreyas Badiger <7680410+hard-fault@users.noreply.github.com> Co-authored-by: Adam Malcontenti-Wilson Co-authored-by: Adam Malcontenti-Wilson Co-authored-by: Sheldon Shao Co-authored-by: Sahil Badla Co-authored-by: Sheldon Shao --- controllers/helpers_test.go | 35 +++++++++++++++++++++++++++++++++-- controllers/upgrade.go | 9 +++++++++ controllers/upgrade_test.go | 27 +++++++++++++++------------ 3 files changed, 57 insertions(+), 14 deletions(-) diff --git a/controllers/helpers_test.go b/controllers/helpers_test.go index 50fe6cfa..99ba610d 100644 --- a/controllers/helpers_test.go +++ b/controllers/helpers_test.go @@ -179,6 +179,20 @@ func createASGInstance(instanceID string, launchConfigName string) *autoscaling. } } +func createEc2Instances() []*ec2.Instance { + return []*ec2.Instance{ + &ec2.Instance{ + InstanceId: aws.String("mock-instance-1"), + }, + &ec2.Instance{ + InstanceId: aws.String("mock-instance-2"), + }, + &ec2.Instance{ + InstanceId: aws.String("mock-instance-3"), + }, + } +} + func createASG(asgName string, launchConfigName string) *autoscaling.Group { return &autoscaling.Group{ AutoScalingGroupName: &asgName, @@ -192,10 +206,23 @@ func createASG(asgName string, launchConfigName string) *autoscaling.Group { } } +func createDriftedASG(asgName string, launchConfigName string) *autoscaling.Group { + return &autoscaling.Group{ + AutoScalingGroupName: &asgName, + LaunchConfigurationName: &launchConfigName, + Instances: []*autoscaling.Instance{ + createASGInstance("mock-instance-1", "different-launch-config"), + createASGInstance("mock-instance-2", "different-launch-config"), + createASGInstance("mock-instance-3", "different-launch-config"), + }, + DesiredCapacity: func(x int) *int64 { i := int64(x); return &i }(3), + } +} + func createASGs() []*autoscaling.Group { return []*autoscaling.Group{ createASG("mock-asg-1", "mock-launch-config-1"), - createASG("mock-asg-2", "mock-launch-config-2"), + createDriftedASG("mock-asg-2", "mock-launch-config-2"), createASG("mock-asg-3", "mock-launch-config-3"), } } @@ -319,5 +346,9 @@ func (m *MockEC2) DescribeInstancesPages(request *ec2.DescribeInstancesInput, ca } func (m *MockEC2) DescribeInstances(*ec2.DescribeInstancesInput) (*ec2.DescribeInstancesOutput, error) { - return &ec2.DescribeInstancesOutput{}, nil + return &ec2.DescribeInstancesOutput{ + Reservations: []*ec2.Reservation{ + &ec2.Reservation{Instances: createEc2Instances()}, + }, + }, nil } diff --git a/controllers/upgrade.go b/controllers/upgrade.go index 4b5ddc19..712f7666 100644 --- a/controllers/upgrade.go +++ b/controllers/upgrade.go @@ -60,6 +60,15 @@ type RollingUpgradeContext struct { } func (r *RollingUpgradeContext) RotateNodes() error { + // set status to running + r.RollingUpgrade.SetCurrentStatus(v1alpha1.StatusRunning) + common.SetMetricRollupInitOrRunning(r.RollingUpgrade.Name) + + // set start time + if r.RollingUpgrade.StartTime() == "" { + r.RollingUpgrade.SetStartTime(time.Now().Format(time.RFC3339)) + } + // discover the state of AWS and K8s cluster. if err := r.Cloud.Discover(); err != nil { r.Info("failed to discover the cloud", "scalingGroup", r.RollingUpgrade.ScalingGroupName(), "name", r.RollingUpgrade.NamespacedName()) diff --git a/controllers/upgrade_test.go b/controllers/upgrade_test.go index 794ff68d..948fd999 100644 --- a/controllers/upgrade_test.go +++ b/controllers/upgrade_test.go @@ -246,33 +246,36 @@ func TestIsScalingGroupDrifted(t *testing.T) { func TestRotateNodes(t *testing.T) { var tests = []struct { - TestDescription string - Reconciler *RollingUpgradeReconciler - AsgClient *MockAutoscalingGroup - ExpectedValue bool - ExpectedStatusValue string + TestDescription string + Reconciler *RollingUpgradeReconciler + AsgClient *MockAutoscalingGroup + RollingUpgradeContext *RollingUpgradeContext + ExpectedValue bool + ExpectedStatusValue string }{ { - "All instances have different launch config as the ASG, RotateNodes() will not mark CR complete", + "All instances have different launch config as the ASG, RotateNodes() should not mark CR complete", createRollingUpgradeReconciler(t), - func() *MockAutoscalingGroup { - newAsgClient := createASGClient() - newAsgClient.autoScalingGroups[0].LaunchConfigurationName = aws.String("different-launch-config") - return newAsgClient + createASGClient(), + func() *RollingUpgradeContext { + newRollingUpgradeContext := createRollingUpgradeContext(createRollingUpgradeReconciler(t)) + newRollingUpgradeContext.RollingUpgrade.Spec.AsgName = "mock-asg-2" // The instances in mock-asg are drifted + return newRollingUpgradeContext }(), true, v1alpha1.StatusRunning, }, { - "All instances have same launch config as the ASG, RotateNodes() will mark CR complete", + "All instances have same launch config as the ASG, RotateNodes() should mark CR complete", createRollingUpgradeReconciler(t), createASGClient(), + createRollingUpgradeContext(createRollingUpgradeReconciler(t)), false, v1alpha1.StatusComplete, }, } for _, test := range tests { - rollupCtx := createRollingUpgradeContext(test.Reconciler) + rollupCtx := test.RollingUpgradeContext rollupCtx.Cloud.ScalingGroups = test.AsgClient.autoScalingGroups rollupCtx.Auth.AmazonClientSet.AsgClient = test.AsgClient From 7a4766d0ae230737d414048a317185c3fcad3559 Mon Sep 17 00:00:00 2001 From: Shreyas Badiger <7680410+shreyas-badiger@users.noreply.github.com> Date: Wed, 21 Jul 2021 12:41:46 -0700 Subject: [PATCH 47/74] upgrade-manager-v2: Add CI github action, fix lint errors. (#276) * upgrade-manager-v2: Fix unit tests (#275) * Delete README.md Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * delete all Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * scaffolding Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * add API Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * initial code Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * add more scaffolding Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * Add kubernetes API calls Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * aws API calls Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * AWS API calls & Drift detection Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * initial rotation logic Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * Implemented RollingUpgrade object validation. (#176) * Validation step to check Nodes and ASG launch configs Signed-off-by: shreyas-badiger * Validating launch definition after a rolling upgrade Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Fix all the "make vet" errors in Controller V2 branch. (#177) * Validation step to check Nodes and ASG launch configs Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Validating launch definition after a rolling upgrade Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Resolve error log message and return statement Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Adding Functional Test (#113) * Adding BDD, workflow and badge * Changing CI workflow job name * Adding make manifests * Clarifying cron time zone comment Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * release 0.13 (#115) * release 0.13 * Update CHANGELOG.md Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * bump version (#116) Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Repo selection for CI and BDD workflows & CI step for releases (#117) * CI-BDD not on forks & Step for releases (#2) * Testing CI-BDD not on forks & Step for releases * Adding step for image with tag git-tag Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Terminate unjoined nodes Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Resolving PR comments Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Set version and update CHANGELOG for version 0.14. (#121) Co-authored-by: Shri Javadekar Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Bump version to 0.15-dev. Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Fix typo in README.md. (#125) Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Ignore the terminated instance during upgrade Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Added WARNING prefix in the logging Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Apply suggestions from code review Co-authored-by: Kevin Downey Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Capitalize sprintf to Sprintf Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Upgrade to Go 1.15 (#128) Signed-off-by: Oleg Atamanenko Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Fix few typos and simplify error returns, remove redundant types (#131) Signed-off-by: Oleg Atamanenko Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Readiness gates implementation for eager mode (#130) Signed-off-by: Oleg Atamanenko Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * Adding Functional Test (#113) * Adding BDD, workflow and badge * Changing CI workflow job name * Adding make manifests * Clarifying cron time zone comment Signed-off-by: sbadiger * Validation step to check Nodes and ASG launch configs (#112) * Validation step to check Nodes and ASG launch configs * Validating launch definition after a rolling upgrade * Resolve error log message and return statement Co-authored-by: Eytan Avisror Signed-off-by: sbadiger * release 0.13 (#115) * release 0.13 * Update CHANGELOG.md Signed-off-by: sbadiger * bump version (#116) Signed-off-by: sbadiger * Repo selection for CI and BDD workflows & CI step for releases (#117) * CI-BDD not on forks & Step for releases (#2) * Testing CI-BDD not on forks & Step for releases * Adding step for image with tag git-tag Signed-off-by: sbadiger * Terminate unjoined nodes (#120) * Validation step to check Nodes and ASG launch configs * Validating launch definition after a rolling upgrade * Resolve error log message and return statement * Terminate unjoined nodes * Resolving PR comments Co-authored-by: Eytan Avisror Signed-off-by: sbadiger * Set version and update CHANGELOG for version 0.14. (#121) Co-authored-by: Shri Javadekar Signed-off-by: sbadiger * Bump version to 0.15-dev. Signed-off-by: sbadiger * Fix bug when switching to launch templates (#136) * Update rollingupgrade_controller.go * Update rollingupgrade_controller.go Signed-off-by: Eytan Avisror * spacing fixes Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * Extract script runner to a separate type; fix work with env. variables (#132) Signed-off-by: Oleg Atamanenko Signed-off-by: sbadiger * Set version and update CHANGELOG for version v0.15 (#137) Signed-off-by: Shri Javadekar Signed-off-by: sbadiger * Bump version to v0.16-dev. Signed-off-by: Shri Javadekar Signed-off-by: sbadiger * Propagate parent env variables to allow to talk with API Server (#144) Signed-off-by: Oleg Atamanenko Signed-off-by: sbadiger * Bump Golang CI action to fix failed CI run (#146) Signed-off-by: Oleg Atamanenko Signed-off-by: sbadiger * Simplify (#145) Signed-off-by: Oleg Atamanenko Signed-off-by: sbadiger * Add Expiration to cache and do not refresh ASG if cache is not expired (#143) Signed-off-by: Oleg Atamanenko Co-authored-by: Shri Javadekar Signed-off-by: sbadiger * Fix documentation for uniform across AZ Update strategy and fix typos (#147) Signed-off-by: Oleg Atamanenko Signed-off-by: sbadiger * Move cluster state from package level to a cluster state impl (#148) Signed-off-by: Oleg Atamanenko Signed-off-by: sbadiger * Simplify work with intstr type. (#149) Signed-off-by: Oleg Atamanenko Signed-off-by: sbadiger * If instance is in standby mode already, just return (#138) Signed-off-by: Oleg Atamanenko Co-authored-by: Shri Javadekar Signed-off-by: sbadiger * Handle terminated instances gracefully. (#150) Signed-off-by: Oleg Atamanenko Signed-off-by: sbadiger * Template version comparison fix (#155) * get template version Signed-off-by: Eytan Avisror * fix tests Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * release 0.16 (#157) Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * bump version to 0.17-dev (#158) Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * Don't uncordon node on failure to run postDrain script when IgnoreDrainFailures set (#151) * Don't uncordon node on failure to run postDrain script when IgnoreDrainFailures set Signed-off-by: Adam Malcontenti-Wilson * Test node uncordon when postDrain / postDrainWait script fails Signed-off-by: Adam Malcontenti-Wilson Signed-off-by: sbadiger * Abort on strategy failure instead of continuing (#152) * Abort on strategy failure instead of continuing Signed-off-by: Adam Malcontenti-Wilson * Remove unformatted error message placeholder Signed-off-by: Adam Malcontenti-Wilson * Explictly specify strategy for tests Signed-off-by: Adam Malcontenti-Wilson Signed-off-by: sbadiger * use NamespacedName (#160) Signed-off-by: Eytan Avisror Co-authored-by: Shri Javadekar Signed-off-by: sbadiger * Set version and update CHANGELOG for version v0.17 (#161) Signed-off-by: Shri Javadekar Signed-off-by: sbadiger * Bump version to v0.18-dev (#162) Signed-off-by: Shri Javadekar Signed-off-by: sbadiger * Move constants to types so that they can be reused (#167) Signed-off-by: Oleg Atamanenko Signed-off-by: sbadiger * Remove separate module for pkg/log (#168) Signed-off-by: Oleg Atamanenko Co-authored-by: Shri Javadekar Signed-off-by: sbadiger * Bump dependencies. (#169) Signed-off-by: Oleg Atamanenko Signed-off-by: sbadiger * use standard fmt.Errorf to format error message; unify error format (#171) Signed-off-by: Oleg Atamanenko Signed-off-by: sbadiger * Fix namespaced name order (#170) Signed-off-by: Oleg Atamanenko Signed-off-by: sbadiger * Add instance id to the logs (#173) Signed-off-by: Oleg Atamanenko Co-authored-by: Shri Javadekar Signed-off-by: sbadiger * Bump golang and busybox (#172) Signed-off-by: Oleg Atamanenko Co-authored-by: Shri Javadekar Signed-off-by: sbadiger * Expose template list and other execution errors to logs (#166) * Log and return wrapped launchtemplate error Signed-off-by: Adam Malcontenti-Wilson * Expose execution error in logs Signed-off-by: Adam Malcontenti-Wilson Signed-off-by: sbadiger * output can contain other messages from API Server, so be more relaxed (#174) Signed-off-by: Oleg Atamanenko Signed-off-by: sbadiger * Delete README.md Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * delete all Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * scaffolding Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * add API Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * initial code Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * add more scaffolding Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * Add kubernetes API calls Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * aws API calls Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * AWS API calls & Drift detection Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * validate() function Signed-off-by: shreyas-badiger Signed-off-by: sbadiger * modified validate() Signed-off-by: sbadiger * modified validate() Signed-off-by: sbadiger * initial rotation logic Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * basic script_runner without any modifications Signed-off-by: sbadiger * Fix all the vet related errors Signed-off-by: sbadiger Co-authored-by: Alfredo Garo <44888596+garomonegro@users.noreply.github.com> Co-authored-by: Eytan Avisror Co-authored-by: Shri Javadekar Co-authored-by: Shri Javadekar Co-authored-by: Shri Javadekar Co-authored-by: Craig Robson Co-authored-by: Kevin Downey Co-authored-by: Oleg Atamanenko Co-authored-by: Shreyas Badiger <7680410+hard-fault@users.noreply.github.com> Co-authored-by: Adam Malcontenti-Wilson Co-authored-by: Adam Malcontenti-Wilson Co-authored-by: Eytan Avisror Signed-off-by: sbadiger * Controller v2: Implementation of Instance termination (#178) * fix make vet errors. Signed-off-by: sbadiger * Terminate instances and run v2 for first time. Signed-off-by: sbadiger * Addressing review comments Signed-off-by: sbadiger * addressing more review comments Signed-off-by: sbadiger * Log error message Signed-off-by: sbadiger * error handling for instance tagging Signed-off-by: sbadiger * Migrate Script Runner (#179) * Basic script runner Signed-off-by: Eytan Avisror * Update upgrade.go Signed-off-by: Eytan Avisror Signed-off-by: sbadiger * Implemented node drain. (#181) Signed-off-by: sbadiger * Eager mode implementation (#183) * Eager mode implementation Signed-off-by: sbadiger * Metrics features (#189) Signed-off-by: xshao Signed-off-by: sbadiger * Process the batch rotation in parallel (#192) * Process the batch rotation in parallel Signed-off-by: sbadiger * addressing review comments Signed-off-by: sbadiger * Move the DrainManager within ReplaceBatch(), to access one per RollingUpgrade CR (#195) Signed-off-by: sbadiger * Refine metrics implementation to support goroutines (#196) * Refine the metrics status Signed-off-by: xshao * Refine the metrics status Signed-off-by: xshao * Fix test case error Signed-off-by: xshao * Use group instead of ASG Signed-off-by: xshao Signed-off-by: sbadiger * Ignore generated code (#201) * Refine the metrics status Signed-off-by: xshao * Refine the metrics status Signed-off-by: xshao * Fix test case error Signed-off-by: xshao * Use group instead of ASG Signed-off-by: xshao * Ignore generated code Signed-off-by: xshao * Ignore generated code Signed-off-by: xshao Signed-off-by: sbadiger * Fix bug in deleting the entry in syncMap (#203) Signed-off-by: sbadiger * Unit tests for controller-v2 (#215) * Unit tests Signed-off-by: sbadiger * minor change in accessing the namespace name Signed-off-by: sbadiger * move helper functions to a differnt file Signed-off-by: sbadiger * #2285: rollup CR statistic metrics in v2 (#218) * #2285: rollup CR statistic metrics in v2 Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 Signed-off-by: sbadiger * #2285: renamed some methods related to metrics (#224) Signed-off-by: sbadla1 Signed-off-by: sbadiger * #2286: removed version from metric namespace (#227) Signed-off-by: sbadla1 Signed-off-by: sbadiger * Create RollingUpgradeContext (#234) * #2285: rollup CR statistic metrics in v2 (#218) * #2285: rollup CR statistic metrics in v2 Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 Signed-off-by: sbadiger * log cloud discovery failure Signed-off-by: sbadiger * Create RollingUpgrade Context Signed-off-by: sbadiger * rollingupgrade context Signed-off-by: sbadiger Co-authored-by: Sahil Badla Signed-off-by: sbadiger * Resolve compile errors caused by merge conflict. (#235) * #2285: rollup CR statistic metrics in v2 (#218) * #2285: rollup CR statistic metrics in v2 Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 Signed-off-by: sbadiger * log cloud discovery failure Signed-off-by: sbadiger * Create RollingUpgrade Context Signed-off-by: sbadiger * rollingupgrade context Signed-off-by: sbadiger * resolve compile errors due to merge conflict Signed-off-by: sbadiger Co-authored-by: Sahil Badla Signed-off-by: sbadiger * upgrade-manager-v2: Move DrainManager back to Reconciler (#236) * #2285: rollup CR statistic metrics in v2 (#218) * #2285: rollup CR statistic metrics in v2 Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 Signed-off-by: sbadiger * log cloud discovery failure Signed-off-by: sbadiger * Create RollingUpgrade Context Signed-off-by: sbadiger * rollingupgrade context Signed-off-by: sbadiger * #2285: rollup CR statistic metrics in v2 (#218) * #2285: rollup CR statistic metrics in v2 Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 Signed-off-by: sbadiger * #2285: renamed some methods related to metrics (#224) Signed-off-by: sbadla1 Signed-off-by: sbadiger * #2286: removed version from metric namespace (#227) Signed-off-by: sbadla1 Signed-off-by: sbadiger * resolve compile errors due to merge conflict Signed-off-by: sbadiger * move drain-manager to reconciler Signed-off-by: sbadiger * initialize RollingUpgrade object Signed-off-by: sbadiger * use bool instead of count for standby function Signed-off-by: sbadiger * refactor in-progress and standby code Signed-off-by: sbadiger * rename instance standby function Signed-off-by: sbadiger * DrainManager changes in unit test files Signed-off-by: sbadiger Co-authored-by: Sahil Badla Signed-off-by: sbadiger * V2 controller metrics concurrency fix (#231) * Refine the metrics status Signed-off-by: xshao * Refine the metrics status Signed-off-by: xshao * Fix test case error Signed-off-by: xshao * Use group instead of ASG Signed-off-by: xshao * Ignore generated code Signed-off-by: xshao * Ignore generated code Signed-off-by: xshao * Fix the concurrent issue Signed-off-by: xshao * Fix the concurrent issue Signed-off-by: xshao * Move metrics related functions into RollingUpgradeContext Signed-off-by: xshao * Move metrics related functions into RollingUpgradeContext Signed-off-by: xshao * Move metrics related functions into upgrade_metrics.go Signed-off-by: xshao * Move metrics related functions into metrics.go Signed-off-by: xshao Signed-off-by: sbadiger * add missing parenthesis (#239) Signed-off-by: sbadiger * metricsMutex should be initialized (#240) Signed-off-by: xshao Signed-off-by: sbadiger * upgrade-manager-v2: Load test fixes (#245) * upgrade-manager-v2: Move DrainManager back to Reconciler (#236) * #2285: rollup CR statistic metrics in v2 (#218) * #2285: rollup CR statistic metrics in v2 Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 Signed-off-by: sbadiger * log cloud discovery failure Signed-off-by: sbadiger * Create RollingUpgrade Context Signed-off-by: sbadiger * rollingupgrade context Signed-off-by: sbadiger * #2285: rollup CR statistic metrics in v2 (#218) * #2285: rollup CR statistic metrics in v2 Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 * #2285: updated metric flags Signed-off-by: sbadla1 Signed-off-by: sbadiger * #2285: renamed some methods related to metrics (#224) Signed-off-by: sbadla1 Signed-off-by: sbadiger * #2286: removed version from metric namespace (#227) Signed-off-by: sbadla1 Signed-off-by: sbadiger * resolve compile errors due to merge conflict Signed-off-by: sbadiger * move drain-manager to reconciler Signed-off-by: sbadiger * initialize RollingUpgrade object Signed-off-by: sbadiger * use bool instead of count for standby function Signed-off-by: sbadiger * refactor in-progress and standby code Signed-off-by: sbadiger * rename instance standby function Signed-off-by: sbadiger * DrainManager changes in unit test files Signed-off-by: sbadiger Co-authored-by: Sahil Badla Signed-off-by: sbadiger * V2 controller metrics concurrency fix (#231) * Refine the metrics status Signed-off-by: xshao * Refine the metrics status Signed-off-by: xshao * Fix test case error Signed-off-by: xshao * Use group instead of ASG Signed-off-by: xshao * Ignore generated code Signed-off-by: xshao * Ignore generated code Signed-off-by: xshao * Fix the concurrent issue Signed-off-by: xshao * Fix the concurrent issue Signed-off-by: xshao * Move metrics related functions into RollingUpgradeContext Signed-off-by: xshao * Move metrics related functions into RollingUpgradeContext Signed-off-by: xshao * Move metrics related functions into upgrade_metrics.go Signed-off-by: xshao * Move metrics related functions into metrics.go Signed-off-by: xshao Signed-off-by: sbadiger * add missing parenthesis Signed-off-by: sbadiger * load test fixes Signed-off-by: sbadiger * handle scaling group not found Signed-off-by: sbadiger * Update upgrade.go Signed-off-by: sbadiger * log one level up * remove double logging Signed-off-by: sbadiger * final push before RC release. (#254) * support IgnoreDrainFailures flag Signed-off-by: sbadiger * add else condition Signed-off-by: sbadiger * set min for maxUnavailable Signed-off-by: sbadiger * calculateMaxUnavailable function Signed-off-by: sbadiger * add a new coloumn (completePercentage) Signed-off-by: sbadiger * disable debug logs by default Signed-off-by: sbadiger * Fix metrics collecting issue (#249) * metricsMutex should be initialized Signed-off-by: xshao * Use InProcessingNode instead of Stringp[] so that it can have the status of steps Signed-off-by: xshao Signed-off-by: sbadiger * Revert "Fix metrics collecting issue (#249)" (#256) This reverts commit f5dd1cb5f76f2b78cb15c53daed14032a2a4c6ec. Signed-off-by: sbadiger * Fix metrics calculation issue (#258) * metricsMutex should be initialized Signed-off-by: xshao * Use InProcessingNode instead of Stringp[] so that it can have the status of steps Signed-off-by: xshao * Make the change backward compatible Signed-off-by: xshao * Make the change backward compatible Signed-off-by: xshao * Add mutex for InProcessingNode deleting Signed-off-by: xshao Signed-off-by: sbadiger * Add a mock for test and update version in Makefile (#262) Signed-off-by: sbadiger * and CR end time (#264) Signed-off-by: sbadiger * upgrade-manager-v2: expose totalProcessing time and other metrics (#265) * and CR end time Signed-off-by: sbadiger * expose totalProcessing time and other metrics Signed-off-by: sbadiger * addressing review comments Signed-off-by: sbadiger * upgrade-manager-v2: remove function duplicate declaration. (#266) * and CR end time Signed-off-by: sbadiger * expose totalProcessing time and other metrics Signed-off-by: sbadiger * addressing review comments Signed-off-by: sbadiger * remove function duplication Signed-off-by: sbadiger * Carry the metrics status in RollingUpgrade CR (#267) * Update metrics status at same time Signed-off-by: xshao * Update metrics status when terminating instance Signed-off-by: xshao * Add terminated step Signed-off-by: xshao * Add terminated step Signed-off-by: xshao * Add terminated step Signed-off-by: xshao Signed-off-by: sbadiger * move cloud discovery after nodeInterval / drainInterval wait (#270) Signed-off-by: sbadiger * upgrade-manager-v2: Add nodeEvents handler instead of a watch handler (#272) * upgrade-manager-v2: remove function duplicate declaration. (#266) * and CR end time Signed-off-by: sbadiger * expose totalProcessing time and other metrics Signed-off-by: sbadiger * addressing review comments Signed-off-by: sbadiger * remove function duplication Signed-off-by: sbadiger * Carry the metrics status in RollingUpgrade CR (#267) * Update metrics status at same time Signed-off-by: xshao * Update metrics status when terminating instance Signed-off-by: xshao * Add terminated step Signed-off-by: xshao * Add terminated step Signed-off-by: xshao * Add terminated step Signed-off-by: xshao Signed-off-by: sbadiger * move cloud discovery after nodeInterval / drainInterval wait Signed-off-by: sbadiger * Add watch event for cluster nodes instead of API calls Signed-off-by: sbadiger * upon node deletion, remove it from syncMap as well Signed-off-by: sbadiger * Add nodeEvents handler instead of watch handler Signed-off-by: sbadiger * Ignore Reconciles on nodeEvents Signed-off-by: sbadiger * Add comments Signed-off-by: sbadiger Co-authored-by: Sheldon Shao Signed-off-by: sbadiger * upgrade-manager-v2: Process next batch while waiting on nodeInterval period. (#273) * upgrade-manager-v2: remove function duplicate declaration. (#266) * and CR end time Signed-off-by: sbadiger * expose totalProcessing time and other metrics Signed-off-by: sbadiger * addressing review comments Signed-off-by: sbadiger * remove function duplication Signed-off-by: sbadiger * Carry the metrics status in RollingUpgrade CR (#267) * Update metrics status at same time Signed-off-by: xshao * Update metrics status when terminating instance Signed-off-by: xshao * Add terminated step Signed-off-by: xshao * Add terminated step Signed-off-by: xshao * Add terminated step Signed-off-by: xshao Signed-off-by: sbadiger * move cloud discovery after nodeInterval / drainInterval wait Signed-off-by: sbadiger * Add watch event for cluster nodes instead of API calls Signed-off-by: sbadiger * upon node deletion, remove it from syncMap as well Signed-off-by: sbadiger * Add nodeEvents handler instead of watch handler Signed-off-by: sbadiger * Ignore Reconciles on nodeEvents Signed-off-by: sbadiger * Add comments Signed-off-by: sbadiger * Set nextbatch to standBy while waiting for terminate * Avoid parallel reconcile operation per ASG * add default requeue time Co-authored-by: Sheldon Shao Signed-off-by: sbadiger * fix unit tests Signed-off-by: sbadiger Co-authored-by: Eytan Avisror Co-authored-by: Alfredo Garo <44888596+garomonegro@users.noreply.github.com> Co-authored-by: Eytan Avisror Co-authored-by: Shri Javadekar Co-authored-by: Shri Javadekar Co-authored-by: Shri Javadekar Co-authored-by: Craig Robson Co-authored-by: Kevin Downey Co-authored-by: Oleg Atamanenko Co-authored-by: Shreyas Badiger <7680410+hard-fault@users.noreply.github.com> Co-authored-by: Adam Malcontenti-Wilson Co-authored-by: Adam Malcontenti-Wilson Co-authored-by: Sheldon Shao Co-authored-by: Sahil Badla Co-authored-by: Sheldon Shao Signed-off-by: sbadiger * add ci.yaml file Signed-off-by: sbadiger * test commit to trigger ci build Signed-off-by: sbadiger * move ci.yaml inside workflows Signed-off-by: sbadiger * delete ci.yaml file from previous place Signed-off-by: sbadiger * address lint issues Signed-off-by: sbadiger * Update ci.yaml Signed-off-by: sbadiger * generate coverage.txt file Signed-off-by: sbadiger * fix golang lint errors Signed-off-by: sbadiger * Delete delete-me.file * generate coverage.txt file Signed-off-by: sbadiger Co-authored-by: Eytan Avisror Co-authored-by: Alfredo Garo <44888596+garomonegro@users.noreply.github.com> Co-authored-by: Eytan Avisror Co-authored-by: Shri Javadekar Co-authored-by: Shri Javadekar Co-authored-by: Shri Javadekar Co-authored-by: Craig Robson Co-authored-by: Kevin Downey Co-authored-by: Oleg Atamanenko Co-authored-by: Shreyas Badiger <7680410+hard-fault@users.noreply.github.com> Co-authored-by: Adam Malcontenti-Wilson Co-authored-by: Adam Malcontenti-Wilson Co-authored-by: Sheldon Shao Co-authored-by: Sahil Badla Co-authored-by: Sheldon Shao --- .github/workflows/ci.yaml | 82 +++++++++++++++++++++++ Makefile | 3 +- controllers/helpers_test.go | 3 - controllers/providers/kubernetes/utils.go | 10 ++- controllers/rollingupgrade_controller.go | 6 +- 5 files changed, 91 insertions(+), 13 deletions(-) create mode 100644 .github/workflows/ci.yaml diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml new file mode 100644 index 00000000..f88c3ecf --- /dev/null +++ b/.github/workflows/ci.yaml @@ -0,0 +1,82 @@ +name: Build-Test + +on: + push: + branches: + - controller-v2 + pull_request: + branches: + - controller-v2 + release: + types: + - published + +jobs: + build: + name: CI # Lint, Test, Codecov, Docker build & Push + runs-on: ubuntu-latest + steps: + + - name: Checkout code + uses: actions/checkout@v2 + + - name: Golangci-lint + uses: golangci/golangci-lint-action@v2 + with: + # Required: the version of golangci-lint is required and must be specified without patch version: we always use the latest patch version. + version: v1.32 + args: --timeout 2m + + - name: Get kubebuilder + env: + version: 1.0.8 # latest stable version + arch: amd64 + run: | + # download the release + curl -L -O "https://github.com/kubernetes-sigs/kubebuilder/releases/download/v${version}/kubebuilder_${version}_linux_${arch}.tar.gz" + # extract the archive + tar -zxvf kubebuilder_${version}_linux_${arch}.tar.gz + mv kubebuilder_${version}_linux_${arch} kubebuilder && sudo mv kubebuilder /usr/local/ + # update your PATH to include /usr/local/kubebuilder/bin + export PATH=$PATH:/usr/local/kubebuilder/bin + - name: Run Tests + run: make test + + - name: Codecov + uses: codecov/codecov-action@v1 + with: + file: ./coverage.txt # optional + flags: unittests # optional + name: codecov-umbrella # optional + fail_ci_if_error: true # optional (default = false) + + - name: Docker build + if: github.event_name == 'pull_request' || (github.repository != 'keikoproj/upgrade-manager' && github.event_name == 'push') + run: make docker-build + + - name: Build and push Docker image with tag controller-v2 # only on pushes to keikoproj/upgrade-manager + if: github.event_name == 'push' && github.repository == 'keikoproj/upgrade-manager' + uses: docker/build-push-action@v1 + with: + username: ${{ secrets.DOCKER_USERNAME }} + password: ${{ secrets.DOCKER_PASSWORD }} + repository: keikoproj/rolling-upgrade-controller + tags: controller-v2 + + - name: Build and push Docker image with tag latest # only on releases of keikoproj/upgrade-manager + if: github.event_name == 'release' && github.repository == 'keikoproj/upgrade-manager' + uses: docker/build-push-action@v1 + with: + username: ${{ secrets.DOCKER_USERNAME }} + password: ${{ secrets.DOCKER_PASSWORD }} + repository: keikoproj/rolling-upgrade-controller + tags: latest + + - name: Build and push Docker image with tag git-tag # only on releases of keikoproj/upgrade-manager + if: github.event_name == 'release' && github.repository == 'keikoproj/upgrade-manager' + uses: docker/build-push-action@v1 + with: + username: ${{ secrets.DOCKER_USERNAME }} + password: ${{ secrets.DOCKER_PASSWORD }} + repository: keikoproj/rolling-upgrade-controller + tag_with_ref: true diff --git a/Makefile b/Makefile index 72bcec3d..628d8c7d 100644 --- a/Makefile +++ b/Makefile @@ -18,7 +18,8 @@ ENVTEST_ASSETS_DIR=$(shell pwd)/testbin test: generate fmt vet manifests mkdir -p ${ENVTEST_ASSETS_DIR} test -f ${ENVTEST_ASSETS_DIR}/setup-envtest.sh || curl -sSLo ${ENVTEST_ASSETS_DIR}/setup-envtest.sh https://raw.githubusercontent.com/kubernetes-sigs/controller-runtime/v0.7.0/hack/setup-envtest.sh - source ${ENVTEST_ASSETS_DIR}/setup-envtest.sh; fetch_envtest_tools $(ENVTEST_ASSETS_DIR); setup_envtest_env $(ENVTEST_ASSETS_DIR); go test ./... -coverprofile cover.out + source ${ENVTEST_ASSETS_DIR}/setup-envtest.sh; fetch_envtest_tools $(ENVTEST_ASSETS_DIR); setup_envtest_env $(ENVTEST_ASSETS_DIR); go test ./... -coverprofile coverage.txt + go tool cover -html=./coverage.txt -o cover.html # Build manager binary manager: generate fmt vet diff --git a/controllers/helpers_test.go b/controllers/helpers_test.go index 99ba610d..b975a7e7 100644 --- a/controllers/helpers_test.go +++ b/controllers/helpers_test.go @@ -158,13 +158,10 @@ type MockAutoscalingGroup struct { } type launchTemplateInfo struct { - data *ec2.ResponseLaunchTemplateData name *string } type MockEC2 struct { ec2iface.EC2API - awsErr awserr.Error - reservations []*ec2.Reservation LaunchTemplates map[string]*launchTemplateInfo } diff --git a/controllers/providers/kubernetes/utils.go b/controllers/providers/kubernetes/utils.go index 2e557416..abeddd92 100644 --- a/controllers/providers/kubernetes/utils.go +++ b/controllers/providers/kubernetes/utils.go @@ -87,12 +87,10 @@ func GetKubernetesLocalConfig() (*rest.Config, error) { } func SelectNodeByInstanceID(instanceID string, nodes []*corev1.Node) *corev1.Node { - if nodes != nil { - for _, node := range nodes { - nodeID := GetNodeInstanceID(node) - if strings.EqualFold(instanceID, nodeID) { - return node - } + for _, node := range nodes { + nodeID := GetNodeInstanceID(node) + if strings.EqualFold(instanceID, nodeID) { + return node } } return nil diff --git a/controllers/rollingupgrade_controller.go b/controllers/rollingupgrade_controller.go index 37d196ef..b57dbd39 100644 --- a/controllers/rollingupgrade_controller.go +++ b/controllers/rollingupgrade_controller.go @@ -78,7 +78,7 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque err := r.Get(ctx, req.NamespacedName, rollingUpgrade) if err != nil { if kerrors.IsNotFound(err) { - r.AdmissionMap.Delete(fmt.Sprintf("%s", req.NamespacedName)) + r.AdmissionMap.Delete(req.NamespacedName.String()) r.Info("rolling upgrade resource not found, deleted object from admission map", "name", req.NamespacedName) return ctrl.Result{}, nil } @@ -113,7 +113,7 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque ) // at any given point in time, there should be only one reconcile operation running per ASG - if _, present := r.ReconcileMap.LoadOrStore(rollingUpgrade.NamespacedName(), scalingGroupName); present == true { + if _, present := r.ReconcileMap.LoadOrStore(rollingUpgrade.NamespacedName(), scalingGroupName); present { r.Info("a reconcile operation is already in progress for this ASG, requeuing", "scalingGroup", scalingGroupName, "name", rollingUpgrade.NamespacedName()) return ctrl.Result{RequeueAfter: v1alpha1.DefaultRequeueTime}, nil } @@ -136,7 +136,7 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque } // store the rolling upgrade in admission map - if _, present := r.AdmissionMap.LoadOrStore(rollingUpgrade.NamespacedName(), scalingGroupName); present == false { + if _, present := r.AdmissionMap.LoadOrStore(rollingUpgrade.NamespacedName(), scalingGroupName); !present { r.Info("admitted new rolling upgrade", "scalingGroup", scalingGroupName, "update strategy", rollingUpgrade.Spec.Strategy, "name", rollingUpgrade.NamespacedName()) r.CacheConfig.FlushCache("autoscaling") } else { From 164725ffaabfaa48d41739167a2b243bf1accf8c Mon Sep 17 00:00:00 2001 From: Shreyas Badiger <7680410+shreyas-badiger@users.noreply.github.com> Date: Thu, 22 Jul 2021 12:04:07 -0700 Subject: [PATCH 48/74] Release v1.0.0 (#277) Signed-off-by: sbadiger --- .github/CHANGELOG.md | 49 +++++++++++++++++++++++++ Makefile | 2 +- config/default/manager_image_patch.yaml | 12 ++++++ 3 files changed, 62 insertions(+), 1 deletion(-) create mode 100644 config/default/manager_image_patch.yaml diff --git a/.github/CHANGELOG.md b/.github/CHANGELOG.md index e1ec48df..ae06411c 100644 --- a/.github/CHANGELOG.md +++ b/.github/CHANGELOG.md @@ -1,6 +1,55 @@ # Change Log All notable changes to this project will be documented in this file. +## [v1.0.0] - 2021-07-21 + 7a4766d (HEAD -> controller-v2, origin/controller-v2) upgrade-manager-v2: Add CI github action, fix lint errors. (#276) + 00f7e89 upgrade-manager-v2: Fix unit tests (#275) + 0e64929 upgrade-manager-v2: Process next batch while waiting on nodeInterval period. (#273) + b2b39a0 upgrade-manager-v2: Add nodeEvents handler instead of a watch handler (#272) + c0a163b move cloud discovery after nodeInterval / drainInterval wait (#270) + b15838e Carry the metrics status in RollingUpgrade CR (#267) + 610f454 upgrade-manager-v2: remove function duplicate declaration. (#266) + a4e0e84 upgrade-manager-v2: expose totalProcessing time and other metrics (#265) + 2390ea0 and CR end time (#264) + 79db022 (tag: v1.0.0-RC1) Add a mock for test and update version in Makefile (#262) + 3eafd00 Fix metrics calculation issue (#258) + 376657f Revert "Fix metrics collecting issue (#249)" (#256) + f5dd1cb Fix metrics collecting issue (#249) + 066731d final push before RC release. (#254) + 18e0e75 upgrade-manager-v2: Load test fixes (#245) + 1fc5847 metricsMutex should be initialized (#240) + a9ac50f add missing parenthesis (#239) + 6fef5fd V2 controller metrics concurrency fix (#231) + a490333 upgrade-manager-v2: Move DrainManager back to Reconciler (#236) + b659e0f Resolve compile errors caused by merge conflict. (#235) + b664fdd Create RollingUpgradeContext (#234) + b8d0e72 #2286: removed version from metric namespace (#227) + c445af9 #2285: renamed some methods related to metrics (#224) + 1f0f075 #2285: rollup CR statistic metrics in v2 (#218) + d5935e3 Unit tests for controller-v2 (#215) + 665c64b Fix bug in deleting the entry in syncMap (#203) + 77f985c Ignore generated code (#201) + 71b310a Refine metrics implementation to support goroutines (#196) + 668c5d8 Move the DrainManager within ReplaceBatch(), to access one per RollingUpgrade CR (#195) + 728dae9 Process the batch rotation in parallel (#192) + 14e950e Metrics features (#189) + 11d3ae6 Eager mode implementation (#183) + 57df5a5 Implemented node drain. (#181) + dd6a332 Migrate Script Runner (#179) + 2c1d8e7 Controller v2: Implementation of Instance termination (#178) + 7cb15b0 Fix all the "make vet" errors in Controller V2 branch. (#177) + 59e9b0d Implemented RollingUpgrade object validation. (#176) + 5cb9efb initial rotation logic + 6b8dad5 AWS API calls & Drift detection + 335fb4f aws API calls + 41bd571 Add kubernetes API calls + 8f33f1e add more scaffolding + 25644a6 initial code + 87afbd6 add API + 2816490 scaffolding + 3ad13b8 delete all + 6ce7953 Delete README.md + ## [v0.17] - 2020-12-11 * aa2b73b - use NamespacedName (#160) diff --git a/Makefile b/Makefile index 628d8c7d..c10efc6c 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -VERSION=1.0.0-RC1 +VERSION=1.0.0 # Image URL to use all building/pushing image targets IMG ?= controller:latest # Produce CRDs that work back to Kubernetes 1.11 (no version conversion) diff --git a/config/default/manager_image_patch.yaml b/config/default/manager_image_patch.yaml new file mode 100644 index 00000000..af209dae --- /dev/null +++ b/config/default/manager_image_patch.yaml @@ -0,0 +1,12 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: controller-manager + namespace: system +spec: + template: + spec: + containers: + # Change the value of image field below to your controller image URL + - image: keikoproj/rolling-upgrade-controller:1.0.0 + name: manager From 5bdc134fd2a3d50fffd12ecd46078268d82d9247 Mon Sep 17 00:00:00 2001 From: Sahil Badla Date: Fri, 30 Jul 2021 11:57:53 -0700 Subject: [PATCH 49/74] Controller v2 bdd changes (#278) * #2286: removed version from metric namespace Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * update bdd for testing Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 --- .github/workflows/bdd.yaml | 44 ++++ coverage.txt | 298 ++++++++++++++++++++++++ test-bdd/bases/kustomization.yaml | 8 + test-bdd/features/01_create.feature | 18 ++ test-bdd/go.mod | 9 + test-bdd/go.sum | 236 +++++++++++++++++++ test-bdd/main_test.go | 98 ++++++++ test-bdd/templates/rolling-upgrade.yaml | 23 ++ testbin/setup-envtest.sh | 96 ++++++++ 9 files changed, 830 insertions(+) create mode 100644 .github/workflows/bdd.yaml create mode 100644 coverage.txt create mode 100644 test-bdd/bases/kustomization.yaml create mode 100644 test-bdd/features/01_create.feature create mode 100644 test-bdd/go.mod create mode 100644 test-bdd/go.sum create mode 100644 test-bdd/main_test.go create mode 100644 test-bdd/templates/rolling-upgrade.yaml create mode 100644 testbin/setup-envtest.sh diff --git a/.github/workflows/bdd.yaml b/.github/workflows/bdd.yaml new file mode 100644 index 00000000..7109cfd2 --- /dev/null +++ b/.github/workflows/bdd.yaml @@ -0,0 +1,44 @@ +name: BDD + +on: + schedule: + - cron: '* /5 * * *' # UTC is being used, 07:00 am would be 12:00am in PT + +jobs: + build: + name: Setup & Run + if: github.repository == 'keikoproj/upgrade-manager' + runs-on: ubuntu-latest + steps: + + - name: Checkout code + uses: actions/checkout@v2 + + - name: Configure AWS Credentials + uses: aws-actions/configure-aws-credentials@v1 + with: + aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} + aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + aws-region: us-west-2 + + - name: Prerequisites + run: | + # Getting kustomize + sudo snap install kustomize + # Generating resources files + make manifests + kustomize build config/default -o test-bdd/bases/base_crd-rbac-deployment.yaml + kustomize build test-bdd/bases -o test-bdd/crd-rbac-deployment.yaml + # Setting kubeconfig + aws eks update-kubeconfig --name upgrademgr-eks-nightly --region us-west-2 + # Deploying + kubectl apply -f test-bdd/crd-rbac-deployment.yaml + + - name: Run BDD + run: | + go get github.com/cucumber/godog/cmd/godog@v0.10.0 + cd test-bdd + $HOME/go/bin/godog + + - name: Cleanup + run: kubectl delete deployment upgrade-manager-controller-manager -n upgrade-manager-system diff --git a/coverage.txt b/coverage.txt new file mode 100644 index 00000000..0266463f --- /dev/null +++ b/coverage.txt @@ -0,0 +1,298 @@ +mode: set +github.com/keikoproj/upgrade-manager/controllers/common/metrics.go:59.20,62.2 2 0 +github.com/keikoproj/upgrade-manager/controllers/common/metrics.go:65.68,67.50 2 0 +github.com/keikoproj/upgrade-manager/controllers/common/metrics.go:89.2,89.37 1 0 +github.com/keikoproj/upgrade-manager/controllers/common/metrics.go:67.50,76.17 3 0 +github.com/keikoproj/upgrade-manager/controllers/common/metrics.go:83.3,85.24 3 0 +github.com/keikoproj/upgrade-manager/controllers/common/metrics.go:76.17,77.75 1 0 +github.com/keikoproj/upgrade-manager/controllers/common/metrics.go:77.75,79.5 1 0 +github.com/keikoproj/upgrade-manager/controllers/common/metrics.go:79.10,81.5 1 0 +github.com/keikoproj/upgrade-manager/controllers/common/metrics.go:86.8,88.3 1 0 +github.com/keikoproj/upgrade-manager/controllers/common/metrics.go:92.115,94.39 2 0 +github.com/keikoproj/upgrade-manager/controllers/common/metrics.go:116.2,116.27 1 0 +github.com/keikoproj/upgrade-manager/controllers/common/metrics.go:94.39,103.17 3 0 +github.com/keikoproj/upgrade-manager/controllers/common/metrics.go:110.3,112.24 3 0 +github.com/keikoproj/upgrade-manager/controllers/common/metrics.go:103.17,104.75 1 0 +github.com/keikoproj/upgrade-manager/controllers/common/metrics.go:104.75,106.5 1 0 +github.com/keikoproj/upgrade-manager/controllers/common/metrics.go:106.10,108.5 1 0 +github.com/keikoproj/upgrade-manager/controllers/common/metrics.go:113.8,115.3 1 0 +github.com/keikoproj/upgrade-manager/controllers/common/metrics.go:119.55,121.2 1 0 +github.com/keikoproj/upgrade-manager/controllers/common/metrics.go:123.68,125.2 1 0 +github.com/keikoproj/upgrade-manager/controllers/common/metrics.go:128.81,129.42 1 1 +github.com/keikoproj/upgrade-manager/controllers/common/metrics.go:129.42,131.3 1 1 +github.com/keikoproj/upgrade-manager/controllers/common/metrics.go:131.8,133.45 2 1 +github.com/keikoproj/upgrade-manager/controllers/common/metrics.go:142.3,143.36 2 1 +github.com/keikoproj/upgrade-manager/controllers/common/metrics.go:165.3,165.38 1 1 +github.com/keikoproj/upgrade-manager/controllers/common/metrics.go:133.45,138.4 4 1 +github.com/keikoproj/upgrade-manager/controllers/common/metrics.go:138.9,140.4 1 1 +github.com/keikoproj/upgrade-manager/controllers/common/metrics.go:143.36,152.18 3 1 +github.com/keikoproj/upgrade-manager/controllers/common/metrics.go:159.4,161.25 3 1 +github.com/keikoproj/upgrade-manager/controllers/common/metrics.go:152.18,153.76 1 1 +github.com/keikoproj/upgrade-manager/controllers/common/metrics.go:153.76,155.6 1 1 +github.com/keikoproj/upgrade-manager/controllers/common/metrics.go:155.11,157.6 1 0 +github.com/keikoproj/upgrade-manager/controllers/common/metrics.go:162.9,164.4 1 1 +github.com/keikoproj/upgrade-manager/controllers/common/metrics.go:169.50,171.2 1 1 +github.com/keikoproj/upgrade-manager/controllers/common/metrics.go:173.46,175.2 1 1 +github.com/keikoproj/upgrade-manager/controllers/common/metrics.go:177.43,179.2 1 1 +github.com/keikoproj/upgrade-manager/controllers/common/utils.go:24.55,25.29 1 0 +github.com/keikoproj/upgrade-manager/controllers/common/utils.go:30.2,30.14 1 0 +github.com/keikoproj/upgrade-manager/controllers/common/utils.go:25.29,26.33 1 0 +github.com/keikoproj/upgrade-manager/controllers/common/utils.go:26.33,28.4 1 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:39.55,43.2 1 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:45.61,56.2 2 1 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:58.86,64.16 5 1 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:67.2,67.25 1 1 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:64.16,66.3 1 1 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:70.65,72.18 2 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:76.2,77.16 2 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:82.2,84.12 2 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:72.18,74.3 1 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:77.16,80.3 2 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:87.60,89.18 2 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:93.2,94.16 2 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:99.2,100.12 2 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:89.18,91.3 1 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:94.16,97.3 2 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:103.61,105.18 2 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:109.2,110.16 2 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:115.2,116.12 2 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:105.18,107.3 1 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:110.16,113.3 2 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:119.60,121.18 2 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:125.2,126.16 2 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:131.2,132.12 2 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:121.18,123.3 1 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:126.16,129.3 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:62.53,68.40 3 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:73.2,73.43 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:80.2,83.59 2 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:86.2,97.32 3 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:104.2,105.57 2 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:109.2,109.12 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:68.40,70.3 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:73.43,78.3 4 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:83.59,85.3 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:97.32,102.3 4 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:105.57,107.3 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:112.95,123.30 5 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:127.2,127.14 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:213.2,221.139 2 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:226.2,226.122 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:231.2,231.69 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:294.2,295.12 2 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:300.2,300.9 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:368.2,368.18 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:123.30,125.3 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:128.40,129.32 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:144.3,146.36 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:167.3,167.32 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:181.3,182.29 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:187.3,187.79 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:189.39,190.32 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:204.3,206.111 3 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:129.32,132.19 3 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:137.4,141.115 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:132.19,134.13 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:146.36,149.112 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:155.4,156.106 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:160.4,161.20 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:149.112,153.5 3 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:156.106,158.5 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:162.9,164.4 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:167.32,170.19 3 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:174.4,177.124 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:170.19,172.13 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:182.29,186.4 3 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:190.32,193.19 3 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:197.4,201.115 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:193.19,195.13 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:206.111,210.4 3 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:221.139,224.3 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:226.122,229.3 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:231.69,232.32 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:232.32,235.19 3 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:239.4,249.14 3 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:235.19,237.13 2 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:249.14,256.65 3 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:261.5,261.99 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:276.5,279.66 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:284.5,287.65 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:256.65,258.6 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:261.99,267.160 3 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:267.160,268.52 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:268.52,271.8 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:279.66,281.6 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:287.65,289.6 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:295.12,298.3 2 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:301.43,305.20 3 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:307.14,311.32 3 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:358.3,358.54 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:360.45,366.19 3 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:311.32,314.19 3 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:318.4,333.59 4 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:340.4,346.69 3 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:351.4,355.135 3 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:314.19,316.13 2 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:333.59,338.5 3 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:346.69,348.5 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:371.104,381.55 4 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:390.2,390.22 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:395.2,395.76 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:429.2,429.16 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:381.55,382.152 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:382.152,384.122 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:384.122,386.5 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:390.22,392.3 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:395.76,396.51 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:401.3,401.36 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:404.3,404.34 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:396.51,397.141 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:397.141,399.5 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:401.36,403.4 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:406.8,406.92 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:406.92,407.51 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:413.3,415.20 3 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:418.3,418.34 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:424.3,424.38 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:427.3,427.36 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:407.51,408.141 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:408.141,410.5 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:415.20,417.4 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:418.34,420.37 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:420.37,422.5 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:424.38,426.4 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:432.88,441.111 2 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:446.2,446.39 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:462.2,462.49 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:508.2,508.14 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:441.111,443.3 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:446.39,448.18 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:452.3,456.51 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:448.18,451.4 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:456.51,459.4 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:462.49,463.46 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:466.3,468.63 3 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:463.46,465.4 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:468.63,470.4 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:471.8,471.47 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:471.47,472.37 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:476.3,483.67 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:472.37,474.4 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:483.67,485.4 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:485.9,485.74 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:485.74,487.4 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:489.8,489.53 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:489.53,490.37 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:494.3,501.67 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:490.37,492.4 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:501.67,503.4 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:503.9,503.74 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:503.74,505.4 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:511.62,519.50 3 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:524.2,524.21 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:529.2,531.14 3 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:519.50,520.36 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:520.36,522.4 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:524.21,528.3 3 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:534.58,543.56 3 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:549.2,549.97 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:557.2,557.41 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:562.2,562.13 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:543.56,546.3 2 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:549.97,550.45 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:550.45,552.187 2 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:552.187,554.5 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:557.41,560.3 2 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:565.80,567.37 2 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:578.2,578.25 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:583.2,583.33 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:587.2,587.23 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:567.37,568.46 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:568.46,570.4 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:570.9,572.4 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:573.8,575.3 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:578.25,580.3 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:583.33,585.3 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:590.81,600.2 6 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:602.49,609.32 4 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:609.32,611.3 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:611.8,617.3 3 1 +github.com/keikoproj/upgrade-manager/controllers/cloud.go:43.97,48.2 1 1 +github.com/keikoproj/upgrade-manager/controllers/cloud.go:50.44,53.16 2 1 +github.com/keikoproj/upgrade-manager/controllers/cloud.go:56.2,59.16 3 1 +github.com/keikoproj/upgrade-manager/controllers/cloud.go:62.2,65.16 3 1 +github.com/keikoproj/upgrade-manager/controllers/cloud.go:68.2,70.12 2 1 +github.com/keikoproj/upgrade-manager/controllers/cloud.go:53.16,55.3 1 0 +github.com/keikoproj/upgrade-manager/controllers/cloud.go:59.16,61.3 1 0 +github.com/keikoproj/upgrade-manager/controllers/cloud.go:65.16,67.3 1 0 +github.com/keikoproj/upgrade-manager/controllers/metrics.go:12.153,15.2 2 1 +github.com/keikoproj/upgrade-manager/controllers/metrics.go:18.104,21.28 3 1 +github.com/keikoproj/upgrade-manager/controllers/metrics.go:24.2,24.47 1 1 +github.com/keikoproj/upgrade-manager/controllers/metrics.go:21.28,23.3 1 0 +github.com/keikoproj/upgrade-manager/controllers/metrics.go:28.100,29.30 1 1 +github.com/keikoproj/upgrade-manager/controllers/metrics.go:29.30,30.26 1 0 +github.com/keikoproj/upgrade-manager/controllers/metrics.go:30.26,32.4 1 0 +github.com/keikoproj/upgrade-manager/controllers/metrics.go:37.84,39.55 1 0 +github.com/keikoproj/upgrade-manager/controllers/metrics.go:48.2,54.4 1 0 +github.com/keikoproj/upgrade-manager/controllers/metrics.go:39.55,40.33 1 0 +github.com/keikoproj/upgrade-manager/controllers/metrics.go:40.33,46.4 3 0 +github.com/keikoproj/upgrade-manager/controllers/metrics.go:58.164,69.2 2 1 +github.com/keikoproj/upgrade-manager/controllers/metrics.go:72.139,75.47 2 1 +github.com/keikoproj/upgrade-manager/controllers/metrics.go:87.2,89.48 3 1 +github.com/keikoproj/upgrade-manager/controllers/metrics.go:75.47,83.3 2 1 +github.com/keikoproj/upgrade-manager/controllers/metrics.go:83.8,85.3 1 1 +github.com/keikoproj/upgrade-manager/controllers/metrics.go:89.48,99.3 8 1 +github.com/keikoproj/upgrade-manager/controllers/metrics.go:99.8,99.50 1 1 +github.com/keikoproj/upgrade-manager/controllers/metrics.go:99.50,102.26 3 1 +github.com/keikoproj/upgrade-manager/controllers/metrics.go:102.26,107.4 4 1 +github.com/keikoproj/upgrade-manager/controllers/metrics.go:113.118,115.2 1 1 +github.com/keikoproj/upgrade-manager/controllers/metrics.go:117.147,119.46 2 1 +github.com/keikoproj/upgrade-manager/controllers/metrics.go:127.2,127.25 1 1 +github.com/keikoproj/upgrade-manager/controllers/metrics.go:119.46,123.3 1 1 +github.com/keikoproj/upgrade-manager/controllers/metrics.go:123.8,126.3 2 1 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:75.106,79.16 4 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:89.2,89.48 1 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:96.2,97.68 2 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:103.2,103.47 1 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:108.2,116.106 3 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:122.2,122.51 1 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:133.2,133.16 1 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:139.2,139.107 1 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:145.2,164.34 5 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:172.2,172.48 1 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:178.2,178.73 1 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:79.16,80.30 1 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:85.3,85.28 1 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:80.30,84.4 3 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:89.48,93.3 3 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:97.68,101.3 3 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:103.47,105.3 1 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:116.106,119.3 2 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:122.51,125.112 3 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:130.3,130.14 1 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:125.112,129.4 3 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:133.16,136.3 1 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:139.107,142.3 2 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:142.8,144.3 1 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:164.34,168.4 3 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:172.48,176.3 3 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:182.77,189.2 1 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:192.76,194.46 1 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:194.46,196.10 2 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:202.4,202.15 1 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:196.10,201.5 4 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:204.46,206.10 2 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:212.4,212.15 1 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:206.10,211.5 4 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:214.46,216.10 2 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:222.4,222.15 1 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:216.10,221.5 4 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:228.58,229.12 1 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:229.12,232.3 2 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:236.90,238.80 2 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:238.80,240.3 1 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:244.69,248.60 3 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:252.2,252.26 1 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:255.2,255.21 1 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:248.60,251.3 2 0 +github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:252.26,254.3 1 0 diff --git a/test-bdd/bases/kustomization.yaml b/test-bdd/bases/kustomization.yaml new file mode 100644 index 00000000..c324cfa7 --- /dev/null +++ b/test-bdd/bases/kustomization.yaml @@ -0,0 +1,8 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: +- base_crd-rbac-deployment.yaml +images: + - name: keikoproj/rolling-upgrade-controller + newTag: master + diff --git a/test-bdd/features/01_create.feature b/test-bdd/features/01_create.feature new file mode 100644 index 00000000..fa02af85 --- /dev/null +++ b/test-bdd/features/01_create.feature @@ -0,0 +1,18 @@ +Feature: UM's RollingUpgrade Create + In order to create RollingUpgrades + As an EKS cluster operator + I need to submit the custom resource + + Background: + Given valid AWS Credentials + And a Kubernetes cluster + And an Auto Scaling Group named upgrademgr-eks-nightly-ASG + + Scenario: The ASG had a launch config update that allows nodes to join + Given the current Auto Scaling Group has the required initial settings + Then 1 node(s) with selector bdd-test=preUpgrade-label should be ready + Given I update the current Auto Scaling Group with LaunchConfigurationName set to upgrade-eks-nightly-LC-postUpgrade + And I submit the resource rolling-upgrade.yaml + Then the resource rolling-upgrade.yaml should be created + When the resource rolling-upgrade.yaml converge to selector .status.currentStatus=completed + Then 1 node(s) with selector bdd-test=postUpgrade-label should be ready diff --git a/test-bdd/go.mod b/test-bdd/go.mod new file mode 100644 index 00000000..e54397c5 --- /dev/null +++ b/test-bdd/go.mod @@ -0,0 +1,9 @@ +module github.com/keikoproj/upgrade-manager/test-bdd + +go 1.16 + +require ( + github.com/cucumber/godog v0.10.0 + github.com/keikoproj/kubedog v0.0.1 + github.com/sirupsen/logrus v1.6.0 +) \ No newline at end of file diff --git a/test-bdd/go.sum b/test-bdd/go.sum new file mode 100644 index 00000000..ecf143ed --- /dev/null +++ b/test-bdd/go.sum @@ -0,0 +1,236 @@ +cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= +cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= +cloud.google.com/go v0.38.0/go.mod h1:990N+gfupTy94rShfmMCWGDn0LpTmnzTp2qbd1dvSRU= +github.com/Azure/go-autorest/autorest v0.9.0/go.mod h1:xyHB1BMZT0cuDHU7I0+g046+BFDTQ8rEZB0s4Yfa6bI= +github.com/Azure/go-autorest/autorest/adal v0.5.0/go.mod h1:8Z9fGy2MpX0PvDjB1pEgQTmVqjGhiHBW7RJJEciWzS0= +github.com/Azure/go-autorest/autorest/date v0.1.0/go.mod h1:plvfp3oPSKwf2DNjlBjWF/7vwR+cUD/ELuzDCXwHUVA= +github.com/Azure/go-autorest/autorest/mocks v0.1.0/go.mod h1:OTyCOPRA2IgIlWxVYxBee2F5Gr4kF2zd2J5cFRaIDN0= +github.com/Azure/go-autorest/autorest/mocks v0.2.0/go.mod h1:OTyCOPRA2IgIlWxVYxBee2F5Gr4kF2zd2J5cFRaIDN0= +github.com/Azure/go-autorest/logger v0.1.0/go.mod h1:oExouG+K6PryycPJfVSxi/koC6LSNgds39diKLz7Vrc= +github.com/Azure/go-autorest/tracing v0.5.0/go.mod h1:r/s2XiOKccPW3HrqB+W0TQzfbtp2fGCgRFtBroKn4Dk= +github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= +github.com/NYTimes/gziphandler v0.0.0-20170623195520-56545f4a5d46/go.mod h1:3wb06e3pkSAbeQ52E9H9iFoQsEEwGN64994WTCIhntQ= +github.com/PuerkitoBio/purell v1.0.0/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0= +github.com/PuerkitoBio/urlesc v0.0.0-20160726150825-5bd2802263f2/go.mod h1:uGdkoq3SwY9Y+13GIhn11/XLaGBb4BfwItxLd5jeuXE= +github.com/aslakhellesoy/gox v1.0.100/go.mod h1:AJl542QsKKG96COVsv0N74HHzVQgDIQPceVUh1aeU2M= +github.com/aws/aws-sdk-go v1.33.8 h1:2/sOfb9oPHTRZ0lxinoaTPDcYwNa1H/SpKP4nVRBwmg= +github.com/aws/aws-sdk-go v1.33.8/go.mod h1:5zCpMtNQVjRREroY7sYe8lOMRSxkhG6MZveU8YkpAk0= +github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw= +github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= +github.com/cucumber/gherkin-go/v11 v11.0.0 h1:cwVwN1Qn2VRSfHZNLEh5x00tPBmZcjATBWDpxsR5Xug= +github.com/cucumber/gherkin-go/v11 v11.0.0/go.mod h1:CX33k2XU2qog4e+TFjOValoq6mIUq0DmVccZs238R9w= +github.com/cucumber/godog v0.10.0 h1:W01u1+o8bRpgqJRLrclN3iAanU1jAao+TwOMoSV9g1Y= +github.com/cucumber/godog v0.10.0/go.mod h1:0Q+MOUg8Z9AhzLV+nNMbThQ2x1b17yYwGyahApTLjJA= +github.com/cucumber/messages-go/v10 v10.0.1/go.mod h1:kA5T38CBlBbYLU12TIrJ4fk4wSkVVOgyh7Enyy8WnSg= +github.com/cucumber/messages-go/v10 v10.0.3 h1:m/9SD/K/A15WP7i1aemIv7cwvUw+viS51Ui5HBw1cdE= +github.com/cucumber/messages-go/v10 v10.0.3/go.mod h1:9jMZ2Y8ZxjLY6TG2+x344nt5rXstVVDYSdS5ySfI1WY= +github.com/davecgh/go-spew v0.0.0-20151105211317-5215b55f46b2/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= +github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ= +github.com/docker/spdystream v0.0.0-20160310174837-449fdfce4d96/go.mod h1:Qh8CwZgvJUkLughtfhJv5dyTYa91l1fOUCrgjqmcifM= +github.com/elazarl/goproxy v0.0.0-20170405201442-c4fc26588b6e/go.mod h1:/Zj4wYkgs4iZTTu3o/KG3Itv/qCCa8VVMlb3i9OVuzc= +github.com/emicklei/go-restful v0.0.0-20170410110728-ff4f55a20633/go.mod h1:otzb+WCGbkyDHkqmQmT5YD2WR4BBwUdeQoFo8l/7tVs= +github.com/evanphx/json-patch v4.2.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk= +github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo= +github.com/ghodss/yaml v0.0.0-20150909031657-73d445a93680/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04= +github.com/go-logr/logr v0.1.0/go.mod h1:ixOQHD9gLJUVQQ2ZOR7zLEifBX6tGkNJF4QyIY7sIas= +github.com/go-openapi/jsonpointer v0.0.0-20160704185906-46af16f9f7b1/go.mod h1:+35s3my2LFTysnkMfxsJBAMHj/DoqoB9knIWoYG/Vk0= +github.com/go-openapi/jsonreference v0.0.0-20160704190145-13c6e3589ad9/go.mod h1:W3Z9FmVs9qj+KR4zFKmDPGiLdk1D9Rlm7cyMvf57TTg= +github.com/go-openapi/spec v0.0.0-20160808142527-6aced65f8501/go.mod h1:J8+jY1nAiCcj+friV/PDoE1/3eeccG9LYBs0tYvLOWc= +github.com/go-openapi/swag v0.0.0-20160704191624-1d0bd113de87/go.mod h1:DXUve3Dpr1UfpPtxFw+EFuQ41HhCWZfha5jSVRG7C7I= +github.com/go-sql-driver/mysql v1.5.0/go.mod h1:DCzpHaOWr8IXmIStZouvnhqoel9Qv2LBy8hT2VhHyBg= +github.com/gofrs/uuid v3.2.0+incompatible h1:y12jRkkFxsd7GpqdSZ+/KCs/fJbqpEXSGd4+jfEaewE= +github.com/gofrs/uuid v3.2.0+incompatible/go.mod h1:b2aQJv3Z4Fp6yNu3cdSllBxTCLRxnplIgP/c0N/04lM= +github.com/gogo/protobuf v1.2.2-0.20190723190241-65acae22fc9d/go.mod h1:SlYgWuQ5SjCEi6WLHjHCa1yvBfUnHcTbrrZtXPKa29o= +github.com/gogo/protobuf v1.3.1 h1:DqDEcV5aeaTmdFBePNpYsp3FlcVH/2ISVVM9Qf8PSls= +github.com/gogo/protobuf v1.3.1/go.mod h1:SlYgWuQ5SjCEi6WLHjHCa1yvBfUnHcTbrrZtXPKa29o= +github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= +github.com/golang/groupcache v0.0.0-20160516000752-02826c3e7903/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= +github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= +github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= +github.com/golang/protobuf v0.0.0-20161109072736-4bd1920723d7/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= +github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= +github.com/golang/protobuf v1.3.2 h1:6nsPYzhq5kReh6QImI3k5qWzO4PEbvbIW2cwSfR/6xs= +github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= +github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= +github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= +github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M= +github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= +github.com/google/gofuzz v0.0.0-20161122191042-44d81051d367/go.mod h1:HP5RmnzzSNb993RKQDq4+1A4ia9nllfqcQFTQJedwGI= +github.com/google/gofuzz v1.0.0 h1:A8PeW59pxE9IoFRqBp37U+mSNaQoZ46F1f0f863XSXw= +github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= +github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs= +github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc= +github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg= +github.com/googleapis/gnostic v0.0.0-20170729233727-0c5108395e2d h1:7XGaL1e6bYS1yIonGp9761ExpPPV1ui0SAC59Yube9k= +github.com/googleapis/gnostic v0.0.0-20170729233727-0c5108395e2d/go.mod h1:sJBsCZ4ayReDTBIg8b9dl28c5xFWyhBTVRp3pOg5EKY= +github.com/gophercloud/gophercloud v0.1.0/go.mod h1:vxM41WHh5uqHVBMZHzuwNOHh8XEoIEcSTewFxm1c5g8= +github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA= +github.com/hashicorp/go-immutable-radix v1.2.0 h1:l6UW37iCXwZkZoAbEYnptSHVE/cQ5bOTPYG5W3vf9+8= +github.com/hashicorp/go-immutable-radix v1.2.0/go.mod h1:0y9vanUI8NX6FsYoO3zeMjhV/C5i9g4Q3DwcSNZ4P60= +github.com/hashicorp/go-memdb v1.2.1 h1:wI9btDjYUOJJHTCnRlAG/TkRyD/ij7meJMrLK9X31Cc= +github.com/hashicorp/go-memdb v1.2.1/go.mod h1:OSvLJ662Jim8hMM+gWGyhktyWk2xPCnWMc7DWIqtkGA= +github.com/hashicorp/go-uuid v1.0.0/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro= +github.com/hashicorp/go-version v1.0.0/go.mod h1:fltr4n8CU8Ke44wwGCBoEymUuxUHl09ZGVZPK5anwXA= +github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= +github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= +github.com/hashicorp/golang-lru v0.5.4 h1:YDjusn29QI/Das2iO9M0BHnIbxPeyuCHsjMW+lJfyTc= +github.com/hashicorp/golang-lru v0.5.4/go.mod h1:iADmTwqILo4mZ8BN3D2Q6+9jd8WM5uGBxy+E8yxSoD4= +github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU= +github.com/imdario/mergo v0.3.5/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA= +github.com/imdario/mergo v0.3.10 h1:6q5mVkdH/vYmqngx7kZQTjJ5HRsx+ImorDIEQ+beJgc= +github.com/imdario/mergo v0.3.10/go.mod h1:jmQim1M+e3UYxmgPu/WyfjB3N3VflVyUjjjwH0dnCYA= +github.com/jmespath/go-jmespath v0.3.0 h1:OS12ieG61fsCg5+qLJ+SsW9NicxNkg3b25OyT2yCeUc= +github.com/jmespath/go-jmespath v0.3.0/go.mod h1:9QtRXoHjLGCJ5IBSaohpXITPlowMeeYCZ7fLUTSywik= +github.com/json-iterator/go v0.0.0-20180612202835-f2b4162afba3/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU= +github.com/json-iterator/go v1.1.8 h1:QiWkFLKq0T7mpzwOTu6BzNDbfTE8OLrYhVKYMLF46Ok= +github.com/json-iterator/go v1.1.8/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4= +github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU= +github.com/keikoproj/kubedog v0.0.1 h1:SKo5g78QvlXx+JniYGvgohsB/5dm6bdj6ccyyUrnLDs= +github.com/keikoproj/kubedog v0.0.1/go.mod h1:8aRJYCL//c+RvycK3qsAfuHqyS1EP7Pa4g8M+t1wO3M= +github.com/kisielk/errcheck v1.2.0/go.mod h1:/BMXB+zMLi60iA8Vv6Ksmxu/1UDYcXs4uQLJ+jE2L00= +github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= +github.com/konsorten/go-windows-terminal-sequences v1.0.3 h1:CE8S1cTafDpPvMhIxNJKvHsGVBgn1xWYf1NbHQhywc8= +github.com/konsorten/go-windows-terminal-sequences v1.0.3/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ= +github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= +github.com/kr/pretty v0.2.0/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI= +github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= +github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= +github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= +github.com/mailru/easyjson v0.0.0-20160728113105-d5b7844b561a/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= +github.com/mitchellh/iochan v1.0.0/go.mod h1:JwYml1nuB7xOzsp52dPpHFffvOCDupsG0QubkSMEySY= +github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= +github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg= +github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= +github.com/modern-go/reflect2 v0.0.0-20180320133207-05fbef0ca5da/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0= +github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0= +github.com/modern-go/reflect2 v1.0.1 h1:9f412s+6RmYXLWZSEzVVgPGK7C2PphHj5RJrvfx9AWI= +github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0= +github.com/munnerz/goautoneg v0.0.0-20120707110453-a547fc61f48d/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ= +github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f/go.mod h1:ZdcZmHo+o7JKHSa8/e818NopupXU1YMK5fe1lsApnBw= +github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e/go.mod h1:zD1mROLANZcx1PVRCS0qkT7pwLkGfwJo4zjcN/Tysno= +github.com/onsi/ginkgo v0.0.0-20170829012221-11459a886d9c/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= +github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= +github.com/onsi/ginkgo v1.10.1/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= +github.com/onsi/gomega v0.0.0-20170829124025-dcabb60a477c/go.mod h1:C1qb7wdrVGGVU+Z6iS04AVkA3Q65CEZX59MT0QO5uiA= +github.com/onsi/gomega v1.7.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY= +github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU= +github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= +github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= +github.com/pmezard/go-difflib v0.0.0-20151028094244-d8ed2627bdf0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= +github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/sirupsen/logrus v1.6.0 h1:UBcNElsrwanuuMsnGSlYmtmgbb23qDR5dG+6X6Oo89I= +github.com/sirupsen/logrus v1.6.0/go.mod h1:7uNnSEd1DgxDLC74fIahvMZmmYsHGZGEOFrfsX/uA88= +github.com/spf13/afero v1.2.2/go.mod h1:9ZxEEn6pIJ8Rxe320qSDBk6AsU0r9pR7Q4OcevTdifk= +github.com/spf13/pflag v0.0.0-20170130214245-9ff6c6923cff/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4= +github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA= +github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= +github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= +github.com/stretchr/testify v0.0.0-20151208002404-e3a8ff8ce365/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= +github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= +github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= +github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4= +github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA= +github.com/stretchr/testify v1.6.1 h1:hDPOHmpOpP40lSULcqw7IrRb/u7w6RpDC9399XyoNd0= +github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= +go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU= +golang.org/x/crypto v0.0.0-20190211182817-74369b46fc67/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= +golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= +golang.org/x/crypto v0.0.0-20190820162420-60c769a6c586 h1:7KByu05hhLed2MO29w7p1XfZvZ13m8mub3shuVftRs0= +golang.org/x/crypto v0.0.0-20190820162420-60c769a6c586/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= +golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= +golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= +golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU= +golang.org/x/lint v0.0.0-20190301231843-5614ed5bae6f/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= +golang.org/x/net v0.0.0-20170114055629-f2499483f923/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= +golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= +golang.org/x/net v0.0.0-20191004110552-13f9640d40b9/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20200202094626-16171245cfb2 h1:CCH4IOTTfewWjGOlSp+zGcjutRKlBEZQ6wTn8ozI/nI= +golang.org/x/net v0.0.0-20200202094626-16171245cfb2/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= +golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= +golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= +golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d h1:TzXSXBo42m9gQenoE3b9BGiEpg5IG2JkU5FkPIawgtw= +golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= +golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sys v0.0.0-20170830134202-bb24a47a89ea/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190209173611-3b5209105503/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20190422165155-953cdadca894/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20190826190057-c7b8b68b1456 h1:ng0gs1AKnRRuEMZoTLLlbOd+C17zUDepwGQBb/n+JVg= +golang.org/x/sys v0.0.0-20190826190057-c7b8b68b1456/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/text v0.0.0-20160726164857-2910a502d2bf/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= +golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= +golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= +golang.org/x/text v0.3.2 h1:tW2bmiBqwgJj/UpqtC8EpXEZVYOwU0yG4iWbprSVAcs= +golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk= +golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= +golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= +golang.org/x/time v0.0.0-20200630173020-3af7569d3a1e h1:EHBhcS0mlXEAVwNyO2dLfjToGsyY4j24pTs2ScHnX7s= +golang.org/x/time v0.0.0-20200630173020-3af7569d3a1e/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= +golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20181011042414-1f849cf54d09/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20181030221726-6c7e314b6563/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY= +golang.org/x/tools v0.0.0-20190312170243-e65039ee4138/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= +google.golang.org/api v0.4.0/go.mod h1:8k5glujaEP+g9n7WNsDg8QP6cUVNI86fCNMcbazEtwE= +google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= +google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= +google.golang.org/appengine v1.5.0 h1:KxkO13IPW4Lslp2bz+KHP2E3gtFlrIGNThxkZQ3g+4c= +google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= +google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc= +google.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= +google.golang.org/genproto v0.0.0-20190418145605-e7d98fc518a7/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= +google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys= +gopkg.in/inf.v0 v0.9.1 h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc= +gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw= +gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw= +gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= +gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= +gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= +gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= +gopkg.in/yaml.v2 v2.3.0 h1:clyUAQHOM3G0M3f5vQj7LuJrETvjVot3Z5el9nffUtU= +gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= +gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c h1:dUUwHk2QECo/6vqA44rthZ8ie2QXMNeKRTHCNY2nXvo= +gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= +honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= +honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= +k8s.io/api v0.17.2 h1:NF1UFXcKN7/OOv1uxdRz3qfra8AHsPav5M93hlV9+Dc= +k8s.io/api v0.17.2/go.mod h1:BS9fjjLc4CMuqfSO8vgbHPKMt5+SF0ET6u/RVDihTo4= +k8s.io/apimachinery v0.17.2 h1:hwDQQFbdRlpnnsR64Asdi55GyCaIP/3WQpMmbNBeWr4= +k8s.io/apimachinery v0.17.2/go.mod h1:b9qmWdKlLuU9EBh+06BtLcSf/Mu89rWL33naRxs1uZg= +k8s.io/client-go v0.17.2 h1:ndIfkfXEGrNhLIgkr0+qhRguSD3u6DCmonepn1O6NYc= +k8s.io/client-go v0.17.2/go.mod h1:QAzRgsa0C2xl4/eVpeVAZMvikCn8Nm81yqVx3Kk9XYI= +k8s.io/gengo v0.0.0-20190128074634-0689ccc1d7d6/go.mod h1:ezvh/TsK7cY6rbqRK0oQQ8IAqLxYwwyPxAX1Pzy0ii0= +k8s.io/klog v0.0.0-20181102134211-b9b56d5dfc92/go.mod h1:Gq+BEi5rUBO/HRz0bTSXDUcqjScdoY3a9IHpCEIOOfk= +k8s.io/klog v0.3.0/go.mod h1:Gq+BEi5rUBO/HRz0bTSXDUcqjScdoY3a9IHpCEIOOfk= +k8s.io/klog v1.0.0 h1:Pt+yjF5aB1xDSVbau4VsWe+dQNzA0qv1LlXdC2dF6Q8= +k8s.io/klog v1.0.0/go.mod h1:4Bi6QPql/J/LkTDqv7R/cd3hPo4k2DG6Ptcz060Ez5I= +k8s.io/kube-openapi v0.0.0-20191107075043-30be4d16710a/go.mod h1:1TqjTSzOxsLGIKfj0lK8EeCP7K1iUG65v09OM0/WG5E= +k8s.io/utils v0.0.0-20191114184206-e782cd3c129f h1:GiPwtSzdP43eI1hpPCbROQCCIgCuiMMNF8YUVLF3vJo= +k8s.io/utils v0.0.0-20191114184206-e782cd3c129f/go.mod h1:sZAwmy6armz5eXlNoLmJcl4F1QuKu7sr+mFQ0byX7Ew= +sigs.k8s.io/structured-merge-diff v0.0.0-20190525122527-15d366b2352e/go.mod h1:wWxsB5ozmmv/SG7nM11ayaAW51xMvak/t1r0CSlcokI= +sigs.k8s.io/yaml v1.1.0 h1:4A07+ZFc2wgJwo8YNlQpr1rVlgUDlxXHhPJciaPY5gs= +sigs.k8s.io/yaml v1.1.0/go.mod h1:UJmg0vDUVViEyp3mgSv9WPwZCDxu4rQW1olrI1uml+o= diff --git a/test-bdd/main_test.go b/test-bdd/main_test.go new file mode 100644 index 00000000..58589c3e --- /dev/null +++ b/test-bdd/main_test.go @@ -0,0 +1,98 @@ +package test + +import ( + "os" + "testing" + "time" + + "github.com/cucumber/godog" + kdog "github.com/keikoproj/kubedog" + "github.com/sirupsen/logrus" +) + +var t kdog.Test + +var log = logrus.New() + +func TestMain(m *testing.M) { + opts := godog.Options{ + Format: "pretty", + Paths: []string{"features"}, + Randomize: time.Now().UTC().UnixNano(), // randomize scenario execution order + } + + // godog v0.10.0 (latest) + status := godog.TestSuite{ + Name: "godogs", + TestSuiteInitializer: InitializeTestSuite, + ScenarioInitializer: InitializeScenario, + Options: &opts, + }.Run() + + if st := m.Run(); st > status { + status = st + } + os.Exit(status) +} + +func InitializeTestSuite(ctx *godog.TestSuiteContext) { + ctx.BeforeSuite(func() { + log.Info("BDD >> trying to delete any existing test RollingUpgrade") + err := t.KubeContext.DeleteAllTestResources() + if err != nil { + log.Errorf("Failed deleting the test resources: %v", err) + } + }) + + ctx.AfterSuite(func() { + log.Infof("BDD >> scaling down the ASG %v", t.AwsContext.AsgName) + err := t.AwsContext.ScaleCurrentASG(0, 0) + if err != nil { + log.Errorf("Failed scaling down the ASG %v: %v", t.AwsContext.AsgName, err) + } + + log.Info("BDD >> deleting any existing test RollingUpgrade") + err = t.KubeContext.DeleteAllTestResources() + if err != nil { + log.Errorf("Failed deleting the test resources: %v", err) + } + }) + + t.SetTestSuite(ctx) +} + +func InitializeScenario(ctx *godog.ScenarioContext) { + ctx.AfterStep(func(s *godog.Step, err error) { + time.Sleep(time.Second * 5) + }) + + ctx.Step(`^the current Auto Scaling Group has the required initial settings$`, theRequiredInitialSettings) + + t.SetScenario(ctx) + t.Run() +} + +func theRequiredInitialSettings() error { + // Making sure the ASG has the pre-test launch config and 1 node with correct config + err := t.AwsContext.UpdateFieldOfCurrentASG("LaunchConfigurationName", "upgrade-eks-nightly-LC-preUpgrade") + if err != nil { + return err + } + err = t.AwsContext.ScaleCurrentASG(0, 0) + if err != nil { + return err + } + err = t.KubeContext.NodesWithSelectorShouldBe(0, "bdd-test=preUpgrade-label", "found") + if err != nil { + return err + } + err = t.KubeContext.NodesWithSelectorShouldBe(0, "bdd-test=postUpgrade-label", "found") + if err != nil { + return err + } + err = t.AwsContext.ScaleCurrentASG(1, 1) + if err != nil { + return err + } + return nil +} diff --git a/test-bdd/templates/rolling-upgrade.yaml b/test-bdd/templates/rolling-upgrade.yaml new file mode 100644 index 00000000..6ad892ec --- /dev/null +++ b/test-bdd/templates/rolling-upgrade.yaml @@ -0,0 +1,23 @@ +apiVersion: upgrademgr.keikoproj.io/v1alpha1 +kind: RollingUpgrade +metadata: + name: test-rollup + namespace: upgrade-manager-system +spec: + asgName: upgrademgr-eks-nightly-ASG + nodeIntervalSeconds: 90 + postDrain: {} + postDrainDelaySeconds: 30 + postDrainScript: | + echo "Hello, postDrain!" + postDrainWaitScript: | + echo "Hello, postDrainWait!" + postTerminate: {} + postTerminateScript: | + echo "Hello, postTerminate!" + preDrain: {} + strategy: + mode: eager + drainTimeout: 600 + type: randomUpdate + maxUnavailable: 1 \ No newline at end of file diff --git a/testbin/setup-envtest.sh b/testbin/setup-envtest.sh new file mode 100644 index 00000000..783f930d --- /dev/null +++ b/testbin/setup-envtest.sh @@ -0,0 +1,96 @@ +#!/usr/bin/env bash + +# Copyright 2020 The Kubernetes Authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -o errexit +set -o pipefail + +# Turn colors in this script off by setting the NO_COLOR variable in your +# environment to any value: +# +# $ NO_COLOR=1 test.sh +NO_COLOR=${NO_COLOR:-""} +if [ -z "$NO_COLOR" ]; then + header=$'\e[1;33m' + reset=$'\e[0m' +else + header='' + reset='' +fi + +function header_text { + echo "$header$*$reset" +} + +function setup_envtest_env { + header_text "setting up env vars" + + # Setup env vars + KUBEBUILDER_ASSETS=${KUBEBUILDER_ASSETS:-""} + if [[ -z "${KUBEBUILDER_ASSETS}" ]]; then + export KUBEBUILDER_ASSETS=$1/bin + fi +} + +# fetch k8s API gen tools and make it available under envtest_root_dir/bin. +# +# Skip fetching and untaring the tools by setting the SKIP_FETCH_TOOLS variable +# in your environment to any value: +# +# $ SKIP_FETCH_TOOLS=1 ./check-everything.sh +# +# If you skip fetching tools, this script will use the tools already on your +# machine. +function fetch_envtest_tools { + SKIP_FETCH_TOOLS=${SKIP_FETCH_TOOLS:-""} + if [ -n "$SKIP_FETCH_TOOLS" ]; then + return 0 + fi + + tmp_root=/tmp + envtest_root_dir=$tmp_root/envtest + + k8s_version="${ENVTEST_K8S_VERSION:-1.19.2}" + goarch="$(go env GOARCH)" + goos="$(go env GOOS)" + + if [[ "$goos" != "linux" && "$goos" != "darwin" ]]; then + echo "OS '$goos' not supported. Aborting." >&2 + return 1 + fi + + local dest_dir="${1}" + + # use the pre-existing version in the temporary folder if it matches our k8s version + if [[ -x "${dest_dir}/bin/kube-apiserver" ]]; then + version=$("${dest_dir}"/bin/kube-apiserver --version) + if [[ $version == *"${k8s_version}"* ]]; then + header_text "Using cached envtest tools from ${dest_dir}" + return 0 + fi + fi + + header_text "fetching envtest tools@${k8s_version} (into '${dest_dir}')" + envtest_tools_archive_name="kubebuilder-tools-$k8s_version-$goos-$goarch.tar.gz" + envtest_tools_download_url="https://storage.googleapis.com/kubebuilder-tools/$envtest_tools_archive_name" + + envtest_tools_archive_path="$tmp_root/$envtest_tools_archive_name" + if [ ! -f $envtest_tools_archive_path ]; then + curl -sL ${envtest_tools_download_url} -o "$envtest_tools_archive_path" + fi + + mkdir -p "${dest_dir}" + tar -C "${dest_dir}" --strip-components=1 -zvxf "$envtest_tools_archive_path" +} From 42abe52551b7ef7506490c8a32c012a5c79becbb Mon Sep 17 00:00:00 2001 From: Sahil Badla Date: Fri, 30 Jul 2021 12:10:04 -0700 Subject: [PATCH 50/74] Controller v2 (#279) * #2286: removed version from metric namespace Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * update bdd for testing Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 --- .github/workflows/bdd.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/bdd.yaml b/.github/workflows/bdd.yaml index 7109cfd2..84b8942c 100644 --- a/.github/workflows/bdd.yaml +++ b/.github/workflows/bdd.yaml @@ -2,7 +2,7 @@ name: BDD on: schedule: - - cron: '* /5 * * *' # UTC is being used, 07:00 am would be 12:00am in PT + - cron: "*/5 * * * *" # UTC is being used, 07:00 am would be 12:00am in PT jobs: build: From 62c2255d5677d75edceeed0e32ad3552bea0c74e Mon Sep 17 00:00:00 2001 From: Sahil Badla Date: Fri, 30 Jul 2021 13:13:52 -0700 Subject: [PATCH 51/74] Controller v2 (#280) * #2286: removed version from metric namespace Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * update bdd for testing Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 --- .github/workflows/bdd.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/bdd.yaml b/.github/workflows/bdd.yaml index 84b8942c..f33a581b 100644 --- a/.github/workflows/bdd.yaml +++ b/.github/workflows/bdd.yaml @@ -2,7 +2,7 @@ name: BDD on: schedule: - - cron: "*/5 * * * *" # UTC is being used, 07:00 am would be 12:00am in PT + - cron: '*/5 * * * *' # UTC is being used, 07:00 am would be 12:00am in PT jobs: build: From 1be8190e9854c59bf233ec9798f09c63ed2deac3 Mon Sep 17 00:00:00 2001 From: Sahil Badla Date: Fri, 30 Jul 2021 13:26:28 -0700 Subject: [PATCH 52/74] Controller v2 (#282) * #2286: removed version from metric namespace Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * update bdd for testing Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 --- .github/workflows/bdd.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/bdd.yaml b/.github/workflows/bdd.yaml index f33a581b..8ac0c134 100644 --- a/.github/workflows/bdd.yaml +++ b/.github/workflows/bdd.yaml @@ -2,7 +2,7 @@ name: BDD on: schedule: - - cron: '*/5 * * * *' # UTC is being used, 07:00 am would be 12:00am in PT + - cron: '*/10 * * * *' # UTC is being used, 07:00 am would be 12:00am in PT jobs: build: From 93626b48cb2f696292abb5f9b47a8ea08f141ea3 Mon Sep 17 00:00:00 2001 From: Sahil Badla Date: Fri, 30 Jul 2021 14:57:13 -0700 Subject: [PATCH 53/74] Controller v2 (#283) * #2286: removed version from metric namespace Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * update bdd for testing Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 --- .github/workflows/bdd.yaml | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/.github/workflows/bdd.yaml b/.github/workflows/bdd.yaml index 8ac0c134..37ae08e3 100644 --- a/.github/workflows/bdd.yaml +++ b/.github/workflows/bdd.yaml @@ -1,8 +1,9 @@ name: BDD on: - schedule: - - cron: '*/10 * * * *' # UTC is being used, 07:00 am would be 12:00am in PT + push: + branches: + - controller-v2 jobs: build: From 3841cc7a7e50833fab45c20125ca4ef979939075 Mon Sep 17 00:00:00 2001 From: Sahil Badla Date: Fri, 30 Jul 2021 16:42:29 -0700 Subject: [PATCH 54/74] #2122: bdd changes for v2 (#284) Signed-off-by: sbadla1 --- test-bdd/bases/kustomization.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test-bdd/bases/kustomization.yaml b/test-bdd/bases/kustomization.yaml index c324cfa7..ea8c537f 100644 --- a/test-bdd/bases/kustomization.yaml +++ b/test-bdd/bases/kustomization.yaml @@ -4,5 +4,5 @@ resources: - base_crd-rbac-deployment.yaml images: - name: keikoproj/rolling-upgrade-controller - newTag: master + newTag: controller-v2 From 998de0ddb3f1351b77cb368def92b82a2dffa791 Mon Sep 17 00:00:00 2001 From: Sahil Badla Date: Mon, 2 Aug 2021 13:02:23 -0700 Subject: [PATCH 55/74] V2 bdd (#285) * #2122: bdd changes for v2 Signed-off-by: sbadla1 * #2122: added makefile for test-bdd module Signed-off-by: sbadla1 * #2122: added makefile for test-bdd module Signed-off-by: sbadla1 * #2122: update rolling-update template Signed-off-by: sbadla1 * #2122: bdd changes for v2 Signed-off-by: sbadla1 * #2122: bdd changes for v2 Signed-off-by: sbadla1 --- .github/workflows/ci.yaml | 8 ++++++++ Makefile | 2 ++ test-bdd/Makefile | 12 ++++++++++++ test-bdd/templates/rolling-upgrade.yaml | 21 ++++++++++----------- 4 files changed, 32 insertions(+), 11 deletions(-) create mode 100644 test-bdd/Makefile diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index f88c3ecf..8791cbb8 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -14,12 +14,20 @@ on: jobs: build: name: CI # Lint, Test, Codecov, Docker build & Push + if: github.repository == 'keikoproj/upgrade-manager' runs-on: ubuntu-latest steps: - name: Checkout code uses: actions/checkout@v2 + - name: Configure AWS Credentials + uses: aws-actions/configure-aws-credentials@v1 + with: + aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} + aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + aws-region: us-west-2 + - name: Golangci-lint uses: golangci/golangci-lint-action@v2 with: diff --git a/Makefile b/Makefile index c10efc6c..a8ed4acd 100644 --- a/Makefile +++ b/Makefile @@ -20,6 +20,8 @@ test: generate fmt vet manifests test -f ${ENVTEST_ASSETS_DIR}/setup-envtest.sh || curl -sSLo ${ENVTEST_ASSETS_DIR}/setup-envtest.sh https://raw.githubusercontent.com/kubernetes-sigs/controller-runtime/v0.7.0/hack/setup-envtest.sh source ${ENVTEST_ASSETS_DIR}/setup-envtest.sh; fetch_envtest_tools $(ENVTEST_ASSETS_DIR); setup_envtest_env $(ENVTEST_ASSETS_DIR); go test ./... -coverprofile coverage.txt go tool cover -html=./coverage.txt -o cover.html + $(MAKE) -C test-bdd/ test + # Build manager binary manager: generate fmt vet diff --git a/test-bdd/Makefile b/test-bdd/Makefile new file mode 100644 index 00000000..7640c4e9 --- /dev/null +++ b/test-bdd/Makefile @@ -0,0 +1,12 @@ +test: fmt vet + go test ./... -coverprofile coverage.txt + go tool cover -html=./coverage.txt -o cover.html + + +# Run go fmt against code +fmt: + go fmt ./... + +# Run go vet against code +vet: + go vet ./... \ No newline at end of file diff --git a/test-bdd/templates/rolling-upgrade.yaml b/test-bdd/templates/rolling-upgrade.yaml index 6ad892ec..2dd84d69 100644 --- a/test-bdd/templates/rolling-upgrade.yaml +++ b/test-bdd/templates/rolling-upgrade.yaml @@ -5,17 +5,16 @@ metadata: namespace: upgrade-manager-system spec: asgName: upgrademgr-eks-nightly-ASG - nodeIntervalSeconds: 90 - postDrain: {} - postDrainDelaySeconds: 30 - postDrainScript: | - echo "Hello, postDrain!" - postDrainWaitScript: | - echo "Hello, postDrainWait!" - postTerminate: {} - postTerminateScript: | - echo "Hello, postTerminate!" - preDrain: {} + nodeIntervalSeconds: 300 + postDrainDelaySeconds: 90 + postDrain: + script: | + echo "Hello, postDrain!" + postWaitScript: | + echo "Hello, postDrainWait!" + postTerminate: + script: | + echo "Hello, postTerminate!" strategy: mode: eager drainTimeout: 600 From 835fd0d41e4e25d4ec9e48557d3a3a0cd8b2607a Mon Sep 17 00:00:00 2001 From: Sahil Badla Date: Tue, 3 Aug 2021 11:34:51 -0700 Subject: [PATCH 56/74] V2 bdd (#286) * #2122: bdd changes for v2 Signed-off-by: sbadla1 * #2122: added makefile for test-bdd module Signed-off-by: sbadla1 * #2122: added makefile for test-bdd module Signed-off-by: sbadla1 * #2122: update rolling-update template Signed-off-by: sbadla1 * #2122: bdd changes for v2 Signed-off-by: sbadla1 * #2122: bdd changes for v2 Signed-off-by: sbadla1 * #2122: bdd changes for v2 Signed-off-by: sbadla1 * #2122: bdd changes for v2 Signed-off-by: sbadla1 --- .github/workflows/bdd.yaml | 4 +++- Makefile | 3 ++- config/default/kustomization.yaml | 5 +++-- config/default/manager_auth_proxy_patch.yaml | 2 -- config/default/manager_image_patch.yaml | 2 +- 5 files changed, 9 insertions(+), 7 deletions(-) diff --git a/.github/workflows/bdd.yaml b/.github/workflows/bdd.yaml index 37ae08e3..0f26d0fa 100644 --- a/.github/workflows/bdd.yaml +++ b/.github/workflows/bdd.yaml @@ -42,4 +42,6 @@ jobs: $HOME/go/bin/godog - name: Cleanup - run: kubectl delete deployment upgrade-manager-controller-manager -n upgrade-manager-system + run: | + kubectl delete deployment upgrade-manager-controller-manager -n upgrade-manager-system + kubectl delete ru test-rollup -n upgrade-manager-system diff --git a/Makefile b/Makefile index a8ed4acd..2493023a 100644 --- a/Makefile +++ b/Makefile @@ -20,8 +20,9 @@ test: generate fmt vet manifests test -f ${ENVTEST_ASSETS_DIR}/setup-envtest.sh || curl -sSLo ${ENVTEST_ASSETS_DIR}/setup-envtest.sh https://raw.githubusercontent.com/kubernetes-sigs/controller-runtime/v0.7.0/hack/setup-envtest.sh source ${ENVTEST_ASSETS_DIR}/setup-envtest.sh; fetch_envtest_tools $(ENVTEST_ASSETS_DIR); setup_envtest_env $(ENVTEST_ASSETS_DIR); go test ./... -coverprofile coverage.txt go tool cover -html=./coverage.txt -o cover.html - $(MAKE) -C test-bdd/ test +# make test target for test-bdd module +test-bdd: $(MAKE) -C test-bdd/ test # Build manager binary manager: generate fmt vet diff --git a/config/default/kustomization.yaml b/config/default/kustomization.yaml index 79e04547..2f84da22 100644 --- a/config/default/kustomization.yaml +++ b/config/default/kustomization.yaml @@ -1,12 +1,12 @@ # Adds namespace to all resources. -namespace: test-system +namespace: upgrade-manager-system # Value of this field is prepended to the # names of all resources, e.g. a deployment named # "wordpress" becomes "alices-wordpress". # Note that it should also match with the prefix (text before '-') of the namespace # field above. -namePrefix: test- +namePrefix: upgrade-manager- # Labels to add to all resources and selectors. #commonLabels: @@ -25,6 +25,7 @@ bases: #- ../prometheus patchesStrategicMerge: +- manager_image_patch.yaml # Protect the /metrics endpoint by putting it behind auth. # If you want your controller-manager to expose the /metrics # endpoint w/o any authn/z, please comment the following line. diff --git a/config/default/manager_auth_proxy_patch.yaml b/config/default/manager_auth_proxy_patch.yaml index 49b1f1ab..a36932d6 100644 --- a/config/default/manager_auth_proxy_patch.yaml +++ b/config/default/manager_auth_proxy_patch.yaml @@ -21,6 +21,4 @@ spec: name: https - name: manager args: - - "--health-probe-bind-address=:8081" - "--metrics-bind-address=127.0.0.1:8080" - - "--leader-elect" diff --git a/config/default/manager_image_patch.yaml b/config/default/manager_image_patch.yaml index af209dae..b6e6d922 100644 --- a/config/default/manager_image_patch.yaml +++ b/config/default/manager_image_patch.yaml @@ -8,5 +8,5 @@ spec: spec: containers: # Change the value of image field below to your controller image URL - - image: keikoproj/rolling-upgrade-controller:1.0.0 + - image: keikoproj/rolling-upgrade-controller:controller-v2 name: manager From 2d8651c24491b0c3e067ea9abbd3c4e14786e1be Mon Sep 17 00:00:00 2001 From: Sahil Badla Date: Tue, 3 Aug 2021 12:11:01 -0700 Subject: [PATCH 57/74] Controller v2 (#288) * #2286: removed version from metric namespace Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * update bdd for testing Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: bdd changes for v2 Signed-off-by: sbadla1 --- config/default/manager_auth_proxy_patch.yaml | 2 +- config/manager/manager.yaml | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/config/default/manager_auth_proxy_patch.yaml b/config/default/manager_auth_proxy_patch.yaml index a36932d6..24d0f5ae 100644 --- a/config/default/manager_auth_proxy_patch.yaml +++ b/config/default/manager_auth_proxy_patch.yaml @@ -21,4 +21,4 @@ spec: name: https - name: manager args: - - "--metrics-bind-address=127.0.0.1:8080" + - "--max-parallel=10" diff --git a/config/manager/manager.yaml b/config/manager/manager.yaml index 70e3f6ac..55627a42 100644 --- a/config/manager/manager.yaml +++ b/config/manager/manager.yaml @@ -48,8 +48,8 @@ spec: resources: limits: cpu: 100m - memory: 30Mi + memory: 300Mi requests: cpu: 100m - memory: 20Mi + memory: 200Mi terminationGracePeriodSeconds: 10 From 86412d543c3e6ed111e8f37b98dbdbbd4591edc4 Mon Sep 17 00:00:00 2001 From: Sahil Badla Date: Tue, 3 Aug 2021 14:44:23 -0700 Subject: [PATCH 58/74] Controller v2 (#289) * #2286: removed version from metric namespace Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * update bdd for testing Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: bdd changes for v2 Signed-off-by: sbadla1 * #2122: bdd changes for v2 Signed-off-by: sbadla1 --- test-bdd/templates/rolling-upgrade.yaml | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/test-bdd/templates/rolling-upgrade.yaml b/test-bdd/templates/rolling-upgrade.yaml index 2dd84d69..6ad892ec 100644 --- a/test-bdd/templates/rolling-upgrade.yaml +++ b/test-bdd/templates/rolling-upgrade.yaml @@ -5,16 +5,17 @@ metadata: namespace: upgrade-manager-system spec: asgName: upgrademgr-eks-nightly-ASG - nodeIntervalSeconds: 300 - postDrainDelaySeconds: 90 - postDrain: - script: | - echo "Hello, postDrain!" - postWaitScript: | - echo "Hello, postDrainWait!" - postTerminate: - script: | - echo "Hello, postTerminate!" + nodeIntervalSeconds: 90 + postDrain: {} + postDrainDelaySeconds: 30 + postDrainScript: | + echo "Hello, postDrain!" + postDrainWaitScript: | + echo "Hello, postDrainWait!" + postTerminate: {} + postTerminateScript: | + echo "Hello, postTerminate!" + preDrain: {} strategy: mode: eager drainTimeout: 600 From b698dd6a35e31c2989c5d1bba2f4e1be99a875a2 Mon Sep 17 00:00:00 2001 From: Sahil Badla Date: Tue, 3 Aug 2021 15:00:42 -0700 Subject: [PATCH 59/74] Controller v2 (#290) * #2286: removed version from metric namespace Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * update bdd for testing Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: bdd changes for v2 Signed-off-by: sbadla1 * #2122: bdd changes for v2 Signed-off-by: sbadla1 * #2122: bdd changes for v2 Signed-off-by: sbadla1 --- .github/workflows/bdd.yaml | 1 - 1 file changed, 1 deletion(-) diff --git a/.github/workflows/bdd.yaml b/.github/workflows/bdd.yaml index 0f26d0fa..0a77e4de 100644 --- a/.github/workflows/bdd.yaml +++ b/.github/workflows/bdd.yaml @@ -44,4 +44,3 @@ jobs: - name: Cleanup run: | kubectl delete deployment upgrade-manager-controller-manager -n upgrade-manager-system - kubectl delete ru test-rollup -n upgrade-manager-system From db54e0bc58ba2eefbdfd83b92ea0f2dfc1b86234 Mon Sep 17 00:00:00 2001 From: Sahil Badla Date: Tue, 3 Aug 2021 15:56:46 -0700 Subject: [PATCH 60/74] Controller v2 (#291) * #2286: removed version from metric namespace Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * update bdd for testing Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: bdd changes for v2 Signed-off-by: sbadla1 * #2122: bdd changes for v2 Signed-off-by: sbadla1 * #2122: bdd changes for v2 Signed-off-by: sbadla1 * #2122: bdd changes for v2 Signed-off-by: sbadla1 --- test-bdd/templates/rolling-upgrade.yaml | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-) diff --git a/test-bdd/templates/rolling-upgrade.yaml b/test-bdd/templates/rolling-upgrade.yaml index 6ad892ec..f0d708ed 100644 --- a/test-bdd/templates/rolling-upgrade.yaml +++ b/test-bdd/templates/rolling-upgrade.yaml @@ -5,17 +5,15 @@ metadata: namespace: upgrade-manager-system spec: asgName: upgrademgr-eks-nightly-ASG - nodeIntervalSeconds: 90 - postDrain: {} - postDrainDelaySeconds: 30 - postDrainScript: | - echo "Hello, postDrain!" - postDrainWaitScript: | - echo "Hello, postDrainWait!" - postTerminate: {} - postTerminateScript: | - echo "Hello, postTerminate!" - preDrain: {} + nodeIntervalSeconds: 300 + postDrainDelaySeconds: 90 + postDrain: + script: echo "Hello, postDrain!" + postWaitScript: echo "Hello, postDrainWait!" + postTerminate: + script: echo "Hello, postTerminate!" + preDrain: + script: echo "Hello, preDrain!" strategy: mode: eager drainTimeout: 600 From c35445dc106bfa020bbe0b9252cef21a645e4012 Mon Sep 17 00:00:00 2001 From: Sahil Badla Date: Tue, 3 Aug 2021 17:43:06 -0700 Subject: [PATCH 61/74] Controller v2 (#292) * #2286: removed version from metric namespace Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * update bdd for testing Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: bdd changes for v2 Signed-off-by: sbadla1 * #2122: bdd changes for v2 Signed-off-by: sbadla1 * #2122: bdd changes for v2 Signed-off-by: sbadla1 * #2122: bdd changes for v2 Signed-off-by: sbadla1 * #2122: added sh for dockerfile Signed-off-by: sbadla1 --- Dockerfile | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/Dockerfile b/Dockerfile index ce816f3b..382fc5e7 100644 --- a/Dockerfile +++ b/Dockerfile @@ -17,6 +17,9 @@ COPY controllers/ controllers/ # Build RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GO111MODULE=on go build -a -o manager main.go +# Add busybox +FROM busybox:1.32.1 as shelladder + # Use distroless as minimal base image to package the manager binary # Refer to https://github.com/GoogleContainerTools/distroless for more details FROM gcr.io/distroless/static:nonroot @@ -24,4 +27,5 @@ WORKDIR / COPY --from=builder /workspace/manager . USER 65532:65532 +COPY --from=shelladder /bin/sh /bin/sh ENTRYPOINT ["/manager"] From 52d80d98cf80b72c0cabeb70c809bf5fe3d04a7c Mon Sep 17 00:00:00 2001 From: Shreyas Badiger <7680410+shreyas-badiger@users.noreply.github.com> Date: Thu, 5 Aug 2021 11:59:24 -0700 Subject: [PATCH 62/74] check for ASG's launch template version instead latest. (#293) Signed-off-by: sbadiger --- controllers/upgrade.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/controllers/upgrade.go b/controllers/upgrade.go index 712f7666..f88519f5 100644 --- a/controllers/upgrade.go +++ b/controllers/upgrade.go @@ -477,7 +477,7 @@ func (r *RollingUpgradeContext) IsInstanceDrifted(instance *autoscaling.Instance launchTemplateName = aws.StringValue(scalingGroup.LaunchTemplate.LaunchTemplateName) instanceTemplateName = aws.StringValue(instance.LaunchTemplate.LaunchTemplateName) instanceTemplateVersion = aws.StringValue(instance.LaunchTemplate.Version) - templateVersion = awsprovider.GetTemplateLatestVersion(r.Cloud.LaunchTemplates, launchTemplateName) + templateVersion = aws.StringValue(scalingGroup.LaunchTemplate.Version) ) if !strings.EqualFold(launchTemplateName, instanceTemplateName) { @@ -495,7 +495,7 @@ func (r *RollingUpgradeContext) IsInstanceDrifted(instance *autoscaling.Instance launchTemplateName = aws.StringValue(scalingGroup.MixedInstancesPolicy.LaunchTemplate.LaunchTemplateSpecification.LaunchTemplateName) instanceTemplateName = aws.StringValue(instance.LaunchTemplate.LaunchTemplateName) instanceTemplateVersion = aws.StringValue(instance.LaunchTemplate.Version) - templateVersion = awsprovider.GetTemplateLatestVersion(r.Cloud.LaunchTemplates, launchTemplateName) + templateVersion = aws.StringValue(scalingGroup.LaunchTemplate.Version) ) if !strings.EqualFold(launchTemplateName, instanceTemplateName) { From 0b22e3af27d855f7992cdc7a097a0cf04325fb1c Mon Sep 17 00:00:00 2001 From: Shreyas Badiger <7680410+shreyas-badiger@users.noreply.github.com> Date: Thu, 5 Aug 2021 14:00:05 -0700 Subject: [PATCH 63/74] controller v2: Fix ci job (#295) * fix CI job * fix Makefile --- .github/workflows/ci.yaml | 7 ------- Makefile | 3 --- 2 files changed, 10 deletions(-) diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index 8791cbb8..0662c98e 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -21,13 +21,6 @@ jobs: - name: Checkout code uses: actions/checkout@v2 - - name: Configure AWS Credentials - uses: aws-actions/configure-aws-credentials@v1 - with: - aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} - aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} - aws-region: us-west-2 - - name: Golangci-lint uses: golangci/golangci-lint-action@v2 with: diff --git a/Makefile b/Makefile index 2493023a..c10efc6c 100644 --- a/Makefile +++ b/Makefile @@ -21,9 +21,6 @@ test: generate fmt vet manifests source ${ENVTEST_ASSETS_DIR}/setup-envtest.sh; fetch_envtest_tools $(ENVTEST_ASSETS_DIR); setup_envtest_env $(ENVTEST_ASSETS_DIR); go test ./... -coverprofile coverage.txt go tool cover -html=./coverage.txt -o cover.html -# make test target for test-bdd module -test-bdd: $(MAKE) -C test-bdd/ test - # Build manager binary manager: generate fmt vet go build -o bin/manager main.go From 3427b371eb30dc952733dd496000dcbf5521101f Mon Sep 17 00:00:00 2001 From: Shreyas Badiger <7680410+shreyas-badiger@users.noreply.github.com> Date: Thu, 5 Aug 2021 14:01:31 -0700 Subject: [PATCH 64/74] Release v1.0.1 (#294) Signed-off-by: sbadiger --- .github/CHANGELOG.md | 16 ++++++++++++++++ Makefile | 2 +- config/default/manager_image_patch.yaml | 2 +- 3 files changed, 18 insertions(+), 2 deletions(-) diff --git a/.github/CHANGELOG.md b/.github/CHANGELOG.md index ae06411c..35ded245 100644 --- a/.github/CHANGELOG.md +++ b/.github/CHANGELOG.md @@ -1,6 +1,22 @@ # Change Log All notable changes to this project will be documented in this file. +## [v1.0.1] - 2021-08-05 +52d80d9 check for ASG's launch template version instead latest. (#293) +c35445d Controller v2: fix BDD template and update Dockerfile with bash (#292) +db54e0b Controller v2: fix BDD template (#291) +b698dd6 Controller v2: remove cleaning up ruObject as BDD already does. (#290) +86412d5 Controller v2: increase memory/CPU limit and update args (#289) +2d8651c Controller v2: update args (#288) +835fd0d V2 bdd (#286) +998de0d V2 bdd (#285) +3841cc7 #2122: bdd changes for v2 (#284) +93626b4 Controller v2: BDD cron update (#283) +1be8190 Controller v2: BDD cron update (#282) +62c2255 Controller v2: BDD cron update (#280) +42abe52 Controller v2: BDD cron update (#279) +5bdc134 Controller v2 bdd changes (#278) + ## [v1.0.0] - 2021-07-21 7a4766d (HEAD -> controller-v2, origin/controller-v2) upgrade-manager-v2: Add CI github action, fix lint errors. (#276) 00f7e89 upgrade-manager-v2: Fix unit tests (#275) diff --git a/Makefile b/Makefile index c10efc6c..0871904c 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -VERSION=1.0.0 +VERSION=1.0.1 # Image URL to use all building/pushing image targets IMG ?= controller:latest # Produce CRDs that work back to Kubernetes 1.11 (no version conversion) diff --git a/config/default/manager_image_patch.yaml b/config/default/manager_image_patch.yaml index b6e6d922..d0d64860 100644 --- a/config/default/manager_image_patch.yaml +++ b/config/default/manager_image_patch.yaml @@ -8,5 +8,5 @@ spec: spec: containers: # Change the value of image field below to your controller image URL - - image: keikoproj/rolling-upgrade-controller:controller-v2 + - image: keikoproj/rolling-upgrade-controller:1.0.1 name: manager From 95afa1b4d3ed9ba062775a4fc9d49e56200b4809 Mon Sep 17 00:00:00 2001 From: Shreyas Badiger <7680410+shreyas-badiger@users.noreply.github.com> Date: Thu, 5 Aug 2021 19:14:33 -0700 Subject: [PATCH 65/74] replace launchTemplate latest string with version number (#296) Signed-off-by: sbadiger --- controllers/upgrade.go | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/controllers/upgrade.go b/controllers/upgrade.go index f88519f5..ade49b48 100644 --- a/controllers/upgrade.go +++ b/controllers/upgrade.go @@ -41,6 +41,9 @@ import ( var ( //DefaultWaitGroupTimeout is the timeout value for DrainGroup DefaultWaitGroupTimeout = time.Second * 5 + + //LaunchTemplate latest string + LaunchTemplateVersionLatest = "$Latest" ) // DrainManager holds the information to perform drain operation in parallel. @@ -480,6 +483,11 @@ func (r *RollingUpgradeContext) IsInstanceDrifted(instance *autoscaling.Instance templateVersion = aws.StringValue(scalingGroup.LaunchTemplate.Version) ) + // replace latest string with latest version number + if strings.EqualFold(templateVersion, LaunchTemplateVersionLatest) { + templateVersion = awsprovider.GetTemplateLatestVersion(r.Cloud.LaunchTemplates, launchTemplateName) + } + if !strings.EqualFold(launchTemplateName, instanceTemplateName) { return true } else if !strings.EqualFold(instanceTemplateVersion, templateVersion) { @@ -498,6 +506,11 @@ func (r *RollingUpgradeContext) IsInstanceDrifted(instance *autoscaling.Instance templateVersion = aws.StringValue(scalingGroup.LaunchTemplate.Version) ) + // replace latest string with latest version number + if strings.EqualFold(templateVersion, LaunchTemplateVersionLatest) { + templateVersion = awsprovider.GetTemplateLatestVersion(r.Cloud.LaunchTemplates, launchTemplateName) + } + if !strings.EqualFold(launchTemplateName, instanceTemplateName) { return true } else if !strings.EqualFold(instanceTemplateVersion, templateVersion) { From 15007b3a05258b2ba8e6c9ee08fd9d5a03bc5fcc Mon Sep 17 00:00:00 2001 From: Shreyas Badiger <7680410+shreyas-badiger@users.noreply.github.com> Date: Thu, 5 Aug 2021 21:22:11 -0700 Subject: [PATCH 66/74] Release v1.0.2 (#297) Signed-off-by: sbadiger --- .github/CHANGELOG.md | 3 +++ Makefile | 2 +- config/default/manager_image_patch.yaml | 2 +- 3 files changed, 5 insertions(+), 2 deletions(-) diff --git a/.github/CHANGELOG.md b/.github/CHANGELOG.md index 35ded245..e6a302fd 100644 --- a/.github/CHANGELOG.md +++ b/.github/CHANGELOG.md @@ -1,6 +1,9 @@ # Change Log All notable changes to this project will be documented in this file. +## [v1.0.2] - 2021-08-05 +d73da1b replace launchTemplate latest string with version number (#296) + ## [v1.0.1] - 2021-08-05 52d80d9 check for ASG's launch template version instead latest. (#293) c35445d Controller v2: fix BDD template and update Dockerfile with bash (#292) diff --git a/Makefile b/Makefile index 0871904c..c9ebde73 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -VERSION=1.0.1 +VERSION=1.0.2 # Image URL to use all building/pushing image targets IMG ?= controller:latest # Produce CRDs that work back to Kubernetes 1.11 (no version conversion) diff --git a/config/default/manager_image_patch.yaml b/config/default/manager_image_patch.yaml index d0d64860..96f37bc2 100644 --- a/config/default/manager_image_patch.yaml +++ b/config/default/manager_image_patch.yaml @@ -8,5 +8,5 @@ spec: spec: containers: # Change the value of image field below to your controller image URL - - image: keikoproj/rolling-upgrade-controller:1.0.1 + - image: keikoproj/rolling-upgrade-controller:1.0.2 name: manager From 1e6d29dcf7a00d16a992dae26f5f464c96bc9208 Mon Sep 17 00:00:00 2001 From: Shreyas Badiger <7680410+shreyas-badiger@users.noreply.github.com> Date: Tue, 17 Aug 2021 15:20:28 -0700 Subject: [PATCH 67/74] Add ignoreDrainFailure and DrainTimeout as controller arguements (#300) Signed-off-by: sbadiger --- controllers/common/utils.go | 7 ++++++ controllers/rollingupgrade_controller.go | 27 ++++++++++++++---------- controllers/upgrade.go | 13 ++++++++---- main.go | 7 ++++++ 4 files changed, 39 insertions(+), 15 deletions(-) diff --git a/controllers/common/utils.go b/controllers/common/utils.go index 4b67ce3f..06af48d6 100644 --- a/controllers/common/utils.go +++ b/controllers/common/utils.go @@ -29,3 +29,10 @@ func ContainsEqualFold(slice []string, s string) bool { } return false } + +func IntMax(a, b int) int { + if a > b { + return a + } + return b +} diff --git a/controllers/rollingupgrade_controller.go b/controllers/rollingupgrade_controller.go index b57dbd39..e8372010 100644 --- a/controllers/rollingupgrade_controller.go +++ b/controllers/rollingupgrade_controller.go @@ -42,17 +42,19 @@ import ( type RollingUpgradeReconciler struct { client.Client logr.Logger - Scheme *runtime.Scheme - AdmissionMap sync.Map - CacheConfig *cache.Config - EventWriter *kubeprovider.EventWriter - maxParallel int - ScriptRunner ScriptRunner - Auth *RollingUpgradeAuthenticator - DrainGroupMapper *sync.Map - DrainErrorMapper *sync.Map - ClusterNodesMap *sync.Map - ReconcileMap *sync.Map + Scheme *runtime.Scheme + AdmissionMap sync.Map + CacheConfig *cache.Config + EventWriter *kubeprovider.EventWriter + maxParallel int + ScriptRunner ScriptRunner + Auth *RollingUpgradeAuthenticator + DrainGroupMapper *sync.Map + DrainErrorMapper *sync.Map + ClusterNodesMap *sync.Map + ReconcileMap *sync.Map + DrainTimeout int + IgnoreDrainFailures bool } // RollingUpgradeAuthenticator has the clients for providers @@ -166,6 +168,9 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque c.ClusterNodes = r.getClusterNodes() return c }(), + + DrainTimeout: r.DrainTimeout, + IgnoreDrainFailures: r.IgnoreDrainFailures, } // process node rotation diff --git a/controllers/upgrade.go b/controllers/upgrade.go index ade49b48..04e7d9a2 100644 --- a/controllers/upgrade.go +++ b/controllers/upgrade.go @@ -60,6 +60,9 @@ type RollingUpgradeContext struct { RollingUpgrade *v1alpha1.RollingUpgrade DrainManager *DrainManager metricsMutex *sync.Mutex + + DrainTimeout int + IgnoreDrainFailures bool } func (r *RollingUpgradeContext) RotateNodes() error { @@ -114,7 +117,8 @@ func (r *RollingUpgradeContext) RotateNodes() error { func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) (bool, error) { var ( - mode = r.RollingUpgrade.StrategyMode() + mode = r.RollingUpgrade.StrategyMode() + drainTimeout = common.IntMax(r.DrainTimeout, r.RollingUpgrade.DrainTimeout()) ) r.Info("rotating batch", "instances", awsprovider.GetInstanceIDs(batch), "name", r.RollingUpgrade.NamespacedName()) @@ -267,10 +271,11 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) // Turns onto NodeRotationDrain r.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationDrain) - if err := r.Auth.DrainNode(node, time.Duration(r.RollingUpgrade.PostDrainDelaySeconds()), r.RollingUpgrade.DrainTimeout(), r.Auth.Kubernetes); err != nil { - if !r.RollingUpgrade.IsIgnoreDrainFailures() { + if err := r.Auth.DrainNode(node, time.Duration(r.RollingUpgrade.PostDrainDelaySeconds()), drainTimeout, r.Auth.Kubernetes); err != nil { + // ignore drain failures if either of spec or controller args have set ignoreDrainFailures to true. + if !r.RollingUpgrade.IsIgnoreDrainFailures() && !r.IgnoreDrainFailures { r.DrainManager.DrainErrors <- errors.Errorf("DrainNode failed: instanceID - %v, %v", instanceID, err.Error()) - //TODO: BREAK AFTER ERRORS? + return } } } diff --git a/main.go b/main.go index 1cd35a51..e899e44b 100644 --- a/main.go +++ b/main.go @@ -84,10 +84,14 @@ func main() { maxParallel int maxAPIRetries int debugMode bool + drainTimeout int + ignoreDrainFailures bool logMode string ) flag.BoolVar(&debugMode, "debug", false, "enable debug logging") + flag.IntVar(&drainTimeout, "drain-timeout", 900, "when the drain command should timeout") + flag.BoolVar(&ignoreDrainFailures, "ignore-drain-failures", false, "proceed with instance termination despite drain failures.") flag.StringVar(&logMode, "log-format", "text", "Log mode: supported values: text, json.") flag.StringVar(&metricsAddr, "metrics-addr", ":8080", "The address the metric endpoint binds to.") flag.StringVar(&probeAddr, "health-probe-bind-address", ":8081", "The address the probe endpoint binds to.") @@ -201,6 +205,9 @@ func main() { DrainErrorMapper: &sync.Map{}, ClusterNodesMap: &sync.Map{}, ReconcileMap: &sync.Map{}, + + DrainTimeout: drainTimeout, + IgnoreDrainFailures: ignoreDrainFailures, } reconciler.SetMaxParallel(maxParallel) From e77431c5317e65572f25f79bce08ede1fdc7a0ba Mon Sep 17 00:00:00 2001 From: Jonah Back Date: Mon, 23 Aug 2021 16:02:02 -0700 Subject: [PATCH 68/74] fix: fix panic when using MixedInstancesPolicy (#298) Signed-off-by: Jonah Back --- controllers/helpers_test.go | 47 +++++++++++++++++++++++++++++++++++++ controllers/upgrade.go | 2 +- controllers/upgrade_test.go | 19 +++++++++++++++ 3 files changed, 67 insertions(+), 1 deletion(-) diff --git a/controllers/helpers_test.go b/controllers/helpers_test.go index b975a7e7..380c23f1 100644 --- a/controllers/helpers_test.go +++ b/controllers/helpers_test.go @@ -176,6 +176,17 @@ func createASGInstance(instanceID string, launchConfigName string) *autoscaling. } } +func createASGInstanceWithLaunchTemplate(instanceID string, launchTemplateName string) *autoscaling.Instance { + return &autoscaling.Instance{ + InstanceId: &instanceID, + LaunchTemplate: &autoscaling.LaunchTemplateSpecification{ + LaunchTemplateName: &launchTemplateName, + }, + AvailabilityZone: aws.String("az-1"), + LifecycleState: aws.String("InService"), + } +} + func createEc2Instances() []*ec2.Instance { return []*ec2.Instance{ &ec2.Instance{ @@ -203,6 +214,40 @@ func createASG(asgName string, launchConfigName string) *autoscaling.Group { } } +func createASGWithLaunchTemplate(asgName string, launchTemplate string) *autoscaling.Group { + return &autoscaling.Group{ + AutoScalingGroupName: &asgName, + LaunchTemplate: &autoscaling.LaunchTemplateSpecification{ + LaunchTemplateName: &asgName, + }, + Instances: []*autoscaling.Instance{ + createASGInstance("mock-instance-1", launchTemplate), + createASGInstance("mock-instance-2", launchTemplate), + createASGInstance("mock-instance-3", launchTemplate), + }, + DesiredCapacity: func(x int) *int64 { i := int64(x); return &i }(3), + } +} + +func createASGWithMixedInstanceLaunchTemplate(asgName string, launchTemplate string) *autoscaling.Group { + return &autoscaling.Group{ + AutoScalingGroupName: &asgName, + MixedInstancesPolicy: &autoscaling.MixedInstancesPolicy{ + LaunchTemplate: &autoscaling.LaunchTemplate{ + LaunchTemplateSpecification: &autoscaling.LaunchTemplateSpecification{ + LaunchTemplateName: &asgName, + }, + }, + }, + Instances: []*autoscaling.Instance{ + createASGInstance("mock-instance-1", launchTemplate), + createASGInstance("mock-instance-2", launchTemplate), + createASGInstance("mock-instance-3", launchTemplate), + }, + DesiredCapacity: func(x int) *int64 { i := int64(x); return &i }(3), + } +} + func createDriftedASG(asgName string, launchConfigName string) *autoscaling.Group { return &autoscaling.Group{ AutoScalingGroupName: &asgName, @@ -221,6 +266,8 @@ func createASGs() []*autoscaling.Group { createASG("mock-asg-1", "mock-launch-config-1"), createDriftedASG("mock-asg-2", "mock-launch-config-2"), createASG("mock-asg-3", "mock-launch-config-3"), + createASGWithLaunchTemplate("mock-asg-4", "mock-launch-template-4"), + createASGWithMixedInstanceLaunchTemplate("mock-asg-5", "mock-launch-template-5"), } } diff --git a/controllers/upgrade.go b/controllers/upgrade.go index 04e7d9a2..510c610f 100644 --- a/controllers/upgrade.go +++ b/controllers/upgrade.go @@ -508,7 +508,7 @@ func (r *RollingUpgradeContext) IsInstanceDrifted(instance *autoscaling.Instance launchTemplateName = aws.StringValue(scalingGroup.MixedInstancesPolicy.LaunchTemplate.LaunchTemplateSpecification.LaunchTemplateName) instanceTemplateName = aws.StringValue(instance.LaunchTemplate.LaunchTemplateName) instanceTemplateVersion = aws.StringValue(instance.LaunchTemplate.Version) - templateVersion = aws.StringValue(scalingGroup.LaunchTemplate.Version) + templateVersion = aws.StringValue(scalingGroup.MixedInstancesPolicy.LaunchTemplate.LaunchTemplateSpecification.Version) ) // replace latest string with latest version number diff --git a/controllers/upgrade_test.go b/controllers/upgrade_test.go index 948fd999..35d8e718 100644 --- a/controllers/upgrade_test.go +++ b/controllers/upgrade_test.go @@ -176,30 +176,49 @@ func TestIsInstanceDrifted(t *testing.T) { TestDescription string Reconciler *RollingUpgradeReconciler Instance *autoscaling.Instance + AsgName *string ExpectedValue bool }{ { "Instance has the same launch config as the ASG, expect false from IsInstanceDrifted", createRollingUpgradeReconciler(t), createASGInstance("mock-instance-1", "mock-launch-config-1"), + aws.String("mock-asg-1"), false, }, { "Instance has different launch config from the ASG, expect true from IsInstanceDrifted", createRollingUpgradeReconciler(t), createASGInstance("mock-instance-1", "different-launch-config"), + aws.String("mock-asg-1"), true, }, { "Instance has no launch config, expect true from IsInstanceDrifted", createRollingUpgradeReconciler(t), createASGInstance("mock-instance-1", ""), + aws.String("mock-asg-1"), + true, + }, + { + "Instance has launch template, expect true from IsInstanceDrifted", + createRollingUpgradeReconciler(t), + createASGInstanceWithLaunchTemplate("mock-instance-1", "mock-launch-template-4"), + aws.String("mock-asg-4"), + true, + }, + { + "Instance has mixed instances launch template, expect true from IsInstanceDrifted", + createRollingUpgradeReconciler(t), + createASGInstanceWithLaunchTemplate("mock-instance-1", "mock-launch-template-5"), + aws.String("mock-asg-5"), true, }, } for _, test := range tests { rollupCtx := createRollingUpgradeContext(test.Reconciler) rollupCtx.Cloud.ScalingGroups = createASGs() + rollupCtx.RollingUpgrade.Spec.AsgName = *test.AsgName actualValue := rollupCtx.IsInstanceDrifted(test.Instance) if actualValue != test.ExpectedValue { t.Errorf("Test Description: %s \n expected value: %v, actual value: %v", test.TestDescription, test.ExpectedValue, actualValue) From df08ab0c7c294480d648e1609bb4e4601f80b79f Mon Sep 17 00:00:00 2001 From: Shreyas Badiger <7680410+shreyas-badiger@users.noreply.github.com> Date: Thu, 26 Aug 2021 15:59:20 -0700 Subject: [PATCH 69/74] Set Instances to StandBy in batches (#303) * Controller v2 (#289) * #2286: removed version from metric namespace Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * update bdd for testing Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: bdd changes for v2 Signed-off-by: sbadla1 * #2122: bdd changes for v2 Signed-off-by: sbadla1 Signed-off-by: sbadiger * Controller v2 (#290) * #2286: removed version from metric namespace Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * update bdd for testing Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: bdd changes for v2 Signed-off-by: sbadla1 * #2122: bdd changes for v2 Signed-off-by: sbadla1 * #2122: bdd changes for v2 Signed-off-by: sbadla1 Signed-off-by: sbadiger * Controller v2 (#291) * #2286: removed version from metric namespace Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * update bdd for testing Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: bdd changes for v2 Signed-off-by: sbadla1 * #2122: bdd changes for v2 Signed-off-by: sbadla1 * #2122: bdd changes for v2 Signed-off-by: sbadla1 * #2122: bdd changes for v2 Signed-off-by: sbadla1 Signed-off-by: sbadiger * Controller v2 (#292) * #2286: removed version from metric namespace Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * update bdd for testing Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: ported bdd project from v1 Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: change cron Signed-off-by: sbadla1 * #2122: bdd changes for v2 Signed-off-by: sbadla1 * #2122: bdd changes for v2 Signed-off-by: sbadla1 * #2122: bdd changes for v2 Signed-off-by: sbadla1 * #2122: bdd changes for v2 Signed-off-by: sbadla1 * #2122: added sh for dockerfile Signed-off-by: sbadla1 Signed-off-by: sbadiger * check for ASG's launch template version instead latest. (#293) Signed-off-by: sbadiger * controller v2: Fix ci job (#295) * fix CI job * fix Makefile Signed-off-by: sbadiger * Release v1.0.1 (#294) Signed-off-by: sbadiger * replace launchTemplate latest string with version number (#296) Signed-off-by: sbadiger * Release v1.0.2 (#297) Signed-off-by: sbadiger * Add ignoreDrainFailure and DrainTimeout as controller arguements Signed-off-by: sbadiger * Sequential AWS API call for setting instances to StandBy Signed-off-by: sbadiger * Set Instances to StandBy in batches Signed-off-by: sbadiger * Set Instances to StandBy in batches Signed-off-by: sbadiger * Set Instances to StandBy in batches Signed-off-by: sbadiger * Set Instances to StandBy in batches Signed-off-by: sbadiger * fix: fix panic when using MixedInstancesPolicy (#298) Signed-off-by: Jonah Back Signed-off-by: sbadiger * fix vet errors Signed-off-by: sbadiger * remove log messagE Signed-off-by: sbadiger * change the logic Signed-off-by: sbadiger * fix vet error Signed-off-by: sbadiger * fix lint errors Signed-off-by: sbadiger * add comment Signed-off-by: sbadiger Co-authored-by: Sahil Badla Co-authored-by: Jonah Back --- controllers/common/utils.go | 16 ++++++ controllers/helpers_test.go | 5 ++ controllers/providers/aws/autoscaling.go | 2 + controllers/upgrade.go | 18 +++++-- controllers/upgrade_test.go | 68 ++++++++++++++++++++++++ 5 files changed, 106 insertions(+), 3 deletions(-) diff --git a/controllers/common/utils.go b/controllers/common/utils.go index 06af48d6..645bc2c8 100644 --- a/controllers/common/utils.go +++ b/controllers/common/utils.go @@ -36,3 +36,19 @@ func IntMax(a, b int) int { } return b } + +func IntMin(a, b int) int { + if a < b { + return a + } + return b +} + +func GetChunks(items []string, chunkSize int) [][]string { + var chunks [][]string + for i := 0; i < len(items); i += chunkSize { + end := IntMin(i+chunkSize, len(items)) + chunks = append(chunks, items[i:end]) + } + return chunks +} \ No newline at end of file diff --git a/controllers/helpers_test.go b/controllers/helpers_test.go index 380c23f1..8e6d961b 100644 --- a/controllers/helpers_test.go +++ b/controllers/helpers_test.go @@ -396,3 +396,8 @@ func (m *MockEC2) DescribeInstances(*ec2.DescribeInstancesInput) (*ec2.DescribeI }, }, nil } + +func (mockAutoscalingGroup MockAutoscalingGroup) EnterStandby(_ *autoscaling.EnterStandbyInput) (*autoscaling.EnterStandbyOutput, error) { + output := &autoscaling.EnterStandbyOutput{} + return output, nil +} diff --git a/controllers/providers/aws/autoscaling.go b/controllers/providers/aws/autoscaling.go index 3d66cee9..e461260a 100644 --- a/controllers/providers/aws/autoscaling.go +++ b/controllers/providers/aws/autoscaling.go @@ -32,6 +32,8 @@ var ( autoscaling.LifecycleStateWarmedTerminatingProceed, autoscaling.LifecycleStateWarmedTerminated, } + // Instance standBy limit is enforced by AWS EnterStandBy API + InstanceStandByLimit = 19 ) func (a *AmazonClientSet) DescribeScalingGroups() ([]*autoscaling.Group, error) { diff --git a/controllers/upgrade.go b/controllers/upgrade.go index 510c610f..c83019e7 100644 --- a/controllers/upgrade.go +++ b/controllers/upgrade.go @@ -160,9 +160,10 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) } // Standby r.Info("setting instances to stand-by", "batch", batchInstanceIDs, "instances(InService)", inServiceInstanceIDs, "name", r.RollingUpgrade.NamespacedName()) - if err := r.Auth.SetInstancesStandBy(inServiceInstanceIDs, r.RollingUpgrade.Spec.AsgName); err != nil { - r.Info("failed to set instances to stand-by", "batch", batchInstanceIDs, "instances(InService)", inServiceInstanceIDs, "message", err.Error(), "name", r.RollingUpgrade.NamespacedName()) + if err := r.SetBatchStandBy(batchInstanceIDs); err != nil { + r.Info("failed to set instances to stand-by", "instances", batch, "message", err.Error(), "name", r.RollingUpgrade.NamespacedName()) } + // requeue until there are no InService instances in the batch r.UpdateMetricsStatus(inProcessingNodes, nodeSteps) return true, nil @@ -187,7 +188,6 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) // Wait for desired nodes r.Info("waiting for desired nodes", "name", r.RollingUpgrade.NamespacedName()) if !r.DesiredNodesReady() { - r.Info("new node is yet to join the cluster", "name", r.RollingUpgrade.NamespacedName()) r.UpdateMetricsStatus(inProcessingNodes, nodeSteps) return true, nil } @@ -634,3 +634,15 @@ func (r *RollingUpgradeContext) endTimeUpdate() { common.TotalProcessingTime(r.RollingUpgrade.ScalingGroupName(), totalProcessingTime) } } + +// AWS API call for setting an instance to StandBy has a limit of 19. Hence we have to call the API in batches. +func (r *RollingUpgradeContext) SetBatchStandBy(instanceIDs []string) error { + var err error + instanceBatch := common.GetChunks(instanceIDs, awsprovider.InstanceStandByLimit) + for _, batch := range instanceBatch { + if err = r.Auth.SetInstancesStandBy(batch, r.RollingUpgrade.Spec.AsgName); err != nil { + return err + } + } + return nil +} diff --git a/controllers/upgrade_test.go b/controllers/upgrade_test.go index 35d8e718..d37f35c5 100644 --- a/controllers/upgrade_test.go +++ b/controllers/upgrade_test.go @@ -4,10 +4,12 @@ import ( "os" "testing" + "k8s.io/apimachinery/pkg/util/intstr" drain "k8s.io/kubectl/pkg/drain" "time" + awsprovider "github.com/keikoproj/upgrade-manager/controllers/providers/aws" corev1 "k8s.io/api/core/v1" //AWS @@ -380,3 +382,69 @@ func TestDesiredNodesReady(t *testing.T) { } } } + +func TestSetBatchStandBy(t *testing.T) { + var tests = []struct { + TestDescription string + Reconciler *RollingUpgradeReconciler + RollingUpgrade *v1alpha1.RollingUpgrade + AsgClient *MockAutoscalingGroup + ClusterNodes []*corev1.Node + ExpectedValue error + InstanceStandByLimit int + }{ + { + "Single Batch", + createRollingUpgradeReconciler(t), + func() *v1alpha1.RollingUpgrade { + rollingUpgrade := createRollingUpgrade() + rollingUpgrade.Spec.Strategy.MaxUnavailable = intstr.IntOrString{StrVal: "100%"} + return rollingUpgrade + }(), + createASGClient(), + createNodeSlice(), + nil, + 3, + }, + { + "Multiple Batches", + createRollingUpgradeReconciler(t), + func() *v1alpha1.RollingUpgrade { + rollingUpgrade := createRollingUpgrade() + rollingUpgrade.Spec.Strategy.MaxUnavailable = intstr.IntOrString{StrVal: "100%"} + return rollingUpgrade + }(), + createASGClient(), + createNodeSlice(), + nil, + 1, + }, + { + "Multiple Batches with some overflow", + createRollingUpgradeReconciler(t), + func() *v1alpha1.RollingUpgrade { + rollingUpgrade := createRollingUpgrade() + rollingUpgrade.Spec.Strategy.MaxUnavailable = intstr.IntOrString{StrVal: "100%"} + return rollingUpgrade + }(), + createASGClient(), + createNodeSlice(), + nil, + 2, + }, + } + for _, test := range tests { + awsprovider.InstanceStandByLimit = test.InstanceStandByLimit + rollupCtx := createRollingUpgradeContext(test.Reconciler) + rollupCtx.RollingUpgrade = test.RollingUpgrade + rollupCtx.Cloud.ScalingGroups = test.AsgClient.autoScalingGroups + rollupCtx.Cloud.ClusterNodes = test.ClusterNodes + rollupCtx.Auth.AmazonClientSet.AsgClient = test.AsgClient + + batch := test.AsgClient.autoScalingGroups[0].Instances + actualValue := rollupCtx.SetBatchStandBy(awsprovider.GetInstanceIDs(batch)) + if actualValue != test.ExpectedValue { + t.Errorf("Test Description: %s \n expected value: %v, actual value: %v", test.TestDescription, test.ExpectedValue, actualValue) + } + } +} From 62527258f860b6aef7713d6954b8ad9d04d11d64 Mon Sep 17 00:00:00 2001 From: Shreyas Badiger <7680410+shreyas-badiger@users.noreply.github.com> Date: Fri, 3 Sep 2021 12:31:03 -0700 Subject: [PATCH 70/74] revert #300 (#305) Signed-off-by: sbadiger --- controllers/common/utils.go | 2 +- controllers/rollingupgrade_controller.go | 27 +- controllers/upgrade.go | 10 +- coverage.txt | 375 ++++++++++++----------- main.go | 7 - 5 files changed, 211 insertions(+), 210 deletions(-) diff --git a/controllers/common/utils.go b/controllers/common/utils.go index 645bc2c8..a8c6b3e6 100644 --- a/controllers/common/utils.go +++ b/controllers/common/utils.go @@ -51,4 +51,4 @@ func GetChunks(items []string, chunkSize int) [][]string { chunks = append(chunks, items[i:end]) } return chunks -} \ No newline at end of file +} diff --git a/controllers/rollingupgrade_controller.go b/controllers/rollingupgrade_controller.go index e8372010..b57dbd39 100644 --- a/controllers/rollingupgrade_controller.go +++ b/controllers/rollingupgrade_controller.go @@ -42,19 +42,17 @@ import ( type RollingUpgradeReconciler struct { client.Client logr.Logger - Scheme *runtime.Scheme - AdmissionMap sync.Map - CacheConfig *cache.Config - EventWriter *kubeprovider.EventWriter - maxParallel int - ScriptRunner ScriptRunner - Auth *RollingUpgradeAuthenticator - DrainGroupMapper *sync.Map - DrainErrorMapper *sync.Map - ClusterNodesMap *sync.Map - ReconcileMap *sync.Map - DrainTimeout int - IgnoreDrainFailures bool + Scheme *runtime.Scheme + AdmissionMap sync.Map + CacheConfig *cache.Config + EventWriter *kubeprovider.EventWriter + maxParallel int + ScriptRunner ScriptRunner + Auth *RollingUpgradeAuthenticator + DrainGroupMapper *sync.Map + DrainErrorMapper *sync.Map + ClusterNodesMap *sync.Map + ReconcileMap *sync.Map } // RollingUpgradeAuthenticator has the clients for providers @@ -168,9 +166,6 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque c.ClusterNodes = r.getClusterNodes() return c }(), - - DrainTimeout: r.DrainTimeout, - IgnoreDrainFailures: r.IgnoreDrainFailures, } // process node rotation diff --git a/controllers/upgrade.go b/controllers/upgrade.go index c83019e7..984738ef 100644 --- a/controllers/upgrade.go +++ b/controllers/upgrade.go @@ -60,9 +60,6 @@ type RollingUpgradeContext struct { RollingUpgrade *v1alpha1.RollingUpgrade DrainManager *DrainManager metricsMutex *sync.Mutex - - DrainTimeout int - IgnoreDrainFailures bool } func (r *RollingUpgradeContext) RotateNodes() error { @@ -117,8 +114,7 @@ func (r *RollingUpgradeContext) RotateNodes() error { func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) (bool, error) { var ( - mode = r.RollingUpgrade.StrategyMode() - drainTimeout = common.IntMax(r.DrainTimeout, r.RollingUpgrade.DrainTimeout()) + mode = r.RollingUpgrade.StrategyMode() ) r.Info("rotating batch", "instances", awsprovider.GetInstanceIDs(batch), "name", r.RollingUpgrade.NamespacedName()) @@ -271,9 +267,9 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) // Turns onto NodeRotationDrain r.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationDrain) - if err := r.Auth.DrainNode(node, time.Duration(r.RollingUpgrade.PostDrainDelaySeconds()), drainTimeout, r.Auth.Kubernetes); err != nil { + if err := r.Auth.DrainNode(node, time.Duration(r.RollingUpgrade.PostDrainDelaySeconds()), r.RollingUpgrade.DrainTimeout(), r.Auth.Kubernetes); err != nil { // ignore drain failures if either of spec or controller args have set ignoreDrainFailures to true. - if !r.RollingUpgrade.IsIgnoreDrainFailures() && !r.IgnoreDrainFailures { + if !r.RollingUpgrade.IsIgnoreDrainFailures() { r.DrainManager.DrainErrors <- errors.Errorf("DrainNode failed: instanceID - %v, %v", instanceID, err.Error()) return } diff --git a/coverage.txt b/coverage.txt index 0266463f..81cf37dc 100644 --- a/coverage.txt +++ b/coverage.txt @@ -38,185 +38,15 @@ github.com/keikoproj/upgrade-manager/controllers/common/utils.go:24.55,25.29 1 0 github.com/keikoproj/upgrade-manager/controllers/common/utils.go:30.2,30.14 1 0 github.com/keikoproj/upgrade-manager/controllers/common/utils.go:25.29,26.33 1 0 github.com/keikoproj/upgrade-manager/controllers/common/utils.go:26.33,28.4 1 0 -github.com/keikoproj/upgrade-manager/controllers/script_runner.go:39.55,43.2 1 0 -github.com/keikoproj/upgrade-manager/controllers/script_runner.go:45.61,56.2 2 1 -github.com/keikoproj/upgrade-manager/controllers/script_runner.go:58.86,64.16 5 1 -github.com/keikoproj/upgrade-manager/controllers/script_runner.go:67.2,67.25 1 1 -github.com/keikoproj/upgrade-manager/controllers/script_runner.go:64.16,66.3 1 1 -github.com/keikoproj/upgrade-manager/controllers/script_runner.go:70.65,72.18 2 0 -github.com/keikoproj/upgrade-manager/controllers/script_runner.go:76.2,77.16 2 0 -github.com/keikoproj/upgrade-manager/controllers/script_runner.go:82.2,84.12 2 0 -github.com/keikoproj/upgrade-manager/controllers/script_runner.go:72.18,74.3 1 0 -github.com/keikoproj/upgrade-manager/controllers/script_runner.go:77.16,80.3 2 0 -github.com/keikoproj/upgrade-manager/controllers/script_runner.go:87.60,89.18 2 0 -github.com/keikoproj/upgrade-manager/controllers/script_runner.go:93.2,94.16 2 0 -github.com/keikoproj/upgrade-manager/controllers/script_runner.go:99.2,100.12 2 0 -github.com/keikoproj/upgrade-manager/controllers/script_runner.go:89.18,91.3 1 0 -github.com/keikoproj/upgrade-manager/controllers/script_runner.go:94.16,97.3 2 0 -github.com/keikoproj/upgrade-manager/controllers/script_runner.go:103.61,105.18 2 0 -github.com/keikoproj/upgrade-manager/controllers/script_runner.go:109.2,110.16 2 0 -github.com/keikoproj/upgrade-manager/controllers/script_runner.go:115.2,116.12 2 0 -github.com/keikoproj/upgrade-manager/controllers/script_runner.go:105.18,107.3 1 0 -github.com/keikoproj/upgrade-manager/controllers/script_runner.go:110.16,113.3 2 0 -github.com/keikoproj/upgrade-manager/controllers/script_runner.go:119.60,121.18 2 0 -github.com/keikoproj/upgrade-manager/controllers/script_runner.go:125.2,126.16 2 0 -github.com/keikoproj/upgrade-manager/controllers/script_runner.go:131.2,132.12 2 0 -github.com/keikoproj/upgrade-manager/controllers/script_runner.go:121.18,123.3 1 0 -github.com/keikoproj/upgrade-manager/controllers/script_runner.go:126.16,129.3 2 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:62.53,68.40 3 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:73.2,73.43 1 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:80.2,83.59 2 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:86.2,97.32 3 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:104.2,105.57 2 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:109.2,109.12 1 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:68.40,70.3 1 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:73.43,78.3 4 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:83.59,85.3 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:97.32,102.3 4 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:105.57,107.3 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:112.95,123.30 5 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:127.2,127.14 1 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:213.2,221.139 2 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:226.2,226.122 1 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:231.2,231.69 1 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:294.2,295.12 2 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:300.2,300.9 1 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:368.2,368.18 1 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:123.30,125.3 1 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:128.40,129.32 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:144.3,146.36 2 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:167.3,167.32 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:181.3,182.29 2 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:187.3,187.79 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:189.39,190.32 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:204.3,206.111 3 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:129.32,132.19 3 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:137.4,141.115 2 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:132.19,134.13 2 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:146.36,149.112 2 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:155.4,156.106 2 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:160.4,161.20 2 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:149.112,153.5 3 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:156.106,158.5 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:162.9,164.4 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:167.32,170.19 3 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:174.4,177.124 2 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:170.19,172.13 2 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:182.29,186.4 3 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:190.32,193.19 3 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:197.4,201.115 2 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:193.19,195.13 2 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:206.111,210.4 3 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:221.139,224.3 2 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:226.122,229.3 2 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:231.69,232.32 1 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:232.32,235.19 3 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:239.4,249.14 3 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:235.19,237.13 2 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:249.14,256.65 3 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:261.5,261.99 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:276.5,279.66 2 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:284.5,287.65 2 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:256.65,258.6 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:261.99,267.160 3 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:267.160,268.52 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:268.52,271.8 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:279.66,281.6 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:287.65,289.6 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:295.12,298.3 2 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:301.43,305.20 3 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:307.14,311.32 3 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:358.3,358.54 1 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:360.45,366.19 3 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:311.32,314.19 3 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:318.4,333.59 4 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:340.4,346.69 3 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:351.4,355.135 3 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:314.19,316.13 2 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:333.59,338.5 3 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:346.69,348.5 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:371.104,381.55 4 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:390.2,390.22 1 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:395.2,395.76 1 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:429.2,429.16 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:381.55,382.152 1 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:382.152,384.122 1 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:384.122,386.5 1 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:390.22,392.3 1 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:395.76,396.51 1 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:401.3,401.36 1 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:404.3,404.34 1 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:396.51,397.141 1 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:397.141,399.5 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:401.36,403.4 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:406.8,406.92 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:406.92,407.51 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:413.3,415.20 3 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:418.3,418.34 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:424.3,424.38 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:427.3,427.36 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:407.51,408.141 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:408.141,410.5 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:415.20,417.4 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:418.34,420.37 2 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:420.37,422.5 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:424.38,426.4 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:432.88,441.111 2 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:446.2,446.39 1 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:462.2,462.49 1 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:508.2,508.14 1 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:441.111,443.3 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:446.39,448.18 2 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:452.3,456.51 2 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:448.18,451.4 2 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:456.51,459.4 2 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:462.49,463.46 1 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:466.3,468.63 3 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:463.46,465.4 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:468.63,470.4 1 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:471.8,471.47 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:471.47,472.37 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:476.3,483.67 2 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:472.37,474.4 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:483.67,485.4 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:485.9,485.74 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:485.74,487.4 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:489.8,489.53 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:489.53,490.37 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:494.3,501.67 2 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:490.37,492.4 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:501.67,503.4 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:503.9,503.74 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:503.74,505.4 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:511.62,519.50 3 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:524.2,524.21 1 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:529.2,531.14 3 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:519.50,520.36 1 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:520.36,522.4 1 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:524.21,528.3 3 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:534.58,543.56 3 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:549.2,549.97 1 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:557.2,557.41 1 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:562.2,562.13 1 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:543.56,546.3 2 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:549.97,550.45 1 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:550.45,552.187 2 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:552.187,554.5 1 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:557.41,560.3 2 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:565.80,567.37 2 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:578.2,578.25 1 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:583.2,583.33 1 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:587.2,587.23 1 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:567.37,568.46 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:568.46,570.4 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:570.9,572.4 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:573.8,575.3 1 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:578.25,580.3 1 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:583.33,585.3 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:590.81,600.2 6 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:602.49,609.32 4 1 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:609.32,611.3 1 0 -github.com/keikoproj/upgrade-manager/controllers/upgrade.go:611.8,617.3 3 1 +github.com/keikoproj/upgrade-manager/controllers/common/utils.go:33.27,34.11 1 0 +github.com/keikoproj/upgrade-manager/controllers/common/utils.go:37.2,37.10 1 0 +github.com/keikoproj/upgrade-manager/controllers/common/utils.go:34.11,36.3 1 0 +github.com/keikoproj/upgrade-manager/controllers/common/utils.go:40.27,41.11 1 0 +github.com/keikoproj/upgrade-manager/controllers/common/utils.go:44.2,44.10 1 0 +github.com/keikoproj/upgrade-manager/controllers/common/utils.go:41.11,43.3 1 0 +github.com/keikoproj/upgrade-manager/controllers/common/utils.go:47.58,49.45 2 0 +github.com/keikoproj/upgrade-manager/controllers/common/utils.go:53.2,53.15 1 0 +github.com/keikoproj/upgrade-manager/controllers/common/utils.go:49.45,52.3 2 0 github.com/keikoproj/upgrade-manager/controllers/cloud.go:43.97,48.2 1 1 github.com/keikoproj/upgrade-manager/controllers/cloud.go:50.44,53.16 2 1 github.com/keikoproj/upgrade-manager/controllers/cloud.go:56.2,59.16 3 1 @@ -296,3 +126,190 @@ github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:25 github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:255.2,255.21 1 0 github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:248.60,251.3 2 0 github.com/keikoproj/upgrade-manager/controllers/rollingupgrade_controller.go:252.26,254.3 1 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:39.55,43.2 1 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:45.61,56.2 2 1 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:58.86,64.16 5 1 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:67.2,67.25 1 1 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:64.16,66.3 1 1 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:70.65,72.18 2 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:76.2,77.16 2 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:82.2,84.12 2 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:72.18,74.3 1 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:77.16,80.3 2 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:87.60,89.18 2 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:93.2,94.16 2 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:99.2,100.12 2 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:89.18,91.3 1 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:94.16,97.3 2 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:103.61,105.18 2 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:109.2,110.16 2 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:115.2,116.12 2 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:105.18,107.3 1 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:110.16,113.3 2 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:119.60,121.18 2 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:125.2,126.16 2 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:131.2,132.12 2 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:121.18,123.3 1 0 +github.com/keikoproj/upgrade-manager/controllers/script_runner.go:126.16,129.3 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:65.53,71.40 3 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:76.2,76.43 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:83.2,86.59 2 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:89.2,100.32 3 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:107.2,108.57 2 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:112.2,112.12 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:71.40,73.3 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:76.43,81.3 4 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:86.59,88.3 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:100.32,105.3 4 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:108.57,110.3 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:115.95,126.30 5 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:130.2,130.14 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:216.2,224.139 2 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:229.2,229.122 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:234.2,234.69 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:298.2,299.12 2 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:304.2,304.9 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:372.2,372.18 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:126.30,128.3 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:131.40,132.32 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:147.3,149.36 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:171.3,171.32 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:185.3,186.29 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:190.3,190.79 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:192.39,193.32 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:207.3,209.111 3 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:132.32,135.19 3 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:140.4,144.115 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:135.19,137.13 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:149.36,152.112 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:158.4,159.62 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:164.4,165.20 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:152.112,156.5 3 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:159.62,161.5 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:166.9,168.4 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:171.32,174.19 3 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:178.4,181.124 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:174.19,176.13 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:186.29,189.4 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:193.32,196.19 3 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:200.4,204.115 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:196.19,198.13 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:209.111,213.4 3 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:224.139,227.3 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:229.122,232.3 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:234.69,235.32 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:235.32,238.19 3 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:242.4,252.14 3 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:238.19,240.13 2 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:252.14,259.65 3 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:264.5,264.99 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:280.5,283.66 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:288.5,291.65 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:259.65,261.6 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:264.99,270.160 3 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:270.160,272.52 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:272.52,275.8 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:283.66,285.6 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:291.65,293.6 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:299.12,302.3 2 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:305.43,309.20 3 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:311.14,315.32 3 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:362.3,362.54 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:364.45,370.19 3 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:315.32,318.19 3 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:322.4,337.59 4 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:344.4,350.69 3 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:355.4,359.135 3 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:318.19,320.13 2 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:337.59,342.5 3 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:350.69,352.5 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:375.104,385.55 4 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:394.2,394.22 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:399.2,399.76 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:433.2,433.16 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:385.55,386.152 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:386.152,388.122 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:388.122,390.5 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:394.22,396.3 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:399.76,400.51 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:405.3,405.36 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:408.3,408.34 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:400.51,401.141 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:401.141,403.5 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:405.36,407.4 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:410.8,410.92 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:410.92,411.51 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:417.3,419.20 3 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:422.3,422.34 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:428.3,428.38 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:431.3,431.36 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:411.51,412.141 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:412.141,414.5 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:419.20,421.4 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:422.34,424.37 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:424.37,426.5 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:428.38,430.4 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:436.88,445.111 2 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:450.2,450.39 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:466.2,466.49 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:522.2,522.14 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:445.111,447.3 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:450.39,452.18 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:456.3,460.51 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:452.18,455.4 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:460.51,463.4 2 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:466.49,467.46 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:470.3,472.63 3 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:467.46,469.4 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:472.63,474.4 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:475.8,475.47 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:475.47,476.37 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:480.3,488.70 2 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:492.3,492.67 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:476.37,478.4 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:488.70,490.4 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:492.67,494.4 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:494.9,494.74 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:494.74,496.4 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:498.8,498.53 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:498.53,499.37 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:503.3,511.70 2 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:515.3,515.67 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:499.37,501.4 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:511.70,513.4 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:515.67,517.4 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:517.9,517.74 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:517.74,519.4 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:525.62,533.50 3 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:538.2,538.21 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:543.2,545.14 3 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:533.50,534.36 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:534.36,536.4 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:538.21,542.3 3 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:548.58,557.56 3 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:563.2,563.97 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:571.2,571.41 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:576.2,576.13 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:557.56,560.3 2 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:563.97,564.45 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:564.45,566.187 2 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:566.187,568.5 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:571.41,574.3 2 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:579.80,581.37 2 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:592.2,592.25 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:597.2,597.33 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:601.2,601.23 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:581.37,582.46 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:582.46,584.4 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:584.9,586.4 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:587.8,589.3 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:592.25,594.3 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:597.33,599.3 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:604.81,614.2 6 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:616.49,623.32 4 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:623.32,625.3 1 0 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:625.8,631.3 3 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:635.77,638.38 3 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:643.2,643.12 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:638.38,639.89 1 1 +github.com/keikoproj/upgrade-manager/controllers/upgrade.go:639.89,641.4 1 0 diff --git a/main.go b/main.go index e899e44b..1cd35a51 100644 --- a/main.go +++ b/main.go @@ -84,14 +84,10 @@ func main() { maxParallel int maxAPIRetries int debugMode bool - drainTimeout int - ignoreDrainFailures bool logMode string ) flag.BoolVar(&debugMode, "debug", false, "enable debug logging") - flag.IntVar(&drainTimeout, "drain-timeout", 900, "when the drain command should timeout") - flag.BoolVar(&ignoreDrainFailures, "ignore-drain-failures", false, "proceed with instance termination despite drain failures.") flag.StringVar(&logMode, "log-format", "text", "Log mode: supported values: text, json.") flag.StringVar(&metricsAddr, "metrics-addr", ":8080", "The address the metric endpoint binds to.") flag.StringVar(&probeAddr, "health-probe-bind-address", ":8081", "The address the probe endpoint binds to.") @@ -205,9 +201,6 @@ func main() { DrainErrorMapper: &sync.Map{}, ClusterNodesMap: &sync.Map{}, ReconcileMap: &sync.Map{}, - - DrainTimeout: drainTimeout, - IgnoreDrainFailures: ignoreDrainFailures, } reconciler.SetMaxParallel(maxParallel) From 4d1afe87fa5f78b6d7ed2c8c9dbefe3daf2fcaa3 Mon Sep 17 00:00:00 2001 From: Shreyas Badiger <7680410+shreyas-badiger@users.noreply.github.com> Date: Fri, 3 Sep 2021 12:55:17 -0700 Subject: [PATCH 71/74] Release v1.0.3 (#306) Signed-off-by: sbadiger --- .github/CHANGELOG.md | 7 +++++++ Makefile | 2 +- config/default/manager_image_patch.yaml | 2 +- 3 files changed, 9 insertions(+), 2 deletions(-) diff --git a/.github/CHANGELOG.md b/.github/CHANGELOG.md index e6a302fd..20d849d2 100644 --- a/.github/CHANGELOG.md +++ b/.github/CHANGELOG.md @@ -1,6 +1,13 @@ # Change Log All notable changes to this project will be documented in this file. +## [v1.0.3] - 2021-09-03 +6252725 revert #300 (#305) +df08ab0 Set Instances to StandBy in batches (#303) +e77431c fix: fix panic when using MixedInstancesPolicy (#298) +1e6d29d Add ignoreDrainFailure and DrainTimeout as controller arguements (#300) + + ## [v1.0.2] - 2021-08-05 d73da1b replace launchTemplate latest string with version number (#296) diff --git a/Makefile b/Makefile index c9ebde73..843a23ed 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -VERSION=1.0.2 +VERSION=1.0.3 # Image URL to use all building/pushing image targets IMG ?= controller:latest # Produce CRDs that work back to Kubernetes 1.11 (no version conversion) diff --git a/config/default/manager_image_patch.yaml b/config/default/manager_image_patch.yaml index 96f37bc2..7dc33693 100644 --- a/config/default/manager_image_patch.yaml +++ b/config/default/manager_image_patch.yaml @@ -8,5 +8,5 @@ spec: spec: containers: # Change the value of image field below to your controller image URL - - image: keikoproj/rolling-upgrade-controller:1.0.2 + - image: keikoproj/rolling-upgrade-controller:1.0.3 name: manager From 995b81b63960167c11f0c93435b2f2d3b719c520 Mon Sep 17 00:00:00 2001 From: Shreyas Badiger <7680410+shreyas-badiger@users.noreply.github.com> Date: Fri, 1 Oct 2021 16:40:05 -0700 Subject: [PATCH 72/74] controller flags for ignoreDrainFailures and drainTimeout (#307) --- api/v1alpha1/rollingupgrade_types.go | 21 +++--- api/v1alpha1/zz_generated.deepcopy.go | 12 ++- ...grademgr.keikoproj.io_rollingupgrades.yaml | 2 - controllers/helpers_test.go | 3 +- controllers/rollingupgrade_controller.go | 27 ++++--- controllers/upgrade.go | 36 +++++++-- controllers/upgrade_test.go | 75 ++++++++++++++++++- main.go | 14 +++- 8 files changed, 149 insertions(+), 41 deletions(-) diff --git a/api/v1alpha1/rollingupgrade_types.go b/api/v1alpha1/rollingupgrade_types.go index 078b1c3e..53e359be 100644 --- a/api/v1alpha1/rollingupgrade_types.go +++ b/api/v1alpha1/rollingupgrade_types.go @@ -37,7 +37,7 @@ type RollingUpgradeSpec struct { PostDrain PostDrainSpec `json:"postDrain,omitempty"` PostTerminate PostTerminateSpec `json:"postTerminate,omitempty"` Strategy UpdateStrategy `json:"strategy,omitempty"` - IgnoreDrainFailures bool `json:"ignoreDrainFailures,omitempty"` + IgnoreDrainFailures *bool `json:"ignoreDrainFailures,omitempty"` ForceRefresh bool `json:"forceRefresh,omitempty"` ReadinessGates []NodeReadinessGate `json:"readinessGates,omitempty"` } @@ -216,7 +216,7 @@ type UpdateStrategy struct { Type UpdateStrategyType `json:"type,omitempty"` Mode UpdateStrategyMode `json:"mode,omitempty"` MaxUnavailable intstr.IntOrString `json:"maxUnavailable,omitempty"` - DrainTimeout int `json:"drainTimeout"` + DrainTimeout *int `json:"drainTimeout,omitempty"` } func (c UpdateStrategyMode) String() string { @@ -232,7 +232,7 @@ func (r *RollingUpgrade) ScalingGroupName() string { return r.Spec.AsgName } -func (r *RollingUpgrade) DrainTimeout() int { +func (r *RollingUpgrade) DrainTimeout() *int { return r.Spec.Strategy.DrainTimeout } @@ -331,7 +331,7 @@ func (r *RollingUpgrade) IsForceRefresh() bool { return r.Spec.ForceRefresh } -func (r *RollingUpgrade) IsIgnoreDrainFailures() bool { +func (r *RollingUpgrade) IsIgnoreDrainFailures() *bool { return r.Spec.IgnoreDrainFailures } func (r *RollingUpgrade) StrategyMode() UpdateStrategyMode { @@ -372,12 +372,13 @@ func (r *RollingUpgrade) Validate() (bool, error) { } // validating the DrainTimeout value - if strategy.DrainTimeout == 0 { - r.Spec.Strategy.DrainTimeout = -1 - } else if strategy.DrainTimeout < -1 { - err := fmt.Errorf("%s: Invalid value for startegy DrainTimeout - %d", r.Name, strategy.MaxUnavailable.IntVal) - return false, err + if strategy.DrainTimeout != nil { + if *strategy.DrainTimeout == 0 { + *r.Spec.Strategy.DrainTimeout = -1 + } else if *strategy.DrainTimeout < -1 { + err := fmt.Errorf("%s: Invalid value for startegy DrainTimeout - %d", r.Name, strategy.MaxUnavailable.IntVal) + return false, err + } } - return true, nil } diff --git a/api/v1alpha1/zz_generated.deepcopy.go b/api/v1alpha1/zz_generated.deepcopy.go index 62489a6f..3057b24e 100644 --- a/api/v1alpha1/zz_generated.deepcopy.go +++ b/api/v1alpha1/zz_generated.deepcopy.go @@ -205,7 +205,12 @@ func (in *RollingUpgradeSpec) DeepCopyInto(out *RollingUpgradeSpec) { out.PreDrain = in.PreDrain out.PostDrain = in.PostDrain out.PostTerminate = in.PostTerminate - out.Strategy = in.Strategy + in.Strategy.DeepCopyInto(&out.Strategy) + if in.IgnoreDrainFailures != nil { + in, out := &in.IgnoreDrainFailures, &out.IgnoreDrainFailures + *out = new(bool) + **out = **in + } if in.ReadinessGates != nil { in, out := &in.ReadinessGates, &out.ReadinessGates *out = make([]NodeReadinessGate, len(*in)) @@ -304,6 +309,11 @@ func (in *RollingUpgradeStatus) DeepCopy() *RollingUpgradeStatus { func (in *UpdateStrategy) DeepCopyInto(out *UpdateStrategy) { *out = *in out.MaxUnavailable = in.MaxUnavailable + if in.DrainTimeout != nil { + in, out := &in.DrainTimeout, &out.DrainTimeout + *out = new(int) + **out = **in + } } // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new UpdateStrategy. diff --git a/config/crd/bases/upgrademgr.keikoproj.io_rollingupgrades.yaml b/config/crd/bases/upgrademgr.keikoproj.io_rollingupgrades.yaml index 3e1012ed..b7c9ef39 100644 --- a/config/crd/bases/upgrademgr.keikoproj.io_rollingupgrades.yaml +++ b/config/crd/bases/upgrademgr.keikoproj.io_rollingupgrades.yaml @@ -115,8 +115,6 @@ spec: type: string type: type: string - required: - - drainTimeout type: object type: object status: diff --git a/controllers/helpers_test.go b/controllers/helpers_test.go index 8e6d961b..cb3dae4e 100644 --- a/controllers/helpers_test.go +++ b/controllers/helpers_test.go @@ -85,8 +85,7 @@ func createRollingUpgrade() *v1alpha1.RollingUpgrade { AsgName: "mock-asg-1", PostDrainDelaySeconds: 30, Strategy: v1alpha1.UpdateStrategy{ - Type: v1alpha1.RandomUpdateStrategy, - DrainTimeout: 30, + Type: v1alpha1.RandomUpdateStrategy, }, }, } diff --git a/controllers/rollingupgrade_controller.go b/controllers/rollingupgrade_controller.go index b57dbd39..e8372010 100644 --- a/controllers/rollingupgrade_controller.go +++ b/controllers/rollingupgrade_controller.go @@ -42,17 +42,19 @@ import ( type RollingUpgradeReconciler struct { client.Client logr.Logger - Scheme *runtime.Scheme - AdmissionMap sync.Map - CacheConfig *cache.Config - EventWriter *kubeprovider.EventWriter - maxParallel int - ScriptRunner ScriptRunner - Auth *RollingUpgradeAuthenticator - DrainGroupMapper *sync.Map - DrainErrorMapper *sync.Map - ClusterNodesMap *sync.Map - ReconcileMap *sync.Map + Scheme *runtime.Scheme + AdmissionMap sync.Map + CacheConfig *cache.Config + EventWriter *kubeprovider.EventWriter + maxParallel int + ScriptRunner ScriptRunner + Auth *RollingUpgradeAuthenticator + DrainGroupMapper *sync.Map + DrainErrorMapper *sync.Map + ClusterNodesMap *sync.Map + ReconcileMap *sync.Map + DrainTimeout int + IgnoreDrainFailures bool } // RollingUpgradeAuthenticator has the clients for providers @@ -166,6 +168,9 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque c.ClusterNodes = r.getClusterNodes() return c }(), + + DrainTimeout: r.DrainTimeout, + IgnoreDrainFailures: r.IgnoreDrainFailures, } // process node rotation diff --git a/controllers/upgrade.go b/controllers/upgrade.go index 984738ef..afc3725f 100644 --- a/controllers/upgrade.go +++ b/controllers/upgrade.go @@ -54,12 +54,14 @@ type DrainManager struct { type RollingUpgradeContext struct { logr.Logger - ScriptRunner ScriptRunner - Auth *RollingUpgradeAuthenticator - Cloud *DiscoveredState - RollingUpgrade *v1alpha1.RollingUpgrade - DrainManager *DrainManager - metricsMutex *sync.Mutex + ScriptRunner ScriptRunner + Auth *RollingUpgradeAuthenticator + Cloud *DiscoveredState + RollingUpgrade *v1alpha1.RollingUpgrade + DrainManager *DrainManager + metricsMutex *sync.Mutex + DrainTimeout int + IgnoreDrainFailures bool } func (r *RollingUpgradeContext) RotateNodes() error { @@ -249,6 +251,24 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) ) r.DrainManager.DrainGroup.Add(1) + // Determine IgnoreDrainFailure and DrainTimeout values. CR spec takes the precedence. + var ( + drainTimeout int + ignoreDrainFailures bool + ) + if r.RollingUpgrade.DrainTimeout() == nil { + drainTimeout = r.DrainTimeout + } else { + drainTimeout = *r.RollingUpgrade.DrainTimeout() + } + + if r.RollingUpgrade.IsIgnoreDrainFailures() == nil { + ignoreDrainFailures = r.IgnoreDrainFailures + } else { + ignoreDrainFailures = *r.RollingUpgrade.IsIgnoreDrainFailures() + } + + // Drain the nodes in parallel go func() { defer r.DrainManager.DrainGroup.Done() @@ -267,9 +287,9 @@ func (r *RollingUpgradeContext) ReplaceNodeBatch(batch []*autoscaling.Instance) // Turns onto NodeRotationDrain r.NodeStep(inProcessingNodes, nodeSteps, r.RollingUpgrade.Spec.AsgName, nodeName, v1alpha1.NodeRotationDrain) - if err := r.Auth.DrainNode(node, time.Duration(r.RollingUpgrade.PostDrainDelaySeconds()), r.RollingUpgrade.DrainTimeout(), r.Auth.Kubernetes); err != nil { + if err := r.Auth.DrainNode(node, time.Duration(r.RollingUpgrade.PostDrainDelaySeconds()), drainTimeout, r.Auth.Kubernetes); err != nil { // ignore drain failures if either of spec or controller args have set ignoreDrainFailures to true. - if !r.RollingUpgrade.IsIgnoreDrainFailures() { + if !ignoreDrainFailures { r.DrainManager.DrainErrors <- errors.Errorf("DrainNode failed: instanceID - %v, %v", instanceID, err.Error()) return } diff --git a/controllers/upgrade_test.go b/controllers/upgrade_test.go index d37f35c5..a74665d8 100644 --- a/controllers/upgrade_test.go +++ b/controllers/upgrade_test.go @@ -45,7 +45,7 @@ func TestDrainNode(t *testing.T) { err := rollupCtx.Auth.DrainNode( test.Node, time.Duration(rollupCtx.RollingUpgrade.PostDrainDelaySeconds()), - rollupCtx.RollingUpgrade.DrainTimeout(), + 900, rollupCtx.Auth.Kubernetes, ) if (test.ExpectError && err == nil) || (!test.ExpectError && err != nil) { @@ -112,7 +112,7 @@ func TestRunCordonOrUncordon(t *testing.T) { Out: os.Stdout, ErrOut: os.Stdout, DeleteEmptyDirData: true, - Timeout: time.Duration(rollupCtx.RollingUpgrade.Spec.Strategy.DrainTimeout) * time.Second, + Timeout: 900, } err := drain.RunCordonOrUncordon(helper, test.Node, test.Cordon) if (test.ExpectError && err == nil) || (!test.ExpectError && err != nil) { @@ -163,7 +163,7 @@ func TestRunDrainNode(t *testing.T) { Out: os.Stdout, ErrOut: os.Stdout, DeleteEmptyDirData: true, - Timeout: time.Duration(rollupCtx.RollingUpgrade.Spec.Strategy.DrainTimeout) * time.Second, + Timeout: 900, } err := drain.RunNodeDrain(helper, test.Node.Name) if (test.ExpectError && err == nil) || (!test.ExpectError && err != nil) { @@ -448,3 +448,72 @@ func TestSetBatchStandBy(t *testing.T) { } } } + +func TestIgnoreDrainFailuresAndDrainTimeout(t *testing.T) { + var tests = []struct { + TestDescription string + Reconciler *RollingUpgradeReconciler + RollingUpgrade *v1alpha1.RollingUpgrade + AsgClient *MockAutoscalingGroup + ClusterNodes []*corev1.Node + ExpectedStatusValue string + }{ + { + "CR spec has IgnoreDrainFailures as nil, so default false should be considered", + createRollingUpgradeReconciler(t), + createRollingUpgrade(), + createASGClient(), + createNodeSlice(), + v1alpha1.StatusComplete, + }, + { + "CR spec has IgnoreDrainFailures as true, so default false should not be considered", + createRollingUpgradeReconciler(t), + func() *v1alpha1.RollingUpgrade { + rollingUpgrade := createRollingUpgrade() + ignoreDrainFailuresValue := true + rollingUpgrade.Spec.IgnoreDrainFailures = &ignoreDrainFailuresValue + return rollingUpgrade + }(), + createASGClient(), + createNodeSlice(), + v1alpha1.StatusComplete, + }, + { + "CR spec has DrainTimeout as nil, so default value of 900 should be considered", + createRollingUpgradeReconciler(t), + createRollingUpgrade(), + createASGClient(), + createNodeSlice(), + v1alpha1.StatusComplete, + }, + { + "CR spec has DrainTimeout as 1800, so default value of 900 should not be considered", + createRollingUpgradeReconciler(t), + func() *v1alpha1.RollingUpgrade { + rollingUpgrade := createRollingUpgrade() + drainTimeoutValue := 1800 + rollingUpgrade.Spec.Strategy.DrainTimeout = &drainTimeoutValue + return rollingUpgrade + }(), + createASGClient(), + createNodeSlice(), + v1alpha1.StatusComplete, + }, + } + for _, test := range tests { + rollupCtx := createRollingUpgradeContext(test.Reconciler) + rollupCtx.RollingUpgrade = test.RollingUpgrade + rollupCtx.Cloud.ScalingGroups = test.AsgClient.autoScalingGroups + rollupCtx.Cloud.ClusterNodes = test.ClusterNodes + rollupCtx.Auth.AmazonClientSet.AsgClient = test.AsgClient + + err := rollupCtx.RotateNodes() + if err != nil { + t.Errorf("Test Description: %s \n error: %v", test.TestDescription, err) + } + if rollupCtx.RollingUpgrade.CurrentStatus() != test.ExpectedStatusValue { + t.Errorf("Test Description: %s \n expected value: %s, actual value: %s", test.TestDescription, test.ExpectedStatusValue, rollupCtx.RollingUpgrade.CurrentStatus()) + } + } +} diff --git a/main.go b/main.go index 1cd35a51..0dc2ea95 100644 --- a/main.go +++ b/main.go @@ -85,6 +85,8 @@ func main() { maxAPIRetries int debugMode bool logMode string + drainTimeout int + ignoreDrainFailures bool ) flag.BoolVar(&debugMode, "debug", false, "enable debug logging") @@ -96,6 +98,8 @@ func main() { flag.StringVar(&namespace, "namespace", "", "The namespace in which to watch objects") flag.IntVar(&maxParallel, "max-parallel", 10, "The max number of parallel rolling upgrades") flag.IntVar(&maxAPIRetries, "max-api-retries", 12, "The number of maximum retries for failed/rate limited AWS API calls") + flag.IntVar(&drainTimeout, "drain-timeout", 900, "when the drain command should timeout") + flag.BoolVar(&ignoreDrainFailures, "ignore-drain-failures", false, "proceed with instance termination despite drain failures.") opts := zap.Options{ Development: true, @@ -197,10 +201,12 @@ func main() { ScriptRunner: controllers.ScriptRunner{ Logger: logger, }, - DrainGroupMapper: &sync.Map{}, - DrainErrorMapper: &sync.Map{}, - ClusterNodesMap: &sync.Map{}, - ReconcileMap: &sync.Map{}, + DrainGroupMapper: &sync.Map{}, + DrainErrorMapper: &sync.Map{}, + ClusterNodesMap: &sync.Map{}, + ReconcileMap: &sync.Map{}, + DrainTimeout: drainTimeout, + IgnoreDrainFailures: ignoreDrainFailures, } reconciler.SetMaxParallel(maxParallel) From c593b0157de1e2910b4479f006bdfdb9b3de46d5 Mon Sep 17 00:00:00 2001 From: Shreyas Badiger <7680410+shreyas-badiger@users.noreply.github.com> Date: Mon, 4 Oct 2021 09:59:14 -0700 Subject: [PATCH 73/74] Release v1.0.4 (#308) controller flags for ignoreDrainFailures and drainTimeout (#307) --- .github/CHANGELOG.md | 4 ++++ Makefile | 2 +- config/default/manager_image_patch.yaml | 2 +- 3 files changed, 6 insertions(+), 2 deletions(-) diff --git a/.github/CHANGELOG.md b/.github/CHANGELOG.md index 20d849d2..908dd819 100644 --- a/.github/CHANGELOG.md +++ b/.github/CHANGELOG.md @@ -1,6 +1,10 @@ # Change Log All notable changes to this project will be documented in this file. +## [v1.0.4] - 2021-10-04 +995b81b controller flags for ignoreDrainFailures and drainTimeout (#307) + + ## [v1.0.3] - 2021-09-03 6252725 revert #300 (#305) df08ab0 Set Instances to StandBy in batches (#303) diff --git a/Makefile b/Makefile index 843a23ed..643368d9 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -VERSION=1.0.3 +VERSION=1.0.4 # Image URL to use all building/pushing image targets IMG ?= controller:latest # Produce CRDs that work back to Kubernetes 1.11 (no version conversion) diff --git a/config/default/manager_image_patch.yaml b/config/default/manager_image_patch.yaml index 7dc33693..a77c96af 100644 --- a/config/default/manager_image_patch.yaml +++ b/config/default/manager_image_patch.yaml @@ -8,5 +8,5 @@ spec: spec: containers: # Change the value of image field below to your controller image URL - - image: keikoproj/rolling-upgrade-controller:1.0.3 + - image: keikoproj/rolling-upgrade-controller:1.0.4 name: manager From 61f4602efec701fd6dd3ec7f85ba0a7b8d18ee57 Mon Sep 17 00:00:00 2001 From: Eytan Avisror Date: Fri, 18 Feb 2022 19:25:08 -0500 Subject: [PATCH 74/74] Update rollingupgrade_controller.go Signed-off-by: Eytan Avisror --- controllers/rollingupgrade_controller.go | 1 + 1 file changed, 1 insertion(+) diff --git a/controllers/rollingupgrade_controller.go b/controllers/rollingupgrade_controller.go index e8372010..f7ae4581 100644 --- a/controllers/rollingupgrade_controller.go +++ b/controllers/rollingupgrade_controller.go @@ -141,6 +141,7 @@ func (r *RollingUpgradeReconciler) Reconcile(ctx context.Context, req ctrl.Reque if _, present := r.AdmissionMap.LoadOrStore(rollingUpgrade.NamespacedName(), scalingGroupName); !present { r.Info("admitted new rolling upgrade", "scalingGroup", scalingGroupName, "update strategy", rollingUpgrade.Spec.Strategy, "name", rollingUpgrade.NamespacedName()) r.CacheConfig.FlushCache("autoscaling") + r.CacheConfig.FlushCache("ec2") } else { r.Info("operating on existing rolling upgrade", "scalingGroup", scalingGroupName, "update strategy", rollingUpgrade.Spec.Strategy, "name", rollingUpgrade.NamespacedName()) }