Opensearch failed when repo list contains two opensearch entries

[marde16]

The playbook setup-component-pack-complete-harbor.yml failed during the opensearch deployment with the following output:

TASK [component-pack-harbor : Found opensearch chart version] ***************************************************************************************************************
ok: [k8.server.local] => {
    "msg": "2.7.0-20230921-162215\n2.7.0-20231006-050009"
}

TASK [component-pack-harbor : Record opensearch chart version] **************************************************************************************************************
skipping: [k8.server.local]

TASK [component-pack-harbor : Upgrade opensearch master] ********************************************************************************************************************
fatal: [k8.server.local]: FAILED! => {"changed": true, "cmd": ["helm", "upgrade", "opensearch-master", "v-connections-helm/opensearch", "-i", "--version", "2.7.0-20230921-162215", "2.7.0-20231006-050009", "-f", "/home/myuser/generated_charts/opensearch_master.yml", "--namespace", "connections", "--set", "common.probe.readinessProbe.timeoutSeconds=60", "--wait", "--timeout", "10m"], "delta": "0:00:00.041915", "end": "2023-10-27 14:43:58.855036", "msg": "non-zero return code", "rc": 1, "start": "2023-10-27 14:43:58.813121", "stderr": "Error: \"helm upgrade\" requires 2 arguments\n\nUsage:  helm upgrade [RELEASE] [CHART] [flags]", "stderr_lines": ["Error: \"helm upgrade\" requires 2 arguments", "", "Usage:  helm upgrade [RELEASE] [CHART] [flags]"], "stdout": "", "stdout_lines": []}

In our Component pack deployment the heml_repo_version_list looks as follows:

/home/myuser/generated_charts/helm_repo_version_list
NAME:CHART
v-connections-helm/bootstrap:0.1.0-20230918-122633
v-connections-helm/connections-env:0.1.40-20230918-122724
v-connections-helm/connections-outlook-desktop:0.1.0-20230918-122643
v-connections-helm/connections-persistent-stora...:0.1.1-20230921-155536
v-connections-helm/huddo-boards-cp:1.1.0-20230124-212221
v-connections-helm/huddo-boards-cp-activity-mig...:1.1.0-20230124-212221
v-connections-helm/infrastructure:0.1.0-20231006-050011
v-connections-helm/k8s-psp:0.1.0-20210909-112534
v-connections-helm/kudos-boards-cp:3.1.1-20221010-011453
v-connections-helm/kudos-boards-cp-activity-mig...:3.1.0-20221010-011453
v-connections-helm/mw-proxy:0.1.0-20230918-122709
v-connections-helm/opensearch:2.7.0-20230921-162215
v-connections-helm/opensearchstack:2.7.0-20231006-050009
v-connections-helm/orientme:0.1.0-20231006-050009
v-connections-helm/tailored-exp:1.0.0-20231006-050009
v-connections-helm/teams:1.0.0-20231006-050009

The following shell script returns both values for opensearch and opensearchstack "2.7.0-20230921-162215\n2.7.0-20231006-050009", why the playbook stops.

connections-automation/roles/hcl/component-pack-harbor/tasks/setup_opensearch.yml

Lines 50 to 53 in f23269a

	- name: Get chart and version
	shell: "cat {{ __ansible_cache_charts }}/helm_repo_version_list | grep opensearch | grep -vE \"opensearch-\" | grep -vE opensearchstack | cut -d ':' -f2"
	register: opensearch_chart_version
	become_user: "{{ __sudo_user }}"

In my opinion the shell script should be modified as follows:

shell: "cat {{ __ansible_cache_charts }}/helm_repo_version_list | grep "opensearch:" | grep -vE \"opensearch-\" | grep -vE opensearchstack | cut -d ':' -f2"

[sabrina-yee]

Hi @marde16, adding the : would work but just wondering if there is more going on here. The existing command seems to work using the sample helm_repo_version_list:

[myuser@mymachine ~]$ cat test.txt
NAME:CHART
v-connections-helm/bootstrap:0.1.0-20230918-122633
v-connections-helm/connections-env:0.1.40-20230918-122724
v-connections-helm/connections-outlook-desktop:0.1.0-20230918-122643
v-connections-helm/connections-persistent-stora...:0.1.1-20230921-155536
v-connections-helm/huddo-boards-cp:1.1.0-20230124-212221
v-connections-helm/huddo-boards-cp-activity-mig...:1.1.0-20230124-212221
v-connections-helm/infrastructure:0.1.0-20231006-050011
v-connections-helm/k8s-psp:0.1.0-20210909-112534
v-connections-helm/kudos-boards-cp:3.1.1-20221010-011453
v-connections-helm/kudos-boards-cp-activity-mig...:3.1.0-20221010-011453
v-connections-helm/mw-proxy:0.1.0-20230918-122709
v-connections-helm/opensearch:2.7.0-20230921-162215
v-connections-helm/opensearchstack:2.7.0-20231006-050009
v-connections-helm/orientme:0.1.0-20231006-050009
v-connections-helm/tailored-exp:1.0.0-20231006-050009
v-connections-helm/teams:1.0.0-20231006-050009
[myuser@mymachine ~]$
[myuser@mymachine ~]$ cat test.txt | grep opensearch | grep -vE "opensearch-" | grep -vE opensearchstack | cut -d ':' -f2
2.7.0-20230921-162215

By any chance there are more than 1 opensearch entries in your full helm_repo_version_list ?

[stoeps13]

I checked my CR4 deployment:

[ansible@cnx7-ora-cp generated_charts]$ cat helm_repo_version_list
v-connections-helm/bootstrap:0.1.0-20230918-122633
v-connections-helm/connections-env:0.1.40-20230918-122724
v-connections-helm/connections-outlook-desktop:0.1.0-20230918-122643
v-connections-helm/connections-persistent-stora...:0.1.1-20230921-155536
v-connections-helm/huddo-boards-cp:1.1.0-20230124-212221
v-connections-helm/huddo-boards-cp-activity-mig...:1.1.0-20230124-212221
v-connections-helm/infrastructure:0.1.0-20231006-050011
v-connections-helm/k8s-psp:0.1.0-20210909-112534
v-connections-helm/kudos-boards-cp:3.1.1-20221010-011453
v-connections-helm/kudos-boards-cp-activity-mig...:3.1.0-20221010-011453
v-connections-helm/mw-proxy:0.1.0-20230918-122709
v-connections-helm/opensearch:2.7.0-20230921-162215
v-connections-helm/opensearchstack:2.7.0-20231006-050009
v-connections-helm/orientme:0.1.0-20231006-050009
v-connections-helm/tailored-exp:1.0.0-20231006-050009
v-connections-helm/teams:1.0.0-20231006-050009
[ansible@cnx7-ora-cp generated_charts]$ cat helm_repo_version_list | grep opensearch | grep -vE \"opensearch-\" | grep -vE opensearchstack | cut -d ':' -f2
2.7.0-20230921-162215

So it returns the opensearch version stamp.

grep -v removes matching strings, so when there is opensearch- or opensearchstack, these do not show up.

The : would be better, because the other two greps can be removed.:

cat helm_repo_version_list | grep opensearch: | cut -d ':' -f2
2.7.0-20230921-162215

[sabrina-yee]

Agreed, we can change it to : in the next push, but wanted to make sure there isn't another issue that caused the original grep to produce unexpected results. Thanks for confirming @stoeps13.

[stoeps13]

One more thing @sabrina-yee , the helm_repo_version_list has 3 charts with abbreviations of ... in the end

v-connections-helm/connections-persistent-stora...:0.1.1-20230921-155536
v-connections-helm/huddo-boards-cp-activity-mig...:1.1.0-20230124-212221
v-connections-helm/kudos-boards-cp-activity-mig...:3.1.0-20221010-011453

So Huddo/Kudos will not use it, when Quay.io is configured, but I think the connections-persistent-storage could fail.

I'm wrong, it doesn't matter because the chart grep is for the shortened name:

connections-automation/roles/hcl/component-pack-harbor/tasks/setup_connections_volumes.yml

Line 2 in f23269a

shell: "cat {{ __ansible_cache_charts }}/helm_repo_version_list | grep connections-persistent-st | cut -d ':' -f2"

[sabrina-yee]

:) yes, we've seen it before so shorten the name. Hopefully these issue will all go away once we switch to using OCI (soon).

[marde16]

Hi @sabrina-yee ,

I further analyzed my issue. The Ansible playbooks are based on 8CR2 version. Therefore the shell script differs between 8CR2 and the current version to get the opensearch version.

8CR2 version

- name:                      Get chart and version
  shell:                     "helm search repo {{ __helm_repository_local_name }} {{ __helm_repo_flag }} | grep opensearch | grep -vE \"opensearch-\" | awk {'print $2'}"
  register:                  opensearch_chart_version
  become_user:               "{{ __sudo_user }}"

since version 8CR3 version July 2023 cnx8 CR3 release

connections-automation/roles/hcl/component-pack-harbor/tasks/setup_opensearch.yml

Lines 50 to 53 in f23269a

	- name: Get chart and version
	shell: "cat {{ __ansible_cache_charts }}/helm_repo_version_list | grep opensearch | grep -vE \"opensearch-\" | grep -vE opensearchstack | cut -d ':' -f2"
	register: opensearch_chart_version
	become_user: "{{ __sudo_user }}"

I have only two opensearch entries in my helm repo list:

v-connections-helm/opensearch:2.7.0-20230921-162215
v-connections-helm/opensearchstack:2.7.0-20231006-050009

So the grep parameters could be modified as @stoeps13 described.