Update cluster by ansible 2.13 to 2.17.1 #170

dzaczek · 2024-10-02T10:07:29Z

Describe the bug
Im trying to use ansible playbook to update opensearch 2.13 to to 2.17.1 on ubuntu 22.04 LTS
2.13 was installed also by playbook few month ago .
Playbook failing during restart open search waiting , geeting timeout.

To Reproduce
Steps to reproduce the behavior:
Install version 2.13 by ansible playbook (iac_enable: false)
Install version 2.17.1 by ansible playbook (iac_enable: true, copy_custom_security_configs: true)
)

Expected behavior
Working able to start cluster

LOGS


[2024-10-02T09:47:29,489][ERROR][o.o.b.Bootstrap          ] [a4211-os-master-1] Exception
java.lang.IllegalStateException: jar hell!
class: org.apache.lucene.queries.CommonTermsQuery
jar1: /usr/share/opensearch/lib/lucene-queries-9.10.0.jar
jar2: /usr/share/opensearch/lib/lucene-queries-9.11.1.jar
        at org.opensearch.bootstrap.JarHell.checkClass(JarHell.java:316) ~[opensearch-common-2.13.0.jar:2.17.1]
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:215) ~[opensearch-common-2.13.0.jar:2.17.1]
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:102) ~[opensearch-common-2.13.0.jar:2.17.1]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) [opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) [opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) [opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) [opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-2.13.0.jar:2.17.1]
        at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-2.13.0.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) [opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) [opensearch-2.17.1.jar:2.17.1]
[2024-10-02T09:47:29,497][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [a4211-os-master-1] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: jar hell!
class: org.apache.lucene.queries.CommonTermsQuery
jar1: /usr/share/opensearch/lib/lucene-queries-9.10.0.jar
jar2: /usr/share/opensearch/lib/lucene-queries-9.11.1.jar
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:185) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.13.0.jar:2.17.1]
        at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.13.0.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) ~[opensearch-2.17.1.jar:2.17.1]
Caused by: java.lang.IllegalStateException: jar hell!
class: org.apache.lucene.queries.CommonTermsQuery
jar1: /usr/share/opensearch/lib/lucene-queries-9.10.0.jar
jar2: /usr/share/opensearch/lib/lucene-queries-9.11.1.jar
        at org.opensearch.bootstrap.JarHell.checkClass(JarHell.java:316) ~[opensearch-common-2.13.0.jar:2.17.1]
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:215) ~[opensearch-common-2.13.0.jar:2.17.1]
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:102) ~[opensearch-common-2.13.0.jar:2.17.1]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.17.1.jar:2.17.1]
        ... 6 more

gc log:


[2024-10-02T09:47:26.855+0000][171247][gc,init] CardTable entry size: 512
[2024-10-02T09:47:26.855+0000][171247][gc     ] Using G1
[2024-10-02T09:47:27.782+0000][171247][gc,init] Version: 21.0.4+7-LTS (release)
[2024-10-02T09:47:27.782+0000][171247][gc,init] CPUs: 2 total, 2 available
[2024-10-02T09:47:27.782+0000][171247][gc,init] Memory: 7937M
[2024-10-02T09:47:27.782+0000][171247][gc,init] Large Page Support: Disabled
[2024-10-02T09:47:27.782+0000][171247][gc,init] NUMA Support: Disabled
[2024-10-02T09:47:27.782+0000][171247][gc,init] Compressed Oops: Enabled (Zero based)
[2024-10-02T09:47:27.782+0000][171247][gc,init] Heap Region Size: 2M
[2024-10-02T09:47:27.782+0000][171247][gc,init] Heap Min Capacity: 4G
[2024-10-02T09:47:27.782+0000][171247][gc,init] Heap Initial Capacity: 4G
[2024-10-02T09:47:27.782+0000][171247][gc,init] Heap Max Capacity: 4G
[2024-10-02T09:47:27.782+0000][171247][gc,init] Pre-touch: Enabled
[2024-10-02T09:47:27.782+0000][171247][gc,init] Parallel Workers: 2
[2024-10-02T09:47:27.782+0000][171247][gc,init] Concurrent Workers: 1
[2024-10-02T09:47:27.782+0000][171247][gc,init] Concurrent Refinement Workers: 2
[2024-10-02T09:47:27.782+0000][171247][gc,init] Periodic GC: Disabled
[2024-10-02T09:47:27.790+0000][171247][gc,metaspace] CDS archive(s) mapped at: [0x00007f83cf000000-0x00007f83cfcae000-0x00007f83cfcae000), size 13295616, SharedBaseAddress: 0x00007f83cf000000, ArchiveRelocationMode: 1.
[2024-10-02T09:47:27.790+0000][171247][gc,metaspace] Compressed class space mapped at: 0x00007f83d0000000-0x00007f8410000000, reserved size: 1073741824
[2024-10-02T09:47:27.791+0000][171247][gc,metaspace] Narrow klass base: 0x00007f83cf000000, Narrow klass shift: 0, Narrow klass range: 0x100000000
[2024-10-02T09:47:28.254+0000][171247][safepoint   ] Safepoint "ICBufferFull", Time since last: 458384473 ns, Reaching safepoint: 3732 ns, Cleanup: 90564 ns, At safepoint: 1474 ns, Total: 95770 ns
[2024-10-02T09:47:28.831+0000][171247][safepoint   ] Safepoint "ICBufferFull", Time since last: 576749324 ns, Reaching safepoint: 3039 ns, Cleanup: 107701 ns, At safepoint: 1756 ns, Total: 112496 ns
[2024-10-02T09:47:29.507+0000][171247][gc,heap,exit] Heap
[2024-10-02T09:47:29.507+0000][171247][gc,heap,exit]  garbage-first heap   total 4194304K, used 60446K [0x0000000700000000, 0x0000000800000000)
[2024-10-02T09:47:29.507+0000][171247][gc,heap,exit]   region size 2048K, 29 young (59392K), 0 survivors (0K)
[2024-10-02T09:47:29.507+0000][171247][gc,heap,exit]  Metaspace       used 15368K, committed 15680K, reserved 1114112K
[2024-10-02T09:47:29.507+0000][171247][gc,heap,exit]   class space    used 1849K, committed 1984K, reserved 1048576K
~

Host/Environment (please complete the following information):

Ansible Version: 2.15.0
Playbook Version: 3c513cf
·

The text was updated successfully, but these errors were encountered:

dzaczek · 2024-10-02T13:09:45Z

This same problem i have when im trying to update from 2.13 to 2.14

2024-10-02T12:54:37,183][ERROR][o.o.b.Bootstrap          ] [a4212-os-master-2] Exception
java.lang.IllegalStateException: jar hell!
class: org.opensearch.plugins.ExtendedPluginsClassLoader
jar1: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.14.0.jar
jar2: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.13.0.jar
        at org.opensearch.bootstrap.JarHell.checkClass(JarHell.java:316) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:215) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:102) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) [opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) [opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) [opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) [opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-2.13.0.jar:2.14.0]
        at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) [opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) [opensearch-2.14.0.jar:2.14.0]
[2024-10-02T12:54:37,190][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [a4212-os-master-2] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: jar hell!
class: org.opensearch.plugins.ExtendedPluginsClassLoader
jar1: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.14.0.jar
jar2: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.13.0.jar
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:185) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.13.0.jar:2.14.0]
        at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) ~[opensearch-2.14.0.jar:2.14.0]
Caused by: java.lang.IllegalStateException: jar hell!
class: org.opensearch.plugins.ExtendedPluginsClassLoader
jar1: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.14.0.jar
jar2: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.13.0.jar
        at org.opensearch.bootstrap.JarHell.checkClass(JarHell.java:316) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:215) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:102) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.14.0.jar:2.14.0]
        ... 6 more

dzaczek · 2024-10-02T13:10:40Z

Sorry i click by mistake close :)

dzaczek · 2024-10-02T13:19:35Z

Im trying to check in logs enabled debug logs

Oct 02 13:15:18 a4212-os-master-2 opensearch[207812]: class: org.opensearch.plugins.ExtendedPluginsClassLoader
class: org.opensearch.plugins.ExtendedPluginsClassLoader
jar1: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.14.0.jar
jar2: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.13.0.jar
        at org.opensearch.bootstrap.JarHell.checkClass(JarHell.java:316) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:215) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:102) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.14.0.jar:2.14.0]
        ... 6 more
[2024-10-02T13:17:31,214][DEBUG][c.s.j.Native             ] [a4212-os-master-2] Looking in classpath from jdk.internal.loader.ClassLoaders$AppClassLoader@4f2410ac for /com/sun/jna/linux-x86-64/libjnidispatch.so
[2024-10-02T13:17:31,217][DEBUG][c.s.j.Native             ] [a4212-os-master-2] Found library resource at jar:file:/usr/share/opensearch/lib/jna-5.13.0.jar!/com/sun/jna/linux-x86-64/libjnidispatch.so
[2024-10-02T13:17:31,219][DEBUG][c.s.j.Native             ] [a4212-os-master-2] Extracting library to /usr/share/opensearch/.cache/JNA/temp/jna15514319990484742182.tmp
[2024-10-02T13:17:31,221][DEBUG][c.s.j.Native             ] [a4212-os-master-2] Trying /usr/share/opensearch/.cache/JNA/temp/jna15514319990484742182.tmp
[2024-10-02T13:17:31,223][DEBUG][c.s.j.Native             ] [a4212-os-master-2] Found jnidispatch at /usr/share/opensearch/.cache/JNA/temp/jna15514319990484742182.tmp
[2024-10-02T13:17:31,264][DEBUG][o.o.b.SystemCallFilter   ] [a4212-os-master-2] Linux seccomp filter installation successful, threads: [all]
[2024-10-02T13:17:31,680][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] java.class.path: /usr/share/opensearch/lib/lucene-queries-9.10.0.jar:/usr/share/opensearch/lib/lucene-spatial-extras-9.10.0.jar:/usr/share/opensearch/lib/lucene-memory-9.10.0.jar:/usr/share/opensearch/lib/opensearch-cli-2.13.0.jar:/usr/share/opensearch/lib/opensearch-plugin-classloader-2.14.0.jar:/usr/share/opensearch/lib/java-version-checker-2.14.0.jar:/usr/share/opensearch/lib/opensearch-compress-2.13.0.jar:/usr/share/opensearch/lib/opensearch-core-2.14.0.jar:/usr/share/opensearch/lib/opensearch-plugin-classloader-2.13.0.jar:/usr/share/opensearch/lib/java-version-checker-2.13.0.jar:/usr/share/opensearch/lib/opensearch-2.14.0.jar:/usr/share/opensearch/lib/jna-5.13.0.jar:/usr/share/opensearch/lib/opensearch-core-2.13.0.jar:/usr/share/opensearch/lib/lucene-grouping-9.10.0.jar:/usr/share/opensearch/lib/opensearch-compress-2.14.0.jar:/usr/share/opensearch/lib/lucene-spatial3d-9.10.0.jar:/usr/share/opensearch/lib/lucene-analysis-common-9.10.0.jar:/usr/share/opensearch/lib/joda-time-2.12.2.jar:/usr/share/opensearch/lib/opensearch-launchers-2.13.0.jar:/usr/share/opensearch/lib/opensearch-secure-sm-2.14.0.jar:/usr/share/opensearch/lib/jakarta.annotation-api-1.3.5.jar:/usr/share/opensearch/lib/protobuf-java-3.22.3.jar:/usr/share/opensearch/lib/jts-core-1.15.0.jar:/usr/share/opensearch/lib/opensearch-cli-2.14.0.jar:/usr/share/opensearch/lib/jackson-dataformat-yaml-2.17.0.jar:/usr/share/opensearch/lib/lucene-join-9.10.0.jar:/usr/share/opensearch/lib/reactive-streams-1.0.4.jar:/usr/share/opensearch/lib/zstd-jni-1.5.5-5.jar:/usr/share/opensearch/lib/opensearch-geo-2.13.0.jar:/usr/share/opensearch/lib/spatial4j-0.7.jar:/usr/share/opensearch/lib/opensearch-common-2.13.0.jar:/usr/share/opensearch/lib/opensearch-launchers-2.14.0.jar:/usr/share/opensearch/lib/opensearch-x-content-2.13.0.jar:/usr/share/opensearch/lib/reactor-core-3.5.15.jar:/usr/share/opensearch/lib/jackson-core-2.17.0.jar:/usr/share/opensearch/lib/lucene-core-9.10.0.jar:/usr/share/opensearch/lib/lucene-queryparser-9.10.0.jar:/usr/share/opensearch/lib/opensearch-telemetry-2.13.0.jar:/usr/share/opensearch/lib/jzlib-1.1.3.jar:/usr/share/opensearch/lib/lucene-highlighter-9.10.0.jar:/usr/share/opensearch/lib/opensearch-secure-sm-2.13.0.jar:/usr/share/opensearch/lib/jopt-simple-5.0.4.jar:/usr/share/opensearch/lib/HdrHistogram-2.1.12.jar:/usr/share/opensearch/lib/opensearch-telemetry-2.14.0.jar:/usr/share/opensearch/lib/log4j-core-2.21.0.jar:/usr/share/opensearch/lib/joda-time-2.12.7.jar:/usr/share/opensearch/lib/lucene-misc-9.10.0.jar:/usr/share/opensearch/lib/log4j-api-2.21.0.jar:/usr/share/opensearch/lib/jackson-dataformat-cbor-2.17.0.jar:/usr/share/opensearch/lib/opensearch-common-2.14.0.jar:/usr/share/opensearch/lib/jackson-dataformat-smile-2.17.0.jar:/usr/share/opensearch/lib/t-digest-3.2.jar:/usr/share/opensearch/lib/opensearch-x-content-2.14.0.jar:/usr/share/opensearch/lib/log4j-jul-2.21.0.jar:/usr/share/opensearch/lib/lucene-sandbox-9.10.0.jar:/usr/share/opensearch/lib/lucene-backward-codecs-9.10.0.jar:/usr/share/opensearch/lib/snakeyaml-2.1.jar:/usr/share/opensearch/lib/opensearch-geo-2.14.0.jar:/usr/share/opensearch/lib/opensearch-2.13.0.jar:/usr/share/opensearch/lib/lucene-suggest-9.10.0.jar
[2024-10-02T13:17:31,681][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] sun.boot.class.path: null
[2024-10-02T13:17:31,684][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] java.home: /usr/share/opensearch/jdk
[2024-10-02T13:17:31,684][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] examining jar: /usr/share/opensearch/lib/lucene-queries-9.10.0.jar
[2024-10-02T13:17:31,686][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] examining jar: /usr/share/opensearch/lib/lucene-spatial-extras-9.10.0.jar
[2024-10-02T13:17:31,687][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] examining jar: /usr/share/opensearch/lib/lucene-memory-9.10.0.jar
[2024-10-02T13:17:31,687][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] examining jar: /usr/share/opensearch/lib/opensearch-cli-2.13.0.jar
[2024-10-02T13:17:31,688][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] examining jar: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.14.0.jar
[2024-10-02T13:17:31,688][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] examining jar: /usr/share/opensearch/lib/java-version-checker-2.14.0.jar
[2024-10-02T13:17:31,689][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] examining jar: /usr/share/opensearch/lib/opensearch-compress-2.13.0.jar
[2024-10-02T13:17:31,689][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] examining jar: /usr/share/opensearch/lib/opensearch-core-2.14.0.jar
[2024-10-02T13:17:31,690][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] examining jar: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.13.0.jar
[2024-10-02T13:17:31,694][ERROR][o.o.b.Bootstrap          ] [a4212-os-master-2] Exception
java.lang.IllegalStateException: jar hell!
class: org.opensearch.plugins.ExtendedPluginsClassLoader
jar1: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.14.0.jar
jar2: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.13.0.jar
        at org.opensearch.bootstrap.JarHell.checkClass(JarHell.java:316) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:215) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:102) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) [opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) [opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) [opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) [opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-2.13.0.jar:2.14.0]
        at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) [opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) [opensearch-2.14.0.jar:2.14.0]
[2024-10-02T13:17:31,699][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [a4212-os-master-2] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: jar hell!
class: org.opensearch.plugins.ExtendedPluginsClassLoader
jar1: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.14.0.jar
jar2: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.13.0.jar
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:185) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.13.0.jar:2.14.0]
        at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) ~[opensearch-2.14.0.jar:2.14.0]
Caused by: java.lang.IllegalStateException: jar hell!
class: org.opensearch.plugins.ExtendedPluginsClassLoader
jar1: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.14.0.jar
jar2: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.13.0.jar
        at org.opensearch.bootstrap.JarHell.checkClass(JarHell.java:316) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:215) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:102) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.14.0.jar:2.14.0]
        ... 6 more
[2024-10-02T13:18:18,905][DEBUG][c.s.j.Native             ] [a4212-os-master-2] Looking in classpath from jdk.internal.loader.ClassLoaders$AppClassLoader@4f2410ac for /com/sun/jna/linux-x86-64/libjnidispatch.so
[2024-10-02T13:18:18,908][DEBUG][c.s.j.Native             ] [a4212-os-master-2] Found library resource at jar:file:/usr/share/opensearch/lib/jna-5.13.0.jar!/com/sun/jna/linux-x86-64/libjnidispatch.so
[2024-10-02T13:18:18,910][DEBUG][c.s.j.Native             ] [a4212-os-master-2] Extracting library to /usr/share/opensearch/.cache/JNA/temp/jna16960683428097951911.tmp
[2024-10-02T13:18:18,912][DEBUG][c.s.j.Native             ] [a4212-os-master-2] Trying /usr/share/opensearch/.cache/JNA/temp/jna16960683428097951911.tmp
[2024-10-02T13:18:18,914][DEBUG][c.s.j.Native             ] [a4212-os-master-2] Found jnidispatch at /usr/share/opensearch/.cache/JNA/temp/jna16960683428097951911.tmp
[2024-10-02T13:18:18,959][DEBUG][o.o.b.SystemCallFilter   ] [a4212-os-master-2] Linux seccomp filter installation successful, threads: [all]
[2024-10-02T13:18:19,373][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] java.class.path: /usr/share/opensearch/lib/lucene-queries-9.10.0.jar:/usr/share/opensearch/lib/lucene-spatial-extras-9.10.0.jar:/usr/share/opensearch/lib/lucene-memory-9.10.0.jar:/usr/share/opensearch/lib/opensearch-cli-2.13.0.jar:/usr/share/opensearch/lib/opensearch-plugin-classloader-2.14.0.jar:/usr/share/opensearch/lib/java-version-checker-2.14.0.jar:/usr/share/opensearch/lib/opensearch-compress-2.13.0.jar:/usr/share/opensearch/lib/opensearch-core-2.14.0.jar:/usr/share/opensearch/lib/opensearch-plugin-classloader-2.13.0.jar:/usr/share/opensearch/lib/java-version-checker-2.13.0.jar:/usr/share/opensearch/lib/opensearch-2.14.0.jar:/usr/share/opensearch/lib/jna-5.13.0.jar:/usr/share/opensearch/lib/opensearch-core-2.13.0.jar:/usr/share/opensearch/lib/lucene-grouping-9.10.0.jar:/usr/share/opensearch/lib/opensearch-compress-2.14.0.jar:/usr/share/opensearch/lib/lucene-spatial3d-9.10.0.jar:/usr/share/opensearch/lib/lucene-analysis-common-9.10.0.jar:/usr/share/opensearch/lib/joda-time-2.12.2.jar:/usr/share/opensearch/lib/opensearch-launchers-2.13.0.jar:/usr/share/opensearch/lib/opensearch-secure-sm-2.14.0.jar:/usr/share/opensearch/lib/jakarta.annotation-api-1.3.5.jar:/usr/share/opensearch/lib/protobuf-java-3.22.3.jar:/usr/share/opensearch/lib/jts-core-1.15.0.jar:/usr/share/opensearch/lib/opensearch-cli-2.14.0.jar:/usr/share/opensearch/lib/jackson-dataformat-yaml-2.17.0.jar:/usr/share/opensearch/lib/lucene-join-9.10.0.jar:/usr/share/opensearch/lib/reactive-streams-1.0.4.jar:/usr/share/opensearch/lib/zstd-jni-1.5.5-5.jar:/usr/share/opensearch/lib/opensearch-geo-2.13.0.jar:/usr/share/opensearch/lib/spatial4j-0.7.jar:/usr/share/opensearch/lib/opensearch-common-2.13.0.jar:/usr/share/opensearch/lib/opensearch-launchers-2.14.0.jar:/usr/share/opensearch/lib/opensearch-x-content-2.13.0.jar:/usr/share/opensearch/lib/reactor-core-3.5.15.jar:/usr/share/opensearch/lib/jackson-core-2.17.0.jar:/usr/share/opensearch/lib/lucene-core-9.10.0.jar:/usr/share/opensearch/lib/lucene-queryparser-9.10.0.jar:/usr/share/opensearch/lib/opensearch-telemetry-2.13.0.jar:/usr/share/opensearch/lib/jzlib-1.1.3.jar:/usr/share/opensearch/lib/lucene-highlighter-9.10.0.jar:/usr/share/opensearch/lib/opensearch-secure-sm-2.13.0.jar:/usr/share/opensearch/lib/jopt-simple-5.0.4.jar:/usr/share/opensearch/lib/HdrHistogram-2.1.12.jar:/usr/share/opensearch/lib/opensearch-telemetry-2.14.0.jar:/usr/share/opensearch/lib/log4j-core-2.21.0.jar:/usr/share/opensearch/lib/joda-time-2.12.7.jar:/usr/share/opensearch/lib/lucene-misc-9.10.0.jar:/usr/share/opensearch/lib/log4j-api-2.21.0.jar:/usr/share/opensearch/lib/jackson-dataformat-cbor-2.17.0.jar:/usr/share/opensearch/lib/opensearch-common-2.14.0.jar:/usr/share/opensearch/lib/jackson-dataformat-smile-2.17.0.jar:/usr/share/opensearch/lib/t-digest-3.2.jar:/usr/share/opensearch/lib/opensearch-x-content-2.14.0.jar:/usr/share/opensearch/lib/log4j-jul-2.21.0.jar:/usr/share/opensearch/lib/lucene-sandbox-9.10.0.jar:/usr/share/opensearch/lib/lucene-backward-codecs-9.10.0.jar:/usr/share/opensearch/lib/snakeyaml-2.1.jar:/usr/share/opensearch/lib/opensearch-geo-2.14.0.jar:/usr/share/opensearch/lib/opensearch-2.13.0.jar:/usr/share/opensearch/lib/lucene-suggest-9.10.0.jar
[2024-10-02T13:18:19,374][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] sun.boot.class.path: null
[2024-10-02T13:18:19,377][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] java.home: /usr/share/opensearch/jdk
[2024-10-02T13:18:19,378][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] examining jar: /usr/share/opensearch/lib/lucene-queries-9.10.0.jar
[2024-10-02T13:18:19,380][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] examining jar: /usr/share/opensearch/lib/lucene-spatial-extras-9.10.0.jar
[2024-10-02T13:18:19,381][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] examining jar: /usr/share/opensearch/lib/lucene-memory-9.10.0.jar
[2024-10-02T13:18:19,381][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] examining jar: /usr/share/opensearch/lib/opensearch-cli-2.13.0.jar
[2024-10-02T13:18:19,382][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] examining jar: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.14.0.jar
[2024-10-02T13:18:19,382][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] examining jar: /usr/share/opensearch/lib/java-version-checker-2.14.0.jar
[2024-10-02T13:18:19,383][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] examining jar: /usr/share/opensearch/lib/opensearch-compress-2.13.0.jar
[2024-10-02T13:18:19,383][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] examining jar: /usr/share/opensearch/lib/opensearch-core-2.14.0.jar
[2024-10-02T13:18:19,384][DEBUG][o.o.b.JarHell            ] [a4212-os-master-2] examining jar: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.13.0.jar
[2024-10-02T13:18:19,390][ERROR][o.o.b.Bootstrap          ] [a4212-os-master-2] Exception
java.lang.IllegalStateException: jar hell!
class: org.opensearch.plugins.ExtendedPluginsClassLoader
jar1: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.14.0.jar
jar2: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.13.0.jar
        at org.opensearch.bootstrap.JarHell.checkClass(JarHell.java:316) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:215) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:102) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) [opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) [opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) [opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) [opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-2.13.0.jar:2.14.0]
        at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) [opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) [opensearch-2.14.0.jar:2.14.0]
[2024-10-02T13:18:19,395][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [a4212-os-master-2] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: jar hell!
class: org.opensearch.plugins.ExtendedPluginsClassLoader
jar1: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.14.0.jar
jar2: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.13.0.jar
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:185) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.13.0.jar:2.14.0]
        at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) ~[opensearch-2.14.0.jar:2.14.0]
Caused by: java.lang.IllegalStateException: jar hell!
class: org.opensearch.plugins.ExtendedPluginsClassLoader
jar1: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.14.0.jar
jar2: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.13.0.jar
        at org.opensearch.bootstrap.JarHell.checkClass(JarHell.java:316) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:215) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:102) ~[opensearch-common-2.13.0.jar:2.14.0]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.14.0.jar:2.14.0]
        ... 6 more
~

gaiksaya · 2024-10-10T16:06:35Z

Adding @saravanan30erd to take a look.

saravanan30erd · 2024-10-13T16:43:23Z

@gaiksaya @dzaczek

Usually jar hell happens due to jar conflicts on same lib and removing the old lib can solve these issues sometimes. But what I didn't understand is why its not the problem until now and after 2.13 we are getting this issues.

@dzaczek Can you please give it a try by removing the old lib version and start the service?

jar2: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.13.0.jar

dzaczek · 2024-10-15T09:44:05Z

@saravanan30erd after remove
jar2: /usr/share/opensearch/lib/opensearch-plugin-classloader-2.13.0.jar

java.lang.IllegalStateException: jar hell!                                                                                                     │
class: org.apache.lucene.queries.CommonTermsQuery                                                                                              │
jar1: /usr/share/opensearch/lib/lucene-queries-9.10.0.jar                                                                                      │
jar2: /usr/share/opensearch/lib/lucene-queries-9.11.1.jar                                                                                      │
        at org.opensearch.bootstrap.JarHell.checkClass(JarHell.java:316) ~[opensearch-common-2.13.0.jar:2.17.1]                                │
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:215) ~[opensearch-common-2.13.0.jar:2.17.1]                              │
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:102) ~[opensearch-common-2.13.0.jar:2.17.1]                              │
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[opensearch-2.17.1.jar:2.17.1]                                        │
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) [opensearch-2.17.1.jar:2.17.1]                                          │
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) [opensearch-2.17.1.jar:2.17.1]                                        │
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) [opensearch-2.17.1.jar:2.17.1]                                     │
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) [opensearch-2.17.1.jar:2.17.1]                 │
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-2.13.0.jar:2.17.1]                            │
        at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-2.13.0.jar:2.17.1]                                                │
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) [opensearch-2.17.1.jar:2.17.1]                                        │
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) [opensearch-2.17.1.jar:2.17.1]                                        │
[2024-10-15T09:39:43,247][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [a4211-os-master-1] uncaught exception in thread [main]             │
org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: jar hell!                                                          │
class: org.apache.lucene.queries.CommonTermsQuery                                                                                              │
jar1: /usr/share/opensearch/lib/lucene-queries-9.10.0.jar                                                                                      │
jar2: /usr/share/opensearch/lib/lucene-queries-9.11.1.jar                                                                                      │
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:185) ~[opensearch-2.17.1.jar:2.17.1]                                       │
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) ~[opensearch-2.17.1.jar:2.17.1]                                    │
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.17.1.jar:2.17.1]                │
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.13.0.jar:2.17.1]                           │
        at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.13.0.jar:2.17.1]                                               │
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) ~[opensearch-2.17.1.jar:2.17.1]                                       │
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) ~[opensearch-2.17.1.jar:2.17.1]                                       │
Caused by: java.lang.IllegalStateException: jar hell!                                                                                          │
class: org.apache.lucene.queries.CommonTermsQuery                                                                                              │
jar1: /usr/share/opensearch/lib/lucene-queries-9.10.0.jar                                                                                      │
jar2: /usr/share/opensearch/lib/lucene-queries-9.11.1.jar                                                                                      │
        at org.opensearch.bootstrap.JarHell.checkClass(JarHell.java:316) ~[opensearch-common-2.13.0.jar:2.17.1]                                │
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:215) ~[opensearch-common-2.13.0.jar:2.17.1]                              │
        at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:102) ~[opensearch-common-2.13.0.jar:2.17.1]                              │
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[opensearch-2.17.1.jar:2.17.1]                                        │
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.17.1.jar:2.17.1]                                         │
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.17.1.jar:2.17.1]                                       │
        ... 6 more                                                                                                                             │
                                                                                                                                               │
                                                                                             ```

saravanan30erd · 2024-10-15T15:16:28Z

@dzaczek looks like conflict continues on other libs. Could you please try to remove all the files in this folder /usr/share/opensearch/lib and do the upgrade? If it works, we can add that step.

dzaczek · 2024-10-16T11:28:09Z

I addded top playbbok remove ../lib dir and after tahat problem with plugins


[2024-10-16T11:20:37,284][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [a4211-os-master-1] Detected cluster change event for destination migration
[2024-10-16T11:20:37,381][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [a4211-os-master-1] Detected cluster change event for destination migration
[2024-10-16T11:20:53,198][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [a4211-os-master-1] Detected cluster change event for destination migration
[2024-10-16T11:20:53,262][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [a4211-os-master-1] Detected cluster change event for destination migration
[2024-10-16T11:21:00,682][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [a4211-os-master-1] Detected cluster change event for destination migration
[2024-10-16T11:21:00,768][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [a4211-os-master-1] Detected cluster change event for destination migration
[2024-10-16T11:21:14,823][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [a4211-os-master-1] Detected cluster change event for destination migration
[2024-10-16T11:21:14,875][INFO ][o.o.p.PluginsService     ] [a4211-os-master-1] PluginService:onIndexModule index:[call_and_sms2024.05.04/t8Zu1AqXRzSBb_qq2i3A9w]
[2024-10-16T11:21:14,897][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [a4211-os-master-1] Detected cluster change event for destination migration
[2024-10-16T11:21:15,442][INFO ][o.o.c.c.Coordinator      ] [a4211-os-master-1] cluster-manager node [{a4213-os-master-3}{igWdVWPPRzy_S4kbug8TKA}{trZYX2KqRJqZXHWGQYN-iw}{XXX.XXX.XXX.213}{XXX.XXX.XXX.213:9300}{dm}{shard_indexing_pressure_enabled=true}] failed, restarting discovery
org.opensearch.transport.NodeDisconnectedException: [a4213-os-master-3][XXX.XXX.XXX.213:9300][disconnected] disconnected
[2024-10-16T11:21:15,460][INFO ][o.o.c.s.ClusterApplierService] [a4211-os-master-1] cluster-manager node changed {previous [{a4213-os-master-3}{igWdVWPPRzy_S4kbug8TKA}{trZYX2KqRJqZXHWGQYN-iw}{XXX.XXX.XXX.213}{XXX.XXX.XXX.213:9300}{dm}{shard_indexing_pressure_enabled=true}], current []}, term: 39, version: 97174, reason: becoming candidate: onLeaderFailure
[2024-10-16T11:21:15,515][WARN ][o.o.c.NodeConnectionsService] [a4211-os-master-1] failed to connect to {a4213-os-master-3}{igWdVWPPRzy_S4kbug8TKA}{trZYX2KqRJqZXHWGQYN-iw}{XXX.XXX.XXX.213}{XXX.XXX.XXX.213:9300}{dm}{shard_indexing_pressure_enabled=true} (tried [1] times)
org.opensearch.transport.ConnectTransportException: [a4213-os-master-3][XXX.XXX.XXX.213:9300] connect_exception
        at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1077) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.core.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:217) ~[opensearch-core-2.13.0.jar:2.13.0]
        at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:57) ~[opensearch-common-2.13.0.jar:2.13.0]
        at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?]
        at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?]
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194) ~[?:?]
        at org.opensearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:72) ~[opensearch-common-2.13.0.jar:2.13.0]
        at org.opensearch.transport.netty4.Netty4TcpChannel.lambda$addListener$0(Netty4TcpChannel.java:83) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[?:?]
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:322) ~[?:?]
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:338) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: XXX.XXX.XXX.213/XXX.XXX.XXX.213:9300
Caused by: java.net.ConnectException: Connection refused
        at java.base/sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
        at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682) ~[?:?]
        at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:973) ~[?:?]
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) ~[?:?]
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:335) ~[?:?]
        ... 7 more
[2024-10-16T11:21:15,428][WARN ][o.o.i.c.IndicesClusterStateService] [a4211-os-master-1] [call_and_sms2024.05.29][3] marking and sending shard failed due to [failed recovery]
org.opensearch.indices.recovery.RecoveryFailedException: [call_and_sms2024.05.29][3]: Recovery failed from {a4213-os-master-3}{igWdVWPPRzy_S4kbug8TKA}{trZYX2KqRJqZXHWGQYN-iw}{XXX.XXX.XXX.213}{XXX.XXX.XXX.213:9300}{dm}{shard_indexing_pressure_enabled=true} into {a4211-os-master-1}{-pONGn5gRba3FaG7NlqnjQ}{v0q3HL2fQaOGwhw8jpQFsg}{XXX.XXX.XXX.211}{XXX.XXX.XXX.211:9300}{dm}{shard_indexing_pressure_enabled=true} ([call_and_sms2024.05.29][3]: Recovery failed from {a4213-os-master-3}{igWdVWPPRzy_S4kbug8TKA}{trZYX2KqRJqZXHWGQYN-iw}{XXX.XXX.XXX.213}{XXX.XXX.XXX.213:9300}{dm}{shard_indexing_pressure_enabled=true} into {a4211-os-master-1}{-pONGn5gRba3FaG7NlqnjQ}{v0q3HL2fQaOGwhw8jpQFsg}{XXX.XXX.XXX.211}{XXX.XXX.XXX.211:9300}{dm}{shard_indexing_pressure_enabled=true})
        at org.opensearch.indices.recovery.RecoveryTarget.notifyListener(RecoveryTarget.java:136) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.replication.common.ReplicationTarget.fail(ReplicationTarget.java:180) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.replication.common.ReplicationCollection.fail(ReplicationCollection.java:212) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.recovery.PeerRecoveryTargetService$RecoveryResponseHandler.onException(PeerRecoveryTargetService.java:756) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.recovery.PeerRecoveryTargetService$RecoveryResponseHandler.handleException(PeerRecoveryTargetService.java:682) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleException(SecurityInterceptor.java:404) [opensearch-security-2.13.0.0.jar:2.13.0.0]
        at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1515) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.transport.InboundHandler.lambda$handleException$5(InboundHandler.java:447) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:854) [opensearch-2.13.0.jar:2.13.0]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: org.opensearch.indices.recovery.RecoveryFailedException: [call_and_sms2024.05.29][3]: Recovery failed from {a4213-os-master-3}{igWdVWPPRzy_S4kbug8TKA}{trZYX2KqRJqZXHWGQYN-iw}{XXX.XXX.XXX.213}{XXX.XXX.XXX.213:9300}{dm}{shard_indexing_pressure_enabled=true} into {a4211-os-master-1}{-pONGn5gRba3FaG7NlqnjQ}{v0q3HL2fQaOGwhw8jpQFsg}{XXX.XXX.XXX.211}{XXX.XXX.XXX.211:9300}{dm}{shard_indexing_pressure_enabled=true}
        ... 9 more
Caused by: org.opensearch.transport.RemoteTransportException: [a4213-os-master-3][XXX.XXX.XXX.213:9300][internal:index/shard/recovery/start_recovery]
Caused by: org.opensearch.transport.SendRequestTransportException: [a4211-os-master-1][XXX.XXX.XXX.211:9300][internal:index/shard/recovery/file_chunk]
        at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1002) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.security.transport.SecurityInterceptor.sendRequestDecorate(SecurityInterceptor.java:267) ~[?:?]
        at org.opensearch.security.OpenSearchSecurityPlugin$6$2.sendRequest(OpenSearchSecurityPlugin.java:847) ~[?:?]
        at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1750) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.transport.TransportService.sendRequest(TransportService.java:885) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.transport.TransportService.sendRequest(TransportService.java:862) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.recovery.RetryableTransportClient$1.tryAction(RetryableTransportClient.java:91) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.RetryableAction$1.doRun(RetryableAction.java:139) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.OpenSearchExecutors$DirectExecutorService.execute(OpenSearchExecutors.java:343) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.RetryableAction.run(RetryableAction.java:117) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.recovery.RetryableTransportClient.executeRetryableAction(RetryableTransportClient.java:106) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.replication.RemoteSegmentFileChunkWriter.writeFileChunk(RemoteSegmentFileChunkWriter.java:117) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.recovery.RemoteRecoveryTargetHandler.writeFileChunk(RemoteRecoveryTargetHandler.java:279) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.replication.SegmentFileTransferHandler$1.executeChunkRequest(SegmentFileTransferHandler.java:148) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.replication.SegmentFileTransferHandler$1.executeChunkRequest(SegmentFileTransferHandler.java:97) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.recovery.MultiChunkTransfer.handleItems(MultiChunkTransfer.java:149) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.recovery.MultiChunkTransfer$1.write(MultiChunkTransfer.java:98) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.AsyncIOProcessor.processList(AsyncIOProcessor.java:131) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.AsyncIOProcessor.drainAndProcessAndRelease(AsyncIOProcessor.java:119) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.AsyncIOProcessor.put(AsyncIOProcessor.java:97) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.recovery.MultiChunkTransfer.addItem(MultiChunkTransfer.java:109) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.recovery.MultiChunkTransfer.lambda$handleItems$3(MultiChunkTransfer.java:151) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) ~[opensearch-core-2.13.0.jar:2.13.0]
        at org.opensearch.core.action.ActionListener$6.onResponse(ActionListener.java:301) ~[opensearch-core-2.13.0.jar:2.13.0]
        at org.opensearch.core.action.ActionListener$4.onResponse(ActionListener.java:182) ~[opensearch-core-2.13.0.jar:2.13.0]
        at org.opensearch.core.action.ActionListener$6.onResponse(ActionListener.java:301) ~[opensearch-core-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.RetryableAction$RetryingListener.onResponse(RetryableAction.java:183) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:70) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleResponse(SecurityInterceptor.java:398) ~[?:?]
        at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1505) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:420) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.transport.InboundHandler.lambda$handleResponse$3(InboundHandler.java:414) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:854) ~[opensearch-2.13.0.jar:2.13.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: org.opensearch.node.NodeClosedException: node closed {a4213-os-master-3}{igWdVWPPRzy_S4kbug8TKA}{trZYX2KqRJqZXHWGQYN-iw}{XXX.XXX.XXX.213}{XXX.XXX.XXX.213:9300}{dm}{shard_indexing_pressure_enabled=true}
        at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:983) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.security.transport.SecurityInterceptor.sendRequestDecorate(SecurityInterceptor.java:267) ~[?:?]
        at org.opensearch.security.OpenSearchSecurityPlugin$6$2.sendRequest(OpenSearchSecurityPlugin.java:847) ~[?:?]
        at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1750) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.transport.TransportService.sendRequest(TransportService.java:885) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.transport.TransportService.sendRequest(TransportService.java:862) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.recovery.RetryableTransportClient$1.tryAction(RetryableTransportClient.java:91) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.RetryableAction$1.doRun(RetryableAction.java:139) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.OpenSearchExecutors$DirectExecutorService.execute(OpenSearchExecutors.java:343) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.RetryableAction.run(RetryableAction.java:117) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.recovery.RetryableTransportClient.executeRetryableAction(RetryableTransportClient.java:106) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.replication.RemoteSegmentFileChunkWriter.writeFileChunk(RemoteSegmentFileChunkWriter.java:117) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.recovery.RemoteRecoveryTargetHandler.writeFileChunk(RemoteRecoveryTargetHandler.java:279) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.replication.SegmentFileTransferHandler$1.executeChunkRequest(SegmentFileTransferHandler.java:148) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.replication.SegmentFileTransferHandler$1.executeChunkRequest(SegmentFileTransferHandler.java:97) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.recovery.MultiChunkTransfer.handleItems(MultiChunkTransfer.java:149) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.recovery.MultiChunkTransfer$1.write(MultiChunkTransfer.java:98) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.AsyncIOProcessor.processList(AsyncIOProcessor.java:131) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.AsyncIOProcessor.drainAndProcessAndRelease(AsyncIOProcessor.java:119) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.AsyncIOProcessor.put(AsyncIOProcessor.java:97) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.recovery.MultiChunkTransfer.addItem(MultiChunkTransfer.java:109) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indices.recovery.MultiChunkTransfer.lambda$handleItems$3(MultiChunkTransfer.java:151) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) ~[opensearch-core-2.13.0.jar:2.13.0]
        at org.opensearch.core.action.ActionListener$6.onResponse(ActionListener.java:301) ~[opensearch-core-2.13.0.jar:2.13.0]
        at org.opensearch.core.action.ActionListener$4.onResponse(ActionListener.java:182) ~[opensearch-core-2.13.0.jar:2.13.0]
        at org.opensearch.core.action.ActionListener$6.onResponse(ActionListener.java:301) ~[opensearch-core-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.RetryableAction$RetryingListener.onResponse(RetryableAction.java:183) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:70) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleResponse(SecurityInterceptor.java:398) ~[?:?]
        at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1505) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:420) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.transport.InboundHandler.lambda$handleResponse$3(InboundHandler.java:414) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:854) ~[opensearch-2.13.0.jar:2.13.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.lang.Thread.run(Thread.java:1583) ~[?:?]
[2024-10-16T11:21:15,551][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [a4211-os-master-1] Detected cluster change event for destination migration
[2024-10-16T11:21:15,643][INFO ][o.o.n.Node               ] [a4211-os-master-1] stopping ...
[2024-10-16T11:21:15,656][WARN ][o.o.c.a.s.ShardStateAction] [a4211-os-master-1] node closed while execution action [internal:cluster/shard/failure] for shard entry [shard id [[call_and_sms2024.05.29][3]], allocation id [OH1e-x34SiqLvDxVjQOfUw], primary term [0], message [failed recovery], failure [RecoveryFailedException[[call_and_sms2024.05.29][3]: Recovery failed from {a4213-os-master-3}{igWdVWPPRzy_S4kbug8TKA}{trZYX2KqRJqZXHWGQYN-iw}{XXX.XXX.XXX.213}{XXX.XXX.XXX.213:9300}{dm}{shard_indexing_pressure_enabled=true} into {a4211-os-master-1}{-pONGn5gRba3FaG7NlqnjQ}{v0q3HL2fQaOGwhw8jpQFsg}{XXX.XXX.XXX.211}{XXX.XXX.XXX.211:9300}{dm}{shard_indexing_pressure_enabled=true} ([call_and_sms2024.05.29][3]: Recovery failed from {a4213-os-master-3}{igWdVWPPRzy_S4kbug8TKA}{trZYX2KqRJqZXHWGQYN-iw}{XXX.XXX.XXX.213}{XXX.XXX.XXX.213:9300}{dm}{shard_indexing_pressure_enabled=true} into {a4211-os-master-1}{-pONGn5gRba3FaG7NlqnjQ}{v0q3HL2fQaOGwhw8jpQFsg}{XXX.XXX.XXX.211}{XXX.XXX.XXX.211:9300}{dm}{shard_indexing_pressure_enabled=true})]; nested: RecoveryFailedException[[call_and_sms2024.05.29][3]: Recovery failed from {a4213-os-master-3}{igWdVWPPRzy_S4kbug8TKA}{trZYX2KqRJqZXHWGQYN-iw}{XXX.XXX.XXX.213}{XXX.XXX.XXX.213:9300}{dm}{shard_indexing_pressure_enabled=true} into {a4211-os-master-1}{-pONGn5gRba3FaG7NlqnjQ}{v0q3HL2fQaOGwhw8jpQFsg}{XXX.XXX.XXX.211}{XXX.XXX.XXX.211:9300}{dm}{shard_indexing_pressure_enabled=true}]; nested: RemoteTransportException[[a4213-os-master-3][XXX.XXX.XXX.213:9300][internal:index/shard/recovery/start_recovery]]; nested: SendRequestTransportException[[a4211-os-master-1][XXX.XXX.XXX.211:9300][internal:index/shard/recovery/file_chunk]]; nested: NodeClosedException[node closed {a4213-os-master-3}{igWdVWPPRzy_S4kbug8TKA}{trZYX2KqRJqZXHWGQYN-iw}{XXX.XXX.XXX.213}{XXX.XXX.XXX.213:9300}{dm}{shard_indexing_pressure_enabled=true}]; ], markAsStale [true]]
[2024-10-16T11:21:15,658][INFO ][o.o.i.r.PeerRecoveryTargetService] [a4211-os-master-1] recovery of [call_and_sms2024.05.04][4] from [{a4216-os-data-6}{RLKK7wQVRlaSWq3ETW0MpA}{i-eMpNxuRySu9fUJbxnT8w}{XXX.XXX.XXX.216}{XXX.XXX.XXX.216:9300}{d}{shard_indexing_pressure_enabled=true}] interrupted by network disconnect, will retry in [5s]; cause: [[a4216-os-data-6][XXX.XXX.XXX.216:9300][internal:index/shard/recovery/start_recovery] disconnected]
[2024-10-16T11:21:15,673][WARN ][o.o.d.HandshakingTransportAddressConnector] [a4211-os-master-1] handshake failed for [connectToRemoteMasterNode[XXX.XXX.XXX.218:9300]]
org.opensearch.transport.SendRequestTransportException: [][XXX.XXX.XXX.218:9300][internal:transport/handshake]
        at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1002) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.security.transport.SecurityInterceptor.sendRequestDecorate(SecurityInterceptor.java:267) [opensearch-security-2.13.0.0.jar:2.13.0.0]
        at org.opensearch.security.OpenSearchSecurityPlugin$6$2.sendRequest(OpenSearchSecurityPlugin.java:847) [opensearch-security-2.13.0.0.jar:2.13.0.0]
        at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1750) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.transport.TransportService.sendRequest(TransportService.java:885) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.transport.TransportService.handshake(TransportService.java:668) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.transport.TransportService.handshake(TransportService.java:640) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.discovery.HandshakingTransportAddressConnector$1$1.onResponse(HandshakingTransportAddressConnector.java:133) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.discovery.HandshakingTransportAddressConnector$1$1.onResponse(HandshakingTransportAddressConnector.java:126) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.core.action.ActionListener$4.onResponse(ActionListener.java:182) [opensearch-core-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.ThreadedActionListener$1.doRun(ThreadedActionListener.java:78) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:913) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.13.0.jar:2.13.0]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: org.opensearch.node.NodeClosedException: node closed {a4211-os-master-1}{-pONGn5gRba3FaG7NlqnjQ}{v0q3HL2fQaOGwhw8jpQFsg}{XXX.XXX.XXX.211}{XXX.XXX.XXX.211:9300}{dm}{shard_indexing_pressure_enabled=true}
        at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:983) ~[opensearch-2.13.0.jar:2.13.0]
        ... 15 more
[2024-10-16T11:21:15,674][WARN ][o.o.d.HandshakingTransportAddressConnector] [a4211-os-master-1] handshake failed for [connectToRemoteMasterNode[XXX.XXX.XXX.216:9300]]
org.opensearch.transport.SendRequestTransportException: [][XXX.XXX.XXX.216:9300][internal:transport/handshake]
        at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1002) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.security.transport.SecurityInterceptor.sendRequestDecorate(SecurityInterceptor.java:267) [opensearch-security-2.13.0.0.jar:2.13.0.0]
        at org.opensearch.security.OpenSearchSecurityPlugin$6$2.sendRequest(OpenSearchSecurityPlugin.java:847) [opensearch-security-2.13.0.0.jar:2.13.0.0]
        at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1750) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.transport.TransportService.sendRequest(TransportService.java:885) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.transport.TransportService.handshake(TransportService.java:668) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.transport.TransportService.handshake(TransportService.java:640) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.discovery.HandshakingTransportAddressConnector$1$1.onResponse(HandshakingTransportAddressConnector.java:133) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.discovery.HandshakingTransportAddressConnector$1$1.onResponse(HandshakingTransportAddressConnector.java:126) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.core.action.ActionListener$4.onResponse(ActionListener.java:182) [opensearch-core-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.ThreadedActionListener$1.doRun(ThreadedActionListener.java:78) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:913) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.13.0.jar:2.13.0]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: org.opensearch.node.NodeClosedException: node closed {a4211-os-master-1}{-pONGn5gRba3FaG7NlqnjQ}{v0q3HL2fQaOGwhw8jpQFsg}{XXX.XXX.XXX.211}{XXX.XXX.XXX.211:9300}{dm}{shard_indexing_pressure_enabled=true}
        at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:983) ~[opensearch-2.13.0.jar:2.13.0]
        ... 15 more
[2024-10-16T11:21:15,677][INFO ][o.o.s.a.r.AuditMessageRouter] [a4211-os-master-1] Closing AuditMessageRouter
[2024-10-16T11:21:15,678][INFO ][o.o.s.a.s.SinkProvider   ] [a4211-os-master-1] Closing InternalOpenSearchSink
[2024-10-16T11:21:15,678][INFO ][o.o.s.a.s.SinkProvider   ] [a4211-os-master-1] Closing DebugSink
[2024-10-16T11:21:18,085][INFO ][o.o.n.Node               ] [a4211-os-master-1] stopped
[2024-10-16T11:21:18,085][INFO ][o.o.n.Node               ] [a4211-os-master-1] closing ...
[2024-10-16T11:21:18,094][INFO ][o.o.s.a.i.AuditLogImpl   ] [a4211-os-master-1] Closing AuditLogImpl
[2024-10-16T11:21:18,107][INFO ][o.o.n.Node               ] [a4211-os-master-1] closed
[2024-10-16T11:21:25,397][ERROR][o.o.b.Bootstrap          ] [a4211-os-master-1] Exception
java.lang.IllegalStateException: codebase property already set: codebase.opensearch-geospatial -> file:/usr/share/opensearch/plugins/opensearch-geospatial/opensearch-geospatial-2.13.0.0.jar, cannot set to file:/usr/share/opensearch/plugins/opensearch-geospatial/opensearch-geospatial-2.17.1.0.jar
        at org.opensearch.bootstrap.Security.addCodebaseToSystemProperties(Security.java:297) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Security.readPolicy(Security.java:258) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Security.getPluginPermissions(Security.java:211) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Security.configure(Security.java:145) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:237) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) [opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) [opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) [opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) [opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-2.17.1.jar:2.17.1]
        at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) [opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) [opensearch-2.17.1.jar:2.17.1]
[2024-10-16T11:21:25,407][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [a4211-os-master-1] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: codebase property already set: codebase.opensearch-geospatial -> file:/usr/share/opensearch/plugins/opensearch-geospatial/opensearch-geospatial-2.13.0.0.jar, cannot set to file:/usr/share/opensearch/plugins/opensearch-geospatial/opensearch-geospatial-2.17.1.0.jar
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:185) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.17.1.jar:2.17.1]
        at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) ~[opensearch-2.17.1.jar:2.17.1]
Caused by: java.lang.IllegalStateException: codebase property already set: codebase.opensearch-geospatial -> file:/usr/share/opensearch/plugins/opensearch-geospatial/opensearch-geospatial-2.13.0.0.jar, cannot set to file:/usr/share/opensearch/plugins/opensearch-geospatial/opensearch-geospatial-2.17.1.0.jar
        at org.opensearch.bootstrap.Security.addCodebaseToSystemProperties(Security.java:297) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Security.readPolicy(Security.java:258) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Security.getPluginPermissions(Security.java:211) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Security.configure(Security.java:145) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:237) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.17.1.jar:2.17.1]

dzaczek · 2024-10-16T11:40:08Z

then i tied remove plugins and lib

 org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) ~[opensearch-2.17.1.jar:2.17.1]
Caused by: java.lang.IllegalStateException: codebase property already set: codebase.jackson-databind -> file:/usr/share/opensearch/modules/ingest-geoip/jackson-databind-2.17.0.jar, cannot set to file:/usr/share/opensearch/modules/ingest-geoip/jackson-databind-2.17.2.jar
        at org.opensearch.bootstrap.Security.addCodebaseToSystemProperties(Security.java:297) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Security.readPolicy(Security.java:258) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Security.getPluginPermissions(Security.java:211) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Security.configure(Security.java:145) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:237) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.17.1.jar:2.17.1]
        ... 6 more
[2024-10-16T11:38:28,798][ERROR][o.o.b.Bootstrap          ] [a4211-os-master-1] Exception
java.lang.IllegalStateException: codebase property already set: codebase.jackson-databind -> file:/usr/share/opensearch/modules/ingest-geoip/jackson-databind-2.17.0.jar, cannot set to file:/usr/share/opensearch/modules/ingest-geoip/jackson-databind-2.17.2.jar
        at org.opensearch.bootstrap.Security.addCodebaseToSystemProperties(Security.java:297) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Security.readPolicy(Security.java:258) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Security.getPluginPermissions(Security.java:211) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Security.configure(Security.java:145) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:237) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) [opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) [opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) [opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) [opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-2.17.1.jar:2.17.1]
        at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) [opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) [opensearch-2.17.1.jar:2.17.1]
[2024-10-16T11:38:28,806][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [a4211-os-master-1] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: codebase property already set: codebase.jackson-databind -> file:/usr/share/opensearch/modules/ingest-geoip/jackson-databind-2.17.0.jar, cannot set to file:/usr/share/opensearch/modules/ingest-geoip/jackson-databind-2.17.2.jar
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:185) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.17.1.jar:2.17.1]
        at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) ~[opensearch-2.17.1.jar:2.17.1]
Caused by: java.lang.IllegalStateException: codebase property already set: codebase.jackson-databind -> file:/usr/share/opensearch/modules/ingest-geoip/jackson-databind-2.17.0.jar, cannot set to file:/usr/share/opensearch/modules/ingest-geoip/jackson-databind-2.17.2.jar
        at org.opensearch.bootstrap.Security.addCodebaseToSystemProperties(Security.java:297) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Security.readPolicy(Security.java:258) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Security.getPluginPermissions(Security.java:211) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Security.configure(Security.java:145) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:237) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.17.1.jar:2.17.1]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.17.1.jar:2.17.1]
        ... 6 more
root@a4211-os-master-1:~#

Ok so propably i will do update manually :)

dzaczek · 2024-10-16T13:56:42Z

I prepared a workaround script to handle each node manually and preserve the config and data directories.

cd /usr/share 
/usr/share/opensearch/bin/opensearch-plugin remove prometheus-exporter
wget https://artifacts.opensearch.org/releases/bundle/opensearch/2.17.1/opensearch-2.17.1-linux-x64.tar.gz
tar -xvzf opensearch-2.17.1-linux-x64.tar.gz
cp /usr/share/opensearch/config/*  /usr/share/opensearch-2.17.1/config/
systemctl stop opensearch 
sleep 5
find /usr/share/opensearch/ -type f -not -path "/usr/share/opensearch/data/*" -not -path "/usr/share/opensearch/config/*" -delete
chown -R opensearch:opensearch /usr/share/opensearch-2.17.1/
sed -i 's|^#path\.data:.*|path.data: /usr/share/opensearch/data|' /usr/share/opensearch-2.17.1/opensearch.yml
rsync -av --exclude='data' /usr/share/opensearch-2.17.1/ /usr/share/opensearch/
chown -R opensearch:opensearch /usr/share/opensearch
/usr/share/opensearch/bin/opensearch-plugin install  https://github.com/aiven/prometheus-exporter-plugin-for-opensearch/releases/download/2.17.1.0/prometheus-exporter-2.17.1.0.zip

And also converted this same script into an Ansible playbook.


---
- name: Update OpenSearch on all hosts serially
  hosts: os-cluster
  serial: 1  # Ensure only one host is processed at a time
  become: true
  tasks:
    - name: Install rsync
      apt:
        name: rsync
        state: present
        update_cache: yes


    - name: Remove Prometheus exporter OpenSearch plugin
      ansible.builtin.command: /usr/share/opensearch/bin/opensearch-plugin remove prometheus-exporter
      ignore_errors: yes

    - name: Download OpenSearch 2.17.1
      ansible.builtin.get_url:
        url: https://artifacts.opensearch.org/releases/bundle/opensearch/2.17.1/opensearch-2.17.1-linux-x64.tar.gz
        dest: /usr/share/opensearch-2.17.1-linux-x64.tar.gz

    - name: Extract OpenSearch tarball
      ansible.builtin.unarchive:
        src: /usr/share/opensearch-2.17.1-linux-x64.tar.gz
        dest: /usr/share/
        remote_src: yes

    - name: Copy existing OpenSearch config to the new version
      ansible.builtin.copy:
        src: /usr/share/opensearch/config/
        dest: /usr/share/opensearch-2.17.1/config/
        owner: opensearch
        group: opensearch
        mode: '0644'
        remote_src: yes

    - name: Stop OpenSearch service
      ansible.builtin.systemd:
        name: opensearch
        state: stopped

    - name: Wait for 5 seconds after stopping OpenSearch
      ansible.builtin.pause:
        seconds: 5

    - name: Delete files from the current OpenSearch folder, except for data and config directories
      ansible.builtin.shell: |
        find /usr/share/opensearch/ -type f -not -path "/usr/share/opensearch/data/*" -not -path "/usr/share/opensearch/config/*" -delete

    - name: Change ownership of the new OpenSearch installation
      ansible.builtin.command: chown -R opensearch:opensearch /usr/share/opensearch-2.17.1/

    - name: Modify opensearch.yml to adjust data path
      ansible.builtin.lineinfile:
        path: /usr/share/opensearch-2.17.1/config/opensearch.yml
        regexp: '^#path\.data:.*'
        line: 'path.data: /usr/share/opensearch/data'

    - name: Synchronize new OpenSearch files (excluding data) with the old directory
      ansible.builtin.command: rsync -av --exclude='data' /usr/share/opensearch-2.17.1/ /usr/share/opensearch/

    - name: Ensure proper ownership of the OpenSearch folder
      ansible.builtin.command: chown -R opensearch:opensearch /usr/share/opensearch

    - name: Set ownership and permissions for OpenSearch config directory
      ansible.builtin.file:
        path: /usr/share/opensearch/config
        owner: opensearch
        group: opensearch
        mode: '0700'


    - name: Install Prometheus exporter plugin
      ansible.builtin.command: /usr/share/opensearch/bin/opensearch-plugin install https://github.com/aiven/prometheus-exporter-plugin-for-opensearch/releases/download/2.17.1.0/prometheus-exporter-2.17.1.0.zip

    - name: Start OpenSearch service
      ansible.builtin.systemd:
        name: opensearch
        state: started

    - name: Wait for 30 seconds for OpenSearch to start
      ansible.builtin.pause:
        seconds: 30


    - name: Check if OpenSearch service is running
      ansible.builtin.command: systemctl is-active opensearch
      register: open_search_status
      ignore_errors: yes

    - name: Debug OpenSearch status
      debug:
        var: open_search_status

    - name: Fail and stop playbook if OpenSearch did not start
      ansible.builtin.meta: end_play
      when: open_search_status.stdout != 'active'

    - name: Wait for 5 minutes if OpenSearch is running
      ansible.builtin.pause:
        minutes: 5
      when: open_search_status.stdout == 'active'

dzaczek added bug Something isn't working untriaged labels Oct 2, 2024

dzaczek closed this as completed Oct 2, 2024

dzaczek reopened this Oct 2, 2024

gaiksaya removed the untriaged label Oct 10, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update cluster by ansible 2.13 to 2.17.1 #170

Update cluster by ansible 2.13 to 2.17.1 #170

dzaczek commented Oct 2, 2024

dzaczek commented Oct 2, 2024 •

edited

Loading

dzaczek commented Oct 2, 2024

dzaczek commented Oct 2, 2024 •

edited

Loading

gaiksaya commented Oct 10, 2024

saravanan30erd commented Oct 13, 2024

dzaczek commented Oct 15, 2024 •

edited

Loading

saravanan30erd commented Oct 15, 2024

dzaczek commented Oct 16, 2024

dzaczek commented Oct 16, 2024

dzaczek commented Oct 16, 2024 •

edited

Loading

Update cluster by ansible 2.13 to 2.17.1 #170

Update cluster by ansible 2.13 to 2.17.1 #170

Comments

dzaczek commented Oct 2, 2024

dzaczek commented Oct 2, 2024 • edited Loading

dzaczek commented Oct 2, 2024

dzaczek commented Oct 2, 2024 • edited Loading

gaiksaya commented Oct 10, 2024

saravanan30erd commented Oct 13, 2024

dzaczek commented Oct 15, 2024 • edited Loading

saravanan30erd commented Oct 15, 2024

dzaczek commented Oct 16, 2024

dzaczek commented Oct 16, 2024

dzaczek commented Oct 16, 2024 • edited Loading

dzaczek commented Oct 2, 2024 •

edited

Loading

dzaczek commented Oct 2, 2024 •

edited

Loading

dzaczek commented Oct 15, 2024 •

edited

Loading

dzaczek commented Oct 16, 2024 •

edited

Loading