add taskCompletionCount in search_backpressure stats

[kaushalmahi12]

Description

With this change the nodeStats API will also return the completionCount of search tasks at shard and coordinator level. Opening this separate PR due to DCO issue in a past commit.
Closed PR : #8701.

new sample response for an API call like curl "localhost:9200/_nodes/stats/search_backpressure?pretty" will look like the following

{
  "_nodes" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
  },
  "cluster_name" : "runTask",
  "nodes" : {
    "dPpgbce2STSxlvH7LDaGiA" : {
      "timestamp" : 1693433903192,
      "name" : "runTask-0",
      "transport_address" : "127.0.0.1:9300",
      "host" : "127.0.0.1",
      "ip" : "127.0.0.1:9300",
      "roles" : [
        "cluster_manager",
        "data",
        "ingest",
        "remote_cluster_client"
      ],
      "attributes" : {
        "testattr" : "test",
        "shard_indexing_pressure_enabled" : "true"
      },
      "search_backpressure" : {
        "search_task" : {
          "resource_tracker_stats" : {
            "cpu_usage_tracker" : {
              "cancellation_count" : 0,
              "current_max_millis" : 0,
              "current_avg_millis" : 0
            },
            "elapsed_time_tracker" : {
              "cancellation_count" : 0,
              "current_max_millis" : 0,
              "current_avg_millis" : 0
            },
            "heap_usage_tracker" : {
              "cancellation_count" : 0,
              "current_max_bytes" : 0,
              "current_avg_bytes" : 0,
              "rolling_avg_bytes" : 0
            }
          },
          "completion_count" : 0,
          "cancellation_stats" : {
            "cancellation_count" : 0,
            "cancellation_limit_reached_count" : 0
          }
        },
        "search_shard_task" : {
          "resource_tracker_stats" : {
            "cpu_usage_tracker" : {
              "cancellation_count" : 0,
              "current_max_millis" : 0,
              "current_avg_millis" : 0
            },
            "elapsed_time_tracker" : {
              "cancellation_count" : 0,
              "current_max_millis" : 0,
              "current_avg_millis" : 0
            },
            "heap_usage_tracker" : {
              "cancellation_count" : 0,
              "current_max_bytes" : 0,
              "current_avg_bytes" : 0,
              "rolling_avg_bytes" : 0
            }
          },
          "completion_count" : 0,
          "cancellation_stats" : {
            "cancellation_count" : 0,
            "cancellation_limit_reached_count" : 0
          }
        },
        "mode" : "monitor_only"
      },
      "is_node_under_duress" : false
    }
  }
}

Related Issues

Resolves #8698

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Compatibility status:

Checks if related components are compatible with change 9f2565a

Incompatible components

Incompatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/ml-commons.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git]

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/25506/
• CommitID: 880d83e
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/25508/
• CommitID: 89498de
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/27336/
• CommitID: 95d62b0
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/27348/
• CommitID: 95d62b0
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/27359/
• CommitID: ad38885
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/27360/
• CommitID: 906bd8f
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/27512/
• CommitID: 906bd8f
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/27513/
• CommitID: 906bd8f
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=pit/10_basic/Delete all}
      1 org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testRefreshSuccessOnSecondAttempt

• URL: https://build.ci.opensearch.org/job/gradle-check/27524/
• CommitID: 9f2565a
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[dblock]

I am seeing this value in a 3.0 build, but not in a 2.x, was this not intended to be backported to 2.x?

opensearch-project/opensearch-api-specification#459

@kaushalmahi12