Hey @mch2 from my understanding I have summarized the following on how we can remove the Peer recovery for the search replicas, please check

Filtering Allocation IDs in Cluster Updates:

In OpenSearch, the cluster manager tracks which shards are in-sync with the primary shard (we can check using /_cluster/state). This helps in ensuring data consistency across the cluster. The proposal suggests filtering out the allocation IDs of search replicas from these updates. Since search replicas are read-only and do not participate in writes, the cluster manager doesn’t need to track their state as in-sync copies like it does for regular replicas.

We can create a seperate child issue to track this?

Search Replica Recovery

Check Local Storage and Recover from Local Disk

The replica checks its own disk to see if it already has the necessary segment files from previous operations or previous shard instantiations. If the required data is on the search replica’s local disk, it uses that to recover the shard, skipping any communication with the primary node.

Recover from Remote Storage (if configured):

If the data is not available on the local disk, the replica can pull the data from remote-backed storage to recover the shard. We can even consider skipping the local disk check if Remote Storage is configured.

Replication Process (optional): Can be triggered after the recovery or can wait for next refresh interval

After initializing the replica via store recovery (local or remote), the process can start to ensure the shard is synchronized with the latest segments.

Thank you
@getsaurabh02

So in example API curl -X GET "http://localhost:9200/_cluster/state/routing_table?filter_path=routing_table.indices.my_sample_index" -H 'Content-Type: application/json' | jq '.' I can see the "state": "STARTED" for "searchOnly": true, is the proposal to remove this so that cluster manager need not track the searchOnly replicas?

{
  "routing_table": {
    "indices": {
      "my_sample_index": {
        "shards": {
          "0": [
            {
              "state": "STARTED",
              "primary": false,
              "searchOnly": false,
              "node": "BfhYEGSERsOMVxXa1BTFFA",
              "relocating_node": null,
              "shard": 0,
              "index": "my_sample_index",
              "allocation_id": {
                "id": "6TMmzm-6S0ip36g5VsWSog"
              }
            },
            {
              "state": "STARTED",
              "primary": false,
              "searchOnly": true,
              "node": "yOfxLMoGQPeVJ1lcHNPMrw",
              "relocating_node": null,
              "shard": 0,
              "index": "my_sample_index",
              "allocation_id": {
                "id": "icMnWUYjTReUN0J2H6UUlA"
              }
            },
            {
              "state": "STARTED",
              "primary": true,
              "searchOnly": false,
              "node": "KCszv1QKSZ-HHqzuAcoYbg",
              "relocating_node": null,
              "shard": 0,
              "index": "my_sample_index",
              "allocation_id": {
                "id": "Y6CRS10hT26SbpkdaiwMsQ"
              }
            }
          ]
        }
      }
    }
  }
}

is the proposal to remove this so that cluster manager need not track the searchOnly replicas?

No, the shards should still be included of the routing table, just not tracked at the primary.

@prudhvigodithi pls see the linked draft for what i'm thinking here. To simplify i've reduced the scope of separation to only remote store enabled domains at least for now.