Audit the use of substrate connections in Zos

[sameh-farouk]

Describe the bug

It seems that the TFChain RPC node is experiencing a higher number of connections than expected. Currently, there are 35 nodes online (as shown on the dashboard) and 179 connections to the TFChain RPC node (source: Bert). While I understand that other services are also connected, I would assume that the majority of usage comes from ZOS nodes. Having five times the number of connections compared to nodes is not expected.

We need to audit the use of substrate connections in Zos and determine:

• How many are used per node,
• Whether they are used efficiently, and
• If they are closed properly.

[coesensbert]

some additional metrics for this:

• devnet backend stack caddy metrics: https://mon.grid.tf/d/TZBbz9TMk/caddy-load-balancer-tfchain-mainnet?orgId=1&from=now-24h&to=now&var-host=metrics.dev.grid.tf:443&var-host=metrics.02.dev.grid.tf:443&refresh=10s
• devnet backend stack tfchain rpc substrate metrics: https://mon.grid.tf/d/vKVuiD9Zk/substrate-networking?orgId=1&var-nodename=metrics.dev.grid.tf:80&var-nodename=metrics.02.dev.grid.tf:80&var-notif_protocol=All&var-request_protocol_in=All&var-data_source=TF-Mon%20Prometheus&var-metric_namespace=substrate&var-request_protocol_out=All
• devnet backend stack tfchain rpc substrate service tasks: https://mon.grid.tf/d/3LA6XNqZz/substrate-service-tasks?orgId=1&var-nodename=metrics.02.dev.grid.tf:80&var-nodename=metrics.dev.grid.tf:80&var-metric_namespace=substrate&var-data_source=TF-Mon%20Prometheus

[Eslam-Nawara]

WIP:
I've been investigating the number of connections used in zos nodes and the sdk

In zos nodes:
there are 3 places a new connection is created at:

• substrate gateway
• zos api
• identityd
  the first one is kept open for the life of the node, and the latter two are closed after they're used

In zos tools:
each tool (certify, farmup, update worker) all connection are closed after the services are done using them.

In SDK-GO:

• activation service: creates and mentains an open substrate connection
• farmer bot: it opens 6 connections, releases all of them after using them.
• mon-bot: starts 5 connections (removed one of them).
• grid client: opens and maintains an open substrate connection, each tf plugin created has one open connection for the life of it.
• rmb go: opened 2 connections at the same time (removed one of them), the one left is remained open for the life of the peer.

[sameh-farouk]

We need to ensure that multiple clients do not use the same Substrate account simultaneously when sending signed transactions. If two transactions are sent from the same account with the same nonce, only one will be included in the block. This happens because clients cannot account for pending transactions in the pool when querying the account's nonce. As a result, if you want to send multiple valid transactions simultaneously (or within the same block), they must be handled by the same client to avoid nonce conflicts (unless the nonce is incremented manually), otherwise, some transactions will fail or be ignored.

[Eslam-Nawara]

I see that the client always uses one connection every time we run the client. so as long as the user doesn't create two separate plugins and use them simultaneously, then this issue wouldn't happen.

this needs the users themselves to be careful on using the client(they should create one plugin at a time or not use the same mnemonics for farmer bot and a deployment in the same time for instance).

[Eslam-Nawara]

Work Completed

• removed an unnecessary connection from rmb
• reused a connection in mon bot so we can minimize the number of connections created

[Eslam-Nawara]

WIP:

• After more investigation with sameh and omar it sounds like nodes starts a number of connections that can be 5 or more.
• we also found that the dashboard opens a variable number of connections every time it's open, sometimes they're not closed if the tap is closed
• sounds like the nodes problem is in rmb sdk go, creating a new router causes the node to open extra 4 connection, so will be investigating the rmb sdk