Idle SSH connections can trigger unwanted activity on the target

[douglas-raillard-arm]

Idle SSH connections maintained for concurrent command execution can end up creating unwanted activity on the target.
This typically is very small amount, but x50 and with "help" from the kernel scheduler, this ends up being boosted and create frequency increase spikes.

The issue was observed with OpenSSH on the target, less so (or not at all) using dropbear.

As a result, we need to introduce an API to close unused connections, in the form of a context manager that people can use if they are running some experiments that needs as little disturbance as possible.

Here is a graph of utilization spikes resulting from this activity.

TODO:

• add a context manager to silence SSH as much as possible (generic, should be exposed as target level and then possibly a no-op for some connection types).
• figure out if we can make SSH a bit more silent (I suspect some sort of watchdog mechanism is triggering activity at regular interval even if the connection is unsed)

[marcbonnici]

This is an interesting finding. Out of curiosity do you know if there are any default keepalive settings enabled on the server side (I'm assuming this is server side as this is not observed with dropbear rather than a paramiko default config)?

[douglas-raillard-arm]

I could not find anything enabled by default, but the default is probably irrelevant since distros typically ship a non-default config. The kinds of keepalive I found are:

• keepalive in the ssh protocol, implemented by the servers and clients.
• TCP layer keepalive, dealt with by the libc/kernel.

We don't know exactly what is happening here