Error: Authentication failed for user

[iamFIREcracker]

It seems like the underlying SSH client is not properly sourcing ~/.ssh/config.

I installed the key on the host:

$ ssh-copy-id -i ~/.ssh/id_rsa-work -f [email protected]
/home/ubuntu/.nix-profile/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/ubuntu/.ssh/id_rsa-work.pub"

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh '[email protected]'"
and check to make sure that only the key(s) you wanted were added.

Added a new entry to ~/.ssh/config:

...
Host 100.122.104.126
    User root
    IdentityFile ~/.ssh/id_rsa-work
...

Confirmed access in the shell:

$ ssh '[email protected]' -- date
Thu Apr 13 07:55:48 UTC 2023

Hoever, when I run mrsk, it outputs the following error message:

$ mrsk audit
  INFO [aeb45e8c] Running /usr/bin/env tail -n 50 mrsk-cnxt-audit.log on 100.122.104.126
  ERROR (Net::SSH::AuthenticationFailed): Authentication failed for user [email protected]

Could it be trying to use a differnt key? Let's find out.

I tried running ssh-copy-id again, without specifying the identity file to copy, and it tried to install a different entry:

$ ssh-copy-id -f [email protected]
/home/ubuntu/.nix-profile/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/ubuntu/.ssh/id_somethingelse.pub"

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh '[email protected]'"
and check to make sure that only the key(s) you wanted were added.

Unfortunately even with this key installed, mrsk was still unable to connect to the host.

Last thing I tried was removing that host entry from the ~/.ssh/config, and that's when things finally started to work.

What's going on? Anybody bumped into this? It seems like the underlying SSH client is indeed capable of sourcing ~/.ssh/config; however, it's failing to use the configured identity key.

[freddyheppell]

I'm also experiencing this trying to use GitHub Actions. I'm able to run ssh -v root@<IP> 'touch test.txt' in my action, but then I get ERROR (Net::SSH::AuthenticationFailed): Authentication failed for user root@<IP> from mrsk.

[luvs]

Same problem

[kveeti]

Same problem here

[russ]

I'm experiencing the same problem. I've installed the correct key into my ssh-agent. I'm able to login normally myself but mrsk fails to do so.

Acquiring the deploy lock...
  Finished all in 1.6 seconds
  ERROR (Net::SSH::Disconnect): Exception while executing on host X.X.X.X: disconnected: Too many authentication failures (2)

I took a look at the server I was logging into and noticed it was failing multiple times on that attempt. For my case, it was trying several other keys first before getting to the valid key on my system.

This got me back up and running.

eval `ssh-agent -s`
ssh-add path/to/keyfile

[andrew-oko-odion]

Checkout this post, it was what solved it for me!

https://stackoverflow.com/questions/41369406/netsshauthenticationfailed-authentication-failed-for-user

[milushov]

Hey guys, could you take a look at this similar problem? Maybe you will have some ideas: #383

[djmb]

@iamFIREcracker - net-ssh reads the IdentityFile setting and #384 shows that working in the integration tests.

I've raised #386 which will allow the log level to be set in net-ssh. Could you try that branch, set the log level to debug and see if there's anything useful in the output?

@freddyheppell, @luvs, @veeti-k - were you having the exact same issue with the IdentityFile setting as @iamFIREcracker? In any case could you also that branch to get some debug information?

[k0va1]

I had the same problem while having ssh key with name different than id_rsa. After I renamed my key to id_rsa the error is gone

[vbalazs]

My issue was that I have a lot of keys added to the agent and they exhausted the MaxAuthTries configured in sshd on the server before it could try the correct key.
For me the workaround was to add IdentitiesOnly yes along IdentityFile in the local SSH config.

The drawback is that this way it doesn't use the SSH agent and always asks for the key's password.

[freddyheppell]

For what it's worth, it's now working for my use case (using tailscale's ssh auth). Unfortunately I jumped a few versions of mrsk into Kamal, plus some Tailscale updates, so I'm not sure exactly what changed to make it now work.

[gioggi]

if you use kamal with alias, u execute a container and you don't have your .ssh. Add -v <home_path>/.ssh:/root/.ssh at alias like: alias kamal='docker run -it --rm -v "${PWD}:/workdir" -v "/run/host-services/ssh-auth.sock:/run/host-services/ssh-au th.sock" -e SSH_AUTH_SOCK="/run/host-services/ssh-auth.sock" -v <home_path>/.ssh:/root/.ssh -v /var/run/docker.sock:/var/run/docker.sock ghcr.io/basecamp/kamal:latest'

[djmb]

I'm going to close this one for now - there's no concrete report here of a Kamal bug and any debugging needs to be done by the reporters.

For anyone with an authentication problem who finds this issue, you should add this to your Kamal config and see if that helps figure it out:

ssh:
  log_level: debug

If you figure out the issue, do post the fix here (unless someone has already reported the same fix). We can use that to document possible causes.

If it points to a Kamal bug, then please raise a new issue.

[stewones]

here's a little improved version of @gioggi 's solution, which interpolates the current home dir. it really works.

alias kamal='docker run -it --rm -v "${PWD}:/workdir" -v "/run/host-services/ssh-auth.sock:/run/host-services/ssh-auth.sock" -e SSH_AUTH_SOCK="/run/host-services/ssh-auth.sock" -v "$HOME/.ssh:/root/.ssh" -v /var/run/docker.sock:/var/run/docker.sock ghcr.io/basecamp/kamal:latest'

someone could put a PR updating the docs 👀

[savtrip]

@stewones @gioggi you guys are absolute legends, thanks for your help. I was banging my head against the computer for an hour until you reminded I was running Kamal in a docker container.

I raised a pull request in the documentation with @stewones solution: basecamp/kamal-site#133