Once again this week, I had issues connecting to a server using SSH. It was with a different user, and something was just not working. When debugging such issues, I always have to search for that debugging mode, so why not write down the solution on my blog, so I’ll find it faster next time.
Starting an additional OpenSSH daemon in debug mode
You can activate debugging mode on the client side, but often times the issue is on the server. To debug this, you have to run the server part in debug mode. While you can activate this for the main SSH daemon, it is often easier and safer, to start an additional SSH daemon, using a different port, and with debugging activate. To do this, run the following command as root:
/usr/sbin/sshd -ddd -D -p 22222
You need to use the full path to sshd to run this command. With the -D parameter, sshd will not detach and doesn’t become a daemon, so you can see the log output in the terminal. With the -ddd parameter, you turn on the maximum debugging level. And with -p 22222 we will use a different port.
Using client side debugging mode
Now that we have the server part running in debugging mode and waiting for incoming connections, let’s connect to it from a client. We will do so with the following command:
ssh -vvv -p 22222 -i ~/.ssh/id_ed25519 [email protected]
In this example, we are using the previously set port 22222. We also use a custom secret key file we want to test with -i ~/.ssh/id_ed25519 and with -vvv we also use the maximum verbosity in debugging for the client.
Two typical errors for SSH server and client
Now we should be able to find the errors for the server and/or client process. Here are two typical ones, I often come across.
Wrong permissions on the server
Issues with SSH are very often due to wrong file and folder permissions. This is a typical one:
debug1: trying public key file /home/user/.ssh/authorized_keys
debug1: Could not open authorized keys '/home/user/.ssh/authorized_keys': Permission denied
When you see this issue, you might have added the public key correctly to the authorized_keys file, but it is not readable by the SSH daemon. To fix this, make sure to change the folder/file permission to something like this:
chmod 700 /home/user/.ssh/
chmod 644 /home/user/.ssh/authorized_keys
If you got an error in the last attempt to connect, the server process will most probably have died, so make sure to restart it, for the next debug-attempt.
Wrong permissions on the client
On the client, the file permissions are also very often the case of the issues. When the permissions are “too open”, you usually see a message like this on the client:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: UNPROTECTED PRIVATE KEY FILE! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0777 for 'id_ed25519' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
Load key "id_ed25519": bad permissions
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
[email protected]: Permission denied (publickey).
The solution to this is similar to the server. You have to make sure, that the permissions are correct, but in this case, you have to limit the access down to only the user for the private key:
chmod 600 ~/.ssh/id_ed25519
chmod 644 ~/.ssh/id_ed25519.pub
The public key can be readable by others as well. So the second command is usually optional.
Conclusion
There are many different issues you may have to face when configuring SSH on a server. It is really strict, when it comes to file and folder permissions, mainly to protect you from sharing secure data with others. If you happen to have a secret key too open, and you share the client/machine with multiple users, it’s probably best to discard the key and create a new one. When you use ssh-keygen to create one, the file permissions are set automatically.
I usually only use SSH nowadays to connect to servers for terminal access and for file transfer. Using key pairs is often the best way to go. But those file permissions can really be a struggle. Hopefully, you now know how to debug those issue – and I do as well, looking up how to start that debugging mode again. 😀