I find the simplest and quickest way to debug ssh key authentication problems is to run the SSH daemon temporarily in debug mode on an alternate port. Of course this presumes that you already have some kind of administrative access to the destination server.
Typically people would run ssh -v[vv] dest date (for example) to try and debug key authentication failures, however this gives you limited useful information, even with full verbosity:
$ ssh -vvv localhost uptime OpenSSH_4.5p1+sftpfilecontrol-v1.1-hpn12v14, OpenSSL 0.9.7l 28 Sep 2006 HP-UX Secure Shell-A.04.50.003, HP-UX Secure Shell version debug1: Reading configuration data /home/username/.ssh/config debug1: Reading configuration data /opt/ssh/etc/ssh_config debug3: RNG is ready, skipping seeding debug2: ssh_connect: needpriv 0 debug1: Connecting to localhost [127.0.0.1] port 22. debug1: Connection established. debug1: identity file /home/username/.ssh/id_rsa type -1 debug3: Not a RSA1 key file /home/username/.ssh/id_dsa. debug2: key_type_from_name: unknown key type '-----BEGIN' debug3: key_read: missing keytype ... <snip> ... debug2: key: /home/username/.ssh/id_rsa (0) debug2: key: /home/username/.ssh/id_dsa (4002f060) debug1: Authentications that can continue: publickey,password,keyboard-interactive debug3: start over, passed a different list publickey,password,keyboard-interactive debug3: preferred publickey,keyboard-interactive,password debug3: authmethod_lookup publickey debug3: remaining preferred: keyboard-interactive,password debug3: authmethod_is_enabled publickey debug1: Next authentication method: publickey debug1: Trying private key: /home/username/.ssh/id_rsa debug3: no such identity: /home/username/.ssh/id_rsa debug1: Offering public key: /home/username/.ssh/id_dsa debug3: send_pubkey_test debug2: we sent a publickey packet, wait for reply debug1: Authentications that can continue: publickey,password,keyboard-interactive debug2: we did not send a packet, disable method debug3: authmethod_lookup keyboard-interactive debug3: remaining preferred: password debug3: authmethod_is_enabled keyboard-interactive debug1: Next authentication method: keyboard-interactive debug2: userauth_kbdint debug2: we sent a keyboard-interactive packet, wait for reply debug2: input_userauth_info_req debug2: input_userauth_info_req: num_prompts 1 Password:
Presumably this is for security reasons; you don't want potential remote attackers to be able to discover the details of your defences and figure out what weapons they need.
Much more useful information can be obtained by running sshd in debug mode on the destination server:
# /usr/sbin/sshd -p1234 -d debug1: Config token is protocol debug1: Config token is kerberosauthentication debug1: Config token is usepam debug1: Config token is x11forwarding debug1: Config token is x11uselocalhost debug1: Config token is hpndisabled debug1: Config token is subsystem debug1: sshd version OpenSSH_4.5p1+sftpfilecontrol-v1.1-hpn12v14 [ HP-UX Secure Shell-A.04.50.003 ] debug1: read PEM private key done: type RSA debug1: private host key: #0 type 1 RSA debug1: read PEM private key done: type DSA debug1: private host key: #1 type 2 DSA debug1: rexec_argv='/usr/sbin/sshd' debug1: rexec_argv='-p1234' debug1: rexec_argv='-d' debug1: Bind to port 1234 on 0.0.0.0. debug1: Server TCP RWIN socket size: 32768 debug1: HPN Buffer Size: 131072 Server listening on 0.0.0.0 port 1234.
Then attempt to connect to that specific port from the source server, no debugging options required:
$ ssh -p1234 localhost uptime Password:
And the reason for the failure becomes immediately apparent on the server side:
debug1: Server will not fork when running in debugging mode. debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8 debug1: inetd sockets after dupping: 4, 4 debug1: audit connection from 127.0.0.1 port 58570 euid 0 Connection from 127.0.0.1 port 58570 debug1: Client protocol version 2.0; client software version OpenSSH_4.5p1+sftpfilecontrol-v1.1-hpn12v14 debug1: match: OpenSSH_4.5p1+sftpfilecontrol-v1.1-hpn12v14 pat OpenSSH* debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_4.5p1+sftpfilecontrol-v1.1-hpn12v14 debug1: permanently_set_uid: 129/105 ... <snip> ... debug1: PAM: initializing for "username" debug1: PAM: setting PAM_RHOST to "localhost" Failed none for username from 127.0.0.1 port 58570 ssh2 debug1: audit event euid 0 user username event 3 (AUTH_FAIL_NONE) debug1: userauth-request for user username service ssh-connection method publickey debug1: attempt 1 failures 1 debug1: test whether pkalg/pkblob are acceptable debug1: temporarily_use_uid: 3661/20 (e=0/3) debug1: trying public key file /home/username/.ssh/authorized_keys Authentication refused: bad ownership or modes for directory /home/username/.ssh debug1: restore_uid: 0/3 debug1: temporarily_use_uid: 3661/20 (e=0/3) debug1: trying public key file /home/username/.ssh/authorized_keys2 debug1: restore_uid: 0/3 Failed publickey for username from 127.0.0.1 port 58570 ssh2 debug1: Entering record_failed_login uid 0 debug1: audit event euid 0 user username event 6 (AUTH_FAIL_PUBKEY) debug1: userauth-request for user username service ssh-connection method keyboard-interactive debug1: attempt 2 failures 2 debug1: keyboard-interactive devs debug1: auth2_challenge: user=username devs= debug1: kbdint_alloc: devices 'pam' debug1: auth2_challenge_start: trying authentication method 'pam' Postponed keyboard-interactive for username from 127.0.0.1 port 58570 ssh2
In this case you can see it was due to a poorly secured /home/username/.ssh directory.