CSD3 Host Keys

The CSD3 SSH host keys will be refreshed on 1st February 2020. This will require all existing users to accept the new keys. Some users will be able to take care of this in advance and minimise the inconvenience. Please see Refreshing the host keys below for more details about the impending key changes and the steps which will be necessary.

Before 1st February 2020

The SSH host keys valid until 31st January 2020 have the following fingerprints (which of these hashes you see when first connecting to the login nodes will depend on your client):

RSA
    MD5:0b:ef:59:90:fb:13:4a:c9:56:82:7b:cd:4b:2b:e1:3b
    SHA256:sSkVfzpwjwiFvxLcdPoDpN8IsN3kt0ZSywhDhPKZPAg

ED25519

    MD5:64:7c:7c:ff:05:9d:0e:dc:06:fe:f1:c2:10:37:7a:85
    SHA256:wq9ljBfPa7lXXpQq+rk5JTBXLJO/kXjOc5A7rp4ENzA

ECDSA

    MD5:34:9b:f2:d2:c6:b3:5c:63:99:b7:27:da:5b:c8:16:fe
    SHA256:HsiY1Oe0M8tS6JwR76PeQQA/VB7r8675BzG5OYQ4h34

From 1st February 2020

The SSH host keys valid from 1st February 2020 have the following fingerprints (which of these hashes you see when first connecting to the login nodes will depend on your client):

RSA
    MD5:fd:5c:6b:7d:49:95:2f:da:7f:5c:50:9a:bb:ef:3f:24
    SHA256:2rl+MXd9rsrDzFZwEItmhhiHTlLTIqN0d3TSGLTgjTI

ED25519

    MD5:eb:e3:a1:f0:64:68:cf:9c:63:da:84:db:2e:ee:15:83
    SHA256:nFVSXK+VRGCaUupQEdhXzO6kp01m2fzzmbgPr0sc2so

Refreshing the host keys

NB If you are a new user logging in for the first time after 1st February 2020, this section is irrelevant to you. Instead please see First-time login.

The first time a user connects to CSD3 from a new computer, they are required to verify and confirm the host key fingerprints (given above). The user’s local computer then stores these keys and, during subsequent connections, checks that they haven’t changed (as a protection against inadvertently supplying passwords and other sensitive information to a malicious server impersonating a CSD3 login node). However this means that when the host keys are legitimately refreshed, existing users need to update the old host keys stored on their local systems so that they may continue to login. The details of this depend on which SSH client application you use to login.

If you are a Linux, MacOS or Cygwin user, then you are probably using OpenSSH, for which a feature exists that may reduce the pain due to host key changes (provided your version is recent enough), and which can be performed before the old host keys stop working. If this applies to you then please see the section OpenSSH 6.8+ (Linux, MacOS, Cygwin - using ssh, scp, sftp, rsync) below and perform the steps described there as soon as possible.

The procedures for other common SSH client applications cannot be performed until the keys are actually replaced. Please see application-specific notes for these common applications below. Please note that each procedure needs to be followed for each local machine connecting to a particular name on CSD3 (e.g. to each distinct destination login.hpc.cam.ac.uk, login-cpu.hpc.cam.ac.uk, login-gpu.hpc.cam.ac.uk, etc).

Should be done prior to the host key change:

NB This will only work for OpenSSH, versions 6.8 or higher. To check which version you have, see the output of ssh -V. For older versions, or UNIX versions of ssh other than OpenSSH, see Other UNIX ssh (includes scp/sftp/rsync).

Can only be done after the host key change:

OpenSSH 6.8+ (Linux, MacOS, Cygwin - using ssh, scp, sftp, rsync)

NB The changes in this section are to be made on your local machine (not CSD3), before 1st February 2020. After the host keys change on this date it will be necessary instead to follow the procedure Other UNIX ssh (includes scp/sftp/rsync), since by that point it will be too late for the steps described here to help.

If your local machine runs Linux or MacOS, or is a Windows machine with Cygwin, then edit the following file:

~/.ssh/config

(create it if necessary) and insert the following directive into the relevant Host section for CSD3 - if you do not have a CSD3-specific Host section then insert it into the Host * section at the end of the file, e.g.

Host *
  UpdateHostKeys yes

The effect of this will be to download the new host keys (which are already on the nodes) to your local machine in advance of the refresh date, at which time the old keys will quietly and automatically be removed. In other words, after this simple edit, and provided you login to CSD3 before the refresh happens, in theory you will need to take no further action.

In practice however, because each .hpc.cam.ac.uk name translates to multiple different nodes and IP addresses, you may find that not all combinations of name and address have been automatically updated by the time the key change occurs, particularly if UpdateHostKeys was enabled very close to the change date. In this case you may still see Host key verification errors and Offending key for IP warnings after the refresh date. The most efficient cure for this is to follow the procedure described at Other UNIX ssh (includes scp/sftp/rsync).

It should be safe to leave UpdateHostKeys enabled after the refresh is complete, and this is recommended in order to facilitate future host key changes. See Checking which host key you have accepted for a simple way of checking whether your ~/.ssh/known_hosts file still contains references to outdated host keys.

Putty (Windows)

Users of Putty on Windows should expect to see a message like the below when connecting to CSD3 for the first time after the host keys are refreshed:

The message expected from Putty post-hostkey refresh.

Provided the new fingerprint matches one of the new key fingerprints listed at From 1st February 2020, press Yes.

WinSCP (Windows)

Users of WinSCP on Windows should expect to see a message like the below when connecting to CSD3 for the first time after the host keys are refreshed:

The message expected from WinSCP post-hostkey refresh.

Provided the new fingerprints match the new key fingerprints listed at From 1st February 2020, press Update.

MobaXterm (Windows)

Users of MobaXterm on Windows should expect to see a message like the below when connecting to CSD3 for the first time after the host keys are refreshed:

The message expected from MobaXterm post-hostkey refresh.

Unfortunately in this case the application does not display the fingerprint (which is bad). Since you expect to see this message, press Yes, but note that you should not see the message again from the same local machine.

X2Go (Windows)

Users of X2Go on Windows should expect to see a message like the below when connecting to CSD3 for the first time after the host keys are refreshed:

The message expected from X2Go on Windows, post-hostkey refresh.

Provided the new fingerprint matches one of the new key fingerprints listed at From 1st February 2020, press No. You will then see another message like the below, to which answer Yes.

The message expected from X2Go on Windows, post-hostkey refresh.

Other UNIX ssh (includes scp/sftp/rsync)

Other versions of ssh on UNIX, e.g. on Solaris, or on older distributions of Linux (with OpenSSH versions earlier than 6.8) will emit the following dire warning when first connecting to CSD3 after the host keys change. You will also see this with OpenSSH versions 6.8 or later if keys for all name and address combinations were not automatically updated in advance of the host key change.

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:nFVSXK+VRGCaUupQEdhXzO6kp01m2fzzmbgPr0sc2so.
Please contact your system administrator.
Add correct host key in /home/sjr20/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /home/sjr20/.ssh/known_hosts:93
ED25519 host key for login.hpc.cam.ac.uk has changed and you have requested strict checking.
Host key verification failed.

In the above case the login attempt has failed and you are still on your local machine. For at least OpenSSH, this is what should happen unless ssh has had the setting StrictHostKeyChecking set to no, which is not recommended. If it has let you in, log out again and consider returning StrictHostKeyChecking to its default value of ask (this is done in ~/.ssh/config, unless for some reason it has been set centrally by the administrators of your machine).

The following commands on your local machine will save a copy of your existing known_hosts file, then apply an edit to strip out all references to CSD3 login nodes, thus returning the ssh configuration on your local machine to a state of complete ignorance as to the correct host keys for CSD3 login nodes:

sed -i.orig -e '/128\.232\.224/d' -e '/.*\.hpc\.cam\.ac\.uk/d' ~/.ssh/known_hosts

Now connect to CSD3 again, and you will see a message similar to that shown in First-time login. From this point, this is the same procedure required for all versions of UNIX ssh when first logging in from a local machine.

Note that the above command may have no effect if you have the HashKnownHosts ssh config option set. If the first form of the sed command appears to do nothing, try the following instead which will strip out any line matching the ends of the obsolete host keys:

sed -i.orig2 -e '/.*An5vPXewQ==$/d' -e '/.*PwTab9DQ\/0n$/d' -e '/.*HCc9q3BNI=$/d' ~/.ssh/known_hosts

First-time login

When connecting via UNIX ssh to a CSD3 server for the first time from a particular local machine, one sees a message similar to this:

The authenticity of host 'login.hpc.cam.ac.uk (128.232.224.51)' can't be established.
ED25519 key fingerprint is SHA256:nFVSXK+VRGCaUupQEdhXzO6kp01m2fzzmbgPr0sc2so.
ED25519 key fingerprint is MD5:eb:e3:a1:f0:64:68:cf:9c:63:da:84:db:2e:ee:15:83.
Are you sure you want to continue connecting (yes/no)?

Note that the details you see may differ from the above, but the fingerprints reported must match the hashes listed at either Before 1st February 2020 or From 1st February 2020 as appropriate, otherwise do not proceed but contact support@hpc.cam.ac.uk. Assuming the fingerprints are as expected, then respond yes (note that here simply typing y will not work).

Having accepted the new host keys, you should not be asked to accept them again when contacting the same named destination from the same local machine (at least until the next time the host keys are refreshed, or unless you delete the relevant lines from your ~/.ssh/known_hosts file). You may still see occasional warnings about keys for different IP addresses being accepted - since each destination name may map to multiple IP addresses, this is to be expected.

Other SSH-based applications such as putty, WinSCP, x2go etc will also request that you explicitly accept the host key during first-time login. Once again, only do this if the fingerprints match those listed on this page.

Checking which host key you have accepted

This section applies if your local machine runs Linux or MacOS.

In order to check which SSH host keys you have previously accepted for a particular destination, e.g. login.hpc.cam.ac.uk, you may use the following command (tested under OpenSSH):

ssh-keygen -l -F login.hpc.cam.ac.uk

This command will print the fingerprints of the relevant SSH host keys currently stored in your ~/.ssh/known_hosts file. The host key update is complete when this command only lists the new host key fingerprints listed at From 1st February 2020. If you see old fingerprints reported on certain line numbers, you can simply open ~/.ssh/known_hosts with a text editor and remove those lines.