TOFU or no TOFU on OpenBSD.Amsterdam

I've just started up my first VM via OpenBSD.Amsterdam, and I was looking for an improvement in ssh connectivity to avoid the unnecessary Trust-On-First-Use acceptance of the VM's ssh host key fingerprints.

The Problem

When you create a VM, you provide a username, and a user ssh public key. When the machine is provisioned, you get :-

You do not get any information about the ssh host keys in use, so you are unable to connect over SSH without TOFU. ssh asks you to validate the previously-unseen host key, but you do not have sufficient data to do so correctly. Therefore you cannot connect to the new VM safely.

You can connect to the VM host, but without a valid username and password you cannot log in via the console. The root user has a password, but that is stored in the user's home directory ...

TL;DR solution

In order to validate the ssh host keys you need to connect to the host server, restart the VM in single-user mode, connect via the console and generate fingerprints yourself. Doing this correctly without further TOFU isn't straightforward, but is possible.

The details ...

Risk Assessment

Let's have a quick dose of reality here. Using the venerable “risk = impact x likelihood” rule of thumb, this is a vanishingly small risk, because it is very very unlikely that some attacker is diverting your ssh connection to a different machine – even if they were, the impact would also be tiny, as a hostile server cannot steal a user's ssh private key, and without that I don't think they'll be able to affect the real VM that you are being kept away from.

It's possible that you would use the fake server for long enough to populate it with some valuable private data – perhaps keys/tokens to access other services – and these could be stolen. But that seems like a long shot for a very expensive attack.

Basically, you would probably file all this in the “overly paranoid” box – but I think that's a mistake. We should highlight little problems like this, because they do have reasonably easy technical solutions, and they should be solved.

First the bad news

The Welcome email with your host details in it does not contain the VM's ssh host key fingerprints. OpenBSD.Amsterdam haven't added anything to a postinstall script that I'm aware of except to provision the initial user, ssh key, generate the root password and run syspatch.

Connecting to the VM's host machine without TOFU

(I'll use my real server name and keys for these examples, but I will leave out a couple of other non-TOFU-related details)

VerifyHostKeyDNS=yes, if you're lucky

By default, you are able to connect to the VM's host machine, and the host keys for these servers are communicated via SSHFP records in the DNS. This should only work when DNSSEC is validated, but this requires you to be using the correct upstream DNS resolvers, which is not as common as I expected.

As per the documentation for OpenBSD.Amsterdam, you can connect to their server using the ssh option -o VerifyHostKeyDNS=yes. If SSHFP and DNSSEC are working, the host keys are automatically validated (but not stored on local disk), and your connection will proceed normally to check user authentication, which uses the same user key as the VM itself does. If SSHFP is present, but DNSSEC doesn't validate, you will receive the comment Matching host key fingerprint found in DNS and will be asked to accept the fingerprint manually ... but that would be TOFU again. This is the time for you to validate DNSSEC yourself.

(There are a number of things that can cause this problem; MacOS doesn't support DNSSEC validation correctly; Windows OpenSSH builds don't support DNS validation; your system's DNS resolver might not support DNSSEC; your upstream resolvers might not support DNSSEC. In any case, you need to manually validate the fingerprints.)

Validating DNSSEC manually

Try dig +dnssec openbsd.amsterdam and look at the returned HEADER data. If you see the flag ad in the response, your DNSSEC is already correctly authenticated, and your most likely problem now is that your ssh client is the one that's misbehaving. In any case, manual validation is the way to go.

;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 24830
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

If the ad flag is missing, you can try the dig command again, but this time go directly to a known-DNSSEC-enabled resolver, such as DNS.Watch – dig +dnssec openbsd.amsterdam @84.200.70.40 or perhaps CloudFlare's 1.1.1.1.

If this looks good, you have a couple of choices :-

Changing your OS's DNS resolution is outside the scope of this article. It may be as simple as a quick edit to /etc/resolv.conf, or it may end up with a total redesign of your Internet connectivity. You probably should take the time to address this, as DNSSEC is a useful capability.

On the other hand, as that's a big job ... all we're trying to do here is to remove TOFU, with the emphasis on “First Use”. It seems reasonable to defer the bigger yak-shaving job and just validate this single connection by hand.

Validating Host Keys via SSHFP manually

First we need to get the host keys from the DNS, while maintaining DNSSEC validation. In this example, the host server name I'm using is server23 – the one I'm using for my VM, which doesn't seem like it needs to be a secret.

dig +dnssec SSHFP server23.openbsd.amsterdam @1.1.1.1

The SSHFP record that comes back is described in RFC 4255 – Using DNS to Securely Publish Secure Shell (SSH) Key Fingerprints, and has been extended to include newer key algorithms in later RFCs.

The three fields after the 'SSHFP' marker are

server23.openbsd.amsterdam. 3596 IN     SSHFP   1 1 BA4570E61B0BBA1DAB2A143E85DA683E03136891
server23.openbsd.amsterdam. 3596 IN     SSHFP   1 2 A0362931933653CC209C2730BEB5B3DF0EF45E0218A7B42DE166BBF9 264936F3
server23.openbsd.amsterdam. 3596 IN     SSHFP   4 1 5976C6AF2E87C32E65CC57E52935747B5D29D3AD
server23.openbsd.amsterdam. 3596 IN     SSHFP   4 2 ED60E846717EC00BE880C5DDC1CC63AA101C797E2FCC6A9F0F47B9E7 6FCDA6EA

There are 2 keys represented with 2 different hashes in the response above, an RSA key and an Ed25519 key. The one we will probably want to use is the SHA-256 encoded Ed25519 key, “SSHFP 4 2”.

In order to compare the SSHFP data with the fingerprint, we need to convert the data from hex into the base64 representation that ssh presents by default. On most unixes, you would have access to xxd and base64 – but if not then you might enjoy the GCHQ CyberChef web tool as an alternative.

$ printf "ED60E846717EC00BE880C5DDC1CC63AA101C797E2FCC6A9F0F47B9E7 6FCDA6EA" | xxd -p -r | base64
7WDoRnF+wAvogMXdwcxjqhAceX4vzGqfD0e552/Npuo=

Now compare that value with the challenge presented by ssh when you try to connect :-

$ ssh [some redacted flags] server23.openbsd.amsterdam.
The authenticity of host 'server23.openbsd.amsterdam. (46.23.91.33)' can't be established.
ED25519 key fingerprint is SHA256:7WDoRnF+wAvogMXdwcxjqhAceX4vzGqfD0e552/Npuo.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])?

Given that the fingerprints match, we can trust that we are talking to the correct server. Answering 'yes' to the fingerprint challenge will normally cause it to be entered into your local known_hosts file for future use. If your ssh presents the RSA key, or uses a different hash type, you can just repeat the xxd | base64 command with the matching data from DNS.

Getting the VM's Host Keys

Now we are logged in to the VM host server, where we can connect to the VM via its console, avoiding ssh.

But, by default the console is running a login prompt, and we need to have a password to proceed – at this stage we don't have a password for anything, not for our user account and not for the root account. So, what to do?

Remembering again that the key to our problem is “First Use”, I'll point out that your server isn't actually doing anything useful for you yet; so it should be acceptable to shut it down, and reboot in single-user mode.

vmctl

Halt the VM, and then restart it while connected to the console. As per normal OpenBSD procedures, you have 5 seconds to start inputting a command to the boot> prompt, so keep your eyes open and repeat the whole process if necessary. The command to enter will be boot -s. Eventually you will be asked for a shell to run – just accept the default.

# vmctl stop -w vm01
stopping vm vm01: terminated vm 1

# vmctl start -c vm01
Connected to /dev/ttyp1 (speed 115200)
Using drive 0, partition 3.
Loading......
probing: pc0 com0 mem[638K 1022M a20=on]
disk: hd0+
>> OpenBSD/amd64 BOOT 3.67
\
com0: 115200 baud
switching console to com0
>> OpenBSD/amd64 BOOT 3.67
boot>  boot -s
booting hd0a:/bsd: 22104021+4674576+449080+0+1351680 [1954133+128+1499280+1192779]=0x1fb19b8
entry point at 0xffffffff81001000
... [boot sequence output omitted] ...
Enter pathname of shell or RETURN for sh:
#

Fingerprinting the keys on the host

Because we've interrupted normal booting to get to single-user mode, we need to manually mount /usr to get the /usr/bin/ssh-keygen command.

# mount /usr

Now we can go to the ssh config directory and list the host key fingerprints.

# for i in /etc/ssh*key
> do
> ssh-keygen -lf $i
> done
256 SHA256:9bm7K1DaioLglzycrsKxuWkji7cF+bsAw4eJEtmWd98 root@mulberry.openbsd.amsterdam (ECDSA)
256 SHA256:3K+nEIym5djtjMzX5kfdEOPPFJ4wPvQvW2OIY6G0K68 root@mulberry.openbsd.amsterdam (ED25519)
3072 SHA256:a5QyTl8DxKUHuII9hjKenW/qms835EU4GgM/hxDwf+Q root@mulberry.openbsd.amsterdam (RSA)

Keep that data available, and then exit from the single-user shell to allow normal boot to proceed. Disconnect from the console with ~. sent to the session.

The Finale! ssh with no TOFU!

We can now finally connect to our VM over ssh, and be able to verify the suggested host fingerprint. The integrity of our data has been protected with DNSSEC.

$ ssh [flags omitted] 46.23.95.11
The authenticity of host '46.23.95.11 (46.23.95.11)' can't be established.
ED25519 key fingerprint is SHA256:3K+nEIym5djtjMzX5kfdEOPPFJ4wPvQvW2OIY6G0K68.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])?

The fingerprint suggested by ssh matches the one we generated on the console of the VM, so we can continue connecting, safely, without any TOFU.

What would I change?

To avoid this problem in the future (I mean, for future VMs, because it's been solved for this one!) I would like to see some way to get the initial ssh host key fingerprints in an automated way; preferably in the Welcome email.

postinstall?

The VM host here wants to keep their access to a customer VM down to as close to nothing as possible, which is a laudable goal. At the same time they want to use as close to stock OpenBSD as possible, which makes it difficult to consider more extensive customisation.

There is a postinstall script of some kind; there has to be, in order to add the customer's user, add their ssh key and generate a root password. However, it's possible that this script finishes well before the ssh host keys are generated, but that shouldn't preclude generating them early.

This postinstall script seems to be the only lever available to pull, and I'd like OpenBSD.Amsterdam to consider using it :–)

Publish fingerprints on the website?

Given the widespread lack of support for DNSSEC, it seems like the excellent SSHFP work provided here for the VM host servers is being undermined. It would probably be a good idea to publish the various key fingerprints on a page on the TLS-validated website documentation, so save the trouble of finding a working DNSSEC resolver and converting the fingerprint representations.