SSH Key Format Issues and Fixing the “invalid format” Error

1. Introduction

Due to the widespread use of asymmetric cryptography, end users can employ and leverage it more easily. Still, even the ubiquitous Secure Shell (SSH) algorithm isn’t resistant to user mistakes.

In this tutorial, we talk about issues with the format of keys when establishing an SSH session. First, we generate a sample key pair. Next, we discuss key authorization and the pitfalls around it. After that, we consider reasons for supplying an incorrect identity by mistake when authenticating. Finally, we discuss proper handling to avoid key format issues.

For brevity and security reasons, we only consider the newest iteration of SSH version 2 (SSHv2) implemented by OpenSSH.

We tested the code in this tutorial on Debian 11 (Bullseye) with GNU Bash 5.1.4. It should work in most POSIX-compliant environments.

2. Generate Key Pair

First, let’s generate a pair of keys with no password via ssh-keygen:

$ ssh-keygen -f /non-standard/id_rsa -N ''
Generating public/private rsa key pair.
Your identification has been saved in /non-standard/id_rsa
Your public key has been saved in /non-standard/id_rsa.pub
The key fingerprint is:
[...]

Notably, we use a non-standard path instead of the usual $HOME/.ssh.

At this point, we have one example set of two keys:

• private key: /non-standard/id_rsa
• public key: /non-standard/id_rsa.pub

Critically, we require the correct permissions for each key:

$ chmod 0700 /non-standard/id_rsa
$ chmod 0700 /non-standard/id_rsa.pub

Following chmod for restricting access to our keys, let’s output them with cat:

$ cat /non-standard/id_rsa
-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAA06660ZQAAAAAAAAABAAABlwAAAAdzc2gtcn
[...]
bb9JibgMdOkkcAAAANYmFlbGR1bmd06660ZAECAwQFBg==
-----END OPENSSH PRIVATE KEY-----
$ cat /non-standard/id_rsa.pub
ssh-rsa AAAAB3[...]Szdz8= baeldung@web

After generating a pair, we can explore potential format issues with the use of our new keys for public key authentication.

3. Authorizing the Correct Key

Usually, we add the public key to the authorized_keys file for our user with a dedicated tool or directly.

3.1. ssh-copy-id

Let’s see how to authorize a key with ssh-copy-id:

$ ssh-copy-id baeldung@fw
/usr/bin/ssh-copy-id: ERROR: No identities found

Due to our non-standard path outside $HOME/.ssh, ssh-copy-id can’t automatically locate the local keys. Hence, we use the -i argument to specify /non-standard/id_rsa as we did during generation:

$ ssh-copy-id -i /non-standard/id_rsa baeldung@fw
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/non-standard/id_rsa.pub"
[...]

Here, ssh-copy-id appends the .pub suffix automatically. Further, when our keys are in their standard location, ssh-copy-id uses the public key (id_rsa.pub) by default.

These mechanisms prevent mistakes when authorizing the key.

3.2. Manual Key Authorization Problems

Unlike ssh-copy-id, the cat command doesn’t perform any checks:

$ cat /non-standard/id_rsa >> $HOME/.ssh/authorized_keys

In this case, it might not be easy to spot the problem: supplying /non-standard/id_rsa doesn’t self-rectify to /non-standard/id_rsa.pub, meaning we incorrectly place our private key in the authorized_keys file, resulting in several problems:

• the format of authorized_keys is now invalid, so public key authentication seizes to work for our user
• our private key is exposed in a public file
• the key isn’t authorized

Because of these potential security holes, it’s highly recommended to use ssh-copy-id and the default SSH key paths.

4. Using the Correct Key

Another common problem with public key authentication is the identity file we supply during session setup.

In particular, we might mix up the private and public keys:

$ ssh -i /non-standard/id_rsa.pub remote
Load key "/non-standard/id_rsa.pub": invalid format

If we supply the public instead of the private key as the identity during session establishment, our SSH client tells us that the format is invalid. Of course, the same would happen for any file that’s not a private key in the proper format:

$ ssh -i /etc/ssh/sshd_config remote
Load key "/etc/ssh/sshd_config": invalid format

Depending on the key pair naming scheme, this error might not be very helpful. Also, we can experience issues even with the proper key file if we tamper with the format.

5. Handling Keys Properly

Of course, mishandling sensitive and strictly structured data such as SSH keys can lead to security issues.

5.1. Key Naming

The default names of public and private keys depend on the exact key type. For example, when using RSA, the defaults are id_rsa and id_rsa.pub, respectively.

In most cases, the public key has the name of the private key but with a .pub extension. In fact, the .pub suffix can be critical for the correct key detection and use. Moreover, preserving the proper naming scheme can be vital in general.

This is especially important when transferring key data.

5.2. Key Transfer and Streaming

When relocating key content, any formatting must be preserved along with the data. In other words, no changes to the key contents are allowed apart from the public key comments.

While the public key is usually a single line, private keys are longer and spread across multiple lines, so they are more prone to issues:

$ cat /non-standard/id_rsa
-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAA06660ZQAAAAAAAAABAAABlwAAAAdzc2gtcn
[...]
bb9JibgMdOkkcAAAANYmFlbGR1bmd06660ZAECAwQFBg==
-----END OPENSSH PRIVATE KEY-----
$ prk=$(cat /non-standard/id_rsa)
$ echo $prk
-----BEGIN OPENSSH PRIVATE KEY----- b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAA06660ZQAAAAAAAAABAAABlwAAAAdzc2gtcn[...]bb9JibgMdOkkcAAAANYmFlbGR1bmd06660ZAECAwQFBg== -----END OPENSSH PRIVATE KEY-----
$ echo $prk > /non-standard/id_rsa_bad
$ ssh -i /non-standard/id_rsa_bad baeldung@fw
Load key "/non-standard/id_rsa_bad": invalid format

After confirming the current format of our private key with cat, we use the same command in a subshell to assign that to a variable. At this point, we lose the newlines since we don’t use quotes around $(). Thus, the $prk variable contains an incorrect value, which we transfer to /non-standard/id_rsa_bad.

Finally, using that key as an identity results in an invalid format error. Naturally, any other reason for bad formatting will result in the same.

6. Summary

In this article, we looked at common causes for malformed SSH keys, as well as their consequences.

In conclusion, although we can generate and use key pairs fairly easily, there are ways to do it incorrectly and methods to avoid such mistakes.