Secure firmware updates with code signing
Previously, we wrote about implementing firmware update for our devices. One important detail we did not cover is firmware update security.
Much can be done - and written - about firmware update security, but perhaps the most important bit is firmware signing. Other security measures are not much use if we cannot verify the authenticity of a firmware update!
In this post, we explain why firmware signing is important, how it works, and what algorithm should be used to implement it. We also detail a full implementation built with open source, cross platform libraries.
If you’d rather listen to me present this information and see some demos in action, watch this webinar recording
Table of Contents
Firmware Signing Explained
Note: the next few sections provide an overview of code signing, why it matters, and how it works. If you’d like to skip straight to implementation, click here.
Code signing is a method of proving a file was created by a trusted source and has not been tampered with. This is achieved by creating a signature for the file: a token which can be verified but not forged.
We all use signatures in our day to day life, by scribbling our name at the bottom of our credit card receipts, contracts, and bank checks. These signatures can be verified quite easily by comparing them to previous signatures, but are hard to forge because they rely on an individual’s muscle memory.
Code signing uses cryptographic algorithms to achieve a similar goal. The resulting signatures can only be generated by people who know a given password, also known as a secret key, but can be verified as genuine by anyone.
When a signature is generated for a firmware binary, it is uniquely tied to that exact firmware. Changing the binary so much as 1 bit would produce a different signature.
With this in mind, if a device is given a signed binary and a signature and can verify both that the binary and signature are valid and that the binary was created by the correct author, then we know the binary has not been tampered with.
Why sign our firmware
By implementing signature verification in our bootloader we can identify whether or not a given firmware update was provided by the manufacturer, or if it has been tampered with. The bootloader can then decide to either warn the user, void the device’s warranty, or simply refuse to run the unauthenticated binary.
With more and more devices connected to the internet, security is an increasingly hot topic in firmware development. A device which accepts firmware updates over the wireless or internet connectivity but does not verify it opens itself to compromise. By feeding it with a malicious firmware image, an attacker might:
- Brick the device, or the whole fleet
- Snoop on end users and compromise their privacy and security
- Strategically malfunction at a critical time
These are highly undesirable outcomes, which can be effected at scale due to the internet of things. In 2020, it is reckless to implement firmware update for our systems without some form of authentication.
What signing is not: code signing is an important component of firmware security, it is not by itself sufficient to build secure systems. Secure coding, static analysis, hardware tamper detection, JTAG locking, and many more techniques should be implemented as well. Code signing also does nothing to protect against reverse engineering. It is a distinct technique from firmware encryption.
ECDSA
Several algorithms can be used to sign firmware, including RSA, DSA, and ECDSA. In this post, we focus on ECDSA for a few reasons:
- Security: ECDSA is the latest and greatest in terms of signature algorithms. While standard DSA is considered broken by most, ECDSA is expected to remain secure until quantum computing becomes widely available.
- Popularity: ECDSA is used extensively in applications ranging from cryptocurrencies (bitcoin) to secure messaging. With popularity comes battle tested implementations and credibility.
- Availability: Open source implementations of ECDSA are available for microcontrollers, including mbedtls1, wolfssl2, and micro-ecc.
- Small Footprint: ECDSA implementations are very small (single digit kB), and require smaller keys than RSA or DSA for similar levels of security. This saves both code space and RAM and makes ECDSA well suited to embedded environments.
Understanding the math behind ECDSA is outside of the scope of this article, but here is a high level overview of how the process works:
- A cryptographic hash of the firmware binary is created. Any cryptographic hashing algorithm should work, though SHA-2 family hashes are recommended. In the case of SHA-256, this yields a 32-byte number.
- A signature is generated using a private key and the cryptographic hash. This signature is distributed alongside the firmware and a public key. The signature may not be deterministic, so don’t fret if multiple invocations of your ECDSA code yield different signatures. This signature is a pair of integers, each 32 bytes long.
- To verify the binary, a SHA-256 hash is once again computed for our firmware binary.
- The public key and the hash can be used to verify the signature was generated using matching inputs.
Firmware Signing Implementation
Our implementation builds upon the code we wrote for our firmware update architecture post. You may find that code on Github at interrupt@20ec4ba.
Setup
Like we did our previous post, the Firmware Update Cookbook, we use Renode to run the examples in this post. The previous post contains detailed instructions, but in short:
# Clone the repository & navigate to the example
$ git clone https://github.com/memfault/interrupt.git
$ cd examples/fwup-signing
# Build project and start Renode
$ make && ./start.sh
One change we did make from previous posts is run in headless mode. This is done by invoking Renode with the following flags:
mono64 $RENODE_EXE_PATH renode-config.resc --port 4444 --disable-xwt
We can then use telnet to connect to the Renode console:
$ telnet localhost 4444
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Renode, version 1.9.0.28660 (cd1a61a4-202006301553)
(monitor) i $CWD/renode-config.resc
(STM32F429) start
Starting emulation...
(STM32F429) q
Renode is quitting
Similarly, we route the device UART to a telnet port rather than a graphical
window by adding two lines to our renode-config.resc:
emulation CreateServerSocketTerminal 4445 "externalUART"
connector Connect sysbus.uart2 externalUART
We can then access our emulated device’s UART via telnet as well:
$ telnet localhost 4445
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Bootloader started
Valid public key
Invalid signature
Booting slot 1
Shared memory uinitialized, setting magic
Loader STARTED - version 1.0.0 (4015f0a)
Booting slot 2
App STARTED - version 1.0.1 (4015f0a) - CRC 0x1b11019d
Architecture
As a reminder, this is what our device firmware update architecture looks like:
With the following functionality for each block:
- Bootloader: a simple program whose sole job is to load the Loader, and fallback to another image if the Loader is invalid.
- Loader: a program that can verify our Application image, load it, and update it.
- Application: our main code, which does not do any updates itself
- Updater: a program that temporarily replaces the Application and can update the Loader.
The Loader, the Application, and the Updater all need to be signed since they can be updated. Components that update other components need to implement our signature verification algorithm, in our case that’s the Loader and the Updater.
Generating public/private key pairs
Before we can begin, we must generate a pair of keys. Our private key, also known as a signing key, is used to create the signatures, and should be kept private. Anyone with access to the key will be able to sign firmware on your behalf. Our public key, also known as a validation key, is used to verify the signature. It can be freely distributed.
Several tools can be used to generate our key pair but the simplest is
openssl, a cross platform cryptography toolset.
First, we generate our private key:
$ openssl ecparam -name secp256k1 -genkey -noout -out private.pem
Then we create a public key to go with this private key:
$ openssl ec -in private.pem -pubout -out public.pem
At this point we should have two files: public.pem and private.pem.
We can test the sign/verify flow with openssl as well. To sign a file we do:
$ openssl dgst -sha256 -sign private.pem -out build/fwup-example.bin.sig build/fwup-example.bin
Where “fwup-example.bin” is the file we want to sign, and “fwup-example.bin.sig” is the signature file we want to create.
The signature can then be verified with:
$ openssl dgst -sha256 -verify public.pem -signature build/fwup-example.bin.sig build/fwup-example.bin
Verified OK
That’s it! We’ve got keys we can use to sign and verify firmware builds!
Bundling signatures with our builds
Next we must add the signing step to our build and add the signatures to our
image. The latter is easy to do: we already have a metadata for our images saved
into the image_hdr_t. We simply add a field for the signature and increment
our metadata version:
typedef enum {
IMAGE_VERSION_1 = 1,
- IMAGE_VERSION_CURRENT = IMAGE_VERSION_1,
+ IMAGE_VERSION_2 = 2,
+ IMAGE_VERSION_CURRENT = IMAGE_VERSION_2,
} image_version_t;
typedef struct __attribute__((packed)) {
@@ -34,10 +35,13 @@
uint32_t vector_addr;
uint32_t reserved;
char git_sha[8];
+ uint8_t ecdsa_sig[64];
} image_hdr_t;
The 64 bytes contain the two integers that make up our signature one after the other.
We then modify patch_image_header.py so that it generates and appends the
signature to our binary. Rather than shell out to openssl, we use a native
Python library that implements ECDSA. Simply enough, it is called ecdsa.
Generating the signature can be accomplished with a few lines of code:
import hashlib
from ecdsa import SigningKey
from ecdsa.util import sigencode_string
from binascii import hexlify
def gen_binary_signature(data, key_filename):
with open(key_filename, "r") as f:
key_pem = f.read()
key = SigningKey.from_pem(key_pem)
sig = key.sign_deterministic(data, hashfunc=hashlib.sha256, sigencode=sigencode_string)
return sig
A few things to note:
- We use SHA-256 as our hashing function, so we must specify it when we generate the signature
- The
sigencodeparameter is used to specify the format of the signature. By default, a DER file a generated (this is what openssl uses as well), but by usingsigencode_stringwe tell the library to generate a flat string that contains the binary representation of both signature integers one after another. This is a simpler format and will save us from implementing a DER parser in our firmware.
We then update our patch_binary_payload function to invoke
gen_binary_signature:
+def patch_binary_payload(bin_filename, pk_filename):
"""
Patch crc & data_size fields of image_hdr_t in place in binary
Raise exception if binary is not a supported type
"""
- IMAGE_HDR_SIZE_BYTES = 32
+ IMAGE_HDR_SIZE_BYTES = 96
IMAGE_HDR_MAGIC = 0xCAFE
- IMAGE_HDR_VERSION = 1
+ IMAGE_HDR_VERSION = 2
with open(bin_filename, "rb") as f:
image_hdr = f.read(IMAGE_HDR_SIZE_BYTES)
@@ -35,6 +47,7 @@
data_size = len(data)
crc32 = binascii.crc32(data) & 0xffffffff
+ signature = gen_binary_signature(data, 'private.pem')
image_hdr_crc_data_size = struct.pack("<LL", crc32, data_size)
print(
@@ -42,11 +55,17 @@
crc32, data_size, bin_filename
)
)
+
with open(bin_filename, "r+b") as f:
# Seek to beginning of "uint32_t crc"
f.seek(4)
# Write correct values into crc & data_size
f.write(image_hdr_crc_data_size)
+ # Seek to beginning of signature
+ f.seek(32)
+ # Write the signature in place
+ f.write(signature)
if __name__ == "__main__":
@@ -54,6 +73,7 @@
description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter,
)
parser.add_argument("bin", action="store")
+ parser.add_argument("pk", action="store")
args = parser.parse_args()
- patch_binary_payload(args.bin)
+ patch_binary_payload(args.bin, args.pk)
Since the script now takes an argument, we must modify our Makefile as well
-$(BUILD_DIR)/$(PROJECT)-%.bin: $(BUILD_DIR)/$(PROJECT)-%.elf
+$(BUILD_DIR)/$(PROJECT)-%.bin: $(BUILD_DIR)/$(PROJECT)-%.elf $(PK_PEM_PATH)
$(ECHO) " OBJCOPY $@"
- $(Q)$(OCPY) $^ $@ -O binary
+ $(Q)$(OCPY) $< $@ -O binary
$(ECHO) " PATCH_IMAGE $@"
- $(Q)$(PYTHON) patch_image_header.py $@ > /dev/null
+ $(Q)$(PYTHON) patch_image_header.py $@ $(PK_PEM_PATH) > /dev/null
Where PK_PEM_PATH is the path to your private key.
We now have a build process that generates signed binaries!
Verifying signatures in our Loader
Now that our firmware builds are signed, let’s verify those signatures in our Loader!
Remember that the process takes two steps:
- Compute the SHA-256 hash of our firmware binary
- Verify the signature with the hash and the public key
First, we’ll need an implementation of the ECDSA algorithm. Remember, the first rule of crypto is “don’t roll your own crypto” so you’ll want a battle-tested library implemented by experts. There are a few good options, but we chose micro-ecc3 as we are most familiar with it and it is extremely small.
micro-ecc is extremely easy to add to a project: it consists of a single
.c (uECC.c) and a single .h (uECC.h) file. Add the source file to your list of sources, and
the header file to your include path.
We then add our public key to our firmware. micro-ecc expects our keys to be
“represented in standard format, but without the 0x04
prefix”4. We can use openssl to get the raw key
data:
(.venv) $ openssl ec -in private.pem -text -noout
read EC key
Private-Key: (256 bit)
priv:
4f:93:5d:53:41:82:21:91:11:e9:fe:9f:33:90:09:
28:da:d0:96:70:f7:5e:26:85:dc:ff:5d:bc:f8:6f:
5f:15
pub:
04:d0:e6:a7:a5:4e:33:0e:bb:d9:9e:e6:8f:59:ff:
b6:c1:19:76:28:60:88:16:6a:17:8b:7b:e0:66:cf:
7b:71:0d:f5:cc:95:76:22:ae:0e:a4:ef:49:bd:07:
2a:71:49:84:49:78:eb:34:e5:78:b3:a7:96:48:89:
7c:4f:d1:7e:a5
ASN1 OID: secp256k1
We take the pub section, remove the leading 04 bytes, and translate it to a
C array:
static const uint8_t PUBKEY[] = {
0xd0, 0xe6, 0xa7, 0xa5, 0x4e, 0x33, 0x0e, 0xbb, 0xd9, 0x9e, 0xe6, 0x8f, 0x59,
0xff, 0xb6, 0xc1, 0x19, 0x76, 0x28, 0x60, 0x88, 0x16, 0x6a, 0x17, 0x8b, 0x7b,
0xe0, 0x66, 0xcf, 0x7b, 0x71, 0x0d, 0xf5, 0xcc, 0x95, 0x76, 0x22, 0xae, 0x0e,
0xa4, 0xef, 0x49, 0xbd, 0x07, 0x2a, 0x71, 0x49, 0x84, 0x49, 0x78, 0xeb, 0x34,
0xe5, 0x78, 0xb3, 0xa7, 0x96, 0x48, 0x89, 0x7c, 0x4f, 0xd1, 0x7e, 0xa5
// 64 bytes
};
We can then verify that uECC is able to read the key correctly by calling
uECC_valid_public_key. We add the following code somewhere in our main
function at boot:
#include <micro-ecc/uECC.h>
const struct uECC_Curve_t *curve = uECC_secp256k1();
if (!uECC_valid_public_key(PUBKEY, curve)) {
printf("Public key is NOT valid\n");
} else {
printf("Public key is valid\n");
}
Note that we had to select the same curve we used to generate the key, here
secp2561k.
Last but not least, we need to verify the signature itself. This requires
computing the SHA-256, and using uECC_verify.
Many MCUs have hardware implementations of SHA-256, but in our case, we used a
software implementation provided by a library called CIFRA5. We only need
to add two files from CIFRA to our build: cifra/src/sha256.c and cifra/src/blockwise.c,
as well as two paths to our include path: cifra/src and cifra/src/ext.
With that, I implemented a simple sha256 wrapper function which computes the hash of a given buffer:
static void prv_sha256(const void *buf, uint32_t size, uint8_t *hash_out)
{
cf_sha256_context ctx;
cf_sha256_init(&ctx);
cf_sha256_update(&ctx, buf, size);
cf_sha256_digest_final(&ctx, hash_out);
}
Note that cf_sha256_update can be called mutliple times, so if you are
validating a binary that isn’t memory mapped you can calculate the hash
iteratively rather than all at once. For example, here’s an implementation that
reads from a POSIX file:
static void prv_sha256(FILE *fp, uint8_t *hash_out)
{
#define READ_BUF_SZ 128
static uint8_t read_buf[READ_BUF_SZ] = {0};
cf_sha256_context ctx;
cf_sha256_init(&ctx);
fseek(fp, sizeof(image_hdr_t), SEEK_SET);
size_t readsize = 0;
while ((readsize = fread(read_buf, 1, READ_BUF_SZ, fp)) > 0) {
cf_sha256_update(&ctx, read_buf, readsize);
}
cf_sha256_digest_final(&ctx, hash_out);
}
Given a hash and a public key, we can then validate the signature found in our
imaget_hdr_t:
if (!uECC_verify(PUBKEY, hash, CF_SHA256_HASHSZ, hdr->ecdsa_sig, curve)) {
printf("Signature is NOT valid\n");
} else {
printf("Signature is valid\n");
}
Putting it all together, here’s our function to verify the signature of a given image:
int image_check_signature(image_slot_t slot, const image_hdr_t *hdr) {
void *addr = (slot == IMAGE_SLOT_1 ? &__slot1rom_start__ : &__slot2rom_start__);
addr += sizeof(image_hdr_t);
uint32_t len = hdr->data_size;
uint8_t hash[CF_SHA256_HASHSZ];
prv_sha256(addr, len, hash);
const struct uECC_Curve_t *curve = uECC_secp256k1();
if (!uECC_valid_public_key(PUBKEY, curve)) {
return -1;
}
if (!uECC_verify(PUBKEY, hash, CF_SHA256_HASHSZ, hdr->ecdsa_sig, curve)) {
return -1;
}
return 0;
}
We can invoke that function as part of our OTA process in
loader_shell_commands.c:
@@ -32,13 +32,19 @@
return -1;
}
shell_put_line("Validating image");
// Check & commit image
if (image_validate(IMAGE_SLOT_2, hdr)) {
shell_put_line("Validation Failed");
return -1;
};
+ shell_put_line("Checking signature");
+ if (image_check_signature(IMAGE_SLOT_2, hdr)) {
+ shell_put_line("Signature does not match");
+ return -1;
+ };
+
shell_put_line("Committing image");
if (dfu_commit_image(IMAGE_SLOT_2, hdr)) {
shell_put_line("Image Commit Failed");
Running our firmware, we can watch it all happen over serial:
$ telnet localhost 4445
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Bootloader started
Booting slot 1
Shared memory uinitialized, setting magic
Loader STARTED - version 1.0.0 (19dcbe5)
Booting slot 2
App STARTED - version 1.0.1 (19dcbe5) - CRC 0x1b11019d
shell> dfu-mode
Rebooting into DFU mode
Bootloader started
Booting slot 1
Loader STARTED - version 1.0.0 (19dcbe5)
Entering DFU Mode
shell> do-dfu
Starting update
Writing data
Validating image
Checking signature <----- **This is our new log line**
Committing image
Rebooting
Bootloader started
Booting slot 1
Loader STARTED - version 1.0.0 (19dcbe5)
Booting slot 2
App STARTED - version 1.0.1 (19dcbe5) - CRC 0x1b11019d
shell>
The full example is available on Github at interrupt@fwup-signing.
Note: While researching this post I came across Tinycrypt, an open source library maintained by Intel. Tinycript combines micro-ecc with additional primitives such as SHA-256. If I were to do it again, I would look into using tinycript rather than two separate libraries.
Additional Considerations
Key Storage
Your firmware signing process is only as secure as your key storage mechanism. Since the private key can be used to create build that look like they are coming from you, it is important to keep it private.
Your key storage system needs to:
- Put the key out of reach of most employees / partners
- Make it easy to sign a release build you actually want signed
- Be resilient. If you lose the key, you won’t be able to distribute additional firmware updates.
Unfortunately, this is not easy to do. Many people simply store their private key alongside their source code, in their git repoitory. Do not do this! It fails to meet rule (1) and is considered poor practice (in fact, Github will warn you if it detects something that looks like a private key in your repository).
Instead, it is better to store your key alongside your CI system and let it handle signing of release builds. Make sure only select users have access to the private key, by restricting permissions in your CI system. Most systems support this use case one way or another. For example, CircleCI offers Restricted Contexts which can be set up such that only administrator level users can see a private key, and their approval is needed to trigger a release build.
You may take key storage a step further and use a specialized Secrets Management Systems such as Hashicorp Vault6 or Amazon Key Management Service7. These are systems that offer secure secret storage with complex access control lists and API integrations.
Last but not least, make sure to securely store a backup of your keys somewhere. Whether on paper, on a smartcard, or on an airgapped system, your offline backups should be inaccessible most of the time.
Key Rotation
In the event a private key gets compromised, we need the ability to rotate it. This involves generating a new public / private key pair, and updating the public key hardcoded in the firmware. Thankfully, this is a relatively simple process with our current architecture. Since the public key is hardcoded in updateable components, we simply update them!
Here are the step-by-step details:
- Generate a new private/public key pair with
openssl - Hardcode the new public key in our Updater and Loader firmware
- Build a new Updater, sign it with the old private key
- Build a new Loader, sign it with the new private key
- Build a new Application, sign it with the new private key
- Load the new Updater on your devices. Because it is signed with the old key, the current Loader will accept it.
- Load the new Loader on your devices via the new Updater
- Load the new Application on your devices via the new Loader
As you see, this is the same process we’d use to update the Loader for any other reason.
Development Keys
Since we’ve taken great pains to make sure our engineers do not have direct access to the key, how can we enable them to load custom firmware images on their development devices?
We need a set of development keys which are used for that use case only. All development systems have the development public key hardcoded in rather than the production key. The development private key can only be used to sign firmware for these development systems, so it isn’t sensitive and can be stored in the repository.
Production systems should continue to use the production key, and will not accept firmware signed with the development key. In the event we want to load a development build on a production system - say, for debugging - we load a special Loader on it which is signed with the production key but contains the development key. This special Loader must be safeguarded: it downgrades verification on a production system to an insecure key.
Closing
Hope this post has convinced you that firmware signing can be implemented relatively simply, and that there are important benefits to doing so.
There is much more we could do to secure our firmware update process. For example, we should make sure we lock our flash when our application is running to avoid an bug in our firmware being used to modify it. We also should consider a more robust mechanism to “unlock” than our special Loader with development keys.
These will perhaps be topics for future blog posts :-).
As always, we’d love to hear from you. How do you secure your firmware update process? Let us know! And if you see anything you’d like to change, don’t hesitate to submit a pull request or open an issue on Github
Interested in learning more device firmware update best practices? Watch this webinar recording
See anything you'd like to change? Submit a pull request or open an issue on our GitHub
References
François Baldassari has worked on the embedded software teams at Sun, Pebble, and Oculus. He is currently the CEO of Memfault.