This week I had the opportunity to implement mutual authentication in a project that I'm working on. I've always been interested in authentication and cryptography, however I'd never explored this topic in any detail before, let alone implemented it myself on both the client side and server side.
Mutual authentication, or mTLS, is simply a way for both a client and a server to authenticate each other when they open a connection or communicate. The server proves to the client, by means of a certificate, that it is who the client expects, and vice versa. This provides a greater level of security, and protects against most person-in-the-middle (PITM) attacks. For a much better and more detailed explanation see the page on the subject by Cloudflare, Mutual TLS authentication.
In the end, it didn't take me that long to get a workable production implementation in place, however a few of the guides I found online seemed to be lacking in small ways, and didn't work for my particular environment and use case. Due to this, I thought it would be worth writing this blog post to help others who may be struggling with similar issues to the ones I encountered.
I'll begin by discussing the topic of certificate generation. This covers the generation of a root certificate, a server certificate, and a client certificate. This post only considers the 'self-signed' case, however it would be very similar when using an actual, recgonised, certificate authority. Following this, I'll discuss web server configuration, specifically using Nginx. I'll then discuss client side implementation in Flutter. Next, I'll discuss some of the common problems that I came across while implementing mutual authentication, and finally I'll wrap up with a summary.
I lent heavily on the information in the blog posts by Muhammet GÜMÜŞ, namely mTLS Client Authentication with NGINX and Client Certificate Authentication (mTLS) with Flutter, and by Darshit Patel, How To Implement Two Way SSL With Nginx. They formed the basis for this blog post, I just made tweaks and improvements to suit my use case.
1. Generate root private key and certificate
-aes256 : This is the encryption algorithm to use when encrypting your private key. You will be required to provide a password. If you leave this parameter off, your private key will not be encrypted, this is NOT recommended. For more info, see this discussion.
2048 : This is the keysize to use for your private key. This is the minimum size you should use, as 1024 is now considered too weak (reference).
-days 365 : The validity of the new ceritificate, one year is a safe default, especially for self-signed certificates.
After running these two commands and entering the required information you will have a self-signed root certificate and its associated private key. Keep these, and the private key password safe, as they will be used to sign server and client certificates in the next steps.
2. Generate server private key and certificate
-sha256 : This specifies the signature digest algorithm to use for the new certificate. Without this, OpenSSL may default to a weak algorithm, see Common Problems for more information.
-CA and -CAkey : These two parameters specify the filename of the root certificate and its corresponding private key, that were generated in step 1.
Important: When entering the CSR information, make sure you enter a DIFFERENT Organisation than you used in the root certificate, in step 1. For example, you could use <COMPANY NAME> for the root certificate, and "<COMPANY NAME> - Server" for the server. If you use the same organisation, verification of the certificate will fail.
You should now have the private key and a newly generated server certificate.
3. Generate client private key and certificate
Important: Make sure you use a different Organisation name from your root certificate, as described in step 2.
At this stage you can verify the client, or server, certificate against the root certificate, by using:
This command will produce a failure if you generated your client certificate with the same Organisation name as your root certificate.
You should now have a private key and client certificate.
Web Server (Nginx) Configuration
Now that you have the required certificates and their private keys, the next step is to setup your webserver in order to utilize and validate them.
In this case I use Nginx, setup as a reverse proxy. It will forward authenticated incoming requests to a backend service to handle them. It's also possible to set this up in different ways depending on your requirements.
Below is a example of a minimal Nginx configuration file that you can use in order to implement mutual authenticattion. You will likely need to incorporate the relevant config options in your existing Nginx file. Make sure you copy server.crt, server.key, and ca.crt, to a location where Nginx can access it, e.g. under /etc/nginx/certs/.
Some of the options in this config were set using guidance from Mozilla, you can customise it however you like.
The most important configuration options for mutual authentication are:
- ssl_certificate <LOCATION OF SERVER CERTIFICATE>
- ssl_certificate_key <LOCATION OF SERVER PRIVATE KEY>
- ssl_password_file <LOCATION OF PRIVATE KEY PASSWORD FILE>
- ssl_client_certificate <LOCATION OF CLIENT CERTIFICATE AUTHORITY>
- ssl_verifiy_client on
- proxy_set_header VERIFIED $ssl_client_verify
- proxy_pass <URL_TO_FORWARD_TO>
Decrypting the server's private key
As you may have noticed above, we set the ssl_password_file config option to the path of a password file. This is necessary because when we created the server private key, we chose to encrypt it and provide a password. In order for Nginx to decrypt the private key, it needs this password.
There are various ways this can be done, but one of the simplest is to create a new file, e.g. ssl_passwords.txt and place the plaintext password for the server private key in it. Then copy it somewhere accessible to Nginx, and ensure you reflect its path in the config. More information on this subject can be found here.
Testing with cURL
At this point, you should be able to test your server configuration for Nginx by utilising cURL and providing the necessary certificates. This is described in more detail in this article.
This command will complete successfully if you've set everything up properly. Be sure to setup another server to handle the proxied request, which Nginx will pass to it (proxy_pass). If it does not work, review the above steps, and consult the Nginx debug error log for more detail on what the problem might be.
Client (Flutter) Configuration
I'll now cover how to implement the client side of mutual authentication in Dart/Flutter. I used the Dio library as it is an interesting project, and I found the implementation to be the easiest. It is discussed in more detail here. The process will be very similar in other languages. The general approach is to include the client certificate and private key in the client project, then read them in before use (providing the client key password), and finally instantiate a new HTTP client, which will be used for authenticated requests to the server.
Here I'll cover some of the problems that I came across while implementing mutual authentication between an Nginx reverse proxy and a Flutter client.
1. Weak Signature Digest Algorithm
As mentioned in the Generating Certificates section, if you leave off the -sha256 parameter in the openssl x509 command, you could run into trouble. This is due to the fact that, by default, openssl will use a signature algorithm that the latest version of Nginx considers weak, resulting in a verification error.
In my case, Nginx would fail with the following error:
[info] 28#28: *3 client SSL certificate verify error: (68:CA signature digest algorithm too weak) while reading client request headers, client: 172.29.0.1, server: localhost, request: "GET / HTTP/1.1", host: "localhost:443"
You don't have to use -sha256 here, des3 is also another acceptably secure option.
This error seems fairly common and is described in more detail here.
2. Nginx HTTP Response 495
This Nginx HTTP response code means there was an error with the SSL certificate. The error returned by Nginx is not very descriptive, even in the error logs. This code will be returned when Nginx fails to verify a client certificate for some reason.
In this case however, this error could mean that you generated your root and client certificates with the same Organisation, because of this Nginx considers it 'self-signed', as usually these two should not be the same.
[info] 21#21: *3 client SSL certificate verify error: (18:self signed certificate) while reading client request headers, client: 172.29.0.1, server: localhost, request: "GET / HTTP/1.1", host: "localhost:443"
This error also seems to be fairly common, as can be seen in this discussion.
In conclusion, a brief summary of all of the steps that I have discussed here, which are required in order to implement mutual authentication:
- Generate a root certificate and a private key.
- Generate server and client certificates and private keys.
- Configure Nginx to use the root and server certificates, and to verify client certificates in incoming requests.
- Implement reading in and sending the client certificate with HTTPS requests from the client
Thanks for reading. Any constructive feedback is most welcome 😊
There are various aspects of mutual authentication between a server and a client that I believe can be optimised further than I have done here. I plan to spend some time researching these topics, and perhaps writing corresponding blog posts, in the future.
- Use of the ECDSA signature algorithm over RSA when generating certificates. Reference: https://blog.cloudflare.com/ecdsa-the-digital-signature-algorithm-of-a-better-internet/
- A more secure storage of the server private key password. For example, this could be fetching the key via a secrets vault distributed via encrypted environmental variables.
- A more secure way of password storage and key and certificate use in the client, for example: fetching the client private key password from a secrets vault or remote config service such as Firebase.
- TLSv1.3 support. For some reason Flutter/Dio does not negotiate a connection properly when only TLSv1.3 is enabled in Nginx. Investigate why this is and remedy it.
- Using certificates that are signed by a non-self-signed root cerificate, e.g. LetsEncrypt. With special consideration paid to the validity period.
A sample Dockerfile for Nginx to help you on your way:
When running your Nginx container, use the following command if you need to debug:
And, add this to your Nginx config file to enable debug level logging: