In this article, I will walk you through the process of automating the issuance and renewal of certificates provided by Let’s Encrypt (and other services) for Kubernetes Ingress using the cert-manager add-on. But first, let me start with a brief introduction to the problem.
HTTPS in a nutshell
The HTTP protocol developed in the early 1990s has become an integral part of our daily life. Today, we cannot live a single day without it. However, it does not even provide a basic level of security when exchanging information between the user and the web server. That is when HTTPS (“S” means “secure” here) comes to the rescue. In HTTPS, the exchanged data is encrypted using SSL/TLS — that family of protocols has proven itself well in protecting privacy and data integrity and is being actively promoted by the industry.
For example, Google has been calling for “HTTPS everywhere” since 2014. When prioritizing search results, it takes into account whether sites use secure, encrypted connections. All this propaganda affects ordinary users as well: modern browsers warn their users about insecure connections and invalid SSL certificates.
A certificate for a personal website might cost tens of dollars. However, buying it is not always justified. Fortunately, since late 2015, there is a free alternative in the form of Let’s Encrypt (LE) certificates. This nonprofit authority was created by Mozilla enthusiasts to make internet-wide hassle-free encryption a reality.
The certificate authority issues domain-validated certificates (the most basic ones available on the market) valid for 90 days, and it is also possible to obtain a so-called wildcard certificate for several subdomains.
The algorithms described in the Automated Certification Management Environment (ACME) protocol (designed specifically for Let’s Encrypt) are used to obtain a certificate. With it, the agent can prove control of the domain either by provisioning an HTTP resource (the so-called “HTTP-01 challenge”) or DNS records (“DNS-01 challenge”) — more information about them you may find below.
Cert-manager
Cert-manager is a Kubernetes-native certificate management controller consisting of a set of CustomResourceDefinitions (hence the restriction on the minimum supported version of K8s, v1.12) for configuring CA (certificate authorities) and obtaining certificates. The installation of CRDs in a cluster is straightforward and boils down to applying a single YAML file:
kubectl create ns cert-manager
kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v0.13.0/cert-manager.yaml
(You can also install it with Helm).
Before being able to request certificates, you must create CA resources: Issuer
or ClusterIssuer
. They are used for signing CSRs (certificate requests). The difference between these resources is that ClusterIssuer
is non-namespaced and can be used in multiple namespaces:
Issuer
is used within a single namespace only;ClusterIssuer
is a global cluster object.
Using cert-manager
Method #1. Self-signed certificate
Let’s start with the simplest case — requesting a self-signed certificate. It is quite common. For example, you can use it in your K8s cluster for testing environments dynamically created for developers needs. It can be also useful in the case of the external load balancer that terminates SSL traffic.
The Issuer
resource would look like this:
apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
name: selfsigned
spec:
selfSigned: {}
To issue a certificate, you have to define the Certificate
resource that determines the issuance (see the issuerRef
section below) and the location of the private key (the secretName
field). Then you need to invoke that key in the Ingress (note the tls
section in the spec
field):
---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
name: selfsigned-crt
spec:
secretName: tls-secret
issuerRef:
kind: Issuer
name: selfsigned
commonName: "yet-another.website"
dnsNames:
- "yet-another.website"
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: app
spec:
tls:
- hosts:
- "yet-another.website"
secretName: tls-secret
rules:
- host: "yet-another.website"
http:
paths:
- path: /
backend:
serviceName: app
servicePort: 8080
The certificate will be issued in a few seconds after these resources are added to the cluster. You can see the confirmation in the output of the command:
kubectl -n app describe certificate selfsigned-crt
...
Normal GeneratedKey 5s cert-manager Generated a new private key
Normal Requested 5s cert-manager Created new CertificateRequest resource "selfsigned-crt-4198958557"
Normal Issued 5s cert-manager Certificate issued successfully
Looking at the secret resource itself, you will see:
- the
tls.key
private key, - the
ca.crt
root certificate, - and the self-signed
tls.crt
certificate.
You can browse the contents of these files using the openssl
utility:
kubectl -n app get secret tls-secret -ojson | jq -r '.data."tls.crt"' | base64 -d | openssl x509 -dates -noout -issuer
notBefore=Feb 10 21:01:59 2020 GMT
notAfter=May 10 21:01:59 2020 GMT
issuer=O = cert-manager, CN = yet-another.website
It is worth noting that clients consuming certificates issued in such a way (i.e., using the self-signed issuer) will not trust them. The reason is simple: this issuer type does not have a CA (see the note on the project website).
To avoid this, specify the path to the secret file containing ca.crt
in the Certificate
resource. For example, you can use the corporate CA. This way, you will be able to sign certificates issued for your Ingress with a key that is already in use by other server services/information systems.
Method #2. Let’s Encrypt certificate with the HTTP validation
As I mentioned before, there are two types of “challenges” available to prove the control of the domain, HTTP-01 and DNS-01.
The first approach (HTTP-01) involves deploying a tiny web server as a separate deployment. It will be serving some information at the http:///.well-known/acme-challenge/
URL per request of the certification server. Therefore, this method implies the accessibility of Ingress from the outer world via port 80 and the publicity of the domain’s DNS record.
The second challenge (DNS-01) makes sense if there is an API you can use to change the DNS records of your domain. The Issuer uses these tokens to create TXT records for your domain. Then, ACME server gets these records during confirmation. Let’s Encrypt easily integrates with various DNS providers, including CloudFlare, AWS Route53, Google CloudDNS, and others (as well as with the LE’s own DNS implementation, acme-dns).
Note: Let’s Encrypt imposes fairly strict limits on requests to ACME servers. To avoid unnecessary load on LE’s production environment, we recommend using the letsencrypt-staging certificate for testing (the difference is in the ACME server only).
So, let’s describe the resources:
apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
name: letsencrypt
spec:
acme:
server: https://acme-staging-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: letsencrypt
solvers:
- http01:
ingress:
class: nginx
---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
name: le-crt
spec:
secretName: tls-secret
issuerRef:
kind: Issuer
name: letsencrypt
commonName: yet-another.website
dnsNames:
- yet-another.website
Note that we use the staging server in acme
’s server
field for our Issuer
. You can replace it with the production one later.
Let’s apply this configuration and trace the entire process of obtaining the certificate:
1. The creation of a Certificate
leads to the emergence of a new CertificateRequest
resource:
kubectl -n app describe certificate le-crt
...
Created new CertificateRequest resource "le-crt-1127528680"
2. In its description there is a notification about an Order
creation:
kubectl -n app describe certificaterequests le-crt-1127528680
…
Created Order resource app/le-crt-1127528680-1805948596
3. The Order
contains the description of parameters of the validation and its current status. The validation is performed by the Challenge
resource:
kubectl -n app describe order le-crt-1127528680-1805948596
…
Created Challenge resource "le-crt-1127528680-1805948596-1231544594" for domain "yet-another.website"
4. And finally, there is information about the status of the validation itself in resource details:
kubectl -n app describe challenges le-crt-1127528680-1805948596-1231544594
...
Reason: Successfully authorized domain
...
Normal Started 2m45s cert-manager Challenge scheduled for processing
Normal Presented 2m45s cert-manager Presented challenge using http-01 challenge mechanism
Normal DomainVerified 2m22s cert-manager Domain "yet-another.website" verified with "http-01" validation
The certificate will be issued in less than a minute if all the prerequisites are met: the domain is accessible from the outer world, the rate limits on the LE side are respected, and so on. If the issuance was successful, you should see the message “Certificate issued successfully” in the output of the describe certificate le-tls
command.
Now you can safely change the ACME server address to the production one (https://acme-v02.api.letsencrypt.org/directory
) and re-issue valid certificates signed by Let's Encrypt Authority X3
instead of Fake LE Intermediate X1
.
But first, you have to delete the Certificate
resource. Otherwise, the issuance procedure will not start because the certificate already exists, and it is valid. Deleting a secret would immediately result in invalidation of the certificate with the following message in the output of the describe certificate
command:
Normal PrivateKeyLost 44s cert-manager Lost private key for CertificateRequest 'le-crt-613810377', deleting old resource
Now it is time to apply the production manifest for the Issuer
with the Certificate
described above (it has not changed):
apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
name: letsencrypt
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: letsencrypt
solvers:
- http01:
ingress:
class: nginx
After receiving the Certificate issued successfully
confirmation, let us check it out:
kubectl -n app get secret tls-secret -ojson | jq -r '.data."tls.crt"' | base64 -d | openssl x509 -dates -noout -issuer
notBefore=Feb 10 21:11:48 2020 GMT
notAfter=May 10 21:11:48 2020 GMT
issuer=C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
Method #3. Validating Wildcard LE over DNS
To make one step further we will issue a certificate for all subdomains of the site using another method of validation — the DNS one. We will use CloudFlare as our DNS provider to change the domain records we need.
First, let’s create a token to use the CloudFlare API:
1. Profile → API Tokens → Create Token.
2. Set access rights as follows:
- Permissions:
- Zone — DNS — Edit
- Zone — Zone — Read
- Zone Resources:
- Include — All Zones
3. Copy the token generated (for example, y_JNkgQwkroIsflbbYqYmBooyspN6BskXZpsiH4M
).
Create a Secret
resource containing the token and describe it in your Issuer
:
apiVersion: v1
kind: Secret
metadata:
name: cloudflare-api-token
type: Opaque
stringData:
api-token: y_JNkgQwkroIsflbbYqYmBooyspN6BskXZpsiH4M
---
apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
name: letsencrypt
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: letsencrypt
solvers:
- dns01:
cloudflare:
email: my-cloudflare-acc@example.com
apiTokenSecretRef:
name: cloudflare-api-token
key: api-token
---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
name: le-crt
spec:
secretName: tls-secret
issuerRef:
kind: Issuer
name: letsencrypt
commonName: yet-another.website
dnsNames:
- "yet-another.website"
- "*.yet-another.website"
(Do not forget to use a staging environment for testing!)
It is time to go through the domain ownership confirmation procedure:
kubectl -n app describe challenges.acme.cert-manager.io le-crt-613810377-1285319347-3806582233
...
Status:
Presented: true
Processing: true
Reason: Waiting for dns-01 challenge propagation: DNS record for "yet-another.website" not yet propagated
State: pending
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Started 54s cert-manager Challenge scheduled for processing
Normal Presented 53s cert-manager Presented challenge using dns-01 challenge mechanism
The TXT record will appear at your DNS dashboard:
… and after a while, the status will change to:
Domain "yet-another.website" verified with "dns-01" validation
Let’s make sure that the certificate is valid for all subdomains:
kubectl -n app get secret tls-secret -ojson | jq -r '.data."tls.crt"' | base64 -d | openssl x509 -dates -noout -text |grep DNS:
DNS:*.yet-another.website, DNS:yet-another.website
The validation over DNS is usually slow since most DNS providers have a so-called “propagation time” — a period showing how long it takes for an updated DNS record to become available on all DNS servers of the provider.
The ACME standard also supports a combination of both types of validation. You can use it to speed up obtaining a certificate for the main domain. In this case, the description of the Issuer
will have the following form:
apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
name: letsencrypt
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: letsencrypt
solvers:
- selector:
dnsNames:
- "*.yet-another.website"
dns01:
cloudflare:
email: my-cloudflare-acc@example.com
apiTokenSecretRef:
name: cloudflare-api-token
key: api-token
- selector:
dnsNames:
- "yet-another.website"
http01:
ingress:
class: nginx
If you apply this configuration, two Challenge
resources will be created:
kubectl -n app describe orders le-crt-613810377-1285319347
…
Normal Created 3m29s cert-manager Created Challenge resource "le-crt-613810377-1285319347-3996324737" for domain "yet-another.website"
Normal Created 3m29s cert-manager Created Challenge resource "le-crt-613810377-1285319347-1443470517" for domain "yet-another.website"
Method #4. Using special Ingress annotations
Besides creating certificates directly, you can use cert-manager’s ingress-shim component. It relieves you of the need to create Certificate
resources explicitly. The idea is to obtain a certificate automatically using the Issuer
specified in the special annotations of Ingress. Here is an example of the respective Ingress resource:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
cert-manager.io/cluster-issuer: letsencrypt
spec:
tls:
- hosts:
- "yet-another.website"
secretName: tls-secret
rules:
- host: "yet-another.website"
http:
paths:
- path: /
backend:
serviceName: app
servicePort: 8080
In this case, the availability of Issuer is enough to get the work done, meaning that we have to create a lesser number of entities.
Also, there is an obsolete kube-lego annotation — kubernetes.io/tls-acme: "true"
. The peculiarity about it is that when deploying cert-manager, the default Issuer
must be specified via Helm arguments (or by appending arguments of cert-manager’s deployment container).
Personally, we do not use these approaches (mentioned in Method #4) and cannot recommend them due to their opacity (and various associated problems). However, it’s good to mention them in the article to provide a fuller picture.
Key takeaways
We have learned how to obtain auto-renewable, self-signed, and free SSL certificates from Let’s Encrypt for website domains that are managed by Ingresses in the Kubernetes clusters.
The article provides example solutions for the most common problems we face. However, cert-manager features are not limited to those described above. On the project’s website, you can find examples of using it with other services. For example, you can use Vault as a certificate authority or set up external Issuers.
Comments