the k8s rabbithole (#5) - Rancher v2.6 with DNS-01 TLS/SSL Certificates (via Let's Encrypt and Cloudflare)

Having learned a bit about Kubernetes, Rancher and K3S, I've decided I don't like how I setup Rancher originally. I'd like to set it up on a single-node k3s installation, with dns-01 based TLS certificates, at, without exposing it to the outside world.

There's precious little guidance on how to set up Rancher 2.6 with Cloudflare/Let's Encrypt DNS-01 TLS/SSL certificates, so this post will serve as a tutorial of sorts for those lucky enough to stumble upon it, too, as well as notes for my own sanity.

First order of business - destroying the old docker-based Rancher installation.

wings@k8s-deployer:~$ nano
wings@k8s-deployer:~$ chmod +x
wings@k8s-deployer:~$ ./
Untagged: rancher/rancher:latest
Untagged: rancher/rancher@sha256:51658639bf2b28f4796e7daa04b0434e29fcf69ce062bf649462be77ae76cd2c
Deleted: sha256:ec97968cb7d3e447dd1b81af4ee1b65b2c087b2db5690697b3ee7e048d867071
Deleted: sha256:7ab8ac7561dfafeb43aba9c5e8ca1601a958295b179def4703f5915ffe1027b3
Deleted: sha256:a1f714becfe15885e5ba34fb715edd675eac6603f3beaaa846589be6bdd8f45d
Deleted: sha256:675da117dc091b2eddb947f2795f68f050256eb03a26bcf158b8ef828a7f92e2
Deleted: sha256:6c67d0c161c9c3fd8b6293ab0d54168353b1fa71a215f39734a07b66b05e772d
Deleted: sha256:7525ef9059e1de55b1059ea44f97dbf721b5c1068f56a5506ee1772fb1fe3322
Deleted: sha256:c6235b48a019cafcfd1fd5d51e90dd20e90e53193a6eb9e05b2676203d280888
Deleted: sha256:96042d6e8f19937ccd2f561f0f6a3f7e95dbc596e0566fce7e71cceddfff7a35
Deleted: sha256:2f18d02fba9ad2e3f22631e32130ad2153d149f0d1e5fdef2282ce95fc4f9b59
Deleted: sha256:535bf824bf55a4ac5ed6253ef3b9dbc0c58039fb49cd88a24e337b653a22761f
Deleted: sha256:379abdd6465bcd1d3ba0adc38ffc3ed3ed0a044c1e727a9fce53148fb4ccb102
Deleted: sha256:3ae2c7bd78216edef203f34ac1e00acada4fd276250a4be754104605f415c5ff
Deleted: sha256:befe51705744cda5d05bbbada2b0aab6bfeacbd33b777af54f33a02e0471c1f3
Deleted: sha256:a5d64011666f260fb5f141961b2f7c0b4056e804eeb4aa4627fd4a94fd650e24
Deleted: sha256:aef36c978c364672828ddf7baff9475e29c0a062b3696f2f2ad667c76dd52c31
Deleted: sha256:fd477dc87caf4ce174b84667fa35b1e8d373c4f9ddd64d139c916ce09f15aad5
Deleted: sha256:e5f0de5d87642426b5a757dc47937baafae721b37388ceec2ff891898b4e97be
Deleted: sha256:e9775a74a86b51447f927aea0f090463e681497b06df5b8cd2bf561bd35f14a8
Deleted: sha256:86e1a24b7189d95bd23bdfa81d1cafe8c0ff0200e40f07f83b2aaea312d73300

With that out of the way, we can install Rancher on Kubernetes directly (as opposed to Rancher Server/Rancher on K3s on Docker). Per, we'll go with a single node k3s installation, because it's easy and doesn't consume a ton of resources.

Easy as:

wings@k8s-deployer:~$ curl -sfL | sh 
wings@k8s-deployer:~$ mkdir ~/.kube/
wings@k8s-deployer:~$ sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config && sudo chown wings:wings ~/.kube/config
wings@k8s-deployer:~$ export KUBECONFIG=~/.kube/config
wings@k8s-deployer:~$ kubectl get nodes
NAME           STATUS   ROLES                  AGE   VERSION
k8s-deployer   Ready    control-plane,master   99s   v1.22.5+k3s1

And even that is far more complicated than it needs to be. For clarity, I installed k3s, then created a copy of its kubeconfig file in my local user, then finally did a "kubectl get nodes".

To get access to the rest of my tooling, I copied the kubeconfig to my M1, and changed the access URL to instead of

wings:~/ $ kubectl get nodes                                          [4:35:11]
NAME           STATUS   ROLES                  AGE     VERSION
k8s-deployer   Ready    control-plane,master   3m39s   v1.22.5+k3s1

And bam, my M1 can suddenly talk to the new K3S single-node "cluster".

Let's install Rancher's prerequisites:

wings:~/ $ helm repo add rancher-stable

"rancher-stable" has been added to your repositories
wings:~/ $ kubectl create namespace cattle-system                     [4:36:38]

namespace/cattle-system created
wings:~/ $ helm repo add jetstack          [4:36:58]

"jetstack" has been added to your repositories
wings:~/ $ helm repo update                                           [4:36:59]

Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "jetstack" chart repository
...Successfully got an update from the "rancher-stable" chart repository
...Successfully got an update from the "halkeye" chart repository
...Successfully got an update from the "common" chart repository
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈Happy Helming!⎈
wings:~/ $ helm install cert-manager jetstack/cert-manager \          [4:37:07]
  --namespace cert-manager \
  --create-namespace \
  --version v1.7.0 \
  --set installCRDs=true

Once that's done, install Rancher.

wings:~/ $ helm install rancher rancher-stable/rancher \                                            [8:04:45]
  --namespace cattle-system \
  --set \
  --set replicas=1 \
  --set ingress.tls.source=secret \
NAME: rancher
LAST DEPLOYED: Fri Jan 28 08:05:24 2022
NAMESPACE: cattle-system
STATUS: deployed
Rancher Server has been installed.

NOTE: Rancher may take several minutes to fully initialize. Please standby while Certificates are being issued, Containers are started and the Ingress rule comes up.

Check out our docs at

If you provided your own bootstrap password during installation, browse to to get started.

If this is the first time you installed Rancher, get started by running this command and clicking the URL it generates:

echo$(kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{.data.bootstrapPassword|base64decode}}')

To get just the bootstrap password on its own, run:

kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{.data.bootstrapPassword|base64decode}}{{ "\n" }}'

Happy Containering!

(Note, the output of the command above was slightly edited. Specifically, some backticks were removed :))

Now let's get a valid TLS/SSL certificate...

We need to create three files, roughly according to - I've left out sensitive information for security reasons, but this should give you a good idea as to what's required. These are cloudflare.yaml, issuer.yaml and certificate.yaml (the naming doesn't matter, but these names made sense to me :))


apiVersion: v1
kind: Secret
  name: cloudflare-api-token-secret
type: Opaque
  api-token: <REDACTED>


kind: ClusterIssuer
  name: letsencrypt-prod
      name: letsencrypt-prod
      - dns01:
              name: cloudflare-api-token-secret
              key: api-token


kind: Certificate
  name: tls-rancher-ingress
  namespace: cattle-system
  secretName: tls-rancher-ingress
    name: letsencrypt-prod
    kind: ClusterIssuer

Once those are created, apply them to create the Cloudflare secret, create a ClusterIssuer that can request and issue certificates, and finally request a certificate from the issuer. Whew!

# Create Cloudflare secret
kubectl apply -f cloudflare.yaml --namespace=cert-manager
# Create a ClusterIssuer
kubectl apply -f issuer.yaml
# Request a certificate
kubectl apply -f certificate.yaml

If you're quick, you can check out the DNS challenge as it happens:

wings:homelab-dns/ $ kubectl describe challenge -n cattle-system             Name:         tls-rancher-ingress-nb2bd-3206036943-634944680
Namespace:    cattle-system
Labels:       <none>
Annotations:  <none>
API Version:
Kind:         Challenge
  Authorization URL:
  Dns Name: 
  Issuer Ref:
    Kind:  ClusterIssuer
    Name:  letsencrypt-prod
  Key:     <redacted>
        API Token Secret Ref:
          Key:   api-token
          Name:  cloudflare-api-token-secret
  Token:         <redacted>
  Type:          DNS-01
  Wildcard:      false
  Presented:   true
  Processing:  true
  Reason:      Waiting for DNS-01 challenge propagation: DNS record for "" not yet propagated
  State:       pending
  Type    Reason     Age   From          Message
  ----    ------     ----  ----          -------
  Normal  Started    16s   cert-manager  Challenge scheduled for processing
  Normal  Presented  13s   cert-manager  Presented challenge using DNS-01 challenge mechanism

After 60-90 seconds, the DNS challenge is complete and the certificate is magically issued:

We have a working TLS certificate, and a secure HTTPS connection! Success!

While this blog post seems relatively straightforward, it wasn't all kittens and rainbows... lots of poking was needed to figure this out completely, and at one point I switched my Rancher host from Ubuntu to Debian to try to solve a mysterious DNS/Let's Encrypt issue (/etc/resolv.conf - maybe something for another post?)

Anyways, that's all the adventures for today.