Failure is Always an Option - Adventures with Gitea on RKE2 + MetalLB

Hey there!

I wanted to get a Gitea installation up and running, but ran into a roadblock - I want to run Git on port 22, and my Kubernetes nodes are already running SSH on port 22.

The solution is "simple" - run MetalLB, bind SSH to port 22 on $someIP and bind the ingress controller to the same IP, right? Well, it didn't turn out to be that simple.

First off, I moved to RKE2 instead of K3s, as it became clear that I'd probably need to disable KlipperLB and switch from Traefik to Nginx Ingress controller to do this. At some point you're doing too many big changes to the environment to keep it. Maybe that was a dumb decision? We'll find out.

MetalLB

The first task is installing MetalLB, which pretty much consists of following the official installation instructions:

MetalLB, bare metal load-balancer for Kubernetes

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.12.1/manifests/namespace.yaml
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.12.1/manifests/metallb.yaml

then creating some IP ranges:

wings:metallb/ (main) $ cat configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 10.1.2.100-10.1.2.110
wings:metallb/ (main) $ kubectl apply -f configmap.yaml

Exposing Nginx Ingress Controller via MetalLB

Append the following to /var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx.yaml to enable the external service for Nginx (thanks for this step go to a GitHub thread about turning off this feature, not turning it on! 🚀)

---
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-ingress-nginx
  namespace: kube-system
spec:
  valuesContent: |-
    controller:
      publishService:
        enabled: true
      service:
        enabled: true

It should bind to a MetalLB IP:

wings:metallb/ (main) $ kubectl get svc --all-namespaces | grep LoadBalancer
kube-system       rke2-ingress-nginx-controller                     LoadBalancer   10.43.70.128    10.1.2.100    80:31876/TCP,443:30998/TCP   5m37s

Deploying Gitea

We'll deploy Gitea with an ingress using Helm.

First, create values.yaml:

memcached:
  enabled: true

postgresql:
  enabled: true

mysql:
  enabled: false

mariadb:
  enabled: false

ingress:
  enabled: true
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
  hosts:
    - host: git.windowpa.in
      paths:
        - path: /
          pathType: Prefix

  tls:
    - secretName: tls-gitea-ingress
      hosts:
        - git.windowpa.in

gitea:
  config:
    APP_NAME: "Windowpa.in Git service"

then use Helm to install Gitea:

helm repo add gitea-charts https://dl.gitea.io/charts/
helm repo update
helm install gitea gitea-charts/gitea -f values.yaml -n gitea --create-namespace

DNS changes

We need to create a DNS record for git.windowpa.in that points at the MetalLB IP. Easy enough - first we'll create an A record for emerald-ingress.windowpa.in pointing to 10.1.2.100, then create a CNAME for git.windowpa.in pointing at emerald-ingress.windowpa.in. I haven't got any screenshots for this process, but hopefully you get the idea.

First login to Gitea

Now navigate to https://git.windowpa.in:

Register a user, then sign in as the default user - gitea_admin with password r8sA8CPHD9!bt6d - to get administrator access.

Open Site Administration -> User Accounts

Find your user, click Edit, turn it into an Administrator, then delete the gitea_admin user to prevent random people from being able to get administrator access to your instance.

Disabling registration

We'll modify our Helm values slightly now, to disable registration on Gitea.

gitea:
  config:
    APP_NAME: "Windowpa.in Git service"
    service:
      DISABLE_REGISTRATION: true

Now we do a Helm upgrade to apply the changes:

wings:gitea/ (main) $ helm upgrade gitea gitea-charts/gitea -f values.yaml -n gitea

After a quick service outage, Gitea is back and registration is disabled.

Exposing SSH

We've got one more hurdle to clear before we can use Gitea properly - we need to tell the Nginx Ingress controller to forward port 22 to gitea-ssh.

Once again, modify /var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx.yaml. This time, we're appending two lines:

---
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-ingress-nginx
  namespace: kube-system
spec:
  valuesContent: |-
    controller:
      publishService:
        enabled: true
      service:
        enabled: true
    tcp:
      22: gitea/gitea-ssh:22

Save the file on each node, and after 30 seconds or so:

wings:gitea/ (main✗) $ kubectl get svc --all-namespaces | grep LoadBalancer
kube-system       rke2-ingress-nginx-controller                     LoadBalancer   10.43.70.128    10.1.2.100    80:31876/TCP,443:30998/TCP,22:30894/TCP   47m

We see the Ingress controller is now listening on port 22.

Testing it all out

Log into your user on Gitea, add your SSH keys, create a repository, check it out in Git and...

wings:gitea/ (main✗) $ git clone git@git.windowpa.in:wings/test.git
Cloning into 'test'...
The authenticity of host 'git.windowpa.in (10.1.2.100)' can't be established.
ED25519 key fingerprint is SHA256:oMRjFyIlHPZwMN2MKRiJX8386/Q9/iujpESbyz1DyoU.
This host key is known by the following other names/addresses:
    ~/.ssh/known_hosts:172: 10.1.1.27
    ~/.ssh/known_hosts:202: emerald02
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'git.windowpa.in' (ED25519) to the list of known hosts.
git@git.windowpa.in's password:

Welp, for some reason our SSH service is going to emerald02's native SSH server, not the Gitea service. Time for more debugging.

(I later went back and decided Gitea in a traditional VM would be a better fit for me anyways).