Failure is Always an Option - Adventures with Gitea on RKE2 + MetalLB
I wanted to get a Gitea installation up and running, but ran into a roadblock - I want to run Git on port 22, and my Kubernetes nodes are already running SSH on port 22.
The solution is "simple" - run MetalLB, bind SSH to port 22 on $someIP and bind the ingress controller to the same IP, right? Well, it didn't turn out to be that simple.
First off, I moved to RKE2 instead of K3s, as it became clear that I'd probably need to disable KlipperLB and switch from Traefik to Nginx Ingress controller to do this. At some point you're doing too many big changes to the environment to keep it. Maybe that was a dumb decision? We'll find out.
The first task is installing MetalLB, which pretty much consists of following the official installation instructions:
then creating some IP ranges:
wings:metallb/ (main) $ cat configmap.yaml apiVersion: v1 kind: ConfigMap metadata: namespace: metallb-system name: config data: config: | address-pools: - name: default protocol: layer2 addresses: - 10.1.2.100-10.1.2.110 wings:metallb/ (main) $ kubectl apply -f configmap.yaml
Exposing Nginx Ingress Controller via MetalLB
Append the following to
/var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx.yaml to enable the external service for Nginx (thanks for this step go to a GitHub thread about turning off this feature, not turning it on! 🚀)
It should bind to a MetalLB IP:
We'll deploy Gitea with an ingress using Helm.
memcached: enabled: true postgresql: enabled: true mysql: enabled: false mariadb: enabled: false ingress: enabled: true annotations: cert-manager.io/cluster-issuer: letsencrypt-prod hosts: - host: git.windowpa.in paths: - path: / pathType: Prefix tls: - secretName: tls-gitea-ingress hosts: - git.windowpa.in gitea: config: APP_NAME: "Windowpa.in Git service"
then use Helm to install Gitea:
helm repo add gitea-charts https://dl.gitea.io/charts/ helm repo update helm install gitea gitea-charts/gitea -f values.yaml -n gitea --create-namespace
We need to create a DNS record for git.windowpa.in that points at the MetalLB IP. Easy enough - first we'll create an A record for emerald-ingress.windowpa.in pointing to 10.1.2.100, then create a CNAME for git.windowpa.in pointing at emerald-ingress.windowpa.in. I haven't got any screenshots for this process, but hopefully you get the idea.
First login to Gitea
Now navigate to https://git.windowpa.in:
Register a user, then sign in as the default user - gitea_admin with password r8sA8CPHD9!bt6d - to get administrator access.
Open Site Administration -> User Accounts
Find your user, click Edit, turn it into an Administrator, then delete the gitea_admin user to prevent random people from being able to get administrator access to your instance.
We'll modify our Helm values slightly now, to disable registration on Gitea.
Now we do a Helm upgrade to apply the changes:
After a quick service outage, Gitea is back and registration is disabled.
We've got one more hurdle to clear before we can use Gitea properly - we need to tell the Nginx Ingress controller to forward port 22 to gitea-ssh.
Once again, modify
/var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx.yaml. This time, we're appending two lines:
--- apiVersion: helm.cattle.io/v1 kind: HelmChartConfig metadata: name: rke2-ingress-nginx namespace: kube-system spec: valuesContent: |- controller: publishService: enabled: true service: enabled: true tcp: 22: gitea/gitea-ssh:22
Save the file on each node, and after 30 seconds or so:
wings:gitea/ (main✗) $ kubectl get svc --all-namespaces | grep LoadBalancer kube-system rke2-ingress-nginx-controller LoadBalancer 10.43.70.128 10.1.2.100 80:31876/TCP,443:30998/TCP,22:30894/TCP 47m
We see the Ingress controller is now listening on port 22.
Testing it all out
Log into your user on Gitea, add your SSH keys, create a repository, check it out in Git and...
Welp, for some reason our SSH service is going to emerald02's native SSH server, not the Gitea service. Time for more debugging.
(I later went back and decided Gitea in a traditional VM would be a better fit for me anyways).