The k8s rabbithole - k3s and Longhorn (#4)

Having run into issues with RKE, I'm going to try K3s instead.

Go back to create a cluster and click the slider to RKE2/K3s.

Select K3s from the dropdown.

Create the cluster, then provision machines as before (but note it's a different command):

Click "insecure", then copy and paste the commands to each node.

wings@k8s-1:~$ curl --insecure -fL https://10.1.1.100/system-agent-install.sh | sudo  sh -s - --server https://10.1.1.100 --label 'cattle.io/os=linux' --token dm7q68w42pbh6lpw2vdmxwtkjc7ckmph5vdzln8rbdbrmf64fl2qsd --ca-checksum 3708a09f524afc306737f7fc2f807db31e719d9727d94bf353971cbededbd78d --etcd --controlplane --worker
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 23735    0 23735    0     0  2317k      0 --:--:-- --:--:-- --:--:-- 2317k
[INFO]  Label: cattle.io/os=linux
[INFO]  Role requested: etcd
[INFO]  Role requested: controlplane
[INFO]  Role requested: worker
[INFO]  Using default agent configuration directory /etc/rancher/agent
[INFO]  Using default agent var directory /var/lib/rancher/agent
[INFO]  Determined CA is necessary to connect to Rancher
[INFO]  Successfully downloaded CA certificate
[INFO]  Value from https://10.1.1.100/cacerts is an x509 certificate
[INFO]  Successfully tested Rancher connection
[INFO]  Downloading rancher-system-agent from https://10.1.1.100/assets/rancher-system-agent-amd64
[INFO]  Successfully downloaded the rancher-system-agent binary.
[INFO]  Generating Cattle ID
[INFO]  Successfully downloaded Rancher connection information
[INFO]  systemd: Creating service file
[INFO]  Creating environment file /etc/systemd/system/rancher-system-agent.env
[INFO]  Enabling rancher-system-agent.service
[INFO]  Starting/restarting rancher-system-agent.service

After a few minutes... success! We have a cluster.

Download the kubeconfig (the button in the top right, to the left of the search button).

Then copy it into place and run our first kubectl command...

wings@trevor ~ % cp ~/Downloads/sandbox.yaml ~/.kube/config
wings@trevor ~ % kubectl get nodes
NAME    STATUS   ROLES                              AGE     VERSION
k8s-1   Ready    control-plane,etcd,master,worker   5m15s   v1.21.6+k3s1
k8s-2   Ready    control-plane,etcd,master,worker   2m45s   v1.21.6+k3s1
k8s-3   Ready    control-plane,etcd,master,worker   2m29s   v1.21.6+k3s1
k8s-4   Ready    control-plane,etcd,master,worker   3m1s    v1.21.6+k3s1

Install Longhorn using the charts...

And open the Longhorn panel to see the state of our cluster.

It looks like we don't have a lot of room! Let's fix that. I'll add 50GB disks to each node:

wings@k8s-1:~$ sudo mkfs.ext4 /dev/sdb && sudo mkdir /mnt/brick1/ && sudo mount /dev/sdb /mnt/brick1
mke2fs 1.45.5 (07-Jan-2020)
Creating filesystem with 13107200 4k blocks and 3276800 inodes
Filesystem UUID: 937bd24b-00c7-44aa-a419-1823c09a95a7
Superblock backups stored on blocks:
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
	4096000, 7962624, 11239424

Allocating group tables: done
Writing inode tables: done
Creating journal (65536 blocks): done
Writing superblocks and filesystem accounting information: done

Edit disks on k8s-1 and add our new disk:

After enabling the disk...

we have storage!

Rinse and repeat for each node...

(and as a sidenote ensure each node will automount its disk)

root@k8s-2:~# echo "/dev/sdb /mnt/brick1 ext4 defaults 0 0" >> /etc/fstab

All done! So much room for activities.

Let's install MetalLB using Helm - https://metallb.universe.tf/installation/

wings@trevor ~ % helm repo add metallb https://metallb.github.io/metallb
"metallb" has been added to your repositories
wings@trevor ~ % helm install metallb metallb/metallb

W1123 22:07:33.787515    7588 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W1123 22:07:33.802226    7588 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W1123 22:07:34.102990    7588 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W1123 22:07:34.103007    7588 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
NAME: metallb
LAST DEPLOYED: Tue Nov 23 22:07:33 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
MetalLB is now running in the cluster.
WARNING: you specified a ConfigMap that isn't managed by
Helm. LoadBalancer services will not function until you add that
ConfigMap to your cluster yourself.

Nice! We'll give it 10.1.50.x...

wings@trevor k8s % cat metallb.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: default
  name: metallb
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 10.1.50.1-10.1.50.250
wings@trevor k8s % kubectl apply -f metallb.yaml
configmap/metallb created

In theory MetalLB is now configured, to test it out, let's get Postgres Operator going and create a loadbalancer:

git clone https://github.com/zalando/postgres-operator.git
cd postgres-operator
wings@trevor postgres-operator % helm install postgres-operator ./charts/postgres-operator

NAME: postgres-operator
LAST DEPLOYED: Tue Nov 23 22:16:16 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
To verify that postgres-operator has started, run:

  kubectl --namespace=default get pods -l "app.kubernetes.io/name=postgres-operator"
wings@trevor postgres-operator %   kubectl --namespace=default get pods -l "app.kubernetes.io/name=postgres-operator"
NAME                                 READY   STATUS              RESTARTS   AGE
postgres-operator-58b75c5587-tqmgj   0/1     ContainerCreating   0          4s
wings@trevor postgres-operator %   kubectl --namespace=default get pods -l "app.kubernetes.io/name=postgres-operator"
NAME                                 READY   STATUS    RESTARTS   AGE
postgres-operator-58b75c5587-tqmgj   1/1     Running   0          28s

And install the operator UI:

wings@trevor postgres-operator % helm install postgres-operator-ui ./charts/postgres-operator-ui

NAME: postgres-operator-ui
LAST DEPLOYED: Tue Nov 23 22:17:29 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
To verify that postgres-operator has started, run:

  kubectl --namespace=default get pods -l "app.kubernetes.io/name=postgres-operator-ui"
wings@trevor postgres-operator % kubectl get pod -l app.kubernetes.io/name=postgres-operator-ui


NAME                                    READY   STATUS              RESTARTS   AGE
postgres-operator-ui-58644cfcff-cdvp9   0/1     ContainerCreating   0          7s
wings@trevor postgres-operator % kubectl get pod -l app.kubernetes.io/name=postgres-operator-ui

NAME                                    READY   STATUS    RESTARTS   AGE
postgres-operator-ui-58644cfcff-cdvp9   1/1     Running   0          43s

And create a port forward:

wings@trevor postgres-operator % kubectl port-forward svc/postgres-operator-ui 8081:80

Forwarding from 127.0.0.1:8081 -> 8081
Forwarding from [::1]:8081 -> 8081

Now go to http://localhost:8081 in a browser.

Create a test cluster called "test" and enable loadbalancers.

And click Create cluster

Had to try again after disabling one of the default StorageClasses (enabling longhorn and disabling the stock one)

Got stuck here, possibly something wrong with my lab.