Fixing Rook Ceph OSD paths when they change Rook paths changing I rebooted monstar and ambellina and the paths for all the OSDs changed. This broke Rook in half with no clear way to fix it. Here’s the fix. Now, ideally you don't break enough OSDs to actually break your cluster, but if you'
CubeFS: A crash course A new distributed filesystem caught my interest recently - CubeFS. If you know anything about me, you'll know I'm a massive distributed storage nerd; with around 2+ petabytes of storage in my lab (RiffLabs Perth) bound together at various times by distributed filesystems such as MooseFS
hello again, world! Haven't been too active lately on this blog. Extremely busy at Protocol Labs working on tech like Estuary.tech and Filecoin. Expect some updates soon though! Latest news is that the lab has moved to a dedicated location at a friend's house, with redundant pathways to
A failed attempt at rescuing Rocky Linux 8 from a failed DNF upgrade... I recently had a DNF upgrade go very, very badly on Rocky Linux. I lost power at a pretty inopportune time, and well, it broke everything including DNF. If you have a borked Rocky Linux install and are looking for help with it, maybe this will help a bit. MAKE
Failure is Always an Option - Adventures with Gitea on RKE2 + MetalLB Hey there! I wanted to get a Gitea installation up and running, but ran into a roadblock - I want to run Git on port 22, and my Kubernetes nodes are already running SSH on port 22. The solution is "simple" - run MetalLB, bind SSH to port
Getting started with AWX - A beginner's guide This post aims to be an introduction to AWX [https://github.com/ansible/awx] - a tool from Ansible that aims to make using Ansible a little easier. AWX can be a complicated subject, and precious little exists on the wider web documenting how to use it, so I thought
Automated provisioning (#4) - The Foreman, part 4 - Foreman and Proxmox Integration I've now got a basic setup for deploying machines via Foreman in Proxmox, but it's a little click-intensive. What if I could create the machine in Foreman and it automatically created the machines in Proxmox? Enter foreman-fog-proxmox [https://github.com/theforeman/foreman_fog_proxmox]. Somewhat strangely
Featured GPU Mining on Proxmox for Fun and Profit I've got two machines running HiveOS at the moment, and a third machine that has a GPU but isn't yet mining. By the end of this post I hope to have at least one mining Ethereum via HiveOS, running in VMs on Proxmox. This will allow
Automated provisioning (#3) - The Foreman, part 3 - Deploying a test machine It's time to test out Foreman properly, and deploy a machine. Preparation Navigate to Hosts -> Provisioning Setup -> Operating Systems. Opening the Operating Systems panel in Foreman.We see there's already an operating system - Debian 11.3 - which was created/discovered
Automated provisioning (#2) - The Foreman, part 2 - Defining and creating a subnet, DHCP proxy In this post I'll be creating a subnet and attempting to setup a DHCP server / "DHCP Proxy" for Foreman to use. It'll exist on the same server as the Foreman host. In Foreman, I navigated to Infrastructure -> Subnets, then clicked Create Subnet.
Virtualised MooseFS - Chunkserver on Proxmox with Physical Disks This'll be a quick one. I've recently bought a new machine from a friend - an Athlon 3000G, Fractal Design 302-based NAS. It has slots for 6x 3.5" hard drives, of which 3 came filled with 14TB drives. A white storage server, sitting next
Mastodon adventures I've joined Mastodon, apparently just before "the big announcement" (spoiler: Elon Musk buying Twitter). Nothing against Twitter! I'll still be there [https://twitter.com/gnomethrower] as @gnomethrower, but I'll also be on Mastodon via aus.social [https://aus.social] - you can
Upgrading to Ubuntu 22.04 - Jammy Jellyfish! Hi there! Quick post about Ubuntu 22.04. I've been running Ubuntu Server on and off for many years now, and have a few Ubuntu 20.04 boxes lying around. I thought I'd "live dangerously" and upgrade them to the still-quite-young Ubuntu 22.04
Cleaning up the lab (#3) - less highly available DHCP Something I don't find myself saying often: time for a downgrade. I contacted the Technitium [https://github.com/TechnitiumSoftware/DnsServer/issues/197] folks about my issues with DHCP, and they helped me debug things a bit further. It led to me installing my first fresh Ubuntu 22.04
PLUG AV Hack Day 2022-04-24 Today I attended a hack session for Perth Linux Users Group, focusing on AV setup and networking. Notes on things that have come up: * Backing up avmaster (Node #1), backing up some old PLUG footage. Initially tried copying over powerline ethernet, then we proposed sneakernet in ther form of a
Automated provisioning - The Foreman in 2022 I've got a need (okay, more of a want) to be able to deploy various operating systems in my home lab easily without too much fuss, to be able to try out various operating systems and distributions and so on more easily and with less hand-holding. Having an
Cleaning up the lab (#2) - highly available DHCP (in theory) Friday's shenanigans [https://raptorswithhats.com/cleaning-up-the-lab-time-for-self-hosted-dns/] started innocently enough - "I'll just rip out the DHCP server and run my own, and completely restructure the subnets, and..." as it started, soon led to a rabbithole. Where the last post left off, my "simple&
Migrating away from Ghost(Pro) (#1) - forward looking statements At some point recently I decided to migrate away from Ghost(Pro). I've been using the service without issue for a year - a year where I had precious little time or energy to focus on hosting a blog, maintaining any sort of permanent infrastructure, or indeed on
Cleaning up the lab - Subnet mask fun, self-hosted DNS and DHCP. My "home lab" is really just my apartment's network, running in a single VLAN. I've considered splitting into multiple VLANs before, but haven't found a proper reason to do so just yet. Today I made a few major changes to my home
the k8s rabbithole (#6) - Starting over I'm going to take the opportunity to wipe the Kubernetes cluster, which I think might be worth doing regularly anyways to prove it can be done. Ideally everything difficult-to-create or "big" should be stored in MooseFS. I went through the same procedure as before to create
the k8s rabbithole (#5) - Rancher v2.6 with DNS-01 TLS/SSL Certificates (via Let's Encrypt and Cloudflare) Having learned a bit about Kubernetes, Rancher and K3S, I've decided I don't like how I setup Rancher originally. I'd like to set it up on a single-node k3s installation, with dns-01 based TLS certificates [https://github.com/rancher/rancher/issues/26850], at rancher.
the k8s rabbithole (#4) - k3s and Longhorn Having run into issues with RKE, I'm going to try K3s instead. Go back to create a cluster and click the slider to RKE2/K3s. Select K3s from the dropdown. Create the cluster, then provision machines as before (but note it's a different command): Click "
the k8s rabbithole (#3) - Dedicated Rancher host I deployed my first cluster and... it didn't quite work, because etcd couldn't bind to its port. Drat. Likely this is because I deployed Rancher to k8s-1, then attempted to deploy a cluster to k8s-1/2/3. So, let's try again, but with a
the k8s rabbithole (#2) - Installing Docker and Rancher Now it's time to get Docker installed. There are plenty of ways to do this, but luckily Rancher provides a handy script that will install it in one line [https://rancher.com/docs/rancher/v2.6/en/installation/requirements/installing-docker/] . Install Docker... wings@k8s-1:~$ curl https://releases.rancher.
k8s Featured the k8s rabbithole - Reviving my homelab Today I revived my home lab, spinning my small Proxmox cluster back up and cleaning things up, including turning off some unwanted hacks on the machines. I removed all the old cruft and started with a shiny empty cluster, ready for new experiments. The lab consists of two NUCs (Gigabyte
