A failed attempt at rescuing Rocky Linux 8 from a failed DNF upgrade...
I recently had a DNF upgrade go very, very badly on Rocky Linux. I lost power at a pretty inopportune time, and well, it broke everything including DNF.
If you have a borked Rocky Linux install and are looking for help with it, maybe this will help a bit. MAKE A BACKUP BEFORE YOU DO ANYTHING. Ahem. And be prepared to break your system even worse than it's currently broken.
Getting DNF working again was a fairly difficult task, which essentially boiled down to finding which half-installed package was killing it. As it turns out python3-hawkey was the culprit. I forget the exact steps to get it working again, but something like this:
wget https://download.rockylinux.org/pub/rocky/8/BaseOS/x86_64/os/Packages/p/python3-hawkey-0.63.0-8.1.el8_6.x86_64.rpm
sudo rpm -ivh python3-hawkey-0.63.0-8.1.el8_6.x86_64.rpm --force
sudo rpm --verify --all
sudo dnf updateOnce DNF was working, I had the interesting situation of having dozens of packages half installed, with the new version conflicting with the old one, and no easy way to fix the situation.
Luckily, we can do some automation to help a bit:
dnf list | cut -f1 -d' ' | uniq -cd > rescue.txt
This gives you a list of all the packages that were affected. With curl, NetworkManager, criu, glibc, kpartx and other critical utilities all affected (even bash#!), it's no wonder this system was a bit unstable after this happened.
With each package identified in rescue.txt, attempt to reinstall it, then uninstall the newer (half-installed) package. Then do a dnf upgrade and complete the upgrade for this package.
However, while going through this process I had another freeze, and had to reboot my system... and it didn't boot. Crap.
[root@podman01 dnf-rescue]# dnf reinstall glibc.x86_64
Last metadata expiration check: 18:35:57 ago on Sun 11 Sep 2022 01:33:36 EDT.
Error:
Problem: cannot install both glibc-2.28-189.5.el8_6.x86_64 and glibc-2.28-189.1.el8.x86_64
- conflicting requests
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
[root@podman01 dnf-rescue]# dnf remove glibc-2.28-189.5.el8_6.x86_64
Dependencies resolved.
==============================================================================================================
Package Architecture Version Repository Size
==============================================================================================================
Removing:
glibc x86_64 2.28-189.5.el8_6 @baseos 6.4 M
Removing dependent packages:
glibc-all-langpacks x86_64 2.28-189.5.el8_6 @baseos 415 M
glibc-common x86_64 2.28-189.5.el8_6 @baseos 8.4 M
glibc-gconv-extra x86_64 2.28-189.5.el8_6 @baseos 6.1 M
Transaction Summary
==============================================================================================================
Remove 4 Packages
Freed space: 436 M
Is this ok [y/N]: y
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Erasing : glibc-gconv-extra-2.28-189.5.el8_6.x86_64 1/4
Running scriptlet: glibc-gconv-extra-2.28-189.5.el8_6.x86_64 1/4
Erasing : glibc-common-2.28-189.5.el8_6.x86_64 2/4
Erasing : glibc-2.28-189.5.el8_6.x86_64 3/4
Erasing : glibc-all-langpacks-2.28-189.5.el8_6.x86_64 4/4
Running scriptlet: glibc-all-langpacks-2.28-189.5.el8_6.x86_64 4/4
Verifying : glibc-2.28-189.5.el8_6.x86_64 1/4
Verifying : glibc-all-langpacks-2.28-189.5.el8_6.x86_64 2/4
Verifying : glibc-common-2.28-189.5.el8_6.x86_64 3/4
Verifying : glibc-gconv-extra-2.28-189.5.el8_6.x86_64 4/4
Removed:
glibc-2.28-189.5.el8_6.x86_64 glibc-all-langpacks-2.28-189.5.el8_6.x86_64
glibc-common-2.28-189.5.el8_6.x86_64 glibc-gconv-extra-2.28-189.5.el8_6.x86_64
Complete!
[root@podman01 dnf-rescue]# dnf upgrade
Last metadata expiration check: 18:36:12 ago on Sun 11 Sep 2022 01:33:36 EDT.
Dependencies resolved.
==============================================================================================================
Package Architecture Version Repository Size
==============================================================================================================
Upgrading:
glibc x86_64 2.28-189.5.el8_6 baseos 2.2 M
glibc-all-langpacks x86_64 2.28-189.5.el8_6 baseos 25 M
glibc-common x86_64 2.28-189.5.el8_6 baseos 1.3 M
Installing weak dependencies:
glibc-gconv-extra x86_64 2.28-189.5.el8_6 baseos 1.5 M
Transaction Summary
==============================================================================================================
Install 1 Package
Upgrade 3 Packages
Total download size: 30 M
Is this ok [y/N]: y
Downloading Packages:
(1/4): glibc-2.28-189.5.el8_6.x86_64.rpm 748 kB/s | 2.2 MB 00:03
(2/4): glibc-gconv-extra-2.28-189.5.el8_6.x86_64.rpm 513 kB/s | 1.5 MB 00:03
(3/4): glibc-common-2.28-189.5.el8_6.x86_64.rpm 3.4 MB/s | 1.3 MB 00:00
(4/4): glibc-all-langpacks-2.28-189.5.el8_6.x86_64.rpm 2.8 MB/s | 25 MB 00:09
--------------------------------------------------------------------------------------------------------------
Total 3.1 MB/s | 30 MB 00:09
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Upgrading : glibc-all-langpacks-2.28-189.5.el8_6.x86_64 1/7
Upgrading : glibc-common-2.28-189.5.el8_6.x86_64 2/7
Running scriptlet: glibc-2.28-189.5.el8_6.x86_64 3/7
Upgrading : glibc-2.28-189.5.el8_6.x86_64 3/7
Running scriptlet: glibc-2.28-189.5.el8_6.x86_64 3/7
Installing : glibc-gconv-extra-2.28-189.5.el8_6.x86_64 4/7
Running scriptlet: glibc-gconv-extra-2.28-189.5.el8_6.x86_64 4/7
Cleanup : glibc-common-2.28-189.1.el8.x86_64 5/7
Cleanup : glibc-2.28-189.1.el8.x86_64 6/7
Cleanup : glibc-all-langpacks-2.28-189.1.el8.x86_64 7/7
Running scriptlet: glibc-all-langpacks-2.28-189.1.el8.x86_64 7/7
Running scriptlet: glibc-all-langpacks-2.28-189.5.el8_6.x86_64 7/7
Running scriptlet: glibc-all-langpacks-2.28-189.1.el8.x86_64 7/7
Running scriptlet: glibc-common-2.28-189.5.el8_6.x86_64 7/7
Verifying : glibc-gconv-extra-2.28-189.5.el8_6.x86_64 1/7
Verifying : glibc-2.28-189.5.el8_6.x86_64 2/7
Verifying : glibc-2.28-189.1.el8.x86_64 3/7
Verifying : glibc-all-langpacks-2.28-189.5.el8_6.x86_64 4/7
Verifying : glibc-all-langpacks-2.28-189.1.el8.x86_64 5/7
Verifying : glibc-common-2.28-189.5.el8_6.x86_64 6/7
Verifying : glibc-common-2.28-189.1.el8.x86_64 7/7
Upgraded:
glibc-2.28-189.5.el8_6.x86_64 glibc-all-langpacks-2.28-189.5.el8_6.x86_64
glibc-common-2.28-189.5.el8_6.x86_64
Installed:
glibc-gconv-extra-2.28-189.5.el8_6.x86_64
Complete!In the end I had to rebuild the node, there was no plausible way to repair it. It was interesting to try, though!