A failed attempt at rescuing Rocky Linux 8 from a failed DNF upgrade...

I recently had a DNF upgrade go very, very badly on Rocky Linux. I lost power at a pretty inopportune time, and well, it broke everything including DNF.

If you have a borked Rocky Linux install and are looking for help with it, maybe this will help a bit. MAKE A BACKUP BEFORE YOU DO ANYTHING. Ahem. And be prepared to break your system even worse than it's currently broken.

Getting DNF working again was a fairly difficult task, which essentially boiled down to finding which half-installed package was killing it. As it turns out python3-hawkey was the culprit. I forget the exact steps to get it working again, but something like this:

wget https://download.rockylinux.org/pub/rocky/8/BaseOS/x86_64/os/Packages/p/python3-hawkey-0.63.0-8.1.el8_6.x86_64.rpm
sudo rpm -ivh python3-hawkey-0.63.0-8.1.el8_6.x86_64.rpm --force
sudo rpm --verify --all
sudo dnf update
Here's some hints, but be careful.

Once DNF was working, I had the interesting situation of having dozens of packages half installed, with the new version conflicting with the old one, and no easy way to fix the situation.

Luckily, we can do some automation to help a bit:

dnf list | cut -f1 -d' ' | uniq -cd > rescue.txt

This gives you a list of all the packages that were affected. With curl, NetworkManager, criu, glibc, kpartx and other critical utilities all affected (even bash#!), it's no wonder this system was a bit unstable after this happened.

With each package identified in rescue.txt, attempt to reinstall it, then uninstall the newer (half-installed) package. Then do a dnf upgrade and complete the upgrade for this package.

However, while going through this process I had another freeze, and had to reboot my system... and it didn't boot. Crap.

[root@podman01 dnf-rescue]# dnf reinstall glibc.x86_64
Last metadata expiration check: 18:35:57 ago on Sun 11 Sep 2022 01:33:36 EDT.
 Problem: cannot install both glibc-2.28-189.5.el8_6.x86_64 and glibc-2.28-189.1.el8.x86_64
  - conflicting requests
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
[root@podman01 dnf-rescue]# dnf remove glibc-2.28-189.5.el8_6.x86_64
Dependencies resolved.
 Package                         Architecture       Version                         Repository           Size
 glibc                           x86_64             2.28-189.5.el8_6                @baseos             6.4 M
Removing dependent packages:
 glibc-all-langpacks             x86_64             2.28-189.5.el8_6                @baseos             415 M
 glibc-common                    x86_64             2.28-189.5.el8_6                @baseos             8.4 M
 glibc-gconv-extra               x86_64             2.28-189.5.el8_6                @baseos             6.1 M

Transaction Summary
Remove  4 Packages

Freed space: 436 M
Is this ok [y/N]: y
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                      1/1
  Erasing          : glibc-gconv-extra-2.28-189.5.el8_6.x86_64                                            1/4
  Running scriptlet: glibc-gconv-extra-2.28-189.5.el8_6.x86_64                                            1/4
  Erasing          : glibc-common-2.28-189.5.el8_6.x86_64                                                 2/4
  Erasing          : glibc-2.28-189.5.el8_6.x86_64                                                        3/4
  Erasing          : glibc-all-langpacks-2.28-189.5.el8_6.x86_64                                          4/4
  Running scriptlet: glibc-all-langpacks-2.28-189.5.el8_6.x86_64                                          4/4
  Verifying        : glibc-2.28-189.5.el8_6.x86_64                                                        1/4
  Verifying        : glibc-all-langpacks-2.28-189.5.el8_6.x86_64                                          2/4
  Verifying        : glibc-common-2.28-189.5.el8_6.x86_64                                                 3/4
  Verifying        : glibc-gconv-extra-2.28-189.5.el8_6.x86_64                                            4/4

  glibc-2.28-189.5.el8_6.x86_64                      glibc-all-langpacks-2.28-189.5.el8_6.x86_64
  glibc-common-2.28-189.5.el8_6.x86_64               glibc-gconv-extra-2.28-189.5.el8_6.x86_64

[root@podman01 dnf-rescue]# dnf upgrade
Last metadata expiration check: 18:36:12 ago on Sun 11 Sep 2022 01:33:36 EDT.
Dependencies resolved.
 Package                          Architecture        Version                       Repository           Size
 glibc                            x86_64              2.28-189.5.el8_6              baseos              2.2 M
 glibc-all-langpacks              x86_64              2.28-189.5.el8_6              baseos               25 M
 glibc-common                     x86_64              2.28-189.5.el8_6              baseos              1.3 M
Installing weak dependencies:
 glibc-gconv-extra                x86_64              2.28-189.5.el8_6              baseos              1.5 M

Transaction Summary
Install  1 Package
Upgrade  3 Packages

Total download size: 30 M
Is this ok [y/N]: y
Downloading Packages:
(1/4): glibc-2.28-189.5.el8_6.x86_64.rpm                                      748 kB/s | 2.2 MB     00:03
(2/4): glibc-gconv-extra-2.28-189.5.el8_6.x86_64.rpm                          513 kB/s | 1.5 MB     00:03
(3/4): glibc-common-2.28-189.5.el8_6.x86_64.rpm                               3.4 MB/s | 1.3 MB     00:00
(4/4): glibc-all-langpacks-2.28-189.5.el8_6.x86_64.rpm                        2.8 MB/s |  25 MB     00:09
Total                                                                         3.1 MB/s |  30 MB     00:09
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                      1/1
  Upgrading        : glibc-all-langpacks-2.28-189.5.el8_6.x86_64                                          1/7
  Upgrading        : glibc-common-2.28-189.5.el8_6.x86_64                                                 2/7
  Running scriptlet: glibc-2.28-189.5.el8_6.x86_64                                                        3/7
  Upgrading        : glibc-2.28-189.5.el8_6.x86_64                                                        3/7
  Running scriptlet: glibc-2.28-189.5.el8_6.x86_64                                                        3/7
  Installing       : glibc-gconv-extra-2.28-189.5.el8_6.x86_64                                            4/7
  Running scriptlet: glibc-gconv-extra-2.28-189.5.el8_6.x86_64                                            4/7
  Cleanup          : glibc-common-2.28-189.1.el8.x86_64                                                   5/7
  Cleanup          : glibc-2.28-189.1.el8.x86_64                                                          6/7
  Cleanup          : glibc-all-langpacks-2.28-189.1.el8.x86_64                                            7/7
  Running scriptlet: glibc-all-langpacks-2.28-189.1.el8.x86_64                                            7/7
  Running scriptlet: glibc-all-langpacks-2.28-189.5.el8_6.x86_64                                          7/7
  Running scriptlet: glibc-all-langpacks-2.28-189.1.el8.x86_64                                            7/7
  Running scriptlet: glibc-common-2.28-189.5.el8_6.x86_64                                                 7/7
  Verifying        : glibc-gconv-extra-2.28-189.5.el8_6.x86_64                                            1/7
  Verifying        : glibc-2.28-189.5.el8_6.x86_64                                                        2/7
  Verifying        : glibc-2.28-189.1.el8.x86_64                                                          3/7
  Verifying        : glibc-all-langpacks-2.28-189.5.el8_6.x86_64                                          4/7
  Verifying        : glibc-all-langpacks-2.28-189.1.el8.x86_64                                            5/7
  Verifying        : glibc-common-2.28-189.5.el8_6.x86_64                                                 6/7
  Verifying        : glibc-common-2.28-189.1.el8.x86_64                                                   7/7

  glibc-2.28-189.5.el8_6.x86_64                      glibc-all-langpacks-2.28-189.5.el8_6.x86_64


In the end I had to rebuild the node, there was no plausible way to repair it. It was interesting to try, though!