System errors during update in TTY

Nalen · 5 January 2020 16:28

I updated today (05.01.2019) and it failed horribly.

Before update, I went into tty(ctrl+alt+F2) because of issues from past when updating with GUI on, and there did "sudo pacman -Syu"
Update was taking a long time, so I left my laptop updating and went away for maybe 20-30 minutes. When I came back, whole screen was filled with some kind of errors I think or something.
The lines I can remember are:

Somewhere at the top was: Watchdog: Bug: soft lockup - CPU#5 stuck for 22s!
At few lines were info about my hardware, like what is my CPU, GPU, etc...
Then many lines I could not understand at all, like: "Process: fdsagfdbsfd" (yeah, it did not have name or anything, just letters, maybe even numbers"
Last line was something similar to this: "Code: 7a 67 e1 b2 r9" (it seems to me like some hex code, it was quite long, these numbers and letter I wrote are made up, dont remember exact thing)

That is all I remember from that screen, even though there was more than maybe 30 lines of something, unfortunately I did not took a picture of it

And the bottom was cursor waiting for something, and then every few seconds (maybe every 30s) whole screen flickered(dont know right word in english, it was like when you open terminal, do system update, and many things are happening, it is so fast you can not read it. But this was like all lines I saw were immidietely removed at one and new ones shown immidietely, not like normal update where you can see new lines coming from bottom to top) with same lines.

I waited for a few minutes, but nothing changed so I shutted down my laptop with power key. Then I started my laptop, chose Manjaro (I dual-boot Manjaro and W10) and only two lines:

error: Invalid magic number
error: You need to load kernel first

I googled those problems, but it seems that people had such a problem only while installing their OS, not when it was already installed. What can I do now? The most important thing is to not loose data I had on disk. But preferably I do not want to reinstall my Linux again because of all the struggle I had while installing it first time(because dual boot, because millions of problems I encountered bcs I have Nvidia + intel gpu, because installing all IDEs and other big apps).

Any help?

manicjaropanic · 6 January 2020 16:48

Can you boot a live media, manjaro-chroot into the existing system, and just do sudo pacman -Syyu from a terminal emulator? That way you will be able to scroll back and read any errors.

manicjaropanic · 6 January 2020 16:53

Just thought about this: as the system was shut down ungracefully you may need to remove the package manager lock:
rm /var/lib/pacman/db.lck
Do that if pacman complains that it's busy or already running.

Nalen · 6 January 2020 19:22

I booted into live Manjaro, chrooted into the existing system, but when I typed sudo pacman -Syyu I got error: GPGME error: Invalid crypto engine . I googled and found this solution from Jonathon. I tried everything he wrote there, even 'nuclear solution' but I can not get it working. Here is what is happening:

[manjaro /]# sudo rm -fr /etc/pacman.d/gnupg
[manjaro /]# sudo pacman-key --init
chmod: cannot access '/etc/pacman.d/gnupg//trustdb.gpg': No such file or directory
==> Updating trust database...
[manjaro /]# sudo pacman-key --populate archlinux manjaro
==> ERROR: You do not have sufficient permissions to read the pacman keyring.
==> Use 'pacman-key --init' to correct the keyring permissions.
[manjaro /]# sudo pacman-key --refresh-keys
==> ERROR: You do not have sufficient permissions to read the pacman keyring.
==> Use 'pacman-key --init' to correct the keyring permissions.
[manjaro /]# sudo pacman -Syyu
warning: Public keyring not found; have you run 'pacman-key --init'?
error: GPGME error: Invalid crypto engine
:: Synchronizing package databases...
 core                                                                                                                                  151.4 KiB   299 KiB/s 00:01 [#####################################################################################################] 100%
 extra                                                                                                                                1780.8 KiB   316 KiB/s 00:06 [#####################################################################################################] 100%
 community                                                                                                                               5.2 MiB   368 KiB/s 00:15 [#####################################################################################################] 100%
 multilib                                                                                                                              176.2 KiB   705 KiB/s 00:00 [#####################################################################################################] 100%
 sublime-text                                                                                                                         1828.0   B  0.00   B/s 00:00 [#####################################################################################################] 100%
 sublime-text.sig                                                                                                                      543.0   B  0.00   B/s 00:00 [#####################################################################################################] 100%
warning: Public keyring not found; have you run 'pacman-key --init'?
error: GPGME error: Invalid crypto engine
error: failed to update sublime-text (invalid or corrupted database (PGP signature))
error: failed to synchronize all databases
[manjaro /]# sudo pacman -U /var/cache/pacman/pkg/{archlinux,manjaro}-keyring*.pkg.tar.xz
warning: Public keyring not found; have you run 'pacman-key --init'?
error: GPGME error: Invalid crypto engine
error: database 'sublime-text' is not valid (invalid or corrupted database (PGP signature))
loading packages...
warning: downgrading package archlinux-keyring (20191219-1 => 20191007-1)
error: '/var/cache/pacman/pkg/archlinux-keyring-20191018-1-any.pkg.tar.xz': duplicate target
error: '/var/cache/pacman/pkg/archlinux-keyring-20191018-2-any.pkg.tar.xz': duplicate target
error: '/var/cache/pacman/pkg/archlinux-keyring-20191219-1-any.pkg.tar.xz': duplicate target
warning: downgrading package manjaro-keyring (20190608-1 => 20190512-1)
error: '/var/cache/pacman/pkg/manjaro-keyring-20190608-1-any.pkg.tar.xz': duplicate target

Any idea?

Nalen · 6 January 2020 20:36

I even tried to update system not from chroot but from live system

[manjaro@manjaro ~]$ sudo pacman --root /dev/sdb5 -Syu
error: failed to initialize alpm library
(could not find or read directory: /dev/sdb5/var/lib/pacman/)

Could not find solution to this either

manicjaropanic · 6 January 2020 23:08

what happens when you do
ls /dev/sdb5/var/lib/pacman/

And what happens when you do
pacman-key --init
Like pacman suggested?

Nalen · 7 January 2020 16:11

[manjaro@manjaro ~]$ ls /dev/sdb5/var/lib/pacman/
ls: cannot access '/dev/sdb5/var/lib/pacman/': Not a directory

If I do this while chrooted then this:

[manjaro /]# ls /var/lib/pacman
local  sync
[manjaro /]#

about that pacman-key --init, I already added this command yesterday in previous post. Here is result:

[manjaro /]# sudo pacman-key --init
chmod: cannot access '/etc/pacman.d/gnupg//trustdb.gpg': No such file or directory
==> Updating trust database...

Marte · 7 January 2020 16:25

I guess you would need to re-install gnupg after the command above. See:

https://wiki.manjaro.org/index.php?title=Pacman_troubleshooting#Errors_about_Keys

Nalen · 7 January 2020 16:34

already tried it

[manjaro /]# sudo rm -r /etc/pacman.d/gnupg 
[manjaro /]# sudo pacman -Sy gnupg archlinux-keyring manjaro-keyring
warning: Public keyring not found; have you run 'pacman-key --init'?
error: GPGME error: Invalid crypto engine
:: Synchronizing package databases...
 core is up to date
 extra is up to date
 community is up to date
 multilib is up to date
 sublime-text                                                                                                                         1828.0   B   446 KiB/s 00:00 [#####################################################################################################] 100%
 sublime-text.sig                                                                                                                      543.0   B  0.00   B/s 00:00 [#####################################################################################################] 100%
warning: Public keyring not found; have you run 'pacman-key --init'?
error: GPGME error: Invalid crypto engine
error: failed to update sublime-text (invalid or corrupted database (PGP signature))
error: failed to synchronize all databases
[manjaro /]# sudo pacman-key --init 
chmod: cannot access '/etc/pacman.d/gnupg//trustdb.gpg': No such file or directory
==> Updating trust database...
[manjaro /]# sudo pacman-key --populate archlinux manjaro 
==> ERROR: You do not have sufficient permissions to read the pacman keyring.
==> Use 'pacman-key --init' to correct the keyring permissions.
[manjaro /]# sudo pacman-key --refresh-keys 
==> ERROR: You do not have sufficient permissions to read the pacman keyring.
==> Use 'pacman-key --init' to correct the keyring permissions.
[manjaro /]# sudo pacman -Sc
warning: Public keyring not found; have you run 'pacman-key --init'?
error: GPGME error: Invalid crypto engine
Packages to keep:
  All locally installed packages

Cache directory: /var/cache/pacman/pkg/
:: Do you want to remove all other packages from cache? [Y/n] y
removing old packages from cache...

Database directory: /var/lib/pacman/
:: Do you want to remove unused repositories? [Y/n] y
removing unused sync repositories...
[manjaro /]# sudo pacman -Syu
warning: Public keyring not found; have you run 'pacman-key --init'?
error: GPGME error: Invalid crypto engine
:: Synchronizing package databases...
 core is up to date
 extra is up to date
 community is up to date
 multilib is up to date
 sublime-text                                                                                                                         1828.0   B   595 KiB/s 00:00 [#####################################################################################################] 100%
 sublime-text.sig                                                                                                                      543.0   B  0.00   B/s 00:00 [#####################################################################################################] 100%
warning: Public keyring not found; have you run 'pacman-key --init'?
error: GPGME error: Invalid crypto engine
error: failed to update sublime-text (invalid or corrupted database (PGP signature))
error: failed to synchronize all databases
[manjaro /]#

Still nothing

nikgnomic · 7 January 2020 17:47

suggest boot from Live ISO USB and recover data

also suggest doing updates from terminal within DE and not TTY
then it would be possible to resolve errors before reboot using accurate information

anon27588196 · 7 January 2020 17:54

Learning the hard way about not having backups.

Nalen · 7 January 2020 18:06

and then reinstall?
Do I need to do something about data, if all data(home) is on HDD and only Manjaro was installed on SSD?

edit: I used to update within DE, but since one bigger update, where was suggested to do the update from TTY because KDE could corrupt something, I started doing it this way. Several people were actually recommending it back then ( maybe half a year ago, not more) that it is more safer
edit2: Here is one example when it was stressed to udpate only within TTY. So I just got used to it. Probably should start updating within DE though...

Nalen · 7 January 2020 17:57

backup of what? system ? is ther such a thing ? because I did not loose any data, only OS stopped working

anon27588196 · 7 January 2020 21:08

manicjaropanic · 8 January 2020 00:11

Did you try to pacstrap? It's supposed to (re)install all packages for when installing arch linux. Manjaro has basestrap

https://wiki.manjaro.org/index.php?title=Manjaro-tools

Boot live usb, then mount the os partition to /mnt
mount /dev/Yourdeviceandpartition /mnt
lsblk if you don't know which one to mount, look for a big partition ext4

then do
sudo basestrap /mnt base linux linux-firmware
sudo basestrap -S $(pacman -Qq)
The last one straps all packages that are on the liveusb. I'm pretty sure this also installs some stuff you don't need and of course it neglects the stuff you have installed yourself... Seems like that's gone anyway as you don't have the .db anymore.

In fact this is a pretty bad idea and will probably lead to a still broken system. Your best bet is to copy /home to an external drive and just reinstall as others have suggested.

For the future you might want to install with the btrfs filesystem since it allows snapshots and thus rollbacks for when things go bad.

Marte · 8 January 2020 18:40

If you are going to reinstall, use manual partitioning and mount your home partition at /home. Make sure to NOT format the partition.

Have a look at an application called Timeshift.
You could also make partition images to be restored if needed.

Nalen · 8 January 2020 18:33

great response...if you have nothing helpful to add why even bother coming here? ego?

Nalen · 8 January 2020 18:40

thank you manicjaropanic and Marte for help and suggestions. I am going to backup /home and reinstall Manjaro.

manicjaropanic · 8 January 2020 21:23

If you do decide to use the btr filesystem, keep in mind to confige tlp (if you use it, probably the case if you are on a laptop) to prevent corruption.

https://wiki.archlinux.org/index.php/TLP#Btrfs

This makes sure tlp's power saving measures don't corrupt the drive.

It might also be worth it to install /home and root on different partitions.

anon27588196 · 8 January 2020 22:09

Yet you didn't/couldn't be bother(ed) to learn about system backups. Couple of commands from a live CD and you'd be back in business.

For the sheer, unadulterated entertainment.