[Addition: Unfortunately new users can only post one picture per post but all I have are pictures. All the pictures included in the original post can be found here]
[Addition 2: I was so confused about how to start a new topic (I guess I couldn't before?) that I already posted a question on Unix Stackexchange, too (https://unix.stackexchange.com/questions/579765/manjaro-fails-to-boot-after-hard-reset-because-it-was-presumably-out-of-memory). I hope that's not a problem. If anyone is in the same situation that I was, then yes, newbies cannot post to the newbie corner right away but only after some magical time limit? I read all the newbie materials but didn't find any information on this]
What's the situation?
My Manjaro installation is not booting properly but throwing some errors (see more detailed below) and leaving me in an emergency console. I have dual-boot so I can boot to Windows and the computer seems to function properly but Manjaro has some problems. I do not have a USB stick for now so I cannot do live USB debugging but I an access a terminal from both GRUB and the emergency shell that Manjaro dumps me in after a failed boot.
I've been running Manjaro without any issues for about a year. Today I had to perform some extensive matrix calculations that eat up all the RAM and sure enough when I ran the calculations on my 16GB RAM + 8 GB swap machine all of that was consumed and the OS killed the process which is fairly standard. I thought I could mitigate this by adding another swapfile so I created a swapfile2 that was 16GB and gave it to the system to use as swap (only did
swapon, I did not mount the swapfile in /etc/fstab because I knew that I don't want it later when the computation is done). Having newly found 24GB of swap + 16 GB RAM I ran the operation again, only to find ~20 minutes later that the computer had completely frozen. I waited a couple of minutes to see if maybe it was just interacting with swap (which is relatively slow and sometimes freezes the UI), but after 5-10 minutes it seemed that it would not recover. I resorted to the easiest solution and did a hard reset to the system. Only to find that it doesn't boot any more.
What do I see?
I've been rummaging around the emergency terminal for a while now. Firstly I deleted the second swapfile but since it wasn't mounted anyways that probably didn't change anything.
For some weird reason I do not have access to
|<> symbols (tried all of my keyboard but nowhere to be found) so it limits what I can do in the shell. However, I have retrieved some logs and uncovered some weird phenomena.
This is the "splash screen" that I see every time I try to boot:
From here on, I have two options. If I press nothing and wait, it logs some more errors and dumps me in an emergency shell. If I press enter, I can select some locale options (out of all the things!) and then after some logging it just hangs in the following screen until after a while it reboots on its own:
Anyways, I think the emergency shell is a bit more interesting. Start sequence for the emergency shell:
journalctl -xb that is recommended returns nothing, as does just
journalctl and all the variations suggested here. Just says
-- no entries --.
systemctl default hangs for a minute or so and then reboots into the same broken Manjaro.
The end of the
dmesg output can be seen here:
Unfortunately there's no scroll or copy-paste option so I don't know how to extract more than that. The
dmesg output points to LVM2 logical volumes failure so after googling that a bit I came across some things that I should log out, so here's the output of
fdisk -l and
fstab file I also noticed that there's some problem with all the files in the
I googled that too but it seems more like a symptom than the underlying problem.
And finally checked the status of the
journald service, which probably is not very informative.
I hope someone can point me in the right direction, because right now I have no clue where to start debugging this mess.