You are in Emergency mode (How to leave and stay out of the ER)

zeddock · 11 September 2019 23:56

Every hard reboot seems to leave me with a screen that complains:

Starting Flush Journal to Persistent Storage
Timed out waiting for device ....672F
Dependency Failed File system check on ...672F
Dependency Failed /boot/efi
Dependency Failed Local File Systems

Then I get a bunch of options to continue.
CTRL-D usually works, though sometimes I have given a PW and looked around, then I exit, and all seems well at the GUI Login.

How can I get this stabilized so I get a quick boot?
Other times I have "knight-rider" asterisks going back and fourth and counting.... but it says (1 of 6) or (1 of 8) but it lies! It gets to the "of" number and just starts counting again at 1 of...

I did a sudo lsblk -f and see this:

 sdc                                                                            
├─sdc1
│    vfat                  C43D-672F                             299.1M     0% /boot/efi
└─sdc2
     ext4                  c07076d8-5e11-43e1-9d62-1afc277124df  193.8G    53% /

I can see the boot is thought to be on the device 672F, which I believe is correct based on the many drives I have. That one is a SSD dedicated solely to Manjaro, and it looks like there is a little, teeney, weeney 300mb partition made as the grub? Am I learning this correctly?

So what is happening and How do I fix it please?

Note: I have seen some similar posts that talk about fixing fstab, but "fixing" is an unknown process for me yet.

Thanx!

My /etc/fstab is:

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a device; this may
# be used with UUID= as a more robust way to name devices that works even if
# disks are added and removed. See fstab(5).
#
# <file system>             <mount point>  <type>  <options>  <dump>  <pass>
UUID=C43D-672F                            /boot/efi      vfat    umask=0077 0 2
UUID=c07076d8-5e11-43e1-9d62-1afc277124df /              ext4    defaults,noatime,discard 0 1
tmpfs                                     /tmp           tmpfs   defaults,noatime,mode=1777 0 0

mbb · 9 September 2019 22:12

Maybe try sudo mkinitcpio -P && sudo update-grub and reboot. Before this check for updates.

zeddock · 11 September 2019 23:50

Checked for updates and did them. Now it wont boot into a GUI. Not sure how to proceed.
Please help me get back to work.

I am taking pictures of what shows on the screens.

UPDATE.. I just got back in, but now really, really worried. This is the last boot from which I was able to get into the GUI via CTRL-D

zeddock · 11 September 2019 23:53

This is a link to a Photos Album for the failures since August. One is a 3 minute video from today. Perhaps it could lead someone to tell me what to do or where to look?

ALBUM

zeddock · 12 September 2019 00:06

I am doing this now.
But, it means I will need to reboot, right? Ah! I dread!
Anyone else wanna weigh in also, before I reboot? I will wait for a while and see.

BTW, the only two updates were a Google Chrome Browser AUR Build for update, and a NordVPN AUR Build. They both seem to have been accomplished without issue.

The command suggested from @mbb finished like this:

zeddock · 12 September 2019 00:24

Argh!
"Timed out waiting for /boot/efi" again!
Goes to Emergency mode. At least it is booting!

Hmmmm.. this time I gave my credentials versus CTRL-D.
Then I typed systemctl default and ENTER.
It went to the GUI.

Not sure what that means. Please help.

I should note: At some point there is a change in the screen resolution! The type on the screen goes from big to small.

Sinister · 12 September 2019 00:42

Screen resolution change is normal.

~~What I don't think is normal is:~~
~~Timed out waiting for device /dev/disk/by-uuid/C43D-672F~~
~~I don't think it's supposed to say that.~~

Could be wrong, I just have never seen that before or I haven't been paying enough attention.

Yep, I'm wrong. It just whips by so fast I never see it.
Sorry for the misinfo. I had to video it with my phone and slow-mo through it to see and make sure.

zeddock · 12 September 2019 01:22

@Sinister Thank you for the response!

I DONT think that is normal because it shouldn't TIMEOUT on my boot device, right?

I will try to do my work and hold-off on a reboot until I see if someone else can help. I really thought/hoped that what @mbb suggested with the GRUB might have worked because it seems to be an issue with how to boot!

@Sinister you could have looked at that 3minute video at the link... It is a good show of some things, I think. (Hope that was the one!)

Anybody else have some ideas?

Sinister · 12 September 2019 01:27

Yeah I know the Timeout isn't normal. I meant the wording /dev/disk/by-uuid thing.
I'd never seen it but only because it whips so quick normally.

zeddock · 12 September 2019 01:56

I will leave it running overnight if needed. I can't take the chance of not being able to work tomorrow if I turn her off!

Thanx all for trying to helpl!

mbb · 12 September 2019 08:41

Hi! Sorry for the delay.

Now that we are sure this isn't a bad init image, try to change the dump and fsck options on the first line of /etc/fstab to 0 0

zeddock · 12 September 2019 16:25

@mbb, I've looked around but don't understand what that does or where to put it in fstab.

Do you mean for me to just change this:
UUID=C43D-672F /boot/efi vfat umask=0077 0 2

To this:
UUID=C43D-672F /boot/efi vfat umask=0077 0 0

???

mbb · 12 September 2019 17:06

Exactly!

zeddock · 13 September 2019 04:21

@mbb
Something changed but not necessarily for better.
I changed fstab.
An update was done too, and I ran the above command that rebuilds the GRUB.

Here is a picture of where it is now stuck:
15683483678364392158966856372569

Different, for sure, but still failing.

Please let me know what to do for in the morning? I am headed to bed now and will just leave it going.

Thanx!

mbb · 13 September 2019 08:56

The boot partition errors are gone, which is good.

As for those errors, the only one to be concerned about is the Authorization Manager. Try to change to tty2 with CTRL+ALT+F2, what a few seconds and then change back to tty1 with CTRL+ALT+F1.

If it fails try again and if it fails the second time login into tty2 and post the output of:

journalctl -xe -p3 -b
cat /var/log/Xorg.0.log | grep EE
systemctl status haveged.service

zeddock · 13 September 2019 23:33

It is going in to a GUI start now eventually.
I was waiting a while... think the timer was up to 3 minutes?!

First came the "Knight Rider" back and forth for a while.
Then it was sitting and so I tried tty2. CTRL-ALT-F2 had no impact.
I just waited longer and it eventually came to the GUI login screen screen.

I know something is wrong because no other OS has taken this long to boot, and this is an SSD drive, so my expectations are framed in seconds, not minutes.

Here is a shot I took while waiting on it:

I do appreciate the help @mbb.
What can I check next please?
Should I do the 3-lines in the terminal still to give you information?

Thanx!

zeddock · 14 September 2019 05:41

@mbb I have done these lines:

journalctl -xe -p3 -b
cat /var/log/Xorg.0.log | grep EE
systemctl status haveged.service

I am not sure how to get you the output so I am posting a picture below for now and maybe you can tell me what else to do to get you information.

Firstly, one time a reboot will work and then it will stop and fail. I hate intermittent as it is hard to diagnose. But that is when I know what I am doing and how to look. Here, I am a noob!

Just for a reference, it is still taking over 5 minutes to boot... and the Knight-Rider red asterisk keeps going back and forth, then will move on, then start again.

Eventually, when I think it is locked up, with no HD light indication, I will do a CTRL-ALT-DEL and the screen will flash a few lines and reboot.

This last time I went into tty2, back to tty1, then to tty2 and gave the commands. I wasn't sure how to parse them and give them so I had to try and retry.

Right now I am setting in Emergency Mode again.
Please help me stabilize this system. Or, if I must reinstall, I guess I need to know that too!

I go out of town on Tuesday, and will be gone for a few weeks. There is no way I could talk my wife into doing all of this if I need remote access to this work machine, so I want to try and get it fixed now.

Thanx for your help.

rylsm · 14 September 2019 14:18

Looks like your drive is dying or there is connection or power issue from the motherboard to your SSD.
You might need to check on the cable connection of your drive (if any) or in the worst scenario you might need to replace it and hope the data on the disk are still salvageable.

AgentS · 14 September 2019 13:52

If you mean "reboot with a button on your PC", instead of using your system's reboot action buttons/commands, you have to make it clear, as it would probably be your problem.

Anyway on boot, use the fallback kernel image, to see if it boots correctly.

But since you disabled fsck check (in fstab to 0) and the errors went away, then it seems this

You might do a manual fsck check or S.M.A.R.T. check if available.

Copy and paste in your post (in code format, use ``` in the lines before and after the output text.)

zeddock · 14 September 2019 15:41

@AgentS I know this sounds silly to you, but I don't understand how to copy what is on the textual screen before booting into the GUI, in order to paste here.

When I am in terminal, I could do a SELECT / CTRL-C to copy, but I cant get to this thread in my browser so not sure how to CTRL-V to get it here.

When I am HERE, in a GUI environment, I don't know how to look at what I saw previously in the Terminal as I was booting!

DON'T LAUGH AT ME!!!

Seriously, I know this must seem silly. At least I know it can be amusing for those who know this stuff very well. Why? Because I am amused at similar things in subjects I know well.

Anyway... beyond responding to what I have typed above, please tell me about doing an fsck? and SMART check?

This is a new SSD drive 860 Samsung.

Two things I know for sure:

Early this morning the house had a power-blip. I had to restart or turn on many appliances.
In the past 2 weeks this drive has been wiped and has seen fresh installs (repartitioned and formated) of Kubuntu, Linux Mint, and Manjaro.

2 indicates to me that it is not the drive, though I have seen them fail at any time in my windows world experience.

I would like to figure this situation out, if you all will continue to try to help. I think it is important so that I understand more and so that I feel comfortable in trusting the OS install.

Standing by.... but I am going to shut down and check connections, etc.... Will also see if I can figure out how to do the Smart check. The fsck I will wait on , for specific instructions.... seems the wrong arguments can thrash a drive!?!?

Lastly, If I get to the point that a fresh install is needed. Can I just move /home directory into an external then put it back? Or should I handle it differently?

I have Timeshift running, but RSYNC to this same drive... and I also have a USB stick with FileZilla I once used to make an IMAGE of a system. I will want to reload my apps and reconfigure the File Manager settings anyway, so I think my only concern is /home docs and files.

Let's see if we can get this fixed though. It will show me how and be encouraging!

NOTE: I went to install GSmartControl to try to do a SMART test, and I get this:

Warning: lib32-systemd: local (243.9-1) is newer than multilib (242.108-1)
Warning: systemd: local (243.9-1) is newer than core (242.108-1)
Warning: systemd-libs: local (243.9-1) is newer than core (242.108-1)
Warning: systemd-sysvcompat: local (243.9-1) is newer than core (242.108-1)

Not sure it is significant, but wanted it in this post in case.

Oh! Updating this post...
@AgentS You had asked specifically about "hard reboot" and suggested using the "fallback kernel image"...
When I say hard reboot, I mean SHUTDOWN versus RESTART. I guess, since I was immediately shutting down and then trying to boot back up, I miscommunicated with "hard reboot". What I maybe should have said was that I used the menu selections to shutdown, and then restarted after the main unit went silent and dark.

For the fallback, is that when I get a selection between Manjaro and (I can't remember the other option) that counts down during a startup?

You folks must get awfully tired of these noob questions! Thanx for persevering!