Unable to boot after latest stable update

[parad0x]

[UPDATED]

inxi -Fxzc0 output
$ Journalctl -b has -- No Entries --
$ tail -n 3000 /var/log/pacman.log

Following the latest update, I am unable to boot and I get the following error message after grub:

[ 4.896409] Initramfs unpacking
failed: read error  
:: running early hook [udev]  
:: running early hook [lvm2] 
:: running hook [udev] 
:: Triggering uevents... 
:: running hook [plymouth] 
:: Loading plymouth... done
:: running hook [plymouth-encrypt] 
:: running hook [resume] 
ERROR: resume: no device specified for hibernation
ERROR: device '' not found. Skipping fsck
:: mounting '' on real root
You are now being dropped into an emergency shell. 
sh: can't access tty; job control turned off
[rootfs  ] #

At the shell, if I type, no letters are appearing on the screen. At grub, I have the option for Manjaro Linux kernel 4.9.34-1 x64 and 4.9.33-1-rt23 x64 (and respective fallback initramfs) which I have tried. If there is any more information that I can provide, I am willing to.

Following suggestions by @thefallenrat, I chrooted into my system from a live USB with the following commands:

$ cryptsetup luksOpen /dev/sda3 cryptdisk
$ vgchange -a y
$ mount /dev/mapper/ManjaroVG-ManjaroRoot /mnt
$ mount /dev/sda2 /mnt/boot
$ mount /dev/sda1 /mnt/boot/efi
$ manjaro-chroot /mnt

and then attempted to install the linux44 kernel, however it resulted in this output with an error which may provide insight into my issues booting. Here is a truncated version of that error:

cannot open file ca
.
.
==> ERROR: module not found: `crc32'
.
.
==> WARNING: errors were encountered during the build. The image may not be complete.
error: command failed to execute correctly
(2/3) Updating Grub-Bootmenu
/usr/bin/grub-editenv: error: cannot open `//boot/grub/grubenv.new': No such file or directory.
/usr/bin/grub-mkconfig: line 237: /boot/grub/grub.cfg.new: No such file or directory
error: command failed to execute correctly

Lastly, I noticed that there is no grub folder in /boot:

$ ls /boot/
EFI  initramfs-4.4-x86_64-fallback.img	initramfs-4.4-x86_64.img  linux44-x86_64.kver  vmlinuz-4.4-x86_64

However, this file/folder exists:

$ ls /boot/EFI/manjaro_grub/            
grubx64.efi

[thefallenrat]

If you still have a manjaro live usb, and know how to chroot. Wanna try switching kernel to kernel44? It won't hurt to try...

In your chroot, run these commands :

sudo pacman-mirrors -f 5 -y
sudo pacman -Syu
sudo mhwd-kernel -i kernel44

Then reboot into your main system, and select kernel44 on your Grub....

Also, while you're on chroot, can you provide the output of this commands?

cat /var/log/pacman.log
journalctl -b

Copy and paste it into pastebin, and link it to here....

[parad0x]

In trying to chroot, after having installed mhwd-chroot and running either mhwd-chroot or mhwd-chroot-shell, a terminal window opens with the text "No linux system found!" could this be due to my hard drive having luks encryption?

Later on in the tutorial, it mentions the partitions from using the assisted installer (which I used):

/dev/sda1: Boot partition
/dev/sda2: Swap partition
/dev/sda3: Manjaro system
/dev/sda4: Space for personal files.

However, when I run lsblk -f, I get the following output:

$ lsblk -f
NAME   FSTYPE      LABEL       UUID                                     MOUNTPOINT
loop0  squashfs                                                     /run/miso/sfs/livefs
loop1  squashfs                                                     /run/miso/sfs/mhwdfs
loop2  squashfs                                                     /run/miso/sfs/desktopfs
loop3  squashfs                                                     /run/miso/sfs/rootfs
sda                                                                 
├─sda1 vfat        UEFI_SYSTEM 349A-EFB4                            
├─sda2 ext2        ManjaroBoot 704a6ab4-7459-4903-b991-f2c69af97b0e 
└─sda3 crypto_LUKS             8e2fc722-32a4-4af5-b259-9fcb9f8a81ec 
sdb    iso9660     MJRO1701    2017-04-02-11-58-17-00               
├─sdb1 iso9660     MJRO1701    2017-04-02-11-58-17-00               /run/miso/bootmnt
└─sdb2 vfat        MISO_EFI    8607-824A     

In this case, which partition would be my Manjaro System partition to mount for chroot? Additionally, just double checking that the "Manjaro Boot" partition should also be mounted as that would be my GRUB bootloader?

[thefallenrat]

@parad0x

could this be due to my hard drive having luks encryption?

Ahh... In that case I think this will suffice ....

How to chroot into an encrypted root partition

Disclaimer: I finally tested the process myself. But I got the ideas initially from the posts in the old Manjaro forum, this one in particular. When an installed encrypted system fails to boot to TTY (command line) or when the (correct) password fails to open the encryption after an update the user might try to repair the system by chrooting it. Chroot stands for change root and means to switch to a different root file system at runtime. First find out the name of the encrypted device with…

[parad0x]

Thank you, although I am currently experiencing issues mounting for the chroot. I have posted a separate question about it to be safe, since I figured that it might be a digression for this topic.

[parad0x]

# sudo tail -n 500 /var/log/pacman.log

# journalctl -b
-- No entries --

---

Also, when I entered:

$ sudo mhwd-kernel -i kernel44

it returned:

Error: Invalid argument.
Please choose one of the available kernels:
   * linux310
   * linux312
   * linux316
   * linux318
   * linux41
   * linux410
   * linux411
   * linux412
   * linux44
   * linux48
   * linux49
   * linux-rt-lts-manjaro
   * linux-rt-manjaro

so instead I entered:

$ sudo mhwd-kernel -i linux44 

and got the following error during installing the kernel:
# sudo mhwd-kernel -i linux44
:: Synchronizing package databases...
core 138.9 KiB 397K/s 00:00 [######################] 100%
extra 1794.2 KiB 1304K/s 00:01 [######################] 100%
community 4.2 MiB 1458K/s 00:03 [######################] 100%
multilib 181.3 KiB 824K/s 00:00 [######################] 100%
archlinuxfr 15.3 KiB 149K/s 00:00 [######################] 100%
warning: linux44-4.4.75-1 is up to date -- reinstalling
warning: linux44-bbswitch-0.8-52 is up to date -- reinstalling
warning: linux44-nvidia-1:375.66-5 is up to date -- reinstalling
warning: linux44-rt3562sta-2.4.1.1_r4-20 is up to date -- reinstalling
warning: linux44-virtualbox-host-modules-5.1.22-6 is up to date -- reinstalling
resolving dependencies...
looking for conflicting packages...

Packages (5) linux44-4.4.75-1  linux44-bbswitch-0.8-52
             linux44-nvidia-1:375.66-5  linux44-rt3562sta-2.4.1.1_r4-20
             linux44-virtualbox-host-modules-5.1.22-6

Total Installed Size:  85.62 MiB
Net Upgrade Size:       0.00 MiB

:: Proceed with installation? [Y/n] y
(5/5) checking keys in keyring                     [######################] 100%
(5/5) checking package integrity                   [######################] 100%
(5/5) loading package files                        [######################] 100%
(5/5) checking for file conflicts                  [######################] 100%
(5/5) checking available disk space                [######################] 100%
:: Processing package changes...
(1/5) reinstalling linux44                         [######################] 100%
>>> Updating module dependencies. Please wait ...
(2/5) reinstalling linux44-bbswitch                [######################] 100%
(3/5) reinstalling linux44-nvidia                  [######################] 100%
(4/5) reinstalling linux44-rt3562sta               [######################] 100%
(5/5) reinstalling linux44-virtualbox-host-modules [######################] 100%
In order to use the new version, reload all virtualbox modules manually.
:: Running post-transaction hooks...
(1/3) Updating linux44 initcpios
==> Building image from preset: /etc/mkinitcpio.d/linux44.preset: 'default'
  -> -k /boot/vmlinuz-4.4-x86_64 -c /etc/mkinitcpio.conf -g /boot/initramfs-4.4-x86_64.img
==> Starting build: 4.4.75-1-MANJARO
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [autodetect]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
  -> Running build hook: [keyboard]
  -> Running build hook: [keymap]
cannot open file ca
  -> Running build hook: [plymouth]
  -> Running build hook: [plymouth-encrypt]
==> ERROR: module not found: `crc32'
  -> Running build hook: [lvm2]
  -> Running build hook: [resume]
  -> Running build hook: [filesystems]
  -> Running build hook: [fsck]
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: /boot/initramfs-4.4-x86_64.img
==> WARNING: errors were encountered during the build. The image may not be complete.
==> Building image from preset: /etc/mkinitcpio.d/linux44.preset: 'fallback'
  -> -k /boot/vmlinuz-4.4-x86_64 -c /etc/mkinitcpio.conf -g /boot/initramfs-4.4-x86_64-fallback.img -S autodetect
==> Starting build: 4.4.75-1-MANJARO
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
  -> Running build hook: [keyboard]
  -> Running build hook: [keymap]
cannot open file ca
  -> Running build hook: [plymouth]
  -> Running build hook: [plymouth-encrypt]
==> ERROR: module not found: `crc32'
  -> Running build hook: [lvm2]
  -> Running build hook: [resume]
  -> Running build hook: [filesystems]
  -> Running build hook: [fsck]
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: /boot/initramfs-4.4-x86_64-fallback.img
==> WARNING: errors were encountered during the build. The image may not be complete.
error: command failed to execute correctly
(2/3) Updating Grub-Bootmenu
/usr/bin/grub-editenv: error: cannot open `//boot/grub/grubenv.new': No such file or directory.
/usr/bin/grub-mkconfig: line 237: /boot/grub/grub.cfg.new: No such file or directory
error: command failed to execute correctly
(3/3) Arming ConditionNeedsUpdate...

When I run manjaro-settings-manager, the new kernel appears as installed, but kernel44 is not available to select from grub on boot.

[thefallenrat]

I wonder if the error you got from installing linux44 also applies to your current kernel. Seeing at this line :

/usr/bin/grub-editenv: error: cannot open `//boot/grub/grubenv.new': No such file or directory.
/usr/bin/grub-mkconfig: line 237: /boot/grub/grub.cfg.new: No such file or directory

Did you by any chance manually editing grub with grub customizer or something? It may have something to do with that

Also I think pacman.log were not complete as I ddin't see what you were installing from previous session. Though I did see that you get image generation error when running post trasanction hooks ( Which is mkinitpio) Which kinda proves my point....

And I also see this when you're trying to upgrade the linux411 :

gzip: stdout: No space left on device

[parad0x]

I think I might have edited grub to change my grub screen background, but that would have been a while back, maybe a year ago. I posted the latest 500 lines of pacman.log because it was a massive file. Would you like me to post the rest? Also, I tried running manjaro-settings-manager from the terminal and uninstalling an unused kernel but it gave me an error. Is it possible to uninstall the old kernel to free up space via terminal?

[fungilife]

pacman -R linux*****
will remove whatever package you want to remove
For kernels you have a linux*** image and headers files that take up space

Why is your partition so small and why can't you enlarge it? Is this an installation on usb-stick?

[parad0x]

Thanks, I'll try that. My installation takes up my entire SSD but I am currently chrooting into it from a USB so that I can repair it.

EDIT:

I ran the following from chroot:

sudo pacman -R linux-rt-manjaro-rt3562sta
sudo pacman-mirrors -f 5 -y
sudo pacman -Syu
sudo mhwd-kernel -i kernel44

but still received the same errors as before while installing kernel44:

cannot open file ca
.
.
==> ERROR: module not found: `crc32'
.
.
==> WARNING: errors were encountered during the build. The image may not be complete.
error: command failed to execute correctly
(2/3) Updating Grub-Bootmenu
/usr/bin/grub-editenv: error: cannot open `//boot/grub/grubenv.new': No such file or directory.
/usr/bin/grub-mkconfig: line 237: /boot/grub/grub.cfg.new: No such file or directory
error: command failed to execute correctly

I have also UPDATED my original post.

[thefallenrat]

I am really suspicious with this line....

[ALPM-SCRIPTLET] gzip: stdout: No space left on device

Can you check in your boot partition if its full or not? I think this is the root of all problems....

[xircon]

Yes, bet the pacman cache needs cleaning...........

[gohlip]

@paradox, what's

ls /etc/grub.d
df -h

[parad0x]

@thefallenrat @gohlip

# ls /etc/grub.d
00_header  10_linux  20_linux_xen  30_os-prober  40_custom  41_custom  60_memtest86+  README

sh-4.4# df -h
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ManjaroVG-ManjaroRoot  227G   79G  137G  37% /
/dev/sda1                           99M   53M   46M  54% /boot
udev                               7.8G     0  7.8G   0% /dev
shm                                7.8G     0  7.8G   0% /dev/shm
run                                7.8G     0  7.8G   0% /run
tmp                                7.8G     0  7.8G   0% /tmp
overlay                             12G   70M   12G   1% /etc/resolv.conf

@xircon
After some googling earlier, I had already cleaned it beforehand as this seemed a likely way to free space. Thanks for the suggestion though!

# paccache -r
==> no candidate packages found for pruning

(also did a pacman -Scc)

[thefallenrat]

Dude, your /boot partiton size is only 100MB :

|-/dev/sda1                               8:1    0   100M  0 part  /boot

I think this is the one that may caused your issue, and it'll be likely a good time to resize that partitition....

[parad0x]

It seems odd to me that I wouldn't have enough space since I uninstalled a kernel already so I only have one kernel installed. Is there any particular way I should go about increasing the size of my root partition, or can I go ahead and do so like this with gparted from the live USB?

Also just double checking that this is the correct partition to increase and that I should increase it to 250MB?

[thefallenrat]

@parad0x

It seems odd to me since I uninstalled a kernel already so I only have one kernel installed. Is there any particular way I should go about increasing the size of my root partition, or can I go ahead and do so like this?

Also just double checking that this is the correct partition to increase and that I should increase it to 250MB?

I don't know. I'm not expert at this since I see that you have both /boot partition. Maybe its better if you wait for the expert to take it from here..

But, what I think that you need to resize the partition that has linux image in it.....

[parad0x]

Alright, thanks for the heads up. I'll wait for someone to confirm that it's the right thing to do then.
Edit: even if I wanted to, gparted from the live USB won't let me resize any partition aside from lowering my boot partition. This happens even if I've unmounted them...

Edit:

I have changed my mounting before the chroot from

$ cryptsetup luksOpen /dev/sda3 cryptdisk
$ vgchange -a y
$ mount /dev/mapper/ManjaroVG-ManjaroRoot /mnt
$ mount /dev/sda1 /mnt/boot
$ manjaro-chroot /mnt

to

$ cryptsetup luksOpen /dev/sda3 cryptdisk
$ vgchange -a y
$ mount /dev/mapper/ManjaroVG-ManjaroRoot /mnt
$ mount /dev/sda2 /mnt/boot
$ mount /dev/sda1 /mnt/boot/efi
$ manjaro-chroot /mnt

And also found lots of extra kernels in /boot which I moved to /home/
This freed up lots of space in my second boot partition and halfed it from previously being 250MiB / 250MiB.
Now when I run sudo mhwd-kernel -i linux44 I get the following errors instead.

[gohlip]

I guess you've found your problem.

@gohlip

@paradox, what's

df -h

@parad0x

It seems odd to me that I wouldn't have enough space since I uninstalled a kernel already so I only have one kernel installed

gparted:
/dev/sda2 /boot size=256 MB used=256 MB unused=0 MB

And also found lots of extra kernels in /boot which I moved to /home/
This freed up lots of space in my second boot partition and halfed it from previously being 250MiB / 250MiB.

Have you fixed it?
If you still have not, chroot properly again (as including mounting /boot/efi as well as /boot - I assume you have partitions sda1 as /boot/efi and sda2 as /boot, mount both),
Remove all other kernels and install a new kernel, say linux411 or linux412
Use mhwd to install/remove kernel, not pacman.

[edit] just to be clear, mount / first (sda3), then /boot (sda2) and last /boot/efi (sda1)
ps: disable fastboot (if you have fastboot enabled)

[parad0x]

Hey, this problem has not yet been fixed. I have been mounting that way like so:

$ cryptsetup luksOpen /dev/sda3 cryptdisk
$ vgchange -a y
$ mount /dev/mapper/ManjaroVG-ManjaroRoot /mnt
$ mount /dev/sda2 /mnt/boot
$ mount /dev/sda1 /mnt/boot/efi
$ manjaro-chroot /mnt

Then using mhwd to install linux44 kernel like so:
# sudo mhwd-kernel -i linux44

After leaving it for a couple days, then chrooting and doing a pacman -Syu for a larger update, the error during the mhwd seems to have changed.

Edit: also, when I book with the Linux 44 or 49 kernel, it prints "starting version 233" then doesn't boot. Linux 411 rts gives a kernel panic