Failed to start and chroot to fix system image

[splatch]

Hi all,
I come into forum as I ran into problem which I can not solve with help of existing materials in forums and wiki. I followed system recovery category but it didn't bring me solution for my issue.

I been running an installation of homegear package which brings some additional kernel modules (libhomegear) which failed (maybe because) a dummy docker thing which acquired access to /lib, /log, and /etc directories (wrong form of docker volume mount). As a final result of these two actions I ended up with destabilized system where I could not execute any command (ie. ls command not found).
After restart my system shows boot manager, it begins to start up but fails to due to missing sbin/init:

ERROR: Boot device mounted successfully, but /sbin/init does not exist.
Bailling out, you are on your own. Good luck.

sh: can't access tty: job control turned off.
[rootfs ]# _

System information

System:    Host: manjaro-gnome Kernel: 5.3.11-1-MANJARO x86_64 bits: 64 compiler: gcc v: 9.2.0 Console: tty 0 wm: gnome-shell 
           dm: GDM 3.34.1 Distro: Manjaro Linux 
Machine:   Type: Laptop System: Dell product: XPS 13 9380 v: N/A serial: <filter> Chassis: type: 10 serial: <filter> 
           Mobo: Dell model: 0KTW76 v: A00 serial: <filter> UEFI: Dell v: 1.7.0 date: 08/05/2019 
Battery:   ID-1: BAT0 charge: 51.6 Wh condition: 51.6/52.0 Wh (99%) volts: 8.6/7.6 model: SMP DELL G8VCF6C type: Li-poly 
           serial: <filter> status: Full 
CPU:       Topology: Quad Core model: Intel Core i7-8565U bits: 64 type: MT MCP arch: Kaby Lake rev: C L1 cache: 256 KiB 
           L2 cache: 8192 KiB L3 cache: 8192 KiB 
           flags: avx avx2 lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx bogomips: 32012 
           Speed: 4069 MHz min/max: 400/4600 MHz Core speeds (MHz): 1: 4080 2: 4080 3: 4084 4: 4097 5: 4096 6: 3994 7: 4068 
           8: 3797 
Graphics:  Device-1: Intel UHD Graphics 620 vendor: Dell driver: i915 v: kernel bus ID: 00:02.0 chip ID: 8086:3ea0 
           Display: server: X.org 1.20.5 driver: i915 compositor: gnome-shell resolution: <xdpyinfo missing> 
           OpenGL: renderer: Mesa DRI Intel UHD Graphics (Whiskey Lake 3x8 GT2) v: 4.5 Mesa 19.2.4 compat-v: 3.0 
           direct render: Yes 
Audio:     Device-1: Intel Cannon Point-LP High Definition Audio vendor: Dell driver: snd_hda_intel v: kernel bus ID: 00:1f.3 
           chip ID: 8086:9dc8 
           Sound Server: ALSA v: k5.3.11-1-MANJARO 
Network:   Device-1: Qualcomm Atheros QCA6174 802.11ac Wireless Network Adapter 
           vendor: Bigfoot Networks Killer 1435 Wireless-AC driver: ath10k_pci v: kernel port: efa0 bus ID: 02:00.0 
           chip ID: 168c:003e 
           IF: wlp2s0 state: down mac: <filter> 
           Device-2: Realtek RTL8153 Gigabit Ethernet Adapter type: USB driver: r8152 bus ID: 4-1.2:3 chip ID: 0bda:8153 
           serial: <filter> 
           IF: ens12u1u2 state: up speed: 1000 Mbps duplex: full mac: <filter> 
Drives:    Local Storage: total: 484.21 GiB used: 120.34 GiB (24.9%) 
           ID-1: /dev/nvme0n1 vendor: Samsung model: PM981 NVMe 512GB size: 476.94 GiB speed: 31.6 Gb/s lanes: 4 
           serial: <filter> rev: EXA74D1Q scheme: GPT 
           ID-2: /dev/sda type: USB model: USB Flash Memory size: 7.27 GiB serial: <filter> rev: PMAP scheme: MBR 
Partition: ID-1: / size: 11.47 GiB used: 374.6 MiB (3.2%) fs: overlay source: ERR-102 
Sensors:   System Temperatures: cpu: 49.0 C mobo: N/A 
           Fan Speeds (RPM): N/A 
Info:      Processes: 224 Uptime: 17h 18m Memory: 15.30 GiB used: 1.33 GiB (8.7%) Init: systemd v: 242 Compilers: gcc: N/A 
           Shell: bash (sudo) v: 5.0.11 running in: gnome-terminal inxi: 3.0.36

After start from livecd I am able to mount system, however I can not run bash. All attempts which I made ends up with the same result. I used arch-chroot but it fails with same message.

chroot /mnt/manjaro
chroot: failed to run command ‘/bin/bash’: Permission denied

File system is mounted properly (I adjusted fstab):

/dev/nvme0n1p7 on /mnt/manjaro type ext4 (rw,relatime)
dev on /mnt/manjaro/dev type devtmpfs (rw,nosuid,relatime,size=7996912k,nr_inodes=1999228,mode=755)
proc on /mnt/manjaro/proc type proc (rw,nosuid,nodev,noexec,relatime)
sys on /mnt/manjaro/sys type sysfs (rw,nosuid,nodev,noexec,relatime)

The bash itself is executable:

[manjaro-gnome manjaro]# stat /mnt/manjaro/usr/bin/bash
  File: /mnt/manjaro/usr/bin/bash
  Size: 903440    	Blocks: 1768       IO Block: 4096   regular file
Device: 10307h/66311d	Inode: 3147892     Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-11-20 12:14:38.180000545 +0000
Modify: 2019-09-03 19:27:18.000000000 +0000
Change: 2019-11-20 12:14:31.853333873 +0000
 Birth: 2019-10-14 14:36:34.492363214 +0000

I can run program with no issues so its not a problem of noexec or anything like that. I failed to obtain any knowledge (so far) on how to fix chroot issue. I execute it as a root user thus I expect that it should work with no issues.

I need to get into chroot to run mkinitcpio. Thank you in advance for your answers.

Kind regards,
Łukasz

[Wollie]

Have a look here, especially paragraph 3:

Can't login - black screen & chroot guide

If you are stuck and you can't log into your system, use one of these steps below. They are sorted from easiest at the top. Start with the first step and if it doesn't work, try with the next one and so on.. Once you get access to the system, then proceed with updating the system or installing/removing packages, checking log files... Steps (you only need one) are: 1 - log to tty terminal 2 - boot without X 3 - chroot auto (with manjaro-chroot) 4 - chroot (with manjaro-chroot) 5 - chroot ol…

[splatch]

Trying that from livecd, still no lock. I am getting following results:

[manjaro-gnome /]# manjaro-chroot -a
==> Detected systems:
 --> 0) ManjaroLinux
==> Select system to mount [0-0] : 
0
==> ERROR: You can't mount 0!

However, when I type 1 I get my system partition (nvme0n1p7), with same results as before:

[manjaro-gnome /]# manjaro-chroot -a
==> Detected systems:
 --> 0) ManjaroLinux
==> Select system to mount [0-0] : 
1
==> Mounting (ManjaroLinux) [/dev/nvme0n1p7]
 --> mount: [/mnt]
 --> mount: [/mnt/boot/efi]
chroot: failed to run command '/bin/bash': Permission denied
 --> umount: [/mnt/boot/efi]
 --> umount: [/mnt]

Main thing which I found confusing is how to really debug chroot and reason why I am getting permission denied on /bin/bash if it can be executed without chroot. Additional question which arises is why my system partition is not detected.

[Keruskerfuerst]

Your system seems heavily damaged.

You can also try to chroot into the system manually.

[SGS]

@splatch

Permission denied

You need sudo

sudo manjaro-chroot -a

[splatch]

All commands are executed from root level. I doubt if sudo makes any change there, does it?

The filesystem itself looks fine. I see no damage in there, most of things is in place. I compared directory layout of livecd and manually mounted /mnt and found no differences. All directories such bin, lib, lib64 are there. After all, if I am able to launch /mnt/bin/bash chroot should be also fine. However for some reason its not.

[SGS]

@splatch

I doubt if sudo makes any change there, does it?

Give it a try

[splatch]

I just did:

[manjaro-gnome manjaro]# id 
uid=0(root) gid=0(root) groups=0(root)
[manjaro-gnome manjaro]# whoami
root
[manjaro-gnome manjaro]# sudo manjaro-chroot -a
==> Detected systems:
 --> 0) ManjaroLinux
==> Select system to mount [0-0] : 
1
==> Mounting (ManjaroLinux) [/dev/nvme0n1p7]
 --> mount: [/mnt]
 --> mount: [/mnt/boot/efi]
chroot: failed to run command '/bin/bash': Permission denied
 --> umount: [/mnt/boot/efi]
 --> umount: [/mnt]

If I select 0 I get no results as pointed earlier. For reference, same happens when I do sudo from manjaro user level. Options might have +1 offset as described in: Chroot not working "==> ERROR: YOU CANT MOUNT 0!".

[SGS]

Ok, seems /dev/nvme0n1 make trouble, is this a dual boot system with windows, maybe it block write access.

[splatch]

That's what I was thinking and struggling for a moment with manual/user mount which automatically been adding "defaults" and noexec option. manjaro-chroot however is trying to get that around. Also I been trying with manual mount just to make sure that filesystem is read-write and exec are ... well executable.

[manjaro-gnome mnt]# mount | grep -i nvme
/dev/nvme0n1p7 on /mnt type ext4 (rw,relatime)
[manjaro-gnome mnt]# pwd
/mnt
[manjaro-gnome mnt]# echo X > foo.txt
[manjaro-gnome mnt]# cat foo.txt 
X
[manjaro-gnome mnt]# rm foo.txt 

[SGS]

I am out of ideas, sorry.

[Keruskerfuerst]

If you have partitioned the SSD correctly, then a reinstallation is easy.

[splatch]

Digging a little bit into my issue and reasons why I get permission denied. The docker volume mounts I had for /lib /etc and /log had :Zsuffix.

According to stackoverflow answer https://stackoverflow.com/questions/35218194/what-is-z-flag-in-docker-containers-volumes-from-option as well as docker docs https://docs.docker.com/storage/bind-mounts/#configure-the-selinux-label it turns out that I did something extremely stupid:

The Z option indicates that the bind mount content is private and unshared. Use extreme caution with these options. Bind-mounting a system directory such as /home or /usr with the Z option renders your host machine inoperable and you may need to relabel the host machine files by hand.

So now I need to dig how to read and remove these labels from files if they were applied.

[splatch]

Fixing the issues, managed to get system into bootable state:

1. livecd, manual mount, chmod +x /lib /etc - this silly thing was necessary despite of all selinux searches I did.
2. manjaro-chroot -a
3. reinstallation of linux419 via mwhd-kernel -r linux419 && mwhd-kernel -i linux419
4. running pacman -S linux513 linux513-headers to fetch modules
5. execution of pacman-permission-fix
6. restart
   Turns out that X is still not in the shape
7. reinstalling all packages (pacman -Qq + pacman -S --noconfirm)

Still for some mysterious reason I am missing /lib/modules after every restart. Not quite sure why.

I was able so far to get back to X interface. The yaourt packages are probably also affected, however these are minor trouble as they can be reinstalled on demand.

[Wollie]

Be aware that yaourt is not supported any more, not recommendable to continue to use it, see here:

Don't Install Yaourt! Use These Alternatives for AUR in Arch Linux