Manjaro does no longer boot. grub broken

tristelune · 31 July 2017 23:47

Hi,

I used to have a Manjaro Cinnamon edition working fine, I had it installed as a dual boot alongside with win10 and was using grub to boot.

For some reason there was no way for me to boot again into Manjaro (booting into win10 was fine).
I've tried reinstalling grub (chroot from liveusb of manjaro) and now I cannot boot into anything.

I'll try to give more details in this post, but I'll be happy to give some more upon request.
When I could no longer boot into Manjaro, the message I would get was

Loading Linux 4.9.34-1-MANJARO x64 ...
error: invalid cluster 0.
Loading initial ramdisk ...
error: you need to load the kernel first

Press any key to continue...

When I updated my grub using the liveusb, I noticed win10 wasn't detected but I proceeded anyway thinking I could fix that later (no comment).
Grub menu was indeed updated but I still got the same error message and as the win10 options were gone, I couldn't boot into win10 anymore.

Here are some details about the partitions:

$ sudo fdisk -l
Disk /dev/sda: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0xb13e7242

Device     Boot     Start        End    Sectors  Size Id Type
/dev/sda1  *         2048     409599     407552  199M  7 HPFS/NTFS/exFAT
/dev/sda2          409600  195722099  195312500 93.1G  7 HPFS/NTFS/exFAT
/dev/sda3       195723262  349247487  153524226 73.2G  5 Extended
/dev/sda4       349247488 1953521663 1604274176  765G  7 HPFS/NTFS/exFAT
/dev/sda5       195723264  204111871    8388608    4G 82 Linux swap / Solaris
/dev/sda6       349042688  349247487     204800  100M  b W95 FAT32
/dev/sda7       204113920  205162495    1048576  512M  b W95 FAT32
/dev/sda8       205164544  246640639   41476096 19.8G 83 Linux
/dev/sda9       246642688  349040639  102397952 48.8G 83 Linux

Partition 3 does not start on physical sector boundary.
Partition table entries are not in disk order.

$ sudo lsblk -f
NAME   FSTYPE   LABEL    UUID                                 MOUNTPOINT
sda                                                           
├─sda1 ntfs     SYSTEM   2CCEBE2DCEBDEEE8                     
├─sda2 ntfs              66705B73705B4947                     
├─sda4 ntfs     STORAGE  2AE584E62A6AEF3F                     
├─sda5 swap              0e3274df-a705-469e-9ea7-719c989179e6 [SWAP]
├─sda6 vfat     HP_TOOLS 2BD0-62B5                            
├─sda7 vfat              C078-4715                            
├─sda8 ext4              9bf08a4e-2800-4e01-a797-8bfe9d7210be 
└─sda9 ext4              e38d00de-52f3-4044-8d1a-b4077ea32278

primary partitions sda1 and sda2 are where win10 is installed

primary partition sda4 is just a storage space.

sda3 is an extended partition that contains my Manjaro instalation :sda8 is the root install, sda9 is the home, sda7 is boot partition, sda5 swap (sda6 contains some laptop utilities from the manufacturer).

I hope there will be someone interested in helping me to solve that.
Thanks!

deathstalker · 1 August 2017 04:59

sudo grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=manjaro --recheck
have you try to run this command?

gohlip · 1 August 2017 09:59

Boot into the installed OS using this. Make sure you boot up in bios-legacy and not uefi. Remember, boot up to installed OS and not livecd OS.

If you cannot boot up using the link's configfile method (because in your chroot, you may have.... messed ..., never mind) do this instead at the grub prompt of the livecd.

grub> search -f /boot/intel-ucode.img --set=root
grub> ls ($root)/boot/

Copy down the kernel and initramfs, like vmlinuz-4.9-x86_64 and initramfs-4.9-x86_64.img

Continue...

grub> probe -u $root --set=abc
grub> linux /boot/vmlinuz-4.9-x86_64 root=UUID=$abc rw
grub> initrd /boot/initramfs-4.9-x86_64.img
grub> boot

use the correct kernel & initramfs you've copied if different.

When booted,

sudo pacman-mirrors -g
sudo pacman -Syyu
sudo pacman -S grub
sudo grub-install --target=i386-pc /dev/sda
sudo update-grub

Check that internal disk is sda, not sdb using
fdisk -l, otherwise use
sudo grub-install --target=i386-pc /dev/sdb
instead of /dev/sda above

ps: do not use "grub-install --target=x86_64-efi"

tristelune · 2 August 2017 19:23

Hi,

Boot into the installed OS using this. Make sure you boot up in bios-legacy and not uefi. Remember, boot up to installed OS and not livecd OS.

The install media versions referenced in this link is 17.0.1, the one I have is 17.0. I guess this is the reason why I couldn't use grub as boot mechanism.
I am now getting 17.0.2 install media, I hope it has that mechanism too.
I will write back when I have tried your instructions.

Thanks

ps: in my previous attempt with chroot I had used grub-install --target==i386-pc ...after having tried with x86_64-efi I must admit but that was returning immediately an error because of a missing boot/EFI directory if I remember well so I don't think (read hope) it did any harm.

tristelune · 2 August 2017 22:36

Hi,

Ok, I got a livecd OS 17.02 that lets me go to grub prompt pressing 'c'.
Then I tried the following:

grub> search -f /boot/intel-ucode.img --set=root

That didn't work... it returned:

error: no such device: /boot/intel-ucode.img

So I tried to look for a place in which that intel-ucode.img would be and found it in (hd1,msdos7)
I found it running this command:

grub> ls (hd1,msdos7)/
grub/ memtest86+/ vmlinuz-4.9-x86_64 intel-ucode.img initramfs-4.9-x86_64.img vmlinuz-4.9-x86_64.kver initramfs-4.9-x86_64-fallback.img System Volume Information/ $recycle.bin/

Previously I had found the (hd1,mdos7) thing running simple ls command.

But although ls of the intel-ucode.img file works, search command for the same file doesn't work so I guess something is wrong or there is a difference between ls and search that I don't pick up...

grub> ls (hd1,msdos7)/intel-ucode.img
intel-ucode.img
grub> search -f (hd1,msdos7)/intel-ucode.img
error: no such device: (hd1,msdos7)/intel-ucode.img

I will stop here and wait for some wise suggestions.
Thanks again

tristelune · 2 August 2017 23:20

Well, I decided to do some more trials before going to bed...

Combining the use of UUIDs and basic scripting

If you like the idea of using UUIDs to avoid unreliable BIOS mappings or are struggling with GRUB's syntax, here is an example boot menu item that uses UUIDs and a small script to direct GRUB to the proper disk partitions for your system. All you need to do is replace the UUIDs in the sample with the correct UUIDs for your system. The example applies to a system with a boot and root partition. You will obviously need to modify the GRUB configuration if you have additional partitions:

I found the following piece of code on the page https://wiki.archlinux.org/index.php/GRUB/Tips_and_tricks

menuentry "Arch Linux 64" {
    # Set the UUIDs for your boot and root partition respectively
    set the_boot_uuid=ece0448f-bb08-486d-9864-ac3271bd8d07
    set the_root_uuid=c55da16f-e2af-4603-9e0b-03f5f565ec4a

    # (Note: This may be the same as your boot partition)

    # Get the boot/root devices and set them in the root and grub_boot variables
    search --fs-uuid $the_root_uuid --set=root
    search --fs-uuid $the_boot_uuid --set=grub_boot

    # Check to see if boot and root are equal.
    # If they are, then append /boot to $grub_boot (Since $grub_boot is actually the root partition)
    if [ $the_boot_uuid == $the_root_uuid ] ; then
        set grub_boot=($grub_boot)/boot
    else
        set grub_boot=($grub_boot)
    fi

    # $grub_boot now points to the correct location, so the following will properly find the kernel and initrd
    linux $grub_boot/vmlinuz-linux root=/dev/disk/by-uuid/$the_root_uuid ro
    initrd $grub_boot/initramfs-linux.img
}

I tried that (not the menu part, just running some of the commands one by one. By some I mean I skipped the comparison and jumped directly to the command I needed to run), basically replacing the UUID with the values in my setting but then ...the linux command also returned the "invalid cluster 0" error message...
That worries me a bit, sounds more like there is an issue with that partition. But hopefully I'm being too pessimistic..

Good night for now.

gohlip · 3 August 2017 02:11

Morning.
Ah.... you have a separate boot partition. (hd1,msdos7)
And it's a vfat partition? Why?
In a msdos disk using bios-legacy boot, a /boot partition should be ext2.
And generally nowadays, a /boot partition is not necessary.

Never mind, lets proceed. The command therefore should be.. (without .... /boot/xxxx) both in the link and in the above command.

grub> insmod fat
grub> search -f /intel-ucode.img --set=root

But now I don't know where your Manjaro root partition is, sda8 or sda9
You should know and lets proceed. I assume it is sda8 here, change to sda9 if that is so.
Also, and important, your 'sda' device can be listed (like your commands above) is (hd1) not (hd0) even though it is sda.
So be very sure it is (hd1) or (hd0) before we proceed. Use "grub> ls" again to be sure.

Now I'll use (hd1,7) as /boot
and (hd1,8) as / root partition. (below as /dev/sdb8)
Change to correct one, if not so.

grub> insmod fat
grub> set root=(hd1,7)
grub> linux /vmlinuz-4.9-x86_64 root=/dev/sdb8 rw
grub> initrd /initramfs-4.9-x86_64.img
grub> boot

tristelune · 3 August 2017 12:48

I will try that when I get home tonight.
I remember from the 'grub> ls' command that my root is sda8 and this is hd1. I will double check before running the command but in case this is like that, shouldn't it be /dev/sda8 in the 'linux' command?
Thanks

gohlip · 3 August 2017 13:20

No! Careful. if a partition is (hd0,8), it is /dev/sda8.
If it is (hd1,8), it is /dev/sdb8.

So at "grub> ls" command,
when your internal device is listed as (h0), your / partition is (hd0,8) and /dev/sda8 and "set root=(hd0,7)"
When it is (hd1), your / partition is (hd1,8) and /dev/sdb8 and "set root=(hd1,7)"

When in doubt, use another command and verify it is the correct Manjaro partition

grub> cat (hd0,8)/etc/lsb-release
grub> cat (hd1,8)/etc/lsb-release

The output will tell you if that is Manjaro partition.
Reminder if set root=(hd1,7), then linux line ==> root=/dev/sdb8
if set root=(hd0,7), then linux line ==> root=/dev/sda8

That's why grub do not use device mapping but UUID.

tristelune · 3 August 2017 22:10

Thanks for the clarification.
I checked that the root and boot are indeed on hd1, hence sdb (I was confused because this disc was mapped as sda when booting from the livecd, see my first post), and here is what I get:

error: invalid cluster 0.

If, right after that, I try the same 3 commands again, I don't even get the error message, I get no response and it seems all is stuck and forced shutdown is the only way out.

tristelune · 3 August 2017 22:39

While waiting for some fast reply and anticipating it might actually be a partition issue, I booted livecd to run fsck on that boot partition:

$ sudo fsck -f /dev/sda7
fsck from util-linux 2.29.2
fsck.fat 4.1 (2017-01-24)
0x41: Dirty bit is set. Fs was not properly unmounted and some data may be corrupt.
1) Remove dirty bit
2) No action
? 2
There are differences between boot sector and its backup.
This is mostly harmless. Differences: (offset:original/backup)
  65:01/00
1) Copy original to backup
2) Copy backup to original
3) No action
? 3
/vmlinuz-4.9-x86_64
  Contains a free cluster (6988). Assuming EOF.
/vmlinuz-4.9-x86_64
  File size is 4909344 bytes, cluster chain length is 0 bytes.
  Truncating file to 0 bytes.
/initramfs-4.9-x86_64.img
  Contains a free cluster (11372). Assuming EOF.
/initramfs-4.9-x86_64.img
  File size is 6338991 bytes, cluster chain length is 0 bytes.
  Truncating file to 0 bytes.
/linux49-x86_64.kver
  Contains a free cluster (8189). Assuming EOF.
/linux49-x86_64.kver
  File size is 21 bytes, cluster chain length is 0 bytes.
  Truncating file to 0 bytes.
/initramfs-4.9-x86_64-fallback.img
  Contains a free cluster (1483). Assuming EOF.
/initramfs-4.9-x86_64-fallback.img
  File size is 22544485 bytes, cluster chain length is 0 bytes.
  Truncating file to 0 bytes.
Reclaimed 8245 unused clusters (33771520 bytes) in 4 chains.
Perform changes ? (y/n) n
/dev/sda7: 370 files, 11432/130812 clusters

As you can see I didn't perform any action.
My impression is that it doesn't look good and I don't feel confident with the suggested changes ("Truncating file to 0 bytes", hmm...)

what do you think?

I ran the same on sda8, I still did not perform any of the suggested actions but they look safer to run.

$ sudo fsck -f /dev/sda8
fsck from util-linux 2.29.2
e2fsck 1.43.4 (31-Jan-2017)
/dev/sda8: recovering journal
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong (2711712, counted=2711559).
Fix<y>? no
Free inodes count wrong (1071644, counted=1071572).
Fix<y>? no
/dev/sda8: 225796/1297440 files (0.2% non-contiguous), 2472800/5184512 blocks

gohlip · 4 August 2017 00:26

Morning. Right. Doesn't look good.

Back up your data (if you can) and be ready to do a reinstall.
Then we'll try to fsck the partitions (answer 'y') and do again the commands.
And always check hd0 or hd1 each time you start computer.

If you have to reinstall and you do not want to lose your windows, leave your sda7 (fat partition) alone (don't touch it) and let the boot be in the root partition (sda8 - ext4).

And good luck.

ps: as to being mapped as sda after booting up either in livecd or in OS, that's the nature of the bios and OS which may not correspond. Has always been like that as the bios was developed long ago for a single disk environment. It's amazing actually the bios has evolved to handle things (other than multi disks) it was not designed to.

tristelune · 5 August 2017 01:50

Hi,

So I backed up my data and ran fsck.
fsck didn't give any error when it ended (sorry I forgot to get a copy of it to paste here) but it wasn't clear if it had ended up well either..

So I ran these commands again:

grub> insmod fat
grub> set root=(hd1,7)
grub> linux /vmlinuz-4.9-x86_64 root=/dev/sdb8 rw

And got this message:
error: premature end of file /vmlinuz-4.9-x86_64.

This looks to me that fsck couldn't fix the issue with the partition.

I went back to livecd to have a look at the partition I tried to fix and it looked like that:

[manjaro@manjaro-cinnamon C078-4715]$ ls -al
total 33968
drwxr-xr-x  2 manjaro manjaro     4096 Mar 31 13:36 '$RECYCLE.BIN'
drwxr-xr-x  6 manjaro manjaro     4096 Jan  1  1970  .
drwxr-x---+ 3 root    root          60 Aug  5 00:57  ..
-rw-r--r--  1 manjaro manjaro  4907008 Jan  1  1980  FSCK0000.REC
-rw-r--r--  1 manjaro manjaro     4096 Jan  1  1980  FSCK0001.REC
-rw-r--r--  1 manjaro manjaro  6328320 Jan  1  1980  FSCK0002.REC
-rw-r--r--  1 manjaro manjaro 22532096 Jan  1  1980  FSCK0003.REC
drwxr-xr-x  2 manjaro manjaro     4096 Mar 31 13:34 'System Volume Information'
drwxr-xr-x  6 manjaro manjaro     4096 Jul 31 22:45  grub
-rw-r--r--  1 manjaro manjaro        0 Jun 26 22:41  initramfs-4.9-x86_64-fallback.img
-rw-r--r--  1 manjaro manjaro        0 Jun 26 22:41  initramfs-4.9-x86_64.img
-rw-r--r--  1 manjaro manjaro   987648 May 13 09:38  intel-ucode.img
-rw-r--r--  1 manjaro manjaro        0 Jun 24 07:08  linux49-x86_64.kver
drwxr-xr-x  2 manjaro manjaro     4096 Aug  3  2016  memtest86+
-rw-r--r--  1 manjaro manjaro        0 Jun 24 07:08  vmlinuz-4.9-x86_64

These FSCK000x.REC files would be what fsck managed to recover so I thought I could rename them to whatever they were previously and see if I had more luck...
At this point the option was to reinstall and I could spare a little more time.

So here's what I've done:

[manjaro@manjaro-cinnamon C078-4715]$ file FSCK0000.REC
FSCK0000.REC: Linux kernel x86 boot executable bzImage, version 4.9.35-1-MANJARO (builduser@manjaro) #1 SMP PREEMPT Fri Jun 30 06:42:12 UTC 2017, RO-rootFS, swap_dev 0x4, Normal VGA
[manjaro@manjaro-cinnamon C078-4715]$ file FSCK0001.REC
FSCK0001.REC: data
[manjaro@manjaro-cinnamon C078-4715]$ file FSCK0002.REC
FSCK0002.REC: gzip compressed data, last modified: Wed Jul  5 20:21:18 2017, from Unix
[manjaro@manjaro-cinnamon C078-4715]$ file FSCK0003.REC
FSCK0003.REC: gzip compressed data, last modified: Wed Jul  5 20:21:22 2017, from Unix
[manjaro@manjaro-cinnamon C078-4715]$ ls grub/
fonts  grub.cfg  grubenv  i386-pc  locale  themes
[manjaro@manjaro-cinnamon C078-4715]$ mv initramfs-4.9-x86_64-fallback.img BKP_initramfs-4.9-x86_64-fallback.img
[manjaro@manjaro-cinnamon C078-4715]$ mv initramfs-4.9-x86_64.img BKP_initramfs-4.9-x86_64.img
[manjaro@manjaro-cinnamon C078-4715]$ mv linux49-x86_64.kver BKP_linux49-x86_64.kver
[manjaro@manjaro-cinnamon C078-4715]$ mv vmlinuz-4.9-x86_64 BKP_vmlinuz-4.9-x86_64
[manjaro@manjaro-cinnamon C078-4715]$ ls -al
total 33968
drwxr-xr-x  2 manjaro manjaro     4096 Mar 31 13:36 '$RECYCLE.BIN'
drwxr-xr-x  6 manjaro manjaro     4096 Aug  5 01:08  .
drwxr-x---+ 3 root    root          60 Aug  5 00:57  ..
-rw-r--r--  1 manjaro manjaro        0 Jun 26 22:41  BKP_initramfs-4.9-x86_64-fallback.img
-rw-r--r--  1 manjaro manjaro        0 Jun 26 22:41  BKP_initramfs-4.9-x86_64.img
-rw-r--r--  1 manjaro manjaro        0 Jun 24 07:08  BKP_linux49-x86_64.kver
-rw-r--r--  1 manjaro manjaro        0 Jun 24 07:08  BKP_vmlinuz-4.9-x86_64
-rw-r--r--  1 manjaro manjaro  4907008 Jan  1  1980  FSCK0000.REC
-rw-r--r--  1 manjaro manjaro     4096 Jan  1  1980  FSCK0001.REC
-rw-r--r--  1 manjaro manjaro  6328320 Jan  1  1980  FSCK0002.REC
-rw-r--r--  1 manjaro manjaro 22532096 Jan  1  1980  FSCK0003.REC
drwxr-xr-x  2 manjaro manjaro     4096 Mar 31 13:34 'System Volume Information'
drwxr-xr-x  6 manjaro manjaro     4096 Jul 31 22:45  grub
-rw-r--r--  1 manjaro manjaro   987648 May 13 09:38  intel-ucode.img
drwxr-xr-x  2 manjaro manjaro     4096 Aug  3  2016  memtest86+
[manjaro@manjaro-cinnamon C078-4715]$ mv FSCK0000.REC vmlinuz-4.9-x86_64
[manjaro@manjaro-cinnamon C078-4715]$ mv FSCK0003.REC initramfs-4.9-x86_64-fallback.img
[manjaro@manjaro-cinnamon C078-4715]$ mv FSCK0002.REC initramfs-4.9-x86_64.img
[manjaro@manjaro-cinnamon C078-4715]$ ls -al
total 33968
drwxr-xr-x  2 manjaro manjaro     4096 Mar 31 13:36 '$RECYCLE.BIN'
drwxr-xr-x  6 manjaro manjaro     4096 Aug  5 01:13  .
drwxr-x---+ 3 root    root          60 Aug  5 00:57  ..
-rw-r--r--  1 manjaro manjaro        0 Jun 26 22:41  BKP_initramfs-4.9-x86_64-fallback.img
-rw-r--r--  1 manjaro manjaro        0 Jun 26 22:41  BKP_initramfs-4.9-x86_64.img
-rw-r--r--  1 manjaro manjaro        0 Jun 24 07:08  BKP_linux49-x86_64.kver
-rw-r--r--  1 manjaro manjaro        0 Jun 24 07:08  BKP_vmlinuz-4.9-x86_64
-rw-r--r--  1 manjaro manjaro     4096 Jan  1  1980  FSCK0001.REC
drwxr-xr-x  2 manjaro manjaro     4096 Mar 31 13:34 'System Volume Information'
drwxr-xr-x  6 manjaro manjaro     4096 Jul 31 22:45  grub
-rw-r--r--  1 manjaro manjaro 22532096 Jan  1  1980  initramfs-4.9-x86_64-fallback.img
-rw-r--r--  1 manjaro manjaro  6328320 Jan  1  1980  initramfs-4.9-x86_64.img
-rw-r--r--  1 manjaro manjaro   987648 May 13 09:38  intel-ucode.img
drwxr-xr-x  2 manjaro manjaro     4096 Aug  3  2016  memtest86+
-rw-r--r--  1 manjaro manjaro  4907008 Jan  1  1980  vmlinuz-4.9-x86_64

I did some file size matching looking at a manjaro install I have on another laptop.
As you can see there is one file I haven't renamed, I don't know what it was, from the size it looks like it could be a directory but no idea what it could be.

After doing that I ran these commands again:

This time no error and it could boot ...until it stopped complaining about /dev/sdb8 that couldn't be found or mounted and it throw me into some strange shell ([rootfs ] if I remember well).
(Sorry I don't have more detail about the error but I didn't feel like writing it down and my memory has its limits..)

Anyway, I had noticed earlier that the livecd had an option to boot to the manjaro install it had found on hd1,8 and I thought I could give this a try (I had tried it before my fsck experiments and it had failed like it did from the grub shell).
This brought me to the grub menu and I could boot successfully!

I am pretty happy about this. Thank you very much for your help @gohlip

I am now updating the system, I hope it was not a stupid idea.
Then I will try another reboot before I can go happily to bed.

The next step is to get the win10 stuff back into my grub menu ..or maybe I will wait a bit for that.
Expect more questions from my side some day.

Cheers

tristelune · 5 August 2017 01:54

I had to send this update: after reboot win10 was back in the grub menu!!
...but I didn't dare to try it yet, I prefer to sleep on the previous good result.

system · 4 September 2017 01:54

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.