UEFI Partion flags $esp boot brainstorming

Continued from [Manual Partitioning - boot, esp flag]

You can have as many as you need and most often you only have one. Even if you are dual-booting several Linux distroes you can get along with one if the size is reasonable.

The problem manifests in dual boot sceanario which includes Windows.

In that case it an advantage to have two because Windows might change the $esp and you loose your Manjaro entry.

That issue can - to great extend - be avoided by having two $esp.

If you boot a live environment and select Detect EFI boot loaders option you will find that it contains entry like (hd0.gpt12) which refers to the loaders location on disk.

What theoreticly - in a Window-Linux dual-boot scenario - can cause issue with two $esp on same drive is when you decide to remove the Windows part and merge the partition with your root partiton. That would be impossible because you cannot move your second $esp without breaking your boot.

But that is a theoretical issue - as Linux does not care how the data is organized - only humans do.

To clarify any misunderstanding, when you flag a second partition as $esp, will you see 2 partitions flagged as $esp in gparted? Check how I explained in my post please, so I ll be sure we are on the same page.

You will - that is correct.

The boot flag just indicates that the partion is intended to boot a system.

The partition mounted as $esp only contains the files/folders below /boot/efi/ and because it is a FAT32 filesystem there is no significance to casing of the folder/file names (seen from efi pov).

/boot/
├── efi
│   └── EFI
│       ├── boot
│       │   └── bootx64.efi
│       └── Manjaro
│           └── grubx64.efi
├── grub
│
1 Like

I don't know what to think but you are on to something I have not thought of.

In UEFI/GPT schema the actual boot order is not written to the disk (as BIOS/MBR does), instead the boot order is residing in the UEFI system variables, which is written in the efivarfs.

The boot related variables can be listed and manipulated using efibootmgr CLI.

The fact remains that the UEFI firmware only holds pointers to the partitions marked as boot/esp which in turn holds the efi stubs available. If you neglect to format those as FAT32 you will get a filesystem unknown trying to boot from such a partition.

In GPT schema you can create 128 partition on a single disk (only 4 primary on MBR disk) and since no information is written on disk in relation to which partition is bootable but partitions is assigned flags for their function it is perfectly valid to have more than one partition flagged as boot and $esp.

The picture above from a different post perfectly illustrates that. The user who took the picture had several Manjaro installations which failed to boot and the user had tried several times to install. That created in the process several partitions marked as boot/esp.

So your bitwise argument does not quite hold as I see it or am I missing something?

In fact in a UEFI firmware you have no disk boot order - but you have named boot entries which hold the pointer to a specific partition and the name of the loader matches the a folder name in said partition eg. efi/manjaro/.

I can't say I am absolutely certain on this, that's why I only comment to get more valuable info/knowledge.
My "sin" is that I followed some discussion on calamares github about $esp flagging and what my thought was :

  • In GPT it is $esp=$boot bit-wise (it's the same bit)
  • Then, since it is sensible to say that it can only be one $boot, so one $esp (unless this is wrong assumption)
  • It is also known that you can have multiple UEFI "boot partitions" on the same disk, containg what we say "/boot/efi"
  • I understand that Linux doesn't actually care of the $esp flag, while checking for UEFI entries which contain partition info to navigate to the correct partition
  • The opposite (almost) I guess is for WinOS which believes (egoistically) "there could be only one" and forces any overwrites it likes when updates boot entries, that's why it messes UEFI old entries of Linux ones.
  • So your idea of having a second "$esp" partition (not necessarily flagged as such, is my opposition) is really good, which escapes from WinOS overwrites, while UEFI still sees it, IF WinOS does not delete (in some way) the old boot UEFI entries.

I hope this discussion may clarify better what happens with this mess and I really wonder why @gohlip has not got in the discussion to enlighten with his wide experience :grin:

@AgentS

I am beginning to think you could be right about the boot flag.

When manipulating your EFI entries with efibootmgr only one can be active - or the first in the list.

~ >>> efibootmgr
BootCurrent: 000A
Timeout: 1 seconds
BootOrder: 000A,0003,0004
Boot0003* Generic Usb Device
Boot0004* CD/DVD Device
Boot000A* UEFI OS

In Windows - under Advanced recovery - you can set which device to boot on after reboot.

I think they use a similar CLI to set the active EFI entry

sudo efibootmgr -b 0004 -a

So while you can have several $esp partitions - only one of them can be active - the one to be loaded at boot.

1 Like

that was the reason of quoting BTW :joy: , valid but not with the $esp flag.

1 Like

Wouldn't that be byte?

Lets image a byte with 8 bits - quite normal - bit setting is arbitrary

  • bit 2 is $boot
  • bit 3 is $esp

Then by setting the flags in Calamares

  • byte is 00000000 this is not $boot (active) nor $esp
  • byte is 00001100 this the active $boot and $esp
  • byte is 00001000 this is $esp but not active $ɓoot

Am I getting you right?

Because my capable partner here is already on it. :grin:
Also, I've said may times before I have 3 to 4 $esp partitions on each of my 3 to 4 disk.
And I had said there is no difference whether any OS should be on different disk or another disk despite many people saying it's best to have on separate disks. Or no difference if $esp should be on the same disk or on different disks.

As to flags, I have (had) all my $esp's flagged.
Then I removed all flags.
Then I put in a flag on a partition where it shouldn't be.
All makes no difference (to me).
And many websites say linux does not require boot flags, only windows boot need it.
But.... some here have problems with booting if the flag is not there.
I myself have helped resolved some issues here where I asked the OP's to put a flag and it resolved na....

I suspect that some (not many, BTW, just a few) here needs flags because their disks are not set up properly (partitioned 'gpt' or 'msdos') in the first place and their bios needs help in determining what the disks are. And suspicion alone is never a conclusive determinant.

petsam seriously, I think you are more capable than you think
and I'm more than happy whenever you help out.

2 Likes

I think this was the discussion that gave me the ideas

Wrong.

in the MSDOS partitioning scheme, boot and esp flags are distinct; boot=1 (bit 0) and esp=131072 (bit 17) in the flag word
in the GPT partitioning scheme, boot and esp are synonyms; boot=1 (bit 0) and esp is too
Source: https://www.gnu.org/software/parted/manual/parted.html#set

KPMCore has an enum for these, which names FlagBoot and FlagEsp according to the MSDOS scheme. Calamares code checks for FlagEsp in various places -- i.e. using the MSDOS flag value (bit 17) even when Esp should alias to Boot (bit 0). So that is why it is not finding your Esp (Boot) partition on a GPT-style partition table -- it is looking at the wrong bit.

Source

1 Like

Thank you for enlightening me.

Most valuable info. :+1:

1 Like

So what is the flags in Calamares used for?

They must somehow tell Calamares that we want to designate the selected partition as $esp.

That selection must be used by Calamares to assign the right partition type.

That partition type is EF00 one of the types available when partitioning using a CLI tool.

But why create all that confusion? That beats me.

But the latest Calamares does not assign EF00 to a partition marked as $esp - in some cases it assigns 0700 Microsoft basic data. Probably caused by the FAT32 format.

But even that partition type does not break boot - It only came to my attention when I several times had my $esp partition set as 0700 and I began to dive into it.

2 Likes

It maybe time for @philm to explain how it goes, since he is one of Calamares devs :point_left:.

1 Like

hi @gohlip
may be @AgentS and @linux-aarhus not have read this

1 Like

The conclusion as I see it for now is that you can have several partitions with the partition type EF00 which defines the partition as $esp.

Which of the $esp partition is the active partion is defined by the BootOrder item listed when efibootmgr without arguments.

The BootOrder can be modified by executing efibootmgr with argument -b nnnn and -a.

I have once tested that I can - in the EFI firmware - remove my disks from all boot sequences and only rely on the firmwares boot entries.

Those boot entries correspond 1 to 1 by the folder name in the $esp folder - usually located in (hdX.gptY)/EFI/foldername/efistub

From my limited experience of calamares installer :

  1. Manual Partitioning would not allow you to progress with installation without adding boot and esp flags- hence i did not use it
  2. I used the other option to 'Replace partition' - it did not ask for those flags to be added. I was triple booting - windows, ubuntu and mint - I replaced ubuntu partition with manjaro but ended with no boot manager entry - which then I added using tutorials on this forum.

In my extensive, nay... exclusive use of manual installation (in all distros), selecting the $esp partition with mount point /boot/efi will always proceed and it will automatically add boot and esp flags to it.

To be clear...

That happens... but as a failed bootloader installation. All else of installation works.

I also almost exclusively use the manual partition method.

It is nice to know that the selection of the mountpoint - triggers the $esp partition type.

That removes a couple of clicks :slight_smile:

Forum kindly sponsored by