On KDE, whenever I decided to try using Wayland for a few days, I could edit a firefox-nightly.desktop file to read
Exec=env GDK_BACKEND=wayland /opt/firefox-nightly/firefox %u and it would work properly. Firefox would use it's native Wayland backend, kwin glitches and all.
This simply doesn't work with Gnome. To begin with, Gnome seems to completely ignore the
~/.local/share/applications folder. I had to manually move any custom .desktop files I had into /usr/share/applications which is a terrible idea but dammit I want this to work.
I checked with menulibre and the Firefox Nightly menu entry definitely has the proper Exec line. Just in case I ran the env variable and the command in a terminal and Firefox launched with the wayland backend like it's supposed to. But every time I launch Firefox via Gnome, X11 backend.
I hope I'm just being an idiot somewhere and my system isn't borked somehow.