Use Nvidia PRIME and dynamic bind/unbind drivers

So, I'm trying to do some very hacky stuff which I'm sure is possible, I just need help figuring it out.
The main question I would like some help to answer is the following: Is there any way to bind/unbind the nvidia drivers from the GPU (to be able to use it for GPU passthrough when needed) while using nvidia-prime?

I've been banging my head for the past 3 days trying to find a way to do it, but no luck so far. If I understand correctly, when you use PRIME there's a secondary Xorg instance running in the background, and frames are copied from there onto the main instance (nvidia-smi confirms the GPU is being used by Xorg). What I assume would need to be done is kill said Xorg instance and then attempt to unbind the drivers, but so far I've not found a good way to do this, and trying to just unbind the drivers fails every time since they are being used.

My system consists of 2 dGPUs, one weak(ish) one (AMD R9 270X) and one high end one (NVIDIA GTX 1070). I could simply pass through the NVIDIA GPU the normal way, but I still want to make use of it on the host when not running any VMs, mainly to run games. This is why I am trying to use the hybrid drivers, to have the low end GPU giving me video output all the time, yet also being able to use a high end GPU to do all the heavy lifting when needed.

Any help would be greatly appreciated!

Forum kindly sponsored by