RTX 5080 Upgrade/Sidegrade from RTX 3090 massive reduction in performance when I open SteamVR Overlay

Hello,

so this issue has been annoying me for a while now. I was hoping that it would get resolved at some point but I also cannot find anyone reporting this specific issue. Im posting here in the hopes that I just have something set up wrong and that it can be resolved.

My game runs fine generally, its at a state where I would say its playable even in heavily crowded worlds, sure, its not ideal at 30fps but thats how it is.
I am not complaining about that part, my issue is, that when I am in a place like that and I have no shield level that when I open the SteamVR overlay everything goes to shit.

I cant even exit the game properly in that state, the SteamVR overlay is unresponsive, my FPS drops to what feels like 1FPS, the delay goes up to 200ms (so thats 5FPS I guess) and I do not know how to resolve this issue at all.

I dont know if its my operating system thats causing this (Windows 11, I turned off Hardware Accelerated GPU scheduling after some stuff I found online but it didnt make a difference)

My specs are as follows

RTX 5080 (I had a 3090 before and never had this issue. Sure the fps would drop as well and sure, I would saturate the VRAM but I was still able to control my PC, even my friend on a 5070 with everyone shown and 16gb of VRAM used doesnt have this problem)
Ryzen 9 7950X3D
64GB DDR-5 6000
nVidia Drivers are on version 591.4 (I know I’m behind but the last 20 updates also didnt fix shit)

So yeah, this is where im at.

This is a video of what it looks like in the HMD.

If anyone has a clue please chime in, I am so close to just selling the 5080 and going for a 3090/4090

Thanks in advance