Performance issue I've had forever. Desktop & VR (LOW CPU AND GPU UTILISATION)

It’s quite the struggle to enjoy VR at 40fps as many people know, so I ended up checking my computers stats on RTSS.

On desktop mode:
In an empty world I get about 150-200fps with 30% utilisation on the GPU and 5-10% on CPU, (no cores maxed out) (no thermal throttling).

Any level with more than 6-10 avatars on my screen stomps the fps down to 40 or lower. The GPU utilisation also lowers to 3-8%. CPU stays about the same.

Things I can rule out:
Thermal throttling, GPU can operate at 100% up to 84C and vrchat is not bringing it over 60C. CPU also doesn’t go over 50C.
VRAM bottleneck, the frame buffer rarely goes much over 6GB out the 12 available. RAM bottleneck, while admittedly slow, its 32GB at 2134MT/s.


image

Specs:
CPU: Ryzen Threadripper 1950X OC’d close to 4GHz
RAM: 32GB 2134MT/s
MB: MSI X399 AC Carbon Pro Gaming
GPU: 2 x Titan X (Pascal) - running in single GPU mode.
Windows 10 and game is running on a competent SSD.

I would love some suggestions please, VR and desktop mode get similar FPS, even though my desktop monitor is 1080p and the Headset is 1832×1920 per eye.