There’s a lot to unpack here.
First of all, Resonite uses libVLC, which is going to be significantly more compatible with a lot of videos, at the cost of having a legally uncertain standing, something which a lot of companies, especially those based in the US, would be keen to avoid. At a technical level, it’ll be hard to beat, but it does carry risk that reasonable companies might seek to avoid.
Second of all, VRChat isn’t serving videos, it’s playing them, so saying “VRChat uses h.265” doesn’t make sense because it’s up to YouTube or whatever other video site to use it. However, h.265 itself is a very tricky codec to adopt, too, because adopting it can mean VRChat, AVPro, Unity, and Microsoft may have to pay fees depending on whether their lawyers feel they are beholden to the up to three patent licensing pools that gatekeep access. Independent projects fly under the radar all the time, but US-based companies can’t afford to do this. The usual workaround is to lean on the system or some framework to do the decoding and never explicitly touch the codec, and this is certainly the case for VRChat. Resonite, with libVLC, is not cautious about this for their own reasons.
VRChat’s usage of yt-dlp runs it as close to the latest version as possible, so to the maximum extent they can, they’re pulling the video links for YouTube URLs in the best way available, as consistently as they can across devices.
The main differentiating factor is which video quality is selected, and whether the GPU is able to support it. Most likely, your GPU and your computer is old enough that it is selecting a video whose codec is not supported, or whose resolution is too high and causes your GPU to run out of VRAM.
I once owned a GeForce GTX 960 as well, and it is significantly below min-spec for VR; together with system graphics buffers as well as VRChat’s it is extremely realistic for it to run out of VRAM. Technically, in some scenarios such as this one, your Quest 2 GPU may literally be superior to your GPU, even though it has more raw compute, because it has an ancient architecture and access to less memory.
tl;dr: your issues might be one or a couple of:
- your GPU is running out of VRAM
- the specific videos you are testing with are too big or maybe even too old or too new - and missing older codecs that work better with AVPro
- Resonite may be able to compensate for the incompatibility in a way VRChat may not be allowed to (i.e. falling back to your CPU instead of GPU, or compensating for system codec issues), but you don’t notice because the point of libVLC is to handle a lot of those issues for you
- yt-dlp may choose a quality of video that your GPU can’t handle, even when one that it could handle might also exist.
This is all super complicated to figure out with high certainty, and some of the issues are legal and out of everyone’s hands to change. By far the most likely issue is low VRAM and old video card causing the video to not play properly, and the most likely fix is to upgrade your GPU.