I am not a VRChat employee but I am a reasonably experienced software developer. I have tried to explain it the best I can, but 3D rendering is a fundamentally complex topic.
tldr: 3D engines are extremely complex and upgrading is not trivial, and upgrading or even changing engines will not make 500k tri 50 material avatars magically faster. The only way to do that is to not make your avatars 500k tri 50 material monstrosities.
VRChat uses Unity’s Builtin Renderer Pipeline (sometimes called the legacy renderer). There’s plenty of in-depth technical documentation out there on how this all works, just google around, but be prepared: many of these expect some experience with graphics programming.
Versions after Unity 2019 focus on its Scriptable Render Pipeline technology. This allows developers to create their own rendering techniques with a great deal of flexibility. Unity maintains two of their own SRPs, High Defenition Rendering Pipeline (HDRP) and Universal Rendering Pipeline (URP, previously known as Lightweight Rendering Pipeline). You can also find plenty of information on these with google (and can even read the source code).
These newer rendering systems operate very differently to the legacy one. This has the possibility to introduce all sorts of compatibility problems; but let’s focus on the biggest: shaders.
Shaders are compiled programs that the GPU executes to determine what to do with all those vertices, bones, and images that make up avatars and worlds.
Shaders are heavily dependent on the rendering environment. They depend not just on the avatar/world mesh/textures, but also on things like the framebuffer, additional buffers (e.g. the G-Buffer and its particular layout), camera projection matrices etc. They also depend on the engine to execute them correctly e.g. the standard forward-lighting shader handles only one light, and additional lights are handled by having the engine execute that shader over and over and adding the result to the framebuffer (traditional forward rendering techniques).
With a radically different rendering environment, those shaders cannot meaningfully execute. For example: URP can handle multiple lights in one pass and doesn’t execute the additive-lighting pass over and over, and HDRP has a very complex framebuffer layout that cannot be simply rendered to.
The result for VRChat is that upgrading to a new rendering technique would break virtually all uploaded content with no automatic way to fix it. If your lucky the shaders might have their fallbacks set up properly and still render something, but many probably don’t and will just turn pink.
Running native 4k , 5k , 8k . is not possible without
Running at super high resolutions should be quite possible with Unity’s old forward renderer that VRChat uses if content creators know exactly how to optimise for it, but many don’t (and can’t reasonably be expected to). So this is more of a pragmatic issue then it is a technological one.
Most of the performance issues related to VRChat are to do with asset optimisation and not the engine. VRChat cannot be expected to magically run dozens of 500k skinned meshes with fifty materials and 400 physics-enabled bones.