Developer Update - 26 August 2022

Hello, and welcome to the Developer Update for the 26th of August, 2022!

You might notice that the “feature updates” section is a little thin. p3 was a doozy to get bugfixed, and we’ve diverted a good bit of team resources over to getting the UI ready. As such, there haven’t been many things worth posting about just yet since the Tuesday Dev Update!

Schedule Update

We’re going to continue the twice-weekly updates for the next week, but starting on the week of 4 September, we will be reverting to one update a week, probably on Thursday.

We don’t want to repeat information in order to “buff up” posts, which we’ve done (accidentally) a few times. Also, “we fixed bugs” isn’t too interesting to post about…

This change in pace also allows us to cover things that are less “active development” related, such as proposals, ideas, long-range plans, or even interviews with team members. We’re also considering good ways to highlight community content that stands out.

Assuming nothing changes, we’ll have future Dev Updates on these days:

  • Tuesday, Aug 30
  • Friday, Sept 2
  • Thursday, Sept 8
  • All following Thursdays

Release Schedule

2022.2.2p3 Release

Yesterday, we got a Live Beta update for p3 out that puts us at build 1228. This build brought Quest up to date, as it had a few issues in previous builds that prevented it from getting updates.

There’s a few more outstanding bugs we need to handle before launching to Live. Once those are done, we’ll push the next build. That updated build will potentially go straight to Live early next week.

Main Menu Live Beta

Soon after p3 goes to Live, we’re going to try to push the Main Menu to Live Beta. We’ve been doing a good bit of internal testing and dogfooding. There’s a few rough spots to polish out, but we’re hopeful we can get the Main Menu into your hands soon.

We want to launch the Live Beta at the start of a week so its easy for us to manage the initial massive influx of feedback without missing anything. As such, its possible if the Live Beta gets pushed too far next week, we’ll push it to the following week.

Feature Updates

Camera Improvements

We’ve continued on iterating on the camera! There’s a TON of features and requests we’ve gotten from both the community and internally on the camera. It’s a big focus point! Here’s what we’ve been actively working on recently.

We don’t have video just yet, but you’ll probably see some video previews in an upcoming Dev Update.

Camera Flying / Camera Drone for VR

In a previous update, we talked about “Camera Flying” or “Camera Drone” (“Drone Mode” is what people seem to keep calling it), which is a feature that allows you to fly your camera lens around in Desktop mode. This is super useful for getting big group shots or setting up cinematic photos.

We’ve been working on adapting this mode for VR, too! In our prototype, you can now control “drone mode” with the joysticks on your VR controllers.

We’re not completely settled on the controls, but we’re iterating on this feature.

Visual Shutter Effect

We’ve added a tiny bit (more?) skeuomorphism to the camera! There is now a shutter effect that will occur when a photo is taken, so you have a visual cue in addition to the audio.

:information_source: This is not a camera flash! It is a visual shutter effect.

Camera Menu Changes and General Fixes

We’ve reorganized the camera’s menu a bit to make it a little more usable.

The “Take Photo” button has been moved to the beginning of the list, and the “Exit” button has been moved to the camera itself.

If you have your camera set to “World” mode and move to a different world or instance, the lens will now reattach itself to the camera.

AMD FSR and nVidia Variable Rate Shading

:warning: Warning: beyond here lie very nerdy dragons! Lots of technical talk. Skip to the “short version” prompts if you want to know the tl;dr version.

AMD FSR, DLSS, and other systems have been requested by our users many times to potentially increase the performance of VRChat while minimizing damage to visual fidelity. We’ve talked about this in previous updates, but we’re ready to give you another update.

Implementing AMD FSR Natively

After a lot of evaluation, we’ve determined that AMD’s FSR system likely cannot be cleanly implemented into VRChat. The proper place to implement this feature is during post-processing, after the tonemapper but before other effects like bloom. Applying FSR at this point also has the benefit of rendering the UI at full resolution, so that text remains clear.

However, there is not a clean way to implement it with Unity’s Built-In Render Pipeline (BIRP). We tried to integrate FSR into the Post Processing stack at the appropriate point, but the BIRP doesn’t support rendering at a lower resolution while still outputting to the screen directly.

Instead, we need to output to a different lower resolution render target, apply part of the Post Processing Stack, upscale with FSR, and then apply the rest of the Post Processing Stack at the full screen resolution. Unfortunately, Unity doesn’t properly handle the switching the render texture during the render process, so the Post Processing Stack ends up trying to operate directly on the screen-- which is blank when it runs. Changing the scene render target this way also breaks worlds that rely on Camera Stacking.

Short version: Implementing FSR as documented and recommended by AMD isn’t possible in VRChat right now.

What about the OpenVR FSR DLL?

There exist other methods like the OpenVR_FSR patch, but these methods aren’t ideal. In particular, the OpenVR DLL replacement employs a “Man-in-the-Middle” approach and modifies the communication between VRChat and SteamVR. This allows it to spoof the render resolution that SteamVR provides to get VRChat to render at a lower resolution, which it then up-scales with FSR and submits to SteamVR. This is not reliable or good practice.

Moreover, this approach also has issues because it runs too late in the pipeline. According to AMD’s FSR Integration Overview (PDF warning), FSR should run immediately following anti-aliasing and tonemapping, and before any effects that introduce noise like bloom, chromatic aberration, film grain, dithering (notably, Post Processing v2 always dithers), and before the HUD and other UI.

The method used by the OpenVR FSR DLL modification runs after post-processing has completed. Applying FSR after the noise-introducing effects in PPSv2 will cause FSR to accentuate that noise, resulting in worse image fidelity and artifacting.

Since many worlds in VRChat use Post Processing and we support it natively, implementing this method lowers the visual quality of VRChat and introduces undocumented visual behavior that we cannot account for.

On top of all of this, this implementation would not work on Rift PC or Oculus Link unless they went through SteamVR, and it doesn’t work at all in Desktop mode.

Short version: The methods employed by the OpenVR FSR DLL come with significant caveats that we are not comfortable with implementing into VRChat. They will cause issues with the rest of our rendering stack and the post processing stack, going against AMD’s recommended implementation methods. The implementation significantly lowers image quality and may cause significant visual issues, while offering only a minimal improvement to performance. Finally, it only works in certain configurations and on certain platforms.

nVidia Variable Rate Shading

nVidia’s Variable Rate Shading technology increases rendering performance and quality by varying the shading rate for different regions of the frame. Any nVidia RTX GPU model 20xx or later can use VRS.

We have a prototype implementation of VRS in VRChat, and it has a significant impact on GPU times.

Testing in an empty Club Orion (a very pretty but GPU-heavy world) with these settings:

Inner: [0.4, 0.33] X1_PER_PIXEL
Middle: [0.5, 0.5] X1_PER_2X1_PIXEL
Peripheral: [1.075, 1.2] X1_PER_2X2_PIXEL


View of the Orion stage with a lot of volumetric lights. Oof ouchie my overdraw.

Blue: X1_PER_PIXEL
Green: X1_PER_2X1_PIXELS
Yellow: X1_PER_2X2_PIXELS 

The normal desktop view is reasonably close to what you see while wearing the HMD but significantly understates the amount of the screen that is covered by the peripheral region with these test settings. Here’s the full HMD render view:

Due to the lens distortion correction, the peripheral region is significantly squished and the centre is stretched. Without Fixed Foveated Rendering, the periphery ends up over-sampled relative to the center of the view. While wearing the Valve Index headset normally with the eye relief dialed in all the way, the blue and green regions cover about 80-90% of your field of view resulting in something closer to the desktop mirror than the full screen view.

So, results. This shows the reduction in GPU frame times before and after enabling VRS FFR:

  • Before FFR: ~10ms
  • After FFR: ~6.5ms

The region radii and resolutions can easily be adjusted to handle different HMD resolutions, screen aspect ratios, etc.

Currently these settings are adjusted via the config.json file, but we want to add the settings for this into our new UI instead if we implement this as a feature.

Short version: Using nVidia’s Variable Rate Shading technology to implement fixed foveated rendering has a significant benefit to lowering GPU load in VRChat with an acceptable impact on image quality. We are prototyping and evaluating this feature.

Community Highlights

A new section to show off some cool stuff we’ve seen the VRChat Community up to lately! This time around, it’s focused on two events in VRChat, but we’re also keeping an eye on other things to feature.

Summer VKet 2022

Summer VKet 2022 is ending today! Congratulations to HIKKY for another successful Summer VKet.

If you’re not familiar with VKet, it’s a series of virtual conventions run by HIKKY. You can browse through booths of hundreds of creators making avatars, worlds, shaders, music, video, art, anything! Both independent artists and larger organizations showcase their work at VKet. Virtual Market events are a mainstay of VRChat events, and we’ve been hosting them for years.

If you didn’t catch Summer VKet 2022, keep an eye out for their future events.

Slyfest: Azuria

Slyfest: Azuria – a virtual music festival held entirely in VRChat – started today!

Slyfest features over 50 DJs and producers who are active in the VRChat music community.

Slyfest isn’t just about them music, though. The folks behind Slyfest always create a series of beautiful stages that really highlight how awesome (and immersive!) musical performances can be in virtual reality. Even if you only hop in for a bit, we strongly suggest checking out what they’ve built!

Slyfest 2022 will be going on from the 26th to the 28th. You can check out their full schedule here!

You can find the Slyfest: Azuria world in its own world row this weekend.

Conclusion

That’s it for this week! Have a good weekend, and we’ll see you on Tuesday for the next Developer Update.

29 Likes

What about its impact on VRam use? Even if its not correct the DLL implementation is what people voted for on canny.

10 Likes

Ah so if things go well with p3, the plan is to get UI 2.0 to Live Beta early next week, otherwise it’ll get pushed to the following week? Just making sure I understand this correctly ^ ^;

3 Likes

There was an issue where i set the camera to auto level, so it went into landscape mode, however, the photos i took were portrait mode, despite the camera being horizontal, thus my images were sideways. Is a bug?

on fsr, sounds like you have a strange need to stick to documentation, who cares if it impacts visuals slightly more than on most games if it makes it playable for more people. let the people choose to accept the lower quality for better perf.

6 Likes

Is there any plans to add a toggle to seeing the camera shutter flash or audio? During some community events these can be distracting and ive seen people wondering if they can disable them in the menu. This would also be a good idea for those with epilepsy if the flash of the camera bothers them.

6 Likes

I agree there should be a toggle. Flashing lights is bad for me

From what I’m being told, the VRAM reduction isn’t nothing, but it also isn’t anything beyond an incidental benefit. It does make the MSAA buffer smaller (which gets resolved before PPSv2), but…

For example, the Index at 100% is roughly 2016x2240 per eye. That makes the stereo eye buffer 4032x2240 (or ~9Mpx). At 64bit/px (or 8B/px), this is ~103MB/MSAA sample, which makes MSAA 4x ~482MB.

FSR’s most aggressive performance setting is 1/4 the size, so that would make the MSAA buffer only ~121MB.

In short, you’re only saving a few hundred megabytes of VRAM at its most aggressive setting. You’d probably also need another non-MSAA buffer for the post-FSR data, so probably another 50-100MB for that, and maybe more for the FSR intermediate buffers.

Overall, VRAM-wise, you save something in the range of ~200MB of VRAM on an Index (or Vive Pro) at 100%, scaling linearly with SteamVR supersampling. For context, 200MB is less VRAM than some popular avatars, if we’re cherrypicking the good ones… :sweat_smile: (shakes in horror recalling some of the 1+GB VRAM avatars they’ve seen recently)

In short, if you’re having VRAM issues, 200MB isn’t really going to do much for you. You’d get better results from simply downscaling your render resolution.

We understand what people voted for, but we also want to make sure that we actually solve the problem people are asking about instead of blindly implementing things in ways that might not be that effective.

Don’t get me wrong: we like feedback and want people to tell us what they want! We’ve got a lot of smart people that play VRChat and give us good feedback! I just want to make sure you know that we also have a ton of smart people that work at VRChat whose job it is to ensure we implement your feedback in a way that solves the problem people are having.

9 Likes

I’ll take a look into this-- as I understood it, it might be a local-only thing, and isn’t a flash. It’s more of a “fake shutter close” visual effect like taking a picture on an iPhone used to work. I don’t know if they still do that… :thinking:

I might be wrong though, I haven’t tried it yet.

3 Likes

finally, some news about FSR!
glad to get an update on that, and i can’t wait to try it out.

Can’t wait to try out the Nvidia VRS FFR and get better performance!

5 Likes

The FSR injection also creates a few intermediate textures at the source and target resolutions. I wouldn’t be surprised if you’re worse off for vram while using it even at very aggressive settings.

2 Likes

Also unrelated but it would be nice to be able to use the Nvidia geforce suite of recording/replay stuff again, its still possible to do the same type of things but you have to use external programs with from what I have seen worse perf.

image

Maybe! Try to make it happen again. If you can make it happen consistently, post about it on our bug report board.

1 Like

For vram usage is there an option to limit amount used by avatars? Personally I don’t mind a large avatar if they’re blowing bandwidth on audio, as I’ve turned avatar audio to zero ages ago. But download size might not be good enough if people are purposefully sneaky with compression…

1 Like

Can you add a feature to allow for better examination of an avatar and it’s expressions? The camera is a bit small to get a good look at how others see my avatar.

why am i losing 100 frames now before the eac update i was hitting about 150s fps what going on i know it anit my pc because other games i hit about 150s to 200???

1 Like

I mean you can just use the Personal Mirror or the new Face Mirror for that. Also if you just wanna see what Gestures you are doing with an Index Controllern for example, you can just use the Gesture Icons overlay. ^^

1 Like

It’s a shame that you can’t work out FSR correctly but I have a proposal. Maybe implement it anyways for people on a beta or an option but put something that says to the user that turning it on may introduce artifacts or other things. As someone who used it for over a year I personally didn’t notice artifacts on my end but I for sure noticed the drop in frame times. For some people a smoother experience is well worth the potential issues it might bring in. This also brings me to the point of while VRSS for sure should be implemented having some kind of option for AMD GPU users would be great instead of nothing. If it’s that bad to implement I will understand But I think maybe you should just implement it and have it be an option through a toggle/separate beta for the people who do really want it. I don’t see the harm in at least giving people an option.

Apologies for the large wall of text.

2 Likes

Well, VRAM usage is exposed in the avatar perf stats in the new UI coming soon, so not sure about that yet but gonna keep an eye out for it!

3 Likes