Developer Update - 29 September 2022

Welcome to the Developer Update for September 29th, 2022!

If you’d like to catch up, you can read our previous Developer Update from Sept 22nd.

Recent Downtime

We had some downtime recently with very poor timing! :cold_sweat:

What Happened?

On the Friday the 23rd, at ~6PM PT, one of the companies that provides a service to us (an “upstream provider”) performed some momentary maintenance at an inopportune time. During this maintenance, the provider also inexplicably down-scaled the servers they use for us. The new machines had less than half the capacity of the original machines. This down-scaling was unexpected and had not been requested.

Due to this down-scaling and the poor timing of the maintenance, several VRChat services that rely on this upstream provider began failing. The upstream provider’s downgraded servers buckled and broke under the load of a typical Fall VRChat weekend during NA peak hours. A short set of dominoes falling later, and VRChat login was down.

We immediately noticed the down-scaled servers and requested a move back to our original infrastructure. Due to various factors, this would take some time, and in fact the service being overloaded made it even harder to migrate back to the original machines.

In the meantime, the VRChat DevOps team moved quickly to mitigate issues and reduce load until the problem passed. Several endpoints had to be turned off to reduce load. Due to a series of bugs, this also resulted in the aforementioned logins failing. (These bugs are being addressed in upcoming patches!)

Eventually, services came back online as we were able to get our service migrated back to the original machines.

However, we weren’t through everything! The next day, complications resulting from the previous downtime “boiled up” and caused even more downtime – again, during peak hours.

These eventually got resolved too, but the downtime did “chop off” our usual weekly peak. Not fun for anyone involved!

What are we doing about it?

A few things!

  • We’re changing infrastructure so it is less likely that we’ll be affected by this particular upstream provider’s issues
  • We’re shifting internal infrastructure so that it is more durable against “domino effects” from the death of other services
  • We are making some changes to the VRChat client to reduce the load that we endure when we are trying to come back up from downtime

We hope that these changes and improvements prevent this particular type of downtime from occuring again. We’ll see! woo live services wooooo our devops team rules and i love them nobody tell them tho im shy :flushed::point_right::point_left:

Spookality 2022

Spookality is starting up soon! We recently made a tweet announcing that the Spookality Jam will begin on October 1st, and close on the 21st. We’ll have a blog post out at launch with more details.

Keep an eye on our Twitter account for more details as they release.

However, let’s talk about something new! We’re going to use a new Jam Submission System we’ve been working on to make it easier to submit your projects to Spookality and other future jams.

Here’s how it looks right now!

tl;dr: You’ll create content for VRChat Jams and submit them using a new section on the VRChat website, submitting the IDs directly. No more Google Forms! yayyyyyyyyyyy

More info on this system later!

Live Beta Updates

Last Friday, we released an update to the Live Beta! 1243 went live with a bunch of changes.

Live Beta Build 1243

Client

Since build 1241

Improvements

  • The Color Filter rainbow is more consistently visible when you want it and not visible when you don’t want it, in various different situations
  • Mirror Resolution no longer changes unlimited back to full between sessions

Changes

  • Open Logs Folder button was removed from all settings pages, and is now a different button in Main Menu/Settings/Debug
  • VRChat’s Color Filter is not supported on Quest, users should utilize the built-in Meta Quest visual adjustment settings instead
    • Access the Meta Quest’s color correction by going to Settings in the Meta Quest system menu, selecting Accessibility, and turning on Color Correction. You can set filters and adjust filter intensity to customize how colors are shown

Fixes

  • Fixed logging spam
  • Fixed avatars showing up at world origin when previewing in menu
  • Fixed an issue causing “Join Me” or “blue” status to not auto-accept invites
  • Sort order and category choices in wings is now saved properly
  • Fixed issues causing random slider changes, let us know if you encounter this again!

New in build 1242

Improvements

  • Fixed many issues with avatar haptics, improved stability, and improved avatar hand rig compatibility
    • Avatar Haptics is not supported on standalone Quest because it causes significant performance issues related to the render pipeline and how the Quest GPU operates, we are re-assessing how we might best enable this feature on Quest standalone
  • Added in many log entries for when settings change to help out with Support issues

Changes

  • A WHOLE BUNCH of options, buttons, settings, and toggles have been either moved or duplicated into the Main Menu from the Quick Menu!
  • MOVED:
    • Fallback Icon: removed from Quick Menu settings, now in Main Menu/Settings/User Interface/Nameplates
    • Tooltips: removed from Quick Menu settings, now in Main Menu/Settings/User Interface/General
    • UI Haptics: removed from Quick Menu settings, now in Main Menu/Settings/User Interface/General Settings and also Menu/Settings/Accessibility/Haptics
    • Avatar Touch Haptics: removed from Quick Menu settings, now in Main Menu/Settings/Accessibility/Haptics
    • Slider Snapping: removed from Quick Menu settings, now in Main Menu/Settings/User Interface/General
    • Gesture Icons: removed from Quick Menu settings, now in Main Menu/Settings/User Interface/HUD
    • Home Instance Type: removed from Quick Menu settings, now in Main Menu/Settings/Comfort & Safety/Safety
    • Auto Disable Cloning: removed from Quick Menu settings, now in Main Menu/Settings/Comfort & Safety/Safety
    • Portal Prompt: removed from Quick Menu settings, now in Main Menu/Settings/Comfort & Safety/Safety
    • Portal Mode: removed from Quick Menu settings, now in Main Menu/Settings/Comfort & Safety/Safety
    • Clipping plane distance: removed from Quick Menu settings, now in Main Menu/Settings/Performance/Advanced Graphics Settings
    • Particle Limiter: removed from Quick Menu settings, now in Main Menu/Settings/Performance/Advanced Graphics Settings
    • Mirror Resolution: removed from Quick Menu settings, now in Main Menu/Settings/Performance/Advanced Graphics Settings
    • Clear Local Profile Data: removed from Main Menu/Settings/Performance, now in Main Menu/Settings/Debug
  • DUPLICATED:
    • Responsive Menu: Still exists in Quick menu, additionally added to Main Menu/Settings/User Interface/General and Menu/Settings/Accessibility/Comfort
    • Chatbox settings: Still exists in Quick menu, additionally added to Main Menu/Settings/Accessibility
    • Display & Visual Adjustments: Still exists in Quick menu, additionally added to Main Menu/Settings/Accessibility

We haven’t updated this week so far, but we might have something for tomorrow!!

Ongoing Development

Downtime Mitigation Client Changes

Last weekend we had a LOT of downtime. We talked about that above!

This week, we also spent a lot of time on changes that will make the VRChat client behave a bit better during downtime, and be nicer to our servers when they’re trying to come back after downtime. Right now, the VRChat application is a little impatient, and as a result we have to be careful in how we turn things “back on” after downtime. These changes make that easier!

Most of these fixes and changes are invisible to users, but there is one that’ll be noticable and we can talk about:

Show and Hide Avatar

“Show Avatar” and “Hide Avatar” are those buttons you hit to override the Safety settings for users. These buttons send what we call a “player moderation” – a relational setting one user has set on another. Things like blocks, mutes, avatar-showing. When you send those, we save that setting on the servers.

Here’s the (vastly oversimplified) problem: We have a lot of player moderations. Internally we’ve done and are continue to do work to make them more efficient to work with. However, due to the sheer number of player moderations, we’re have to make a change to help reduce the load.

In an upcoming update to both Live and the Beta, “Show Avatar” and “Hide Avatar” player moderations will be saved locally instead of saved on the server. This seemingly minor change will save us a BUNCH of headaches in the long run.

When the update rolls out, we’ll migrate your saved stuff on the server to local storage. After a short period of time (about a week), this migration system will be turned off, and the player moderations stored on the server will be deleted.

This has two consequences that players should be aware of:

  • Show and Hide Avatar player moderations will not transfer between devices, and can be cleared if you delete all VRChat local data
  • If you log on after the migration period has ended, your Show/Hide Avatar player moderations have already been deleted on the server. As such, there will be no Server → Local transfer, and you will not have any Show or Hide Avatar player moderations

As a final side-note, “Block Avatar by ID” / “Block Avatar Globally” is not the same thing as Show/Hide Avatar. Those are still being stored on the server.

We’ll keep you up to date and inform users on a wider scale once this update goes live! If Beta is still a thing when it goes out, we’ll update both Live and the Beta simultaneously.

Even More IK Improvements

We’ve got some more improvements on the way for IK – in particular, for those funny --launch-option-only features!

We’ve added a Tracking & IK section to the Main Menu in the Comfort and Safety section. Most of the IK-related launch options can now be toggled via the menu. So far we have:

  • Legacy Calibration
  • Disable Shoulder Tracking
  • Freeze Tracking on Disconnect
  • IK Debug Logging

That’s not all! There will be more options coming to this section in the future. Wrist and knee rotation offsets will probably be next up.

We’ve also added some more options and improved the User Real Height setting!

You can now swap to Metric for height adjustment! Using this option, the arrows will increment by centimeters instead of inches.

Additionally, User Real Height has a fancy new “Measure” button. Using this button will measure your height by using the distance between your HMD and the floor of your playspace. Make sure you’re standing up and looking straight ahead when you use it! The measure button is available in both the Quick Menu and the Main Menu.

Managing Favorites on the Website

If you’ve got a VRChat account, you can now manage your Favorite Avatars and Worlds on the VRChat Home website!

You can move worlds and avatars between groups, rename groups, remove favorites, or add worlds as favorites! Get started by clicking the “Edit” button in the top right.

You can also rename the favorite groups! This works for both Avatars and Worlds, although the name only reflects in client for the Worlds section right now.

When you look at one of your favorite lists, here’s what you’ll see:

Clicking Edit in the top right puts you into Edit mode!

You can also add worlds to your playlists from the world page!

Avatars work in a very similar way! You can move around avatars between groups, rename the groups, or remove avatars from favorite groups with ease.

Give it a shot over on the VRChat Home website!

New Main Menu UI Polish

A lot of what we worked on this week was a ton of miscellaneous Main Menu UI and UX polish. More than any of the other Live Beta updates so far!

We don’t have a lot of visuals for these changes yet, but they should be coming into the Live Beta soon.

We also fixed this super-important most-impactful-bug-of-all-time, thank goodness. Good catch, Table. We never would lived it down. :upside_down_face:

Avatar Menu Preview

While working on the Avatar Menu in the Live Beta, one of our engineers noticed that a particular frog in a particular car did some funny stuff when shown in the main menu “turntable” preview:

So, we fixed it up with the power of math. Oversized avatars no longer break the preview!

Well, I say that, but there’s almost certainly some weird and crazy avatar out there that’ll break it, probably… :sob:

Hide Avatar by Distance Performance Improvement

Hide Avatar by Distance is getting a performance improvement! We dug in and found a way to improve it even more without any impact to animator state.

These optimizations are best described with our (somewhat hamfisted) test method: in a capped (80/80) instance, the old version would only save you about 5-10 frames with most of the avatar in the instance hidden.

In the new version? We saw improvements of 20-30 frames, depending on your settings and your specs. Potentially a big deal if you use that system often!

This change will make it into an upcoming Live Beta.

Smooth First Person Camera Improvements

Did you know that, in the Camera tab, you can smooth the motion of your “point of view” as shown on your desktop? This is super useful for streaming, so you don’t make your viewers sick from the jitter when you’re streaming from your viewpoint.

Previously, this mode wasn’t too useful in stations – especially fast moving ones. It’d lag behind by a frame, resulting in rather violent and counterintuitively nauseating jitter.

That’s NO GOOD, so we fixed it.

This’ll be in an upcoming Live Beta update!

Toxicity and Toxic Behavior in VRChat

So! Hard topic switch.

We get a lot of feedback from many different people saying that parts of VRChat can be quite toxic – Public instances in particular. Toxic language and behavior is a big contributor to bad experiences in VRChat. It drives off new VRChat players trying things out, and pushes experienced users out of spaces they’d like to participate in, find new friends in, etc.

Even though we’ve observed a drop in reported and observed toxic behavior since the middle of the summer, the problem isn’t gone. We still want to do things to help make VRChat a safe and fun place for everyone, and do our best to curb toxicity overall.

Let’s talk about it.

Toxicity is a difficult problem to solve! It is a rampant issue in ANY online community. If you can name an online platform or game, there’s toxicity somewhere in varying amounts. However, despite the difficulty of the problem, we’ve got some ideas.

For example, we believe that Groups will help reduce the “surface area” of toxicity by a large amount. Groups allows users to self-organize and self-moderate in various ways. The Groups system, which has been in design and development for quite some time, has features that we believe will help community leaders and community members create spaces that are ideal for new and experienced users to go and find like-minded friends.

On top of that, improving VRChat onboarding (the “introduction” experience for new users) will also help, as we can help direct people to experiences and worlds more relevant to their expressed goals and interests in VRChat. We’re also looking into improving our self-moderation systems so that they’re taught to new users more quickly.

However, we don’t think those are the only solutions. Public instances are a discovery melting pot in many ways. We all have stories of going to public instances and finding a new friend or an amazing experience, and we want to preserve that.

With all that in mind, we’re looking into ways to reduce toxicity and harassment on our platform at scale. Some of these methods are pretty big things, so we want to talk to you about them first.

Over the next few months of Dev Updates, we’ll ocassionally bring this topic up and see what kind of discussion results. For now, we just wanted to get your thoughts going.

Conclusion

That’s all for this week! The next Dev Update is due out on October 6th. Don’t forget to keep an eye on our Twitter for Spookality updates!

byyyyeeeeeeeeeeeee

36 Likes

I do hope the safety systems get a new coat of paint. The current systems are bypassed often, despite what I have set. Audio from avatars owners that are not on my friends list, animations playing that shouldn’t, ect.

1 Like

I see people reporting this often, but no successful reproducible cases have been delivered! If you observe it happening, grab a video/screenshot, note the time, grab your logs, and if you can get a screenshot of the avatar’s detail page (so we can see the avatar’s name and author), send it to us. That’d help a TON.

4 Likes

I’ll try to catch it, but I rarely record anymore, and more times than not it’s just a glitch as it only randomly happens. it’s not always, as the same people that I chill with in our game community use the same avi and it’s only happened a few times. It is unfortunate that it’s been an odd thing to nail down, but glad it’s known at the very least.

2 Likes

“Known” is a strong word, haha. We know that people say it happens, certainly! We’ve just never observed it ourselves.

Tons of people have stated that something bypassed their Safety settings, but we have yet to receive any kind of avatar ID or etc that lets us reproduce the problem on command. As such, I don’t have anything to hand to the engineering team.

Video is… only kinda useful, in this case. Avatar name, author name is SUPER useful. The most useful!

2 Likes

Yeah, I gotcha. I’ll see if I can catch it again, as some of the times it happens, it’s not use maliciously. Just somehow slips the cracks that has been put in place. If I catch it, I’ll forward it to the team via a ticket.

While I don’t code and what not, my personal guess is that when an avatar initializes, it doesn’t always get caught by a user’s settings, specially if there was a hitch the game before it loaded (which I’ve seen happen with audio on avatar swaps despite my settings). This is my only theory and angle I would attempt to look at while just poking things.

1 Like

I’m going to guess most likely the reports are a majority about corrupted asset bundles. Which really need to be addressed as they’re the major flaw as the safety setting can’t help with that unless you’ve download size blocked the avatar or hidden all avatars globally to not download it.

1 Like

Very excited to see the options on the site for organizing worlds and avatars, will have to think about how best to use the folders available.

For player moderations of show/hide avatar, I know from personal experience that often it is used to reset someone else’s avatar if it gets into an unexpected state from some bad sync or an animator issue. Never knew that it was hitting an API each time, I feel like a dedicated “reset avatar” button could have also been helpful, but removing the storage entirely will no doubt help more long term even if dropping functionality. (Maybe a way to migrate this list easily manually, a single file documented to copy over?)

This might be an inside baseball question, but also given the recent downtimes during the end of week peak, why does it seem that so many updates are landing on Fridays? Does the team work a different week that has them on Saturday and Sunday to address issues if they arise, or is this more just a case of open beta getting updates when they pass QA?

Would also be interested to know the % of users in the beta, I know my perspective is different since I usually jump on them right away.

4 Likes

That’s good to show more report details like Crashers Avatar and other stuff is not good or something not supposedly be there. :thinking:

1 Like

I don’t know the exact implementation but if you’re talking about backing your local copy up, yeah, that’ll probably work fine. I just don’t know how it’ll be stored. Maybe just a text file! Not very useful for Quest users, sadly…

I’m not quite sure what you’re referring to here, because we’ve only have had two Live updates on a Friday in the last year, and both were minor single-item patches that were required to ensure ongoing service stability.

We try to avoid major Live updates on Thursdays, and never do them on Fridays.

Additionally, none of our downtime that I can recall has been caused by a client update. I think the last time that happened was… 2020, maybe? And it was minor. It’s been a bit, I’d have to go digging.

I know this number, but I don’t want to share it. I will state that Live Betas (like this one) tend to have MUCH higher participation numbers, usually 3-5x more.

1 Like

Will look forward to seeing the specifics shared when it’s ready, then!

Ah, it’s going to be a personal perspective thing in that case then, I’m recalling mostly updates in betas like IK2 before or main menu now, since the last beta update landed last Friday and the next one seems to be for tomorrow, so recency might be a big part of what’s brought it to mind. I know I’d run into issues myself in those updates, but not major outages across platform.

Well if it’s something that can be disclosed some time, would love to see it put out there in a future update post or made available somewhere for folks to see. Interesting to know this one isn’t getting the same rates, which tracks with my own friend groups not giving it a try in the same numbers as before.

2 Likes

You misunderstood my post. Live Betas get a ton more participation, including the current one.

3 Likes

A ha, thanks for the clarification. I just haven’t woo’d my friends to the new menu just yet, even with the camera drone…

1 Like

Toxicity is a hard one to solve, though recognition of those that are and continue to be good and solid pillars of the community that leads with a good example can surely be a good way to get started. The current trust rank system seems to mostly be “gamed” a lot to get the highest rank as soon as possible, to the point where it sadly has lost its value.

I suppose the hard part is figuring out a good way to recognize these great players without that recognition system being exploited in one way or another.

3 Likes

I don’t have much details I just will say try your best your teammates VR chat and take your time.

2 Likes

Is future support for Cardboard VR/Android phone devices supported? I asked tupper :slightly_smiling_face:

1 Like

Agreed. Personally speaking, I think the Trust system does well in some things, but not all. Establishing that someone is toxic or not is not something it does well.

Not at this time, sorry!

3 Likes

This might cause some headache considering how most people tend to utilize showAvatar but I definitely get the need to offload a lot of it.

Unrelated but I hope my suggestion for changes to ‘Block Avatar Globally’ is still being considered internally. :sweat_smile:

Nice that we’re finally starting to get rid of all those launch options also. ^^

All of this is very good, especially rotation offsets! :clap:t3:

This is absolutely massive if it works out especially since I have this on basically at all times. The performance loss from allowing animators to run regardless has been a noticeable hit but if this helps out a lot that’ll be really great!

1 Like

You could probably get away with having steam cloud save just save a file that has hidden avatars saved onto it, wouldn’t work between PC and Quest but it’s a tad better than just per each system

5 Likes

Tupper, The Avatar Rescaling

8 Likes