Welcome to the Developer Update for 26 October 2023!
Today’s Dev Update thumbnail contains even more incredible avatars and a world that were submitted to Spookality 2023! Hop on VRChat now to check them out!
If you’d like to catch up, you can read our previous Developer Update from 12 October.
Important Info / Announcements
VRChat 2023.4.1 and Unity 2022 is OUT!
We’ve just released Unity VRChat 2023.4.1, the first version of VRChat running on Unity 2022!
If you want to learn more, we’ve got a patchnote video for ya.
We’ve said it a bunch of times, and we’ll keep saying it – don’t upgrade your projects yet.
Preparing for this release kinda dominated our last two weeks, hence the short Dev Update.
We’ll have a SDK update in the coming weeks. Thanks to the VRChat Creator Companion, the upgrade process should be a lot smoother. You’re using VCC, right???
Spookality 2023!
Hop on VRChat and check out the Spookality world categories. We’ve got two: one category for SCARY stuff, and one more for themed spots for hanging out, chilling out, that kinda deal.
For the avatars, head on over to the VRChat Home to find a whole new section of the world dedicated to the event!
A Note from Release Team: What happened with 2023.3.3?
As you might have noticed, the recent 2023.3.3 release of the VRChat client (the “world preloading, content gating and performance” release) was not the smoothest we’ve ever had, even needing to be reverted shortly after its initial launch! 
We want to provide a bit of a post-mortem on what made it so tricky to ship, and what caused some people to experience the issues they did.
First of all, an apology for everyone who was unable to play or had major issues during the time some of these builds were out in public! It’s important to note that we will never be able to ship a “bug-free” version of VRChat – that concept simply does not exist in the land of software development. Our goal is to minimize the amount of bugs and their impact, and to make sure the client will always be in a state where it can be played and enjoyed by everyone in our audience.
Details of what happened
The general timeline goes as follows:
- 2023-10-04:
2023.3.3ships to live on all platforms, having been on open-beta for a while prior - 2023-10-06: Live channel is reverted back to
2023.3.2p2, due to several major issues being discovered on2023.3.3 - 2023-10-16:
2023.3.3p1ships to live, as the second attempt at bringing2023.3.3to live - 2023-10-20:
2023.3.3p2ships to live, as followup patch addressing some issues on p1 - 2023-10-24:
2023.3.3p3ships to live finalizing our current plans for the2023.3.3cycle
There were 4 bugs that we classified as major enough to warrant a revert:
- Sometimes seeing other players frozen in place, while voice still worked (this was known as the “Frozen Avatar Issue” once under investigation)
- Being unable to log in under certain network conditions
- Crashing upon leaving an instance
- Quest users experiencing out-of-memory issues way more frequently
What caused these, and why did the revert not fix all of them?
The first two issues were caused by an internal change to our network format. Since reverting this fully would have had other implications, we decided on making new builds based on the previous live version (2023.3.2p1) and named them 2023.3.2p2. This also allowed us to keep compatibility with the newly released SDK that changed the way Udon programs are encoded.
Unfortunately, since some of the issues were caused by that specific change to networking, they persisted through to the reverted p2 builds.
The memory-usage was caused by some changes to how we make builds, and not the actual code itself (I have to be a bit handwavy here, forgive me). This meant that since we needed to generate new builds for 2023.3.2p2 to include the necessary backports, the issue also made it into the revert.
As a side-note, all of this wasn’t helped by the fact we were basically maintaining 2 different client versions at this point, with the beta being on Unity 2022 and live still on Unity 2019.
How did these bugs slip through, and how will we prevent this in the future?
There are a few reasons, and they all combined to create the perfect storm in this case. They are listed below, together with steps we will be (or are already) taking to mitigate such issues in the future.
Statistics
The first 3 issues from the list above were very rare, or only happened in certain unlikely network conditions. This meant, strictly through statistical unlikeliness, none of our testing team was able to reproduce them initially. But given the scale of our playerbase, even unlikely issues can affect many players in absolute numbers.
=> Our team is constantly expanding testing capabilities, both manual and automated, reducing the chance of such issues slipping through the cracks.
Prioritization
The “Frozen Avatar Issue” and the crashing one were known ahead of time (via our feedback boards). Given we were unable to reproduce them ourselves, and the initial observation made them look like local issues (e.g. bad network setups), they were prioritized lower than they should have been.
=> Going forward, even unlikely issues reported on a beta will be given a more thorough examination, especially if their impact could be as high as in these cases.
Unclear Data
The memory usage issue only occured on Meta Quest devices, which see very little adoption in our open-beta programs (thank you to everyone who does test betas on Quest!). Memory issues had been noted before internally, but the amount and quality of data didn’t lead us to conclusive evidence that there was a tangible problem.
=> Over the last couple months we’ve already been working on greatly expanding our automated testing and data collection systems. Starting from 2023.3.3p1 we have made the call to ensure anything reaching Live will go through a rigorous memory and performance test first.
Timelines and a note on the difficulty of fixing bugs
Some of these bugs truly played buzzword-bingo with our engineering team – from race conditions to use-after-free, requiring native debugging and memory minidumps. It was quite the ordeal! Specific test plans also needed to be constructed by our QA team to verify that a fix worked (or didn’t).
The crashing issue for example turned out to be a race condition triggering a bug deep inside the Unity engine. This bug actually existed the whole time, but was incredibly unlikely to happen. However, we optimized world loading speed so much in 2023.3.3 that it suddenly started happening enough to be a problem.
Having bugs that require this much time and effort to fix is brutal for managing timelines. Other teams have certain expectations on when their cool new feature or change will make it out to users, and sometimes shipping something a tad broken and fixing it up later is the most efficient way forward. If a bug turns out to be of this caliber however, that plan falls apart.
=> We will try to be stricter in enforcing stability over delaying timelines. This is not an easy task, even if it may sound like it on first glance. Delaying releases can have a cascading effect on planning future releases, and thus new features and changes that you have been waiting for.
Some final words
We are still in the process of fixing up some smaller issues with 2023.3.3p3, but it is stable enough that we can now defer such changes to upcoming releases (like 2023.4.1, which just shipped!).
All of this isn’t a guarantee that we will never ship something broken to live again. As nice as that would be, making such a promise would be lying no matter how you spin it. Rather, it is to ensure you that we are aware of such issues, and constantly working on improving our strategy. And maybe serve as a soft reminder that we are all just human. Okay the furries might disagree here, and the robot people, and… right, everyone can be what they want, but you get the point.
Ongoing Development
As noted above, most of our efforts over the past two weeks were spent on the Unity 2022 upgrade!
Other projects, including (but not limited to) Creator Economy, VRChat on Android Mobile, Imposters, and many others are continuing onwards. Expect more news on them soon!
Conclusion
That’s it for this Dev Update!
Our next update is scheduled for November 9th, 2023. See you then!
+Indeed you trying to be promised you doing the hard work to do with your VRChat Teams to fix really slow things around to be focus + you’re right everyone can be what they want I know people are special really good artists or more special they are + okie dokie I will see you next time next week VRChat Teams be healthy try your best your work next week 
^-^)
))
