A greate idea to use AI translation to increase VRCPlus subscriptions

General Discussion
,
1
  1. Summary first:

VRCHAT Build their own AI server for translation. VRCHAT offical has separate voice streams for each person. Can accurately translate everyone’s voice. The current VRCHAT Plus really doesn’t have much needed functions. This function officially produced by vrchat can significantly increase the number of subscript for vrchatplus.

  1. Idea Backgrund:
    a. The current VRCHAT Plus really doesn’t have much needed functions.
    b. VRCHAT is a world-wide virtual social game. People from different places on the earth communicate and play together. But language is the biggest barrier, preventing people from different regions from communicating.
    c.There are also ready-made translation software such as VRCT. However, it uses speaker audio, the sound is mixed with the background music of the world, and it is very noisy when everyone is talking in a crowded place, resulting in a very poor speech-to-text effect, which in turn leads to a very poor translation effect.

  2. How to implement:
    In principle. Each person’s voice stream is actually transmitted individually during transmission. VRCHAT official can split each person’s audio stream, and convert each person’s audio stream from voice to text separately, and perform separate translations. This will not be affected by multiple people talking at the same time and ambient background music.

Officials can further package this capability into the VRCHAT PLUS subscript.
and provides the following functions:
a. VRCP Subscribers can translate their own voice to anyone in the room.
b. VRCP Subscribers can translate the voice of anyone in the room into their own native language.

Translation only needs to be done once, and the same translation data can be shared between different people

  1. Others:
    If the official is not interested in merging this function into vrcplus. You can also consider opening the split voice steam API to provide developers with clearer and more accurate local translations.

I am not currently a subscriber of vrchat plus. If the official makes this feature a good experience, I will definitely subscribe to vrchat plus.

2

If you’re in school, like highschool or a college I would suggest thinking if this is necessary, and even important enough to repeat.

If you’re employed in sales, good job? Keep at it.

3

I am a programmer.
The ideas I mentioned can definitely be realized.
If the vrchat official can provide a separate voice steam api for me to implement, that’s fine too.
I think our social circle should not be limited to native languages ​​only.

If this kind of translation of separate voice streams can be implement. Wouldn’t it be more convenient to make friends with people with different language and culture in vrchat than in reality?

4

The cost is too high, do you know the cost of API calls?

Also the accuracy of the translations is not very good at the moment, there are many problems.

LLM seems to be much more accurate, but sometimes it may deviate too much from the original meaning, but the cost is too high.

NMT(Neural machine translation) on the other hand are stable, not very accurate, and inexpensive.

And to recognize speech accurately do you want to upload or process it locally? Bandwidth issues?

The accuracy and performance of different programs vary greatly in cost and compatibility.

5

  1. translations is not very good at the moment
    a. Chatgpt 4o voices chat demo and already good enough.
    b. Even a real person would have a hard time distinguishing one person’s voice from the noise. But each individual audio stream can be distinguished in VRCHAT. Of curse there have a lots of tings to do. But this is the first step in getting a good translation by using split voice audio steam.

  2. Cost too much.
    a. As the technology evolves AI arithmetic will instead become cheaper.
    b. VRCHAT’s voice chat is based on server relayed audio, which VRCHAT itself can easily access in the cloud. It doesn’t take a lot of bandwidth.

If VRCHAT is willing to open up the api for separate voice stays. With more and more computer chips with built-in AI acceleration units. There’s even a chance that you can do real-time speech-to-text and translation locally

If VRCHAT officially provides this service. Or even for example provide this service for free, and for free users their audio can be used as training. With a huge amount of high-definition standalone audio, VRCHAT itself could even become the strongest voice translator in the world!