State of RTMP / RTSP / RTSPT support

Ostensibly, VRChat supports RTSP / RTMP protocol streams through AVPro support, introduced in VRChat 2020.3.3p1 . However, I suspect from both my own testing and comments in the VRChat Discord that such streams don’t actually work. This thread attempts to summarize the current state of things.

Using GitHub - aler9/rtsp-simple-server: ready-to-use RTSP / RTMP server and proxy that allows to read, publish and proxy video and audio streams and an RTMP-muxed H264/AAC input (from OBS), I’m able to open an RTSP and RTMP stream with VLC, at rtsp://example.com/stream.

In vrchat, if I use rtsp://example.com/stream with the VRCAvProVideoPlayer component’s Video URL field, the video fails to load; I think this is because AVPro assumed rtsp means “RTSP over UDP”.

If I use rtspt://example.com/stream, the video loads fine. However, the AAC audio doesn’t play, and I instead hear weird pops and clicks every so often. Searching the VRChat discord for “AAC” shows a few other people mentioning the same problem, with no known remedies. There is a mention of changing the AAC sample rate from 44.1k to 48k might work, but 48k resulted in the same breakage for me.

Toggling the ‘Use Low Latency’ property in the VRCAvProVideoPlayer component doesn’t seem to change anything. Other Discord comments mention that the latency seems to drift to 5-10 seconds anyway, which makes RTSP no better than HLS streams (which work fine with AVPro).

If I use rtmp://example.com/stream, the video doesn’t load at all.

If anybody does have a working RTSP/RTMP stream setup with working audio, please reply to this thread with details of your setup.

tried without defining the protocol?

Hiina -
I am seeing the same issues, using a variety of RTSP servers:

I took a look at open issues in the AVPro Issue tracker, and immediately found a pinned issue highlighting the excessive delay problem seen in the player with rtsp streams. You can take a look at it here:

That being said, I haven’t found any issues raised with these audio problems we’re both having.

I recently came across this project, which claims to stream music at low latency and uses RTSP over TCP to do it, apparently using the gstreamer library:

Japanese documentation only unfortunately. I still haven’t tried it myself, but I’m curious what differs about their encoding parameters, or if gstreamer is somehow more compatible.

I threw together a test world with the topaz player and had similar issues as before - the magic behind the scenes is still AVPRO for this prefab, set to run in “Low Latency Mode”.

That being said - I’ll take a look at their gstreamer profile and report back if I’m able to get stable audio. It’s entirely possible this is sensitive to tuning.

Update: Working Setup

I have spent the last few days on-and-off digging deep into this, and not only found the source of the problem but fixed it for the aler9/rtsp-simple-server implementation.

What Happened

The specification for creating an SDP seems to indicate that the MIME types for data transport are case sensitive. This is contrary to the spec of what a MIME type is, leading to confusion. As a result the Windows Media Foundation API, the backbone of AVPRO’s rendering, bugs out when it hits the all-caps “mpeg4-generic” MIME for AAC audio transport, produced by FFMPEG and most other streaming software.
All it takes is changing it to lowercase, or detecting if it’s uppercase in an incoming relay stream and then changing it to lowercase.

Why use this aler9 server?

This server can take an incoming RTMP stream, such as made by OBS, or can take and relay an RTSP stream (such as generated by FFMPEG)

These changes have yet to be merged into the master branch and have a full release, but if you’re desperate you can use this here: Note this build is for Linux 64-bit.
Google Drive Link - rtsp-simple-server.tar.xz (Linux 64 bit)

For the Curious

Here’s the original issue brought to light by the wonderful user “jazzy348”
RenderHeads/UnityPlugin-AVProVideo/issues/653

Here are my findings as put into the library behind the scenes, written by the same developer
aler9/gortsplib/issues/35
Should my changes be applied, you’ll see this issue closed.

Latency

Testing with my server in the US and a friend in Finland, we were both able to experience latency of 1 second or less while using the “Dynamic Theater (UDON)” world.
This makes it ideal to stream time-sensitive games, like Jackbox, into VR in the future!

1 Like

After trying a few different configurations and servers, I think RTSP relay in combination with rtsp-simple-server and AVPro/vrchat is still broken; the audio either doesn’t play or some sort of buffer overrun occurs and your speakers will emit pops and screeches. However, rtsp-simple-server’s RTMP input does work fine including audio in combination with RTSP output (to vrchat). There must be something subtle about how rtsp-simple-server remuxes rtmp → rtsp vs the straight rtsp → rtsp relay. The rtsp → rtsp relay does still work with VLC, audio and all.

RTMP → RTSP seems to have about the same latency as RTSP → RTSP anyway and ffmpeg can speak both, so this isn’t really a practical concern.