I talked to jynbug directly, but I guess I should probably post some information here as well.
Capturing a RenderTexture in Udon isn’t too hard - you need to attach an UdonBehaviour to the same GameObject as a Camera; in the OnPostRender event the render texture attached to the camera will be active, so you can do ReadPixels there. To have control of when you read the texture, set the camera component inactive, and call Render explicitly. But… that’s the easy part.
The difficult parts about syncing textures are 1) the limited bandwidth available, 2) the issue of desyncs when the texture changes during sync, and 3) the issue of multiple writers. Drawing Retreat has done a first stab at 1), but it’s far from solved (and these limitations are the primary reason I’ve not packaged up the texture sync system as it is today).
Bandwidth and Compression
First, the limited bandwidth issue. There’s currently about 11 kilobytes/s of bandwidth available per sender for Udon manually-synced objects. Without compression, a 512x512 rendertexture will take about 93 seconds to transmit, best case - but in practice, you’re competing for bandwidth with other objects, and it’s necessary to implement a backoff mechanism to avoid latency growing without bound. So it could take something closer to 2 minutes for a 512x512 image.
To improve on this for Drawing Retreat, I implemented a modified version of DXT5 compression, based on the technique described in the paper “Compressing Dynamically Generated Textures on the GPU” (Alexanderson 2006). On top of this I layered a run-length encoding. The DXT5 compression was modified by rearranging its fields to place all alpha data first, followed by base color data, followed by indices; this helps improve RLE encoding efficiency.
The RLE encoding was performed on pairs of pixels; on the encode side, a shader identifies, for each pixel, how many pixels following are either part of a run starting at that pixel, or not part of any run (longer than a threshold). An Udon script uses this information to skip over most pixels, reducing Udon overhead. Then, on the receive side, an Udon script generates a lookup table recording the input and output offsets of each repeating or non-repeating run, and a shader performs binary searches into this table to decode the RLE-encoded data.
In practice, the DXT5 encoding is reasonably effective (4:1 fixed compression ratio), but the RLE encoding isn’t very effective on anything other than fully-transparent areas. This is because the brushes in the world have an noise texture, which is beyond what RLE can efficiently compress. A Huffman or arithmetic code would probably work much better, but would be much more complex to implement in a fragment shader.
Having access to a built-in compression method that can quickly pack a Color32 to a byte would make most of this work unnecessary and would enable syncing much larger textures over Udon.
Transmitting image data takes quite some time, even with compression. If a player draws while the transmission is in progress, then a desync occurs due to the transmitter continuing to transmit an older image, while the local brush rendering is disabled on the receiver. There’s some logic to detect this (using a collider on the brush) and restart, but in practice either this doesn’t happen (because of a bug where the collider is only on the handle, not the brush), or it causes the sync to never complete.
Further, if you’ve been watching for a while, Object Sync has been replicating brush positions and they’ve been rendered locally; this works okay but in practice there’s a significant loss of fidelity; there’s a “Send” button to force a resync when you’re done, but people often don’t know to press it (and doing it while you’re drawing is a bad idea for the reasons above).
In the future, I would like to break the canvas up into smaller segments so that dirty segments can be continually retransmitted as they’re drawn to, but that requires a more-or-less complete rewrite of the texture sync system…
This isn’t so much an issue for Drawing Retreat, but for very large canvases (such as you might find on a graffiti world), there will be multiple players drawing on the same canvas at the same time. Keeping things in sync becomes very complex; whose view is authoritative for late joiners, for example? Can we do segmented retransmission between two players who are drawing at the same time, but on different areas of the texture? It’s not an easy problem.