Parsing large JSON structures

Help!
, , , ,
1

What’s the most efficient way to parse large data structures that also contain sub-objects. Here’s a shortened example:

{
  "architectEnabled": false,
  "architectStructures": [
    {
      "id": "arch_184rkma139o0kaq1",
      "name": "Cube",
      "desc": "Just a typical Unity Cube.",
      "pos": {
        "x": 0,
        "y": 0,
        "z": 0
      },
      "rot": {
        "x": 0,
        "y": 0,
        "z": 0
      },
      "sca": {
        "x": 1,
        "y": 1,
        "z": 1
      },
      "uniqueProperties": {

      }
    }
  ],
  "collectionId": "pbc_3105992525",
  "collectionName": "worlds",
  "config": "prod",
  "created": "2025-10-24 01:39:08.808Z"
}

Parsing and converting every single field with DataTokens is very time consuming, hard to debug, and inefficient, as it would require hundreds upon hundreds of lines of code. Is there a way to just do something similar like Newtonsoft.Json does?

2

Hey, how many elements do you think you will go through?

If it’s less than 1000, you can do it in a loading frame, but I recommend splitting loading meshes and objects like that into chunks based on framerate. There is a lot of data you can already load into Udon with the current system, and if it doesn’t fit in 16 ms, continue it in the next frame using SendCustomEventDelayedFrame.

Also, for loading metadata, I recommend leaving it in its original format until you actually need it.

If your testing and it’s still too slow in the current udon, I recommend showing a loading bar or using a binary format to speed things up a bit.

1 Like
3

Hey, thanks for the response.

These are the full data structures I am dealing with (from my own server and DB):

https://vrc-net.srckat.me/api/collections/config/records/prod

https://vrc-net.srckat.me/api/collections/worlds/records/wrld_9s857lg2lb3y2u60?expand=config

Also, what would be the best way to show a loading bar that represents the actual progress?

From what I have heard, there is a 5 second cooldown before I can send another string downloader request, so I thought it’d be more efficient just to grab it all at once (hence the expand in the second link that includes the global config).

I’m unsure how many total architectStrutures there will be, but it’ll definitely be less than 1000.

4

Do you have hiccups when loading, or do you just find it tedious to load all the data into your objects? Because if I look, this should not be a problem at all to load this all in.
I believe that if you are doing this as your first string load, it will be loaded before the fade even ends.

5

Loading it has no issue, it’s just tedious to load it all where it needs to go. I have to parse basically everything by hand into DataTokens and such, which can get very annoying to do with large amounts of data.

Also, I would like to add here that what I am trying to do is only possible with custom server code, which requires untrusted URLs. I’m assuming the main reason is to prevent IP grabbing. If this is the case, I’m not sure if you know about this, but Discord has a “proxy” system that is extremely easy to use, which is used for activities. Basically, you create an application in the Discord Developer Portal, then go to URL mappings, and add your server URL there. Then, you can access it through a proxy at the URL “[APPLICATION_ID].discordsays.com”.

Adding “*.discordsays.com” to the URL allow list would allow us to run custom server code for worlds, while also not being able to grab IPs (as it’s a proxy and it would just return the IP of Discord’s server).

I’m not sure if that is possible, but just a little suggestion relating to this conversation, as I’ll eventually need a way to have players load it without issue.

6

One thing I do to help with the need to bring all the data into the world is section it by what it needs to do. Then, you can send each specific dictionary to each component that needs it. Each component then only needs to read its own data dictionary object.

So I would split up your file into sections like:

{

“architecture“:{}
”generalData”:{}
”MusicDisks”:{}
…

}

Then send these dictionaries to their respective classes. This helps with organization at least for these things, and yes, we currently don’t have a way to load data into a custom object a la Newtonsoft.Json.

If you want a way to load these files if they’re not changing that often, you can use GitHub.

7

The files do change a lot. The whole idea is that they can be updated with a few clicks of a button.

image
image1900×901 72.1 KB

Also, would it be allowed if I try to make a request to my server, track if it fails, and then put users in a “box” that prompts them to allow untrusted URLs?

8

You are allowed to do that yes.

9

Alright, thanks!

I did end up coming up with a system where I can do something like DataParser.GetString(“test”);, and I have methods for all of the other data types.

Also, to prevent having to redo the request on different objects, I just have one “cache” behavior, which downloads the data, then sets a bool on itself to show that the data can be grabbed by other behaviors.

If I want to grab updated data, would making a new request to get a (max 10kb) blob of JSON data every 5 seconds or so be allowed? I manage the server myself, so I can handle proper rate limiting on my end (it’s a dedicated machine).

10

You can do up to 100mb of stringloading every 5s, how enoyable that is for users is for you to decide. String Loading | VRChat Creation

11

Alright, thanks for the help!

12

To answer your question about the amount of work required to navigate through datatokens: the reason there are so many steps is because datatokebs don’t have strict types, but they exist inside of an environment which does have strict types.

Datatokens are a concept that straddle across multiple different places - udon graph compiler, U# compiler, json input data, udon assembly, and C#. The only way to do that is to support the broadest possible interpretation of types. As not all environments have strict typing, it is impossible for datatokens themselves to have strict types.

However…. U# is powerful enough that you absolutely have all the tools you need to add strict types on top of datatokens, but it is something that must exist inside of U#, which is firmly outside the realm of what datatokens are capable of doing. Instead, it is up to you to create those definitions and mappings.

One way to do this is with a darkclass generation tool: Improved U# DarkClass (Generation tool available) - チカラの技術

This tool allows you to interact with objects that appear to be strictly typed classes, but under the hood it ends up creating a set of helper libraries for your context-specific set of classes that access the datatokens for you, and exposes convenient access points.

Alternatively, if you are interested in diving into things a bit more manually, you can also use the same underlying concept to create custom classes that inherit from datalist and static extension methods.

Regardless of the path you take, the end result is the same - create a helper library which exists to handle navigating datatokens and exposing operations to your strongly typed c# environment.

It’s worth mentioning that I would only recommend doing this if you feel strongly that you want to uphold a strongly typed c# environment. Some people who come from places like JavaScript or Python are perfectly happy operating with strings and loose types. There is no right answer, its just different paradigms. In this case, if you feel strongly that you want strong typing and clean, simple syntax, you do have to put the work in to establish that for yourself

1 Like