Doshik language custom compiler

Working on custom compiler for Udon VM.
GitHub project link: https://github.com/UltimaBeaR/DoshikLang

Status: almost usable, but you should build it from sources if you want test this thing out. I intend to do bare minimum as fast as i can to make first release of this.

Language is called Doshik (russian Дошик - slang/short name for instant noodles trademark), it’s very simple - no user-defined data types, generics, lambdas and other funky things. Just ability to code things in text format instead of visual code blocks.

Syntax is similar to C# but way simplier and less powerful. I intend to add functionality later and start with simple things first. I want it to look as close to C# as it can be, but getting rid of complex stuff and adding udon specific syntax sugar on top of it.
If you did some Unity3D C# code before, it would be very easy to learn it as it is very similar to writing unity’s component (MonoBehaviour) classes.

Demonstration on how it works:

Compiler is written on C# as .net standard 2.0 project, and will work inside of Unity3D editor, so there are no external dependencies on that. I am also making udon custom program source asset for the language, so the final package will likely to be just another unity’s “import custom package” thing to be up and running.

I’m also IDE guy, so i’m making VS Code extension for this language: syntax coloring, compile errors display and autocomplete stuff (for udon api available data types, methods etc)
compilation_errors

Language lexer and parser are made with use of Antlr4 tool (it generates c# lexer and parser code). I’m also using calls to udon/vrchat sdk’s dlls to get stuff like available types and methods to call (native udon graph editor script does the same).

List of currently implemented stuff:

  • Compilation errors output
  • Compile time type checking (strongly typed language). Method call overload choosing (but types must be exact, no implicit downcast for the moment)
  • Unity3D editor integration: very basic program asset and script importer for .doshik extension
  • public/private variable declaration for whole module(script). No initializers support yet (Could be initialized on Start() event)
  • Built-in (Start(), Update(), etc) and custom events declaration
  • Aliases for privitive types - string, int, etc (full C# list of primitive aliases is supported, but literals only work for most common ones for the moment)
  • Statements
    • Local variable declaration, with or without initializer: int a = 5;
    • Expression statement. For example assignment: a = 5;
    • Block statement, empty ; statement
    • If statement
    • While statement (also continue, break statements)
  • Expressions
    • Literals for primitive types (bool, int, string, etc)
    • Existing variable reference
    • Assignment operator = (no complex +=, *=, etc yet)
    • Explicit static method calls: System::Debug.Log(params), instance method calls: variableName.ToString(params)
    • Operator overloading for +, -, *, / and more
    • new T(params), default(T), typecast (T)expression, typeof(T)
    • GetThis<T>() built-in function for getting “this value” for certain types (GameObject, Transform, etc). Mayby i’ll change it to keyword (thisof(T) for example). I don’t want keyword this to be used for that as it is usually related to current class instance (no classes yet) and i want to reserve it for that purpose.

I will edit this post to sync it with available features for the moment, and i’ll be adding separate posts to comment current development progress

7 Likes

I finally came to first event body code generation…

It took me around 2 weeks to make simple variable assignment expression. But handling expressions are one of the most painful things in making compiler and now it’s seems that i know how to do it.

It’s literally the only thing that is working for now. local variable declaration statement and 3 expression types (literal, get variable, static method call). Now when i have grasp of how it all fits together it wouldn’t be that hard to implement everything else.

I also changed syntax a little. I planned to use C#-like typenames with dot delimited namespaces and subtypes, but these things are pretty hard to parse (it’s context sensitive), so i decided to use :: (C++ way) as namespace separator, so .net type System.Int32 becomes System::Int32 (udon’s SystemInt32).

So for given Doshik code

float a;

event void Start()
{
    int result = 5;
}

event void Update()
{
    float result = System::Single.op_Addition(System::Single.op_Multiplication(a, 5.0f), 1.0f);
}

i can now have output

generating external api...
compiling...
compiled.
code:

###############################################################
# compiled with UAssemblyBuilder

.data_start


    constant_0: %SystemInt32, null
    constant_1: %SystemSingle, null
    constant_2: %SystemSingle, null
    global_private_a_0: %SystemSingle, null
    local_event_Start_result_0: %SystemInt32, null
    local_event_Update_result_0: %SystemSingle, null
    temp_0: %SystemSingle, null
    temp_1: %SystemSingle, null

.data_end

.code_start

    .export _start
    .export _update

    _start:
        PUSH, constant_0  # 0x000000
        PUSH, local_event_Start_result_0  # 0x000005
        COPY  # 0x00000A
        JUMP, 0xFFFFFF  # 0x00000B

    _update:
        PUSH, global_private_a_0  # 0x000010
        PUSH, constant_1  # 0x000015
        PUSH, temp_0  # 0x00001A
        EXTERN, "SystemSingle.__op_Multiplication__SystemSingle_SystemSingle__SystemSingle"  # 0x00001F
        PUSH, temp_0  # 0x000024
        PUSH, constant_2  # 0x000029
        PUSH, temp_1  # 0x00002E
        EXTERN, "SystemSingle.__op_Addition__SystemSingle_SystemSingle__SystemSingle"  # 0x000033
        PUSH, temp_1  # 0x000038
        PUSH, local_event_Update_result_0  # 0x00003D
        COPY  # 0x000042
        JUMP, 0xFFFFFF  # 0x000043


.code_end

###############################################################

variable default values:

constant_0 = 5 (System.Int32)
constant_1 = 5 (System.Single)
constant_2 = 1 (System.Single)

I never tried to run anything yet though, but comparing it to what udon graph language generate, that should work

UPD: Today i added implementations for variable assignment expression, expression statement, typecast operator. Tried to run generated assembly in unity - works as expected (prints 47):

float a;

event void Start()
{
    a = 18.5f;
}

event void Update()
{
    float result = System::Single.op_Multiplication(System::Single.op_Addition(a, 5.0f), 2.0f);
    UnityEngine::Debug.Log((object)result);
}
1 Like

Comparison between doshik code and udon graph on simple example:

Doshik code (it’s compiling and working):
simple_code

Udon graph (i composed final image in image editor as it doesn’t fit into screen):

Both output “6 - six, 10 - ten”.
Algorythm in doshik and graph is exact same thing (it’s not 100% identical but i would say 95%)

2 Likes

It is now somewhat usable. Still no support for custom events, out params (edit: it’s working now), event params, and some other stuff. But it’s already possible to do alot using it.
Just random example of spinning cubes i tried to do:

public UnityEngine::Color colorA;
public UnityEngine::Color colorB;
public float speed;

private UnityEngine::Transform _transform;
private UnityEngine::MeshRenderer _renderer;

private UnityEngine::Vector3 _directionX;
private UnityEngine::Vector3 _directionY;
private UnityEngine::Vector3 _directionZ;

private float _rotationRateX;
private float _rotationRateY;
private float _rotationRateZ;

private float _lerpCoef;
private bool _lerpCoefIsGrowing;

event void Start()
{
    _transform = GetThis<UnityEngine::Transform>();
    _renderer = (UnityEngine::MeshRenderer)GetThis<UnityEngine::GameObject>().GetComponent("MeshRenderer");

    _directionX = new UnityEngine::Vector3(1.0f, 0.0f, 0.0f);
    _directionY = new UnityEngine::Vector3(0.0f, 1.0f, 0.0f);
    _directionZ = new UnityEngine::Vector3(0.0f, 0.0f, 1.0f);

    _lerpCoefIsGrowing = true;

    _rotationRateX = 50.0f * speed;
    _rotationRateX = 30.0f * speed;
    _rotationRateZ = 8.0f * speed;
}

event void Update()
{
    _renderer.get_material().set_color(
        UnityEngine::Color.Lerp(colorA, colorB, _lerpCoef)
    );

    float deltaTime = UnityEngine::Time.get_deltaTime();

    _transform.Rotate(_directionX, deltaTime * _rotationRateX);
    _transform.Rotate(_directionY, deltaTime * _rotationRateY);
    _transform.Rotate(_directionZ, deltaTime * _rotationRateZ);
}

event void FixedUpdate()
{
    if (_lerpCoefIsGrowing)
        _lerpCoef = _lerpCoef + 0.01f * speed;
    else
        _lerpCoef = _lerpCoef - 0.01f * speed;

    if ((_lerpCoefIsGrowing && _lerpCoef >= 1.0f) || (!_lerpCoefIsGrowing && _lerpCoef < 0.0f))
        _lerpCoefIsGrowing = !_lerpCoefIsGrowing;
}

Example scene has 3 cubes connected by gameobject hierarchy, each cube has instance of this script with different public variable values applied from udonbehaviour component (2 colors and speed).
Uploaded it on vrchat beta. Works as expected:

2 Likes

Last couple of days i was investigating VS Code’s language server stuff.
Language server is background process that VS Code (and couple of other code editors) is using for autocomplete, compilation errors, goto definition and other things for certain language. Doshik is not 100% based on C# syntactically, so i have to make my own implementation of this if i want some nice IDE experience.

I’ve done some restructurization of project libraries, so i can reuse compiler’s code analyzing part in language server.

I also made udon API json formatted cache (available events, data types and methods that can be used from code), so it doesn’t rebuild it every time on doshik file compilation (it takes couple of seconds).

And i made very basic language server implementation, but it’s far from perfect for now.
What i want from it as 1st priority is showing compile time errors and being able to see all available udon API. I think i could achieve that in couple of days, but full-feature language server is no easy task and it would require more complex structure of code analysis library. I will eventually do that (after compiler itself is stable) but now it’s not in priority.

For now language server is capable of showing compile-time errors, but it’s always highlights 1st symbol in file and messages itself are not very useful, but at least you could know if it compiles or not. Compilation errors update on editor’s open document changes.
compilation_error
(from screenshot - it has to be a = 5.0f, not 5)

It’s also shows all available types in autocomplete window (it’s not dependent on place where autocomplete is called), it’s what i’m going to improve in next couple of days
type_autocomplete

UPD: added gif on compilation errors output demonstration in main(first) post

1 Like

Loving this, too; you folks are creating awesome things.

1 Like

Finished language server for now. It’s not working as it should and further features require internal code refactoring, but now i’m gonna focus on compiler itself (alot to do there).

For now i temporary made some kind of built-in udon types library info. It’s done in form of autocomplete and works like that:
autocomplete_temporary
As i said, it’s temporary. Better than looking for documentation but still not enough. I want to do full support of autocomplete (context-related types, methods, method params and such), but for now i gonna focus on compiler itself.

Syntax colorizing is from C# now, and it doesnt fit doshik language all the time (for example :: types notation) so maybe i do this next. Found a tool to do syntax colorizing from scratch - looks simple enough - https://eeyo.io/iro/ and is able to generate files for vs code.

EDIT:
Made syntax colorization, but it’s VERY basic, and looks like it’s gonna stay this way as it require to redo whole parser logic to make coloring close to semantics (C# colorization file is VERY complex). I’m looking for ways to reuse compiler’s logic for coloring, but found no easy ways (it’s possible by using some language server protocol extension, but i really don’t wanna turn this road).

Also i added events to autocomplete and built-in events parameters now work. also parameters names doesn’t have to match udon api declaration, only signature (types) has to match
autocomplete_temporary_2

I was following spinning cube example video #4 (Program variables) but that didn’t work. Maby i’m missing something but in video there is no explicit “speedProgram” variable declaration and it works somehow. I tried in graph mode, it doesn’t. Maby this is due to newer SDK, but that only work if i explicitly set program variable.

I managed to make it all work from doshik code, but public variable declaration is required.
I also tested getting this program variable from different udonbehaviour (which is made on udon graph) - that also works

// this is program variable that is being used in script (it could be used from different udon behaviour script)
public string programVar;

event void Start()
{
    VRCUdon::UdonBehaviour thisBehaviour = GetThis<VRCUdon::UdonBehaviour>();

    thisBehaviour.SetProgramVariable("programVar", (object)"some string value");

    UnityEngine::Debug.Log(thisBehaviour.GetProgramVariable("programVar")); //< that would print "some string value"
}

Finally added typeof(T) operator (does same thing as Type nodes in graph) and custom events support so now i was able to do last video example #5 from Spinning Cube Example Series but completely in doshik.
Code:

// Script for Cube

// should be set from unity editor. this is cube that is being rotated by mouse clicking
// This object has to have VRC udon behaviour component with "SmallCube" script attached
public UnityEngine::GameObject cubeToRotate;

// this is udon behaviour of cubeToRotate
private VRCUdon::UdonBehaviour udonOnRotateTarget;

event void Start()
{
    // gets udon behaviour from public variable's game object
    udonOnRotateTarget = (VRCUdon::UdonBehaviour)cubeToRotate.GetComponent(typeof(VRCUdon::UdonBehaviour));
}

// when mouse is clicked on THIS object
event void OnMouseDown()
{
    // Invokes rotating cube's custom function to switch it's isRotating state
    udonOnRotateTarget.SendCustomEvent("doRotate");
}

// Script for both Cube and SmallCube
// Copy-paste following code to separate script for SmallCube

private bool isRotating;

// that is custom event that is being invoked by mouse clicking (see OnMouseDown event)
event void doRotate()
{
    // toggle THIS object's isRotating state
    isRotating = !isRotating;
}

// gets called on every graphics frame redraw
event void Update()
{
    if (isRotating)
    {
        // Do rotate THIS object if isRotating state is set to true

        // up vector axis
        UnityEngine::Vector3 axis = new UnityEngine::Vector3(0.0f, 1.0f, 0.0f);
        // rotation angle is dependent on update's delta time to make rotation smooth
        float angle = UnityEngine::Time.get_deltaTime() * 60.0f;

        GetThis<UnityEngine::Transform>().Rotate(axis, angle);
    }
}

cube5