Is there some kind of bug with memory?

It’s happened a few times now, only recently within the past week. I will go into a populated world, start lagging (of course), but when I leave and go anywhere else my memory usage for VR chat stays high. I have 32GB of ram and It devours 80% of it when this happens. I can be in the VRC default world with no one around and still continue to experience this issue like I’m still in a crowded room. Anyone know what happening?