Excessive latency

I live in Norway, and am playing VRChat hanging out a lot in USW and USE based worlds. I have noticed an excessive amount of latency (normally around 5 seconds, often as bad as 15 seconds, which I have tested with my friends), though my ping will normally only lie around 130 ms to USE worlds.
I’m in the process of trying out if a VPN can help alleviate this problem in any way, but I have come to understand that VRChat does not look kindly upon VPNs, and most servers I have tried up to now have been completely banned from accessing VRC.
I am not sure what can be causing such an extreme amount of latency, is this a bug or something like that?
Is there anything else I can do to help combat this issue? Having a conversation with such latency is near impossible, and extremely annoying, as I keep interrupting other people constantly, and is just generally not a good experience