Sleeveless Hoody/Hoodie?

Anyone had any joy in making some kind of sleeveless jacket?
I took the VRoid hoody mesh and alpha-ed out the sleeves with a separate T-shirt underneath
Looks great in VRoid


But taking it into Blender get’s immediately ruined by the T-pose there.

In reality a sleeveless hoody would only start to move/deform when you arms are raised above horizontal. But in the T-pose it is already deformed.
I have tried using weight paint but since the issue is already present in the T-pose it doesn’t help and mostly just introduces horrendous clipping.
I also tried alpha-ing out more of the sleeve but it made it look kinda rubbish.
I put a lot of time and effort into the textures and skin mask. I’m frustrated enough with it, I’d pay someone to fix it for me but no one seems to be advertising clothing fixing on Fiverr - only an avatar from scratch or rigging.

I found an actually sleeveless hoody texture for sale.
But it has the same shape as the one I edited together and so suffers from the same problem: