Maybe make the band wider, with two straps/ buckles around their waist that keeps the waist band snug, rather than the wrap attached to the sleeve. That way the sleeve is in place because itās sewn to the secure body piece, not the other way around.
I would recommend, with the current design, a thicker material like thin neoprene for the wrap part, and keep the sleeve soft with the current material.