If a guy is seeing hits to the right at less than 400yrds, it's likely you are a right hand shooter, it is recoil pushing your body.
Spin drift is real, but not a concern at 6-800 or less, and not much at the thousand.
Shooting a magnum a lot myself, I have learned to get my body more directly behind the recoil, as much as possible.
It affects people more if they are less than 180lb or so in body weight.
I have shared the range with many that claim to get drift to the right because of spin.
Placing the recoil pad more on your peck muscle, just at the shoulder crease helps me dramatically.