Depends if you can stand to wear a wool turtleneck (over another shirt that keeps the wool only touching your neck).

I can’t stand any turtleneck, personally, for exactly the same reason I can’t do tags. It’s there touching me, and it feels wrong but won’t go away, and the wrongness bothers me. Even super fluffy soft fabrics are wrong on my neck.