I’ve found having a cotton t-shirt as a base to actually help when wearing layers, as the cotton wicks away any sweat from my skin and keeps me feeling generally drier and therefor warmer. Also helps if you find wool or other outer layers itchy. You’re right to not wear additional layers of cotton fabric, though.