One thing that helped me more than I expected was an insulated face mask. They’re usually marketed to hunters. A good one sits comfortably below glasses or safety goggles, and it stands just off your face except for the edge.

Adding this allowed me to be more comfortable in colder temperatures and strong wind with less layers overall.