those are supposed to be a pressure relief, for both doors and the HVAC. When you put the system on re-circ, the cabin pressure is neutral. When you have it set to fresh air, the cabin pressure is positive. You have to have a place for the air to go, and that's what those vents are for.
You'll probably notice a drop in heater performance come winter.
2005 Outback FMT