By default im inclined to tell you to save your money and keep the Subaru. My mom had an X5 and probably spent more on repairs than her car payment over the 2 years she had it. I've heard the 3-series is a nice car, but I don't see the appeal besides a 335 or an M3. I'd rather keep my Subaru and buy a nicer house, or put that money into an IRA... But that's just me. You're clearly set on getting your 328xi and probably won't read into half the things we "normies" (you know, because you drive a Bimmer and everybody is a peon to you now) have to say based on our observations.
Enjoy your Bimmer.