Should have thoroughly read your problem lulz.
I believe the swaybar bolt hole isn't exactly flush with the bolt on the Whiteline bar neither. But the rear doesn't bang around. Make sure you have your washers and bolt aligned properly. Whiteline's install had the bolt head oriented on the inside of the system, followed by a small washer, the swaybar, endlink, a large washer, and then finally the nut on the outside of system.
I would also double check your mounts and mount bushings.