I just purchased a 2013 Subaru Legacy, 3.6R Limited with NAV. I have an ancient, non-smart phone (Motorola Razor, model W755). When I tried loading my contacts using the NAV touchscreen, I almost always received the message that it failed. I must have tried dozens of times. My hope was kept up by the fact that it would sometimes go as far as displaying the message "downloading contacts, please wait" before the "failed" message.
Tonight I finally got it to work by shutting off the phone and then turning it back on while the car was running and with "phone" selected on the audio screen. When the car re-recognized my phone, I pushed the transfer button on the NAV screen, and all my contacts loaded into the system. Victory is sweet.