My '06 does this. It mostly happens when it's cold/rainy outside. I've even been standing right next to my car and the FOB hasn't worked.
It's not a low battery issue in the FOB because it's done this even with new batteries. I've also adjusted the connectors in the FOB to ensure they are making good contact with the battery and it hasn't resolved the problem.
'13 Prius Five