Thanks for clearing that up.
I tried using regular leather cleaners with no difference, I heard the tougher ones can wear down the leather and get rid of the color, makes me a bit nervous. It does make me wonder because I rarely use that arm rest and I never use any kind of lotion or cream. If anything my center console should be oily because I use it all the time. However, I did notice that in my parking spot when the sun is hitting my car around noon, it directly hits the door's armrest which maybe what's wearing it.
I checked with Subaru OEM parts, the panel is actually surprisingly not too expensive considering it is brand new leather parts, however the installation process maybe a PITA as I tried to take apart a door in a 2015 Legacy before and broke some clips in the process. The only reason I'm interested in replacement is that I want to sell the vehicle and this may de-value the vehicle.