I just turned 40 this year, and I can tell you I have some serious fear. I too look around and don't want to have ride a scooter when I go to Disneyland.
Skin does lose its elasticity and there really isn't a whole lot you can do about that. I am starting to get a bit of the neck thing, and I do SUDDENLY have squint lines around my eyes. I think some of the fear is fear of death really, because all of these little changes just show you the inevitability of growing older and dying, which is really the scary part.
Some people don't take care of themselves, and this is the biggest problem. If you eat healthy, stay active, and keep your mind active, you will be far ahead of the game.
I too think that appearance does matter less. I mostly just really don't want to lose vitality, and I think sometimes we judge a persons vitality by the way they appear.