I watch Food TV a lot. Chances are, if you turn on my TV, it's already on that channel. I cook a lot and have since I was about 5 years old, so it's quite a passion for me.
The only chef I have a hard time watching is Mario Batali. Emeril, Paula, Ina - well, they're a little heavy, but well put together. Mario just looks sloppy and dirty. The scraggly beard, hairy legs, and clogs are just ICK!! His food is incredible, but I just can't stand to watch him. I'm not sure I could actually eat anything he put in front of me because he looks so skanky. Paul Prudhomme, who is equally large, does not have the same effect on me. He is well dressed and a classy entertainer. So I know it's not just the weight.
Now all that said, I agree with the posters who have trouble with some chefs because of what and how they cook. Sorry, but Paula Deen is not a "chef". She is a cooking personality. Period. And her food, while quite tasty, is very troubling from a wellness perspective. Same concerns about the foods promoted by Emeril and, sad to say, the late, great Julia Child. Butter, fats, and heavy creams are staples that will not lead to long-term health.
I LOVE watching Giada because she reflects her style of cooking so well. Lots of fresh, healthy ingredients, with minimal fats and processed foods. I feel the same way about Micheal Chiarello - the Napa chef. These are people who care about what they eat and it shows.
Ah - I could write a book, but I'll leave it there. To answer the OP question, it's not just the physique, but the attitude as well. Interestingly enough, one seems to lead to the other.