If something is naturally 'fat free" (like veggies) there is no problem, but as others have said, if something that naturally has fat in it (though muffins aren't really 'natural'!) is made non-fat, it usually has lots of sugars and/or refined flours (which your body treats the same as sugars) in. Remember those "Snackwell" cookies? Very low fat, but tons of white flour, sugar, etc.
Rather than looking at one specific nutrient (fat), look at the entire nutritinal picture of a food. Does it contain naturally-occuring vitamins? minerals? fiber? antioxidants? Or does it contain a lot of unpronounceable chemicals, refined flours, artificial ingredients?
Which type of food do you think your body actually thrives on? ;-)