Yeah, I would definitely have been offended. It's seriously out of line, but I just can't believe some of the things people have said to me. I probably would have reacted the same as you did, though. I never tell anybody that they've offended me.
Hmmm...perhaps this is an indication that your company is due for some sensitivity training.
There are a lot of highly inappropriate comments that aren't mean to offend but 1.) they are still inappropriate and 2.) they are still offensive.
Generally speaking, I think that if a person does not feel comfortable speaking to the offender directly they are perfectly within their rights to speak to their supervisor.