Vacations are supposed to be about what makes you happy and relaxes you. Your workouts do that, just as his surfing does for him. So, where's the difference?
I remember once talking to a doctor about a minor operation and he said I was lucky, while recuperating, I wouldn't have to work out for a while. I said, "but I like working out", and he looked at me so strangely.....
If he is that supportive of you, what exactly is his problem?
Just ignore any more comments and go ahead and do what you want to do.
Clare