It doesn't bother me. In fact, I think 'going with the beat of the music' is somewhat counterproductive for muscle and strength gains. I got very used to not following the music with P90X.
HOWEVER, I wish Cathe would have said how many reps we're doing at the beginning of the not-with-the-music sets, so we can pace ourselves, especially on moves where we are not facing the screen.
Now, it REALLY bothers me when an instructor is off the beat of the music on cardio workouts (and Cathe has done this very occasionally, in some 'side-to-side' moves, like in either IMAX or Bootcamp---or maybe both--where the side that was on beat 1 and 3 transitions into being on beat 2 and 4 when she gets off beat, but she doesn't do it habitually, thank goodness!)