I take one day off a week no matter what. Most of the time I use that time to run errands after work, etc. But my body needs it, too.
I also have unscheduled recovery weeks because I'm prone to injury. This has just started within the past year or so, and I'm learning to accommodate.
A word of advice - if you start to feel a little run down, or if you feel persistent aches that are not DOMS, take a break. Otherwise you'll end up like me and be forced to stop - probably for longer than a week - to recover from more serious injuries.