I have lower back problems, too. I see a chiropractor on a regular basis, and that has really helped. More than my regular chiropractic adjustments, though, what has REALLY helped me is a series of lower back stretches I do before every single workout. The first stretch is to lie on my back and pull each leg into my chest (bent, with my hands holding down on my shin). I stay like that for at least 30 seconds on each side. The second stretch: I'm still lying on my back and I take my left foot and place it on the outside of my right knee (both legs are bent). Then, I carefully lower my legs to the left, keeping my upper body facing up. Hold for at least 30 seconds (sometimes I hold this for a minute), then switch sides (right foot on outside of left knee and drop legs to the right). The last stretch is again in a lying-on-back position. Bend the legs, as if you're going to do a crunch, then lift one leg straight up and take a hold of it with your hands and pull it toward you, keeping it straight (I sometimes have a slight bend in my knee). This stretches the hamstring (see next paragraph), which will help your lower back. Hold for 30 seconds and then switch legs. After this, I usually drop my legs over to each side once or twice more, then carefully get up.
Quadricep and hamstring stretches are excellent for the lower back, too. This is why: If you don't stretch your quads and hamstrings, they contract and pull on the muscles in the lower back. If you stretch out those large leg muscle groups, they are nice and limber and don't pull on the lower back. So, being that I have a highly vulnerable lower back (I have 3 chipped vertebrate in my lower back), I started doing lots of hamstring and quad stretches, as well, before and after my workouts.
I have REALLY noticed a big improvement since I started doing this. About a year ago, I was having all kinds of lower back problems and sometimes was unable to even walk. I started doing these stretches, and I haven't had one problem with my lower back since.
The other problem you might be having is that when you do leg work, you aren't holding your core in. If you aren't tightening your core, this will put undue stress on your lower back.
And, about TTOM: Oh, dear goodness gracious! I have the same problem! I don't even touch a leg workout at TTOM, because my lower back hurts so badly that I don't want to aggravate it further. So, you're not alone there! :7
I hope this helps you! Start stretching, and if that doesn't work, see a chiropractor. You might have a slight curvature in your spine or something, but that would usually hurt one or both of your knees, as well.