Calee -
I think all your "fears" are normal. Financially, emotionally and physically it can be hard.
For me, financially, with each of my kids we were never in a good place. With the first, we weren't ready; the second, we weren't ready; and with the third, after finally getting ourselves into a better place, it was a hard decision to make, knowing that for at least a few months I wouldn't be working and trying to figure out if we could budget ourselves on one salary and for how long we could make it work!
Physically, with my first two I never thought about it. With the third I did. And to be honest, this time around, things didn't bounce back quite as easily but they're getting there. I was very lucky to have gained under 30 pounds each time, and with nursing my weight came off very easily. My abs look like a Shar Pei, but I'm hoping that that will improve over time, too!
My older two, DS 8 and DD 5, fight like cats and dogs; they both adore my 8 month old. My son is smitten with her, totally in love. My MIL told me one day that if I had not had the third, I would never have seen that side of his personality and she's right. He would move heaven and Earth for her!
I wish you luck in your decision! Kids are a challenge every day. There are times when I want to pack them up and ship them to the moon, but after a day (or two ;-) )I'd be in a rocket going to get them!