I got my plates, dumbbells, and stability ball from Wal-Mart. They're also selling medicine balls now. I don't know about your local Wal-Mart, but the one near where I live is expanding its fitness aisle. I buy almost everything I need there, including weighted gloves for kickboxing, a weightlifting belt, push-up bars and ankle weights.
I used to use a Fanny Lifter in place of a high step, but I got the high step eventually. It's more versatile since you can adjust the height in 2-inch increments. If you want something that stays a certain height at all times, you can use a stool.
Other places I use are fitness1st.com and fwonline.com. I also find good deals in the local Oshman's and Sports Authority sometimes.
Pinky