After my first surgery, I think I started working out again maybe two months later? I had the cast on for 6 weeks. But the second surgery, I only wore the cast for 3 weeks and I was allowed to bear weight on it. I wasn't exercising at the time, because I had an 18 month old, so I can't answer about how long it took with that one.
If you have a stationary bike or something that's no-impact, that may be your option for exercising after you get the cast off.
Was it your podiatrist that said to not have the surgery? I would disagree on the "so much pain you can't stand it" thing, because they will only get worse and the older you are, the longer your recovery will be.
Get the surgery. You'll be glad you did!