Personally, I have just had to learn to accept my limitations in this area (something I tend not to do gracefully). I ride almost daily and have for five years. I can ride as far as I want now, and regularly ride twenty, thirty, forty miles. (I started out winded from riding six blocks to the grocery store.) But I still can't reach those higher speeds. I have asthma, was a lifelong couch potato, AND I smoked three packs of cigarettes a day for seventeen years (quit about eight years ago). No matter how much I push on the bike, no matter how much Cathe cardio I do, no matter how much running I (try to) do, there are just some things I'll never be able to do, and sustaining speeds over about fifteen mph on my Trek hybrid is one of them, but I'm finally okay with that.
For years, I pushed and pushed, always beating myself up for not being able to do more. But I've finally reached acceptance now, I think. I'm doing worlds more than I was five years ago. I'm 180 degrees healthier and happier, and you know, that's enough. Oh, sure, I still push. I still try to reach the top of my limitations. But I've finally decided to acknowledge that I do HAVE physical limitations, and those limitations do not make me an evil person OR a slacker.
So, my advice to you is not to give up without giving it a helluva fight, but if you don't reach the goals you've set, give yourself a little credit for your progress.
Shari