Yes, you can lose fitness, but even if you a year off the bike, it still takes you less time to regain fitness if you have been fit, than it does to acieve the same level of fitness if you are new to it.

So your hard work is not for nothing, because your body remembers and will help you get back to where you want to be...