Can you get in, oh let's say, 4 or more hours of riding on weekends? 2 or more on saturday and sunday?

If you can and do some other stuff during the week like spoke suggested, then I can't see you losing your general aerobic fitness. You won't get faster on the bike per se, but you should not go backwards.