I started swimming again after a very long (15+ years) hiatus about 8 years ago. It felt horrible for a really long time, and that's swimming 2-3x/week for at least 5 years before it started feeling really good and enjoyable.

The thing about swimming is it's so technical. Most anyone can learn to float and go some distance, but you can tweak every single aspect of the sport and you can be an Olympic swimmer! I don't think there are many sports you pick up and instantly are good at. They're all skills one has to learn and practise to get better. Don't be discouraged, every little bit is a step in the right direction!