fishdr - No you are not being selfish you do need a little time just for you.
I have three kids (my youngest is not at school yet) so I only get out to ride on the two days he is at Preschool. On the weekends I try to go for one longer ride and one short ride so that I still have plenty of time left to spend with the family. At the moment the weather is not being very bike friendly so its not really an issue but it is hard trying to balance riding and family and for you a job too. I guess the key is flexibility - when you have kids you just have to take each day at a time because one never knows what drama will be waiting just around the corner.