Have to agree with teigyr. It's not like you called in sick and took a walk or anything. Just kidding, Jane.
You had a legitimate reason for taking the afternoon off, and it so happens that you'll have some free time to enjoy a nice ride. It's no different than having extra time after the appt. and spending the time reading, or knitting, or napping. It's your time, and you don't abuse taking off work- so GO RIDE and enjoy it, dadnabit!!!