my road bike is my commuter bike. It's a 90's steel Bianchi. It has a rack and fenders. I also use it to do group rides and short tours.
IMO, it would be a PITA to take all that stuff off when you want to ride fast. I mean, I guess you could get a seatpost rack (but that won't hold much weight at all) and ez clip on/off fenders (but they won't allow for full coverage if that's important).
You can make modifications to make it suitable for both for sure. I ride it with the empty rack and fenders when I go on group rides. I don't care about the extra weight- it doesn't bother me. I always feel much faster when I don't have the panniers on the bike- I *feel* like I'm flying- even if I'm not.