I think that's fine. As a fellow civil servant (state), my suit only makes an appearance in court or if I'm a speaker. I would probably feel comfortable wearing a nice cardigan over that dress as an alternative to a jacket. My usual "meeting with management" or "attending a conference" attire is slacks/skirt with a nice top or sweater and dress shoes. I might occasionally wear a coordinating blazer, but rarely a matched suit.

(I confess, I do love suits. Takes a lot of the getting dressed decisions out of the picture! But man, would I look out of place around here!)