I'm guessing your saddle is too narrow, no wiggle room and the bones aren't completely supported.
Stress to the pelvic floor leads to pain at the boney attachments, and pain on the "insides" of the sit bones.
Do what you want, but keep in mind that the saddle might not be wide enough for your riding style. (I assume you already did the measurements, and your saddle is wider than your sit bone outer span by a couple cms, and so is adequate for your anatomy.)
"If Americans want to live the American Dream, they should go to Denmark." - Richard Wilkinson