Again, it's whatever works for you. I too just hop on my bike to get to the library, grocery, doctor's, stop by a friend's, and will often just ride in the jeans or cargo shorts I have on, no problem on the short rides. But you mentioned your bike shorts are maybe a little too small. That could be part of your problem. I found my difficulties in your area of concern eased up when I went to a much thinner pad in my bike shorts. I've experimented with several pairs, and found a favorite, Pearl Izumi. I wear the others, but when I have a long ride I read for the PI's. And because I do ride around town I often slip a skort over the shorts, found I get more use that way than out of padded skorts (got one of those too, it does to me what your shorts do to you). Plus the skort has pockets!

good luck figuring out the problem, if anything's personal and individual, this is.