I've put 50+ pounds on my back in a canoeing packbasket, plus 200+ pounds of me, and been just fine. Getting on and off requires a good high curb (or to be riding my folder with its low top tube), because the basket means I can't really pitch forward to swing my leg over. I don't go fast, though -- slow, steady, using all my granny gears. And I use the packbasket with smaller loads all the time for potlucks, because I can carry delicate things without any worry of squishing them.

Possibly what I really need is a decent kickstand on my grocery shopper -- I've found loading panniers without accidentally toppling the bike to be the hard part, not riding with the weight.

I've broken spokes on an old road bike with just my body weight, but I got a sturdier wheel (and, later, a new bike also with sturdy wheels). Pump your tires up a bit higher if you run them on the low side due to being light under normal operation -- I pretty much always run max PSI.