While you aren't going to break a bike from loading the panniers, make sure you have a good secure rack to hold the panniers. Check the weight limit on the rack and check how much it sways under your load. A swaying rack will make riding feel treacherous. The economy racks under $30 may not be adequate. The weight distribution on the bike can also be a factor. If the weight of the panniers is behind the rear hub, you may feel the bike sway and feel unmanagable - hence the long wheelbases on touring bikes. And too much weight on the rear wheel (especially behind the rear hub) can leave the front end too light and affect steering - adding front panniers can compensate for this. Trailers are one good solution, as LPH says.