The way I breathe doing pups is more like Pilates breathing than belly breathing, I think. Abs are engaged, so I'm breathing more into my ribcage.
If you don't do Pilates, try this Stott Pilates breathing practice: sit on a mat, hug your knees in and curl over them, then just practice inhaling and exhaling. The only way for your lungs to expand is into your ribcage. Doing that will give you an idea of how it feels to breathe into your sides and back.
Yup, forceful exhale on the way up. You can subvocalize counting as you exhale, if it helps to count out loud.
HTH.
Speed comes from what you put behind you. - Judi Ketteler