If you're not going to be switching back and forth frequently, I would just get slicks for the wheels you have. Extra tires take up less room than wheels in addition to costing less.
Depending on the type of off-road riding you plan to do, "light trail" tires might work for pavement as well as off-road. I currently have them on my mountain bike. They work on easy (non-technical) single track and are still decent for pavement.
- Gray 2010 carbon WSD road bike, Rivet Independence saddle
- Red hardtail 26" aluminum mountain bike, Bontrager Evoke WSD saddle
- Royal blue 2018 aluminum gravel bike, Rivet Pearl saddle
Gone but not forgotten:
- Silver 2003 aluminum road bike
- Two awesome worn out Juliana saddles