I'd highly recommend a chest pack from Hill People Gear. You wear it under your backpack (or daypack) and it holds your firearm and other items you want to access. You can drawn quickly from it and it does not interfere with your backpack's hipbelt. Also, you can use two straps they provide to dock it to your backpack so the chest pack is essentially weightless.
I never go in the woods without my chest pack and, being a bit of a gear junkie, can say without question that it is the best piece of outdoor gear, or clothing, of any type that I've ever owned in my life.
I got the largest one as I can fit a Ruger Super Redhawk revolver (Toklat) in it. If you are carrying a smaller handgun maybe a smaller chest pack would work. The video below discusses and displays all of the chest packs they offer.
https://hillpeoplegear.com/Products/CategoryID/1
I've always carried a neck knife so even when my backpack is off I have access to a knife. There are molle attachment points on the bottom of the chest pack to attach a full-size knife and sheath.
In the chest pack, I carry two small flashlights, a compass, map, leukotape, car keys, butane lighter, and my phone so they are always with me when the backpack is off.