Ya, Triple Seven fouling is really easy to clean with just a wet patch, but shotgun primer residue is not. It is sticky and stubborn. Some people will dry fire a couple of primers to clear the flash hole before loading, but this is a bad idea. Doing this actually lays down a layer of sticky primer fouling for everything else to stick to. Just keep the breach plug clean, and use a nipple pick to clear the flash hole once in a while if needed.
Here is something for you to try. Completely clean and dry your gun inside and out. No lube, no bore butter, no oil, nothing, just a clean dry barrel. Now take a piece of plain white paper and hold it up against the muzzle and fire just a shotgun primer (do this in a safe manner). You'll be amazed at how much fouling you get from just the shotgun primer. Now try cleaning that primer crud out of your barrel. It's pretty tough and you might need to use a solvent. Once you get it clean and dry again, wipe the bore down good with Bore Butter. If you have a light layer of Bore Butter on the surface of the bore, the fouling doesn't stick as well, and usually flakes off when you seat another bullet (if you haven't wet wiped the bore).
So, now that we know that the primer fouling is what's causing our problems, try the CCI shotgun primers. They are still pretty dirty, but they seem to be better than the others (Winchester is the worst). If you can find them, the Veri-Flame primer adapters with CCI standard pistol primers are really the best I've found - by a large margin. I have never had a mis-fire or hang-fire. In fact, I was worried they weren't hot enough so I chronographed them. They shot exactly the same velocity and impacted in the same group as the shotgun primers, but there was practically no fouling - even with three 50 grain 777 pellets. I actually shoot 2"-3" average groups at 200 yards with a high power scope, and just about the same with my 1x Simmons.
I use the 300 grain T/C Shock Wave Super Glide Sabots because they're cheap, accurate, flat shooting, and easier to load. And if you're planning a Muzzle Loader Elk Hunt, you could use the Bonded versions for extra confidence.
I sound like a salesman, but I'm not. These things work great for me, and they should work great for you too.