I don't think your experience is unique for the unit. It's a tough unit to hunt with all the private and added competition from the number of other hunters found on the mountain.
Quality seems to have declined each year since 2012. There will always be good bulls found on the unit though due to being able to hide on private.
I would dare say just from others hunting experience from other units that fish lake does not provide the same type of rutting action. I remember how disappointed I was the first time I hunted big bulls here in 2010 with my dad. I just about switched units as a result, but stuck with it.
I was lucky enough to come home with a great bull when I finally got the tag. Would I burn 15 years again to hunt it? Probably not, but considering point creep, just maybe. I would love to hunt the unit again with my own tag, but I am weighing my options during the waiting period.
No unit is necessarily a gimme, but I would recommend anybody wanting a good shot at a 320+ bull to look else where than fish lake. That will make you happy and me happy