I have hunted 35b a number of times. Saw people out on the desert nearly every day. Was also checked by the Border Patrol nearly every day. There have been times that I have been a little worried, but they just say hello and walk past.
Once met a group of thirty way out in the woods. They had me if they wanted me even if I had a handgun. Waved and walked past. I was followed another morning by an individual. Later the Border Patrol told me I was in an area that was known for drug drops. That the person was probably assigned to see who picked up the package.
If it is legal to carry a handgun it would be good to have. I wouldn't camp out on the desert, but I know people that do and just leave water out and have never had trouble.
You are going to see people. If that bothers you or worries you I would hunt as far north as you can.