I can't see why it would be a big problem. But worry about your health first, aviation second.
I noticed a drop in sex drive when the stress piled up, no "performance" issues thankfully but I did some research and made some changes which helped noticeably...
Cardio: I tend to manage stress with cardio. I would do at least an hour cardio if I put on the shoes or got in the pool. Turns out cardio is good for you health overall and will help with stress, but a whole lot of it will increase cortisol, which will drives down your T. Three or four 30 minute cardio sessions / week shouldn't have much impact on T, the cortisol boost starts after 40-45 minutes cardio. Long term, I would actually advocate more cardio than what I suggested here, for overall health, but this might deal with the T issue.
Weights: Must do this to increase T, also has fringe benefit of helping prevent random injuries as you age. With weights you want to focus on activating the large muscle groups...all the insecure little boys at the gym who do nothing but blast their biceps are not really generating much T. Go for somewhat lower reps (ex 10-8-6-4), shoot for 80-90% of max reps before fail. You MUST do legs...squats, deadlift, hamstrings, etc. Pullups, rows, bench, etc all get multiple muscle groups. Get the back involved too.
Zinc: zinc supplements may help, look for good quality about 30 mg/day in the evening (more than about 30mg may not be healthy in the long run). Surprisingly this can have a noticeable affect within an hour or so. Some foods have zinc too.
Body weight: If not already, get fit. You may have to do a lot of cardio to get there but after you get near fighting trim you can ramp up weights and reduce cardio.
Caffeine: Puts your body in an elevated state which, while useful, is similar to a stress response (cortisol again). Try limiting to one cup of joe (or soda equivalent) per day. If you're over age 25 and doing monster drinks or 5-hour energy, that's most likely your problem right there.
Last thing, gym membership can definitely improve your sex drive...so buy one for your wife today
