The USCG (or any other military branch) will provide far better compensation than any other entry-level aviation job. You should be able to make any reasonable loan payments, and even get ahead of the ball after a few years.
The weapon question raises a red flag in my mind...ultimately the USCG is an armed service, even if many of their peacetime missions don't involve the use of force. In time of war, the ENTIRE USCG can be transferred into the US Navy. Any USCG officer could get assigned to a wide variety of missions, although I think aviators mostly stick to flight-related jobs.
I don't think they will (or should) accept you if you are a conscientious objector of some sort. Civilian aviation has no such issues of course.