It depends who you fly for. UPS and FEDEX are good companies, but there are bad (worse) cargo companies too.

Keep your current job and go to college at night or part time. Get the degree (not an aviation degree). Learn to fly now while you have an income. Get your Private pilot certificate and see how you like flying. I got hooked on it and never stopped. I love my job (I fly passengers by the way). There are many good and bad flying jobs out there, but they are all better than most non-flying jobs to me.